DistinguishingTokenStage

[Robin Linacre (RobinL)]

This extends the local distinguishing-token work in #444 by exposing the same
signal as an optional deterministic DistinguishingTokenStage.

The stage matches a non-empty, locally distinguishing canonical prefix at the
start of an address, followed by the next two canonical tokens in order. It
allows at most two safe gap tokens, requires an exact postcode, and only emits
a match when one distinct canonical ID qualifies.

Canonical preparation is opt-in via
derive_distinguishing_wrt_adjacent_records=True; existing defaults are
unchanged. This PR does not modify the packaged Splink model.

Results

Both arms independently process records left unmatched by ExactMatchStage.

Area	Peeled precision	Peeled recall	Distinguishing-token precision	Distinguishing-token recall	Precision delta	Recall delta
Hackney	99.9699%	35.0459%	99.9422%	57.7591%	-0.0277 pp	+22.7132 pp
Rhondda	99.7506%	65.4609%	99.6680%	83.4219%	-0.0826 pp	+17.9610 pp
Aberdeenshire	99.5392%	80.0353%	99.3932%	88.2785%	-0.1461 pp	+8.2432 pp
Mid Sussex	92.9688%	6.3365%	93.0108%	23.0298%	+0.0420 pp	+16.6933 pp
Weighted total	99.6855%	60.0428%	99.6010%	76.1465%	-0.0845 pp	+16.1037 pp

This gives substantially higher recall than peeled matching, while retaining
high precision with a small overall reduction.

Validation

• Packaged and experimental implementations produced identical matched
  (messy ID, canonical ID) pairs across all four benchmark areas.
• Tests cover safe-gap rules, ambiguity handling, canonical preparation across
  chunks, public stage discovery, and missing-preparation errors.
• Full suite: 304 passed.

[github-actions]

🚨 Benchmark Check Required Before Merge

Before merging this PR, make sure that you run the following check to test for performance regression:

bash ./shell/compare_commits.sh \
  -b main \
  -c 1af7d0e0f133c651a41f72c5abfb4930f5ea4b54 \
  -d rhondda,aberdeenshire,hackney,mid_sussex

This comment updates automatically whenever a new commit is pushed to the PR.

Note

If you have changed the cleaning, you will need to rerun cleaning prep in order to test these changes successfully.