feat: range-aware inverted index

[Tom Hepworth (ThomasHepworth)]

Operational Change

The cleaning pipeline now converts numeric TO ranges into dash ranges:

• 55 TO 57 becomes 55-57

The inverted-index pipeline then:

• Uses extracted numeric_tokens containing - as a cheap candidate filter.
• Expands valid numeric ranges up to a maximum span of 20.
• Adds scalar keys for old ranges (55-57 OLD) -> 55 OLD, 56 OLD, and 57 OLD.
• Retains the original range keys.
• Applies the same key generation to canonical and messy addresses.
• Preserves one canonical record identity; only additional index keys and postings are generated.
• Avoids Cartesian expansion when an address contains multiple ranges.

Additional Canonical Records

No additional canonical address rows are created. The canonical row count remains unchanged.

The persisted inverted index changed as follows:

Artefact	Baseline	Range-aware	Difference
Index rows	61,575,577	64,499,824	+2,924,247

For the verified blocking-miss sample:

• Historical blocking recovered 0/143 rows.
• Range-aware blocking recovered 10/143 rows.
• This covered 3/80 distinct messy records.

Accuracy Results

The change was tested on the pooled councils dataset:

• Hackney
• Rhondda
• Aberdeenshire
• 348,687 messy input rows

The controlled comparison used the same matching stages and compared the old canonical index with the range-aware canonical index:

Metric	Baseline	Range-aware	Change
Precision	98.9577%	98.9575%	-0.0002 pp
Recall	96.8000%	96.8066%	+0.0066 pp
F1	97.8670%	97.8702%	+0.0032 pp
Correct matches	337,529	337,552	+23

[github-actions]

🚨 Benchmark Check Required Before Merge

Before merging this PR, make sure that you run the following check to test for performance regression:

bash ./shell/compare_commits.sh \
  -b main \
  -c 3c53663600bc51a899f8073b986ef064fd3b4bda \
  -d rhondda,aberdeenshire,hackney,mid_sussex

This comment updates automatically whenever a new commit is pushed to the PR.

Note

If you have changed the cleaning, you will need to rerun cleaning prep in order to test these changes successfully.

[Tom Hepworth (ThomasHepworth)]

Changes end up looking like so (where the baseline is our new build!):

So we get some very marginal gains to precision/recall and capture some additional records at the cost of 3m more records in the canonical idx. This is around ~100mb on disk, which might be too expensive given the marignal gains involved.