[ArthurZucker]

Stacks on #2241 (perf/bpe-merge).

The bug

merge_word picks its merge engine from a table indexed by the word's first byte:

let gate: u16 = self.byte_to_mode[sequence.as_bytes()[0] as usize];
if sequence.len() > gate as usize { /* two-tier queue */ } else { /* multipass */ }

But a pre-tokenizer hands the model its delimiter attached to the front, so byte 0 describes the delimiter, not the word. ByteLevel gives "Ġword" (Ġ = U+0120 = C4 A0); Metaspace gives "▁word" (▁ = U+2581 = E2 96 81). Both are ≥ 0x80, so both take GATE_MULTI (8) — even when the word behind them is pure ASCII and belongs on GATE_ASCII (24).

build_byte_to_gate already special-cases a literal leading space down to GATE_MULTI, with a comment observing that the first byte "says nothing about the script of the rest". So the hazard was already known for " word"; the multibyte spellings of the same delimiter just never got the same treatment — and forcing them to MULTI is the wrong direction anyway. The fix is to classify the content.

The change

content_start skips one leading delimiter (▁, Ġ, or ASCII whitespace) and merge_word indexes the gate there. A delimiter-only word (a run of spaces) keeps offset 0, where MULTI stays the right conservative answer.

Reach

200 kB of corpora/english.txt under the gpt2 pre-tokenizer: 30.6% of words are both inside the gate's decision band (9..24 bytes) and Ġ-prefixed with ASCII content, so they change engine. On code.txt only 0.9% are.

Measured

gpt2/english, 10 kB chunks, best-of-3 warm, three interleaved A/B rounds so machine drift cannot masquerade as the effect:

	round 1	round 2	round 3
with fix	97.2	98.3	98.7 MB/s
without	89.4	91.0	85.3 MB/s

Non-overlapping ranges: +10.7%. code, dense, russian, chinese are flat, consistent with their much smaller affected fraction.

Byte-exactness

Guaranteed by construction — the gate only chooses between two merge engines that must already produce identical output — and confirmed empirically: identical token counts on every model × corpus probed (llama-2, gpt2, llama-3 × english/code/dense/russian/chinese).

cargo test -p tk-encode: 311 passed, 1 failed. The failure (normalizers::precompiled::tests::pipeline_precompiled_matches_legacy, a missing data fixture) reproduces unchanged with this commit stashed, so it is pre-existing.

What this is not

Not a fix for llama-2. Its config has pre_tokenizer: None, so a 10 kB chunk arrives as a single word whose length exceeds both gates — it always took the two-tier queue and still does. Measured: 31.4 vs 31.3 MB/s, i.e. unchanged. llama-2's cost is the merge over one unbounded span, and closing that needs a derived word split (e.g. proving from the vocab that no piece crosses a ▁ boundary, then splitting there), which is a separate change.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.