[ArthurZucker]

Same change as #2301, ported onto perf/bpe-merge (#2241) so it lands on the branch that already carries the content-byte gate from #2294.

The idea

ignore_merges lets a pretoken that is itself a vocabulary entry be emitted as that entry, skipping the merge loop. It is declared in the config, so llama-3 gets it and gpt2 does not — even though 93.1% of English pretokens are a single vocabulary entry that provably cannot encode to anything else.

The property is checkable, so it need not be assumed. At load, run the merge engine on each entry's own text and record whether it reduces back to itself.

fn fold_id(&self, sequence: &str) -> Option<u32> {
    let id = self.vocab.get_bytes(sequence.as_bytes())?;
    match &self.fold_bits {
        None if self.ignore_merges => Some(id),   // config asked for it: fold every hit
        None => None,
        Some(bits) => (bits.at(id as usize / 64) >> (id % 64) & 1 == 1).then_some(id),
    }
}

Configs that already declare ignore_merges are untouched — the proof only runs when the config did not ask, so this can only enable a case that was already safe.

Why a bitset and not a bool

The bool version measured exactly flat. On gpt2 one entry of 50,257 disagrees: <|endoftext|> decomposes to ['<','|','end','of','text','|','>']. A global flag therefore reads "not provable" and the other 50,256 entries keep paying for it. That entry is a special token consumed by the added-vocabulary frame before the model, so in practice it never reaches this path — but "in practice" is not a proof, and a bitset does not need one. 6.3 KB for gpt2; a disagreeing entry costs only itself.

No new table

One probe into the vocabulary MPHF already held, plus one bit test. The alternative for this workload is a large pretoken cache — gigatoken's is ~64 MB (1.3M entries, 99.4% hit rate) against these 6.3 KB.

Numbers

Measured on the #2279 encode path (identical change), gpt2, ns/byte, 10 kB documents each encoded once with the cache carried. Median of 7 interleaved rounds; chinese/hindi/thai over 11.

corpus	before	after	speedup	pretokens that are one vocab entry
chat-mistral	4.01	2.65	1.51×	94.8%
chat-chatml	3.88	2.74	1.42×	—
english	5.47	4.01	1.36×	93.1%
math-latex	4.69	3.97	1.18×	—
code	3.11	2.69	1.16×	88.5%
agentic-swe	3.16	2.92	1.08×	80.6%
chinese	10.99	10.89	1.01×	37.6%
hindi	2.61	2.61	1.00×	11.9%
thai	5.48	5.93	0.92×	—

Load cost: +7 ms median (61 → 68 ms), one merge per entry.

Thai loses 8% to probes that miss. Gating the probe on the content-byte gate was tried and is a wash (0.96× against the ungated fold): the miss path then computes the gate twice, once in fold_id and once in merge_word, which costs about what the skipped probe saved. A cheaper reject is still open.

Exactness

Full benchmark matrix on the #2279 path, 7 models × 30 corpora: 174 cells byte-exact against tokenizers 0.23.1, 29 mismatched — the same 29 albert (Unigram) cells that already fail before the change. No new mismatch. tk-encode suite passes here (338 tests) with a new test pinning ids for strings that mix folded words, merged words and a literal <|endoftext|>.

Two approaches this replaces

Both measured and rejected:

• Dense 512×512 grid (inline u64 instead of u16 index + values): 0.89–1.00×. Only ~4% of cells are live, so the value array stays hot in L2 and the indirection is already right.
• Incremental rank array (8.0 lookups per merge → ~2): 1.10× on english; removing the 7 panic_bounds_check branches it introduced added nothing. 86.6% of lookups were already grid hits from a ~100 KB table, so they were cheap. The merges were the cost, and the fold removes them rather than speeding them up.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.