Repository navigation
poc(encode): target encode path — bitsplit + batched model + fused cache probe - #2279
Closed
ArthurZucker wants to merge 133 commits into
Closed
ArthurZucker wants to merge 133 commits into
ArthurZucker wants to merge 133 commits into
Conversation
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
new `UnsafeCell` with atomic impl should allow us to scale linearly with core count Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
Signed-off-by: Luc Georges <luc.sydney.georges@gmail.com>
At 10 kB a large share of each pass is per-call overhead -- scratch checkout, added-token scan, post-process -- which rates implementations for work that is not tokenizing. The throughput phase now encodes one ~10 MB input per fixture, repeating a fixture that is shorter than that. The scaling sweep parallelises across chunks, so it re-slices that same input into ~512 kB pieces rather than encoding more of them: every thread still has units to take, and the phase costs exactly what it did before. The memory child keeps its 10 kB prefix -- its `rss0` is supposed to measure the tokenizer, not a corpus sitting next to it. Note this makes phase 1 encode 10x the bytes it used to; the released-crate baseline is the slow side of that.
Brings in the BPE merge work from #2241 and #2275: the multipass and two-tier-queue engines, the rank tables and the pretoken->rank conversion. `models/bpe/model.rs` is gone -- #2241 split it into `bpe_model.rs`, `bpe_scratch.rs` and the merge engines -- so the word cache and the batched `tokenize_spans` this branch added to it are re-wired onto the new engine: `BpeScratch` carries `word_cache` again, and both pipeline entry points emit through `tables.unmap` instead of `Word::get_chars_iter`.
Brings in the parallel encode runtime from #2213: the `ParallelPlan` ladder, the claim-cursor job core, `EncodeHandle`, and the fork-safe pool. Both branches had rewritten `tokenizer/pipeline.rs`, so #2213's runtime is taken whole and this branch's work is grafted back onto it: - `PipelineInner` carries `normalizers: Vec<_>` (metaspace-as-normalizer) and the wired `PipelinePostProcessor` instead of the single normalizer and the unused `_post_processor`. `stride_boundary` now asks every normalizer; `PipelineNormalizer::Metaspace` answers no, so t5/albert take the `Normalized` plan rather than a raw cut. - The model kernels call the batched `tokenize_spans` instead of looping `tokenize_pipeline` per span -- the point of this branch. - Special-token framing is applied per *input*, where an input's chunks are joined (`take_result` / `encode_serial`), never per chunk. `encode` frames by default; `encode_with` takes the flag. `encode_one` is the single-sequence synchronous path, and `encode_generic::<STAGE>` keeps the bench ablation ladder. - The thread-local `EncodeState` replaces this branch's per-tokenizer `ScratchPool`, but the scratch holds a word cache keyed on word bytes alone -- so it is now rebuilt when the thread's last encode used a *different* tokenizer, not merely a different model kind. Two BPE tokenizers sharing a thread traded ids without this. Also: - `BucketVocabStore::build` returns the empty store instead of panicking on an empty vocab; `BPE::default()` is one, and the plan tests build from it. - #2213's oracle keeps its own file (`pipeline_parallel_oracle.rs`) next to the per-model one, and compares against `add_special_tokens = true` now that the post-processor runs -- which also checks the framing lands once per input. - `bpe_pipeline_oracle` skips a model whose *legacy* reference can't encode without `fancy-regex` (deepseek): with the backend on, all four are exact.
The merge had four ways to silently drop parallelism (plan selection, the scratch identity check, the framing hook, the pool), and none of them show up as a test failure. `par_check` prints serial vs one-document-parallel vs batch-parallel MB/s per model, so a regression is one command away. Also clears the two clippy warnings the merged tree inherited.
…gh encode_generic `ab_giga` mirrors gigatoken's own `encode_st` protocol (one repeated buffer, pass 0 cold / later passes warm, best-of over the warm passes, token count printed) so the two can be compared on the same inputs. Token counts match gigatoken exactly on all 8 model x corpus pairs, so the numbers are apples-to-apples. Single-thread warm, 4 MB, interleaved best-of-3 (this box swings ~20% by machine state, so runs must be interleaved -- a single run cannot resolve 10%): gpt2 english 1145/1488 .77x | code 619/776 .80x | chinese 776/493 1.57x | russian 698/602 1.16x llama-3 english 1139/1354 .84x | code 714/780 .92x | chinese 953/503 1.89x | russian 1001/627 1.60x Geomean vs gigatoken: gpt2 1.03x, llama-3 1.24x. `encode_one` now goes through the fused `encode_generic` instead of the worker kernel `encode_one_with`, which splits the pipeline across two functions so a worker can enter at either. Worth ~3% on a whole sequence; it is not the whole story (see below). Measured cost of the merge itself: same harness on the pre-merge tree is 1-8% faster on 7 of 8 pairs (~3% mean). That is the `Arc<PipelineInner>` pointer hop the parallel runtime needs for its 'static workers, and it buys 4-10x on multiple threads.
One 4 MB document per thread through the parallel `encode`, matching gigatoken's `hf_mt` (`encode_docs_ragged` over the same docs) in thread count, work per thread and MiB/s. 14 threads, best-of-3 interleaved, ours/giga: gpt2 english 6240/9229 .68x | code 4252/3056 1.39x | chinese 1181/1251 .94x | russian 4463/1185 3.77x llama-3 english 5075/8766 .58x | code 3380/4071 .83x | chinese 1380/1926 .72x | russian 4894/1638 2.99x Geomean: gpt2 1.35x, llama-3 1.01x. Scaling against each side's own 1-thread encode is the more useful read: english 5.4x/4.5x (giga 6.2x/6.5x), code 6.8x/4.9x (3.9x/5.2x), russian 6.4x/4.9x (2.0x/2.6x), but **chinese only 1.6x on both models** (giga 2.5x/3.8x) -- the whitespace-boundary striding limit already noted for single-document CJK, which is why gigatoken overtakes us at 14 threads on chinese despite our 1.5-1.7x single-thread lead there.
The harness returned a fresh `Vec` per encode (`encode_one`) while gigatoken's reserves once
and `clear()`s per pass. That cost an allocation plus a first touch of the whole token array
every pass -- and it scales with TOKENS, not bytes, so it quietly penalised exactly the
token-dense corpora: chinese emits 2.9 M tokens for a 4 MB input against english's 0.8 M.
Chinese was reading 1.52x when it is 1.87x. Corrected, single thread, warm, ours/giga:
gpt2 english .81x | code .83x | dense 1.02x | russian 1.36x | greek 1.35x
arabic 1.33x | hindi 1.26x | thai 1.33x | korean 1.49x | chinese 1.87x geomean 1.23x
llama-3 english .91x | code .95x | dense .99x | russian 1.62x | greek 1.59x
arabic 1.72x | hindi 1.59x | thai 1.70x | korean 1.84x | chinese 1.91x geomean 1.43x
Token counts match gigatoken exactly on all 20 pairs. Also sweeps every corpus now, not four,
and takes deepseek-v4 when present (0.55-0.92x there -- that model wants the char-fold, which
is not in this tree).
`plan()` chose `Raw` whenever the *config* exposed a stride boundary, without asking whether the *input* contains one. Punctuation-terminated CJK contains none at all: Chinese prose has no space anywhere, and a newline after `。` is not a legal cut because the punct rule ` ?[…]+[\r\n]*` absorbs it into the punct token (`NEWLINE_PREV` excludes punctuation for exactly that reason). `data/corpora/chinese.txt` is 100% such newlines -- 722 of them, every one preceded by `。`, and zero spaces. Striding then did what `stride_range`'s own doc comment warns about: "text with no boundaries at all degrades to stride 0 owning the whole input". Measured on a 4 MB doc: 513 strides tiled, **1 resolved, covering all 4199874 bytes**, after scanning the document twice looking for cuts that do not exist. 0.28x the plain serial encode, and flat across thread counts because there is only ever one work unit. `plan()` now probes the longest input for an actual cut (bounded at 32 KB, once per encode) and falls through to `Pretokenized` when there is none -- which pre-tokenizes serially and parallelises the model over span groups. That prefix runs the normal pre-tokenizer, so it is `bitsplit` wherever bitsplit is wired (gpt2, cl100k), which is what makes it cheap enough to prefer over a degenerate stride. gpt2 chinese, 14 x 4 MB docs, MB/s ours/giga -- was 1193/1243 (0.94x), now 3004/1120 (2.68x); on a cool box the same fix measured 4798/1243 (3.86x). Single thread on one doc: 260 -> 732. Token counts unchanged (23531424), and every plan was already byte-exact, so plan choice only ever trades throughput. Spaced text still probes as cuttable and takes `Raw` unchanged. `cut_exists` passes `lo = 1`, not 0: `boundary_in_window` reads one byte of left context via `block_lo - 1`, so 0 underflows to `usize::MAX` and spins forever in the char-boundary walk-back (it hangs, it does not panic, in release). Byte 0 is never a cut anyway.
SBrandeis
force-pushed
the
feat/bpe-cache-min
branch
3 times, most recently
from
August 4, 2026 23:14
8195f43 to
4b47341
Compare
This was referenced Aug 5, 2026
ArthurZucker
added a commit
that referenced
this pull request
Aug 6, 2026
…g the flag (onto #2241) (#2302) * perf(bpe): prove the whole-pretoken fold per entry instead of trusting the flag `ignore_merges` lets a pretoken that is itself a vocabulary entry be emitted as that entry, skipping the merge loop. It is declared in the config, so models trained expecting it (llama-3) get it and gpt2 does not -- even though ~93% of English pretokens are a single vocabulary entry that provably cannot encode to anything else. The property is checkable, so it need not be assumed. At load, run the merge engine on each entry's own text and record whether it reduces back to itself. Configs that already declare `ignore_merges` are untouched; the proof only runs when the config did not ask, so this can only enable a case that was safe. A bitset, not a bool. A single flag is all-or-nothing and one entry is enough to lose it: on gpt2 exactly one of 50,257 disagrees -- `<|endoftext|>`, which decomposes to seven tokens -- so a global flag reads "not provable" and the other 50,256 keep paying for it. Measured with a bool first: exactly flat. Per-entry bits cost 6.3 KB for gpt2 and a disagreeing entry costs only itself. Encoding is one probe into the vocabulary MPHF already held, then one bit test. No new table: the alternative for this workload is a large pretoken cache, and gigatoken's is ~64 MB against these 6.3 KB. Measured on the #2279 encode path (same change, ported here), gpt2, ns/byte, 10 kB documents each encoded once with the cache carried, median of 7 interleaved rounds: chat-mistral 4.01 -> 2.65 1.51x chat-chatml 3.88 -> 2.74 1.42x english 5.47 -> 4.01 1.36x math-latex 4.69 -> 3.97 1.18x code 3.11 -> 2.69 1.16x agentic-swe 3.16 -> 2.92 1.08x chinese 10.99 -> 10.89 1.01x hindi 2.61 -> 2.61 1.00x thai 5.48 -> 5.93 0.92x The gain tracks how often a pretoken is a whole vocabulary entry: 94.8% on chat-mistral and 93.1% on english against 37.6% on chinese and 11.9% on hindi. Thai loses 8% to probes that miss; gating the probe on the content-byte gate was tried and is a wash, because the miss path then computes the gate twice. Load cost is +7 ms median (61 -> 68 ms), the proof being one merge per entry. Exactness: the fold changes what the model emits, so it is the one thing that must be checked. Full benchmark matrix on the #2279 path, 7 models x 30 corpora: 174 cells byte-exact against tokenizers 0.23.1 and 29 mismatched -- the same 29 albert (Unigram) cells that already fail before this change. * perf(bpe): pack the foldable flag into the id, dropping the side bitset The proof's answer was kept in a separate 6.3 KB bitset consulted after the vocabulary probe. It does not need to exist: an id needs 31 bits at most, the lookup already loads the entry to verify the key, and this crate already packs flags beside an id -- a merge value is a product id in the low 30 bits with SAFE_MASK above it. Same convention here: FOLD_BIT is bit 31 of the stored id, VOCAB_ID_MASK is the rest. So the fold answer arrives on a load that already happened. No side table to allocate, no second structure to keep in step with the vocabulary, and no second lookup on the encode path. `get_bytes_foldable` returns both halves; `get_bytes` masks, so no caller sees the flag. `content`/`byte_content` mask too -- the bit must never escape this type, and an unmasked export would have leaked it into `get_vocab()`. Build asserts every id fits under the flag. The model keeps one `fold_by_flag` bool to tell "proven per entry" from "the config declared ignore_merges, so fold every hit". No behaviour change: 338 tests pass, including the test that pins ids for strings mixing folded words, merged words and a literal `<|endoftext|>`. * Apply suggestion from @ArthurZucker
…elimiter (#2300) `merge_word` picks between multipass and the two-tier queue with a per-byte gate indexed by the word's first byte. With a ByteLevel pre-tokenizer the leading space arrives as `Ġ` (C4 A0) and with Metaspace as `▁` (E2 96 81) -- both >= 0x80, so every space-prefixed word was read as multibyte and given the CJK gate of 8. Any English word longer than eight bytes went to the queue, which is the wrong engine for a short word. `build_byte_to_gate` already makes this argument for the ASCII space and maps " \t\n\r" to the multibyte gate, precisely because a leading delimiter says nothing about the script of the rest. The conclusion was right and the remedy was backwards: what follows the delimiter should choose, so look past it. Measured on gpt2, ns/byte, 10 kB documents each encoded once with the cache carried across them, median of 3 interleaved rounds: english 9.02 -> 7.13 1.27x chat-mistral 6.28 -> 5.04 1.25x chat-chatml 6.07 -> 4.95 1.23x math-latex 7.04 -> 5.76 1.22x code 4.03 -> 3.54 1.14x agentic-swe 4.27 -> 3.92 1.09x thai 7.12 -> 6.65 1.07x chinese 13.75 -> 13.19 1.04x hindi 2.97 -> 2.86 1.04x No corpus regresses. The gain is largest exactly where the vocabulary merges hardest -- English prose, chat templates and LaTeX run 0.72-0.79 merges per byte, against 0.28 for Chinese -- because that is where picking the wrong merge engine is paid most often. The gate only decides which engine runs, so this must not move a single id. Verified across the full benchmark matrix, 7 models x 30 corpora: 174 cells byte-exact against tokenizers 0.23.1 and 29 mismatched, the same 29 albert (Unigram) cells that already failed before this change. The new test pins the case that was wrong, with words straddling the 8-byte boundary in both directions.
ArthurZucker
added a commit
that referenced
this pull request
Aug 6, 2026
* wip / ai: scratch pool * scratch pool iteration * implement word cache * lint * change defaults + max length + capacity bump * more sensible values * rewrite without hashmap * iteration * iteration * fmt * review comments * draft first commit * nits * proper bpe only benches * proper features * update * update what we balckbox * comments * commetn * first drafts * fix glue * draft table builds * start mphf draft * impl get * fix mphf * nits * fixes * slowly but surely * fixes * table tests pass * create top merges * update * up * progress in the conversion table build * update * fixes * dman * ... * eager flag was missing * update * design a manual example * with / without thief * remove some shit * more simplifications * single test move it * draft engine * remove complication * remove one flag that was not worth it * current status * monomorphize for perfs * fixes * rename * start drafting the two tier merge * todo is getting fixed * nit * SparseFold for table! * move stuff around * multipass bpe * scratches * isolate pipeline bpe * remove unused mergemap * u16 is faster * add convert.rs * update * less branches * update * it was not worth it * finally * nits, renaming and moving files here and there * renaming * fix more of the renaming * pipeline vs legacy * first fix for the gate * update the local bench * up * bit better benches * update * revert unrelated changes * more reverts * more reversion! * cleanup * nits * more clippy * perf(merge bpe): improvement suggestions (#2275) * rewrite + document multipass * rename * perf: save a lookup when there is no merge * perf: compiler gotchas * Idiomacy: - Use a trait generic instead of a const generic - 2 Specialized struct instead of mangling all into SymbolSink - fn that had no reason to live in impl PipelineBpe are moved oout * review: address comments on merge_multipass - reword the copy/merge cursor step, drop the redundant invariant line - show the trailing dead slots in the pass diagrams instead of shrinking the boxes, with a legend for the padding - apply the suggested SAFE-bit wording - move MergeState above merge_once, its only user - has_merged -> was_merged, known_pair_value -> cached_pair_value - inline copy_last into merge_once - reframe the step doc around the lookup it wins back --------- Co-authored-by: Arthur <arthur.zucker@gmail.com> * fix(bpe): the byte gate classifies the delimiter, not the word (#2294) * fix(bpe): the byte gate classifies the delimiter, not the word `merge_word` indexes `byte_to_mode` on `sequence.as_bytes()[0]`, but a pre-tokenizer hands the model its delimiter attached to the front, so byte 0 describes the delimiter rather than the word. ByteLevel gives `"Ġword"` (`Ġ` = U+0120 = `C4 A0`), Metaspace gives `"▁word"` (`▁` = U+2581 = `E2 96 81`) — both land above 0x7F and take GATE_MULTI(8), even when the word behind them is pure ASCII and belongs on GATE_ASCII(24). `build_byte_to_gate` already special-cases a literal leading space down to GATE_MULTI, with a comment noting that the first byte "says nothing about the script of the rest" — so the hazard was known. The multibyte spellings of the same delimiter never got the same treatment, and forcing them to MULTI is the wrong direction anyway: the answer is to classify the content. `content_start` skips one leading delimiter (`▁`, `Ġ`, or ASCII whitespace) and `merge_word` indexes the gate there. A delimiter-only word keeps offset 0, where MULTI remains the right conservative answer. Reach, on 200 kB of the english corpus with the gpt2 pre-tokenizer: 30.6% of words sit in the gate's decision band (9..24 bytes) AND are Ġ-prefixed with ASCII content, so they change engine. On code.txt only 0.9% do. Measured, gpt2/english, 10 kB chunks, best-of-3 warm, three interleaved A/B rounds to cancel drift: with fix 97.2 98.3 98.7 MB/s without 89.4 91.0 85.3 MB/s Non-overlapping: +10.7%. code/dense/russian/chinese are flat, consistent with their much smaller affected fraction. Byte-exact by construction — the gate only chooses between two merge engines that must already agree — and confirmed: identical token counts on every model/corpus probed. `cargo test -p tk-encode` 311/313, the one failure (`normalizers::precompiled::pipeline_precompiled_matches_legacy`, a missing data fixture) reproduces unchanged without this commit. NOT a fix for llama-2. Its config has `pre_tokenizer: None`, so a 10 kB chunk arrives as a single word whose length exceeds both gates; it always took the two-tier queue and still does. llama-2's cost is the merge over one unbounded span, which is a separate problem. * Update tokenizers/tk-encode/src/models/bpe/merge_hot_cold_queue.rs * Update tokenizers/tk-encode/src/models/bpe/merge_hot_cold_queue.rs * Perf/bpe merge review (#2297) * chore(node): sync Cargo.lock with tk-encode's libc dependency * refactor(bpe): flatten the filename prefixes, group the released-crate BPE under legacy/ * refactor(bpe): split the tables file, document the packed merge value once * refactor(bpe): move code next to its users * refactor(bpe): one name per concept * refactor(bpe): collapse the three copies of the char-as-bytes push * perf(bpe): prove the whole-pretoken fold per entry instead of trusting the flag (onto #2241) (#2302) * perf(bpe): prove the whole-pretoken fold per entry instead of trusting the flag `ignore_merges` lets a pretoken that is itself a vocabulary entry be emitted as that entry, skipping the merge loop. It is declared in the config, so models trained expecting it (llama-3) get it and gpt2 does not -- even though ~93% of English pretokens are a single vocabulary entry that provably cannot encode to anything else. The property is checkable, so it need not be assumed. At load, run the merge engine on each entry's own text and record whether it reduces back to itself. Configs that already declare `ignore_merges` are untouched; the proof only runs when the config did not ask, so this can only enable a case that was safe. A bitset, not a bool. A single flag is all-or-nothing and one entry is enough to lose it: on gpt2 exactly one of 50,257 disagrees -- `<|endoftext|>`, which decomposes to seven tokens -- so a global flag reads "not provable" and the other 50,256 keep paying for it. Measured with a bool first: exactly flat. Per-entry bits cost 6.3 KB for gpt2 and a disagreeing entry costs only itself. Encoding is one probe into the vocabulary MPHF already held, then one bit test. No new table: the alternative for this workload is a large pretoken cache, and gigatoken's is ~64 MB against these 6.3 KB. Measured on the #2279 encode path (same change, ported here), gpt2, ns/byte, 10 kB documents each encoded once with the cache carried, median of 7 interleaved rounds: chat-mistral 4.01 -> 2.65 1.51x chat-chatml 3.88 -> 2.74 1.42x english 5.47 -> 4.01 1.36x math-latex 4.69 -> 3.97 1.18x code 3.11 -> 2.69 1.16x agentic-swe 3.16 -> 2.92 1.08x chinese 10.99 -> 10.89 1.01x hindi 2.61 -> 2.61 1.00x thai 5.48 -> 5.93 0.92x The gain tracks how often a pretoken is a whole vocabulary entry: 94.8% on chat-mistral and 93.1% on english against 37.6% on chinese and 11.9% on hindi. Thai loses 8% to probes that miss; gating the probe on the content-byte gate was tried and is a wash, because the miss path then computes the gate twice. Load cost is +7 ms median (61 -> 68 ms), the proof being one merge per entry. Exactness: the fold changes what the model emits, so it is the one thing that must be checked. Full benchmark matrix on the #2279 path, 7 models x 30 corpora: 174 cells byte-exact against tokenizers 0.23.1 and 29 mismatched -- the same 29 albert (Unigram) cells that already fail before this change. * perf(bpe): pack the foldable flag into the id, dropping the side bitset The proof's answer was kept in a separate 6.3 KB bitset consulted after the vocabulary probe. It does not need to exist: an id needs 31 bits at most, the lookup already loads the entry to verify the key, and this crate already packs flags beside an id -- a merge value is a product id in the low 30 bits with SAFE_MASK above it. Same convention here: FOLD_BIT is bit 31 of the stored id, VOCAB_ID_MASK is the rest. So the fold answer arrives on a load that already happened. No side table to allocate, no second structure to keep in step with the vocabulary, and no second lookup on the encode path. `get_bytes_foldable` returns both halves; `get_bytes` masks, so no caller sees the flag. `content`/`byte_content` mask too -- the bit must never escape this type, and an unmasked export would have leaked it into `get_vocab()`. Build asserts every id fits under the flag. The model keeps one `fold_by_flag` bool to tell "proven per entry" from "the config declared ignore_merges, so fold every hit". No behaviour change: 338 tests pass, including the test that pins ids for strings mixing folded words, merged words and a literal `<|endoftext|>`. * Apply suggestion from @ArthurZucker * docs(bpe): say where the hot queue's heap comes from hot_push/hot_pop are std's BinaryHeap sift_up/sift_down_range, vendored and specialised: min-order over a raw u64 key, so no Reverse wrapper around the packed (rank, index). Note the "hole" form too -- shift into the hole, write once -- since it reads like an unrolled swap otherwise. --------- Co-authored-by: SBrandeis <33657802+SBrandeis@users.noreply.github.com>
Brings the branch up to the clean base (15 commits, including the WordCache merged as af5a3e3 / #2262) without rewriting history, so the PRs stacked on this branch stay valid. The PR diff now shows this branch's own changes rather than re-adding files the base already has. Both branches had reorganised the same modules since the fork on 31 Jul, so the conflicts were structural rather than line-level. How each was resolved: - `word.rs`, `serialization.rs` -> the base moved them into `legacy/`, this branch renamed them to `legacy_word.rs` / `legacy_serialization.rs`. Verified the base's move is content-identical to the fork point, so the flat names are kept and the `legacy/` copies dropped: a rename collision, nothing lost. - `model.rs` -> the base rewrote it (+228/-1562) while this branch split it into `bpe_model.rs` and `legacy_model.rs`. This branch's split is kept. - `word_cache.rs` -> the two copies have ZERO shared commits (the squash-merge of #2262 removed the common ancestry) and differ by 939/782 lines. This branch's version is kept: its pipeline, model and merge code are written against it, and it is the version every benchmark on this branch measured. - The remaining 12 conflicted files are this branch's own work and keep its side. Worth knowing: keeping this branch's `word_cache.rs` means any fix that landed in #2262 during review and is absent here is NOT picked up by this merge. The two implementations need reconciling properly before this branch merges down, and this commit does not do that. Verified after the merge: builds clean, 391 tests pass, and the full benchmark matrix is unchanged at 203 cells byte-exact against tokenizers 0.23.1 with 29 mismatched -- the same albert (Unigram) cells that already failed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Supersedes #2272. This is the same intent — one tree with the optimization work in it — but
built on the current cache stack and measured end to end against gigatoken on this machine.
Not for merge as-is:
bitsplitis vendored here astokenizers/bitsplit. It belongs in its ownlibrary; this PR exists so the end state can be measured instead of argued about, and so the
individual PRs underneath it can be reviewed knowing what they add up to.
What is in it
memcpyMeasured
Full encode, 4 MB inputs, single thread, warm cache, aarch64,
lto = "fat". gigatoken 0.10 builtand run on the same machine, same corpora, same tokenizer.json — token counts match it exactly on
all three models, so this is like for like.
Geomean vs gigatoken: gpt2 1.14×, llama-3 1.24×. Ahead overall; within 7–10% on its two best
cells, well ahead on everything with long pre-tokens.
For scale,
feat/bpe-cache-minon the same bench is english 368 / code 178, so this is 3.3× thebase on English and 3.8× on code.
Reproduce with
cargo run --release --example stage_bench(included), which prints the marginalns/B of each stage.
On the CI benchmark
The first push of this branch regressed it: the transpose was NEON-only, the CI runner is x86_64,
so the routing handed it bitsplit's byte-at-a-time fallback (split 0.89 ns/B) in place of the FSM
(0.64). Fixed twice over — there is now an SSSE3 builder, and
bitsplit::fast_builder()gates therouting so any target without a SIMD build keeps the FSM.
Where the remaining gap is
With the model stubbed to answer every pre-token with one token, the per-pre-token call structure
measured 0.64 ns/B before #2278 — gigatoken's entire encode — and is 0.09 now. What is left
on English is split 0.33 + model 0.47; matching gigatoken outright needs the split cheaper still,
which is bitsplit's own roadmap (o200k and the class-run family have no bitstream grammar yet, so
atomsplit::fsmcannot be retired).Dead ends, measured, so nobody repeats them
batched. The table is L1-resident — 512 slots performs the same as 262144.
disable it and English collapses 1554 → 32 MB/s. Our merge is 1.8–6.0× faster than theirs uncached.
PipelineTokenizer benchmark
9 / 10 models supported — PipelineTokenizer vs
tokenizersv0.23.1 (latest release) · ~10 kB inputs · add_special_tokens on · single thread + 1/2/4/8/max-thread sweep86bd5faa7 · 2026-08-04 11:15 UTC· Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 32 coresvs base branch (
e87a72c56) — per-model geomean ×speedup of this PR's PipelineTokenizer against the base branch's; regressions in red.Decode
Round-trip: v0.23.1
encode_fastproduces the id streams (same fixtures,add_special_tokens=true); both implementations decode those SAME ids withskip_special_tokens=false. MB/s counts decoded text bytes.bert-base-uncased — normalizer-heavy WordPiece · ×8.72 vs v0.23.1 · ×1.11 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 12+0 (peak 12) · Pipeline 8+2 (peak 17)
deepseek-v4 — deepseek 3-regex split-heavy byte-level BPE · ×25.03 vs v0.23.1 · ×3.99 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 62+0 (peak 68) · Pipeline 82+0 (peak 82)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gemma-4 — byte-fallback BPE, Metaspace-style split (gemma-4) · ×1.85 vs v0.23.1 · ×0.41 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 304+0 (peak 371) · Pipeline 275+0 (peak 371)
gpt2 — gpt2 ByteLevel regex · ×35.09 vs v0.23.1 · ×3.03 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 25+2 (peak 27) · Pipeline 27+0 (peak 28)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).gpt-oss — o200k-regex byte-level BPE (gpt-oss) · ×20.66 vs v0.23.1 · ×2.74 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 241+0 (peak 315) · Pipeline 234+0 (peak 316)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).glm-5.2 — cl100k-variant regex byte-level BPE (glm-5.2) · ×24.40 vs v0.23.1 · ×2.70 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 169+0 (peak 231) · Pipeline 170+0 (peak 231)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).llama-2 — model-bounded BPE, no pre-tokenizer · ×4.66 vs v0.23.1 · ×0.59 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 16+0 (peak 20) · Pipeline 24+0 (peak 24)
llama-3 — cl100k-regex byte-level BPE (llama-3), single regex · ×29.54 vs v0.23.1 · ×2.79 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 73+0 (peak 95) · Pipeline 92+0 (peak 95)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).mistral-small-4 — tekken byte-level BPE, 1k added specials (mistral-small-4) · ×9.15 vs v0.23.1 · ×1.01 vs base · decode pending
Memory (RSS MB, load+encode): v0.23.1 152+0 (peak 193) · Pipeline 109+0 (peak 195)
Pre-tokenize:
classify + fsmvs regex engines — ns/byte, lower better. The fsm is the scalar jump-table in both pipe columns; SIMD / scalar is the classify pass (regex pre-tokenizers have no SIMD fsm).×vs= engine ÷ our pipeline (SIMD / scalar classify);onig&pcre2(JIT) are C,fancyis pure-Rust fancy-regex,logosis a compile-time DFA lexer (approximate grammar; n/a for deepseek).Not yet supported:
t5-base