[nov3o]

A compact version of @Yikai-Liao's parallel BPE trainer in #2501, for the review question there of how small the engine can be: ~470 lines of code (+544 with comments) instead of ~14,500, with the same merges. It keeps the three ideas that give the speed:

1. Fixed coordinates: every symbol has one slot in a flat corpus, and a token stores its ID at its first and last slot. A merge rewrites four slots and nothing in the word moves.
2. Per-pair occurrence lists that are checked against the corpus when used, never cleaned. Counts are exact, and a lazy max-heap gives the same merge order as do_train: highest count first, ties to the smaller pair.
3. Ordered-prefix batching (perf(bpe trainer): 10–32× faster with 8 threads #2501 DESIGN.md, "Why batching preserves rule priority"). Consecutive rules are merged together when they have new IDs, no (A, A) rule and no crossed endpoints. The corpus is cut at word boundaries, one chunk per thread, and pair changes are committed by shard in parallel.

Everything else in #2501 is left out: compressed lists, radix grouping, active-ID reuse mode, the parallel feed, affixes, the alphabet limit and the birth length rule. With an affix, limit_alphabet or max_token_length, do_train falls back to the existing trainer, unchanged. A rule whose result already exists, or (A, A), runs alone.

It also adds do_train_entries(Vec<(CompactString, u64)>), to train from counts the caller already made, without a map.

Exact: vocab and merges are identical to main and to #2501 on 48 settings (4 datasets × 11 pre-tokenizers, plus NFKC + lowercase, 100 MB each) and in all 40 runs of #2501's harness. Existing tk-train tests pass.

Results

These use #2501's own harness (tokenizers-bpe-benchmarks @ 0fa8d08): 512 MiB per corpus, vocab 100,000, min_frequency = 2, 8 workers, median of 3 runs. Machine: 2× AMD EPYC 7742, pinned cores.

Feed + train, seconds (lower is better):

workload	main	#2501	this PR	vs main	vs #2501
Code / ByteLevel	30.64	4.66	8.72	3.5× faster	1.9× slower
English / ByteLevel	13.79	2.86	3.91	3.5× faster	1.4× slower
Chinese / ByteLevel	471.8	13.89	24.96	19× faster	1.8× slower
Chinese / Whitespace	211.4	7.47	21.04	10× faster	2.8× slower

Train only, seconds:

workload	main	#2501	this PR	vs main	vs #2501
Code / ByteLevel	23.63	1.36	2.07	11× faster	1.5× slower
English / ByteLevel	10.83	0.78	1.17	9.3× faster	1.5× slower
Chinese / ByteLevel	463.5	11.37	17.68	26× faster	1.6× slower
Chinese / Whitespace	205.5	5.42	14.61	14× faster	2.7× slower

Peak memory, GiB:

workload	main	#2501	this PR	vs main	vs #2501
Code / ByteLevel	2.48	0.50	1.02	−59%	2.0×
English / ByteLevel	1.24	0.24	0.51	−59%	2.1×
Chinese / ByteLevel	36.96	5.10	14.44	−61%	2.8×
Chinese / Whitespace	18.75	2.70	8.68	−54%	3.2×

Memory growth with input size, 8 workers (main: one point, at 512 MiB):

The gap to #2501 has three causes:

• this PR keeps main's serial feed, where perf(bpe trainer): 10–32× faster with 8 threads #2501 counts words in parallel;
• the occurrence lists are plain Vec<u32>, where perf(bpe trainer): 10–32× faster with 8 threads #2501 compresses them;
• committing pair changes after each batch is slower.

That is the trade: under 4% of the code for 35–75% of #2501's speed.

Broader benchmarks (threads, vocab, 11 pre-tokenizers, 10 normalizers, 1 and 10 GB, 4 datasets) are in nov3o/toki.

[nov3o]

@ArthurZucker hey, could you revisit this one please?