The Python bindings' standalone pretokenization (pretokenizer(), pretokenized_counts()) is hardwired to the r50k (GPT-2) scheme, while the crate already implements and dispatches every other family internally (cl100k, o200k, qwen2/3.5, olmo3, deepseek_v3, nemotron, kimi). Exposing the scheme selection makes gigatoken usable as a pure pretokenization library — my use case: splitting text exactly the way gpt-oss/GPT-4o (o200k) do, without going through encode.
I have this implemented and tested on a branch, but PR creation is currently limited to collaborators, so filing an issue instead:
It is deliberately lean (+128/−30, of which +56 is tests) and adds one optional argument rather than new surface:
pretokenizer(text, pretokenizer="gpt2") and pretokenized_counts(text, separator=None, pretokenizer="gpt2") — same argument name, PretokenizerType::from_name parsing, and ValueError as BPETokenizer.from_tiktoken_model (the parse is now a shared helper, so lib.rs shrinks).- Backward compatible: default
"gpt2" keeps existing callers byte-identical (covered by a test).
- Dispatch reuses the existing
FastPretokenizerDispatch: a new PretokenizerType::pretokenize_at(bytes, pos) + FastPretokenizerDispatch::pos() preserve the binding iterator's resume-at-offset pattern, and pretokenize delegates to pretokenize_at(bytes, 0) so no match arms are duplicated.
pretokenize_par_bytes gains the scheme parameter; internal train/file-source callers pass GPT2 explicitly, unchanged in behavior.- Tests validate o200k output against the verbatim o200k_base split regex from openai/gpt-oss-20b's tokenizer.json (contractions, \p{N}{1,3}, unicode, whitespace/newline runs, the
/ tail absorption), plus default-unchanged and unknown-name error tests; type stubs updated. Full pytest suite: 1445 passed / 16 skipped (the pre-existing local-data skips); clippy warning count unchanged.
Happy to open the PR if you lift the restriction or add me as a collaborator — or feel free to cherry-pick the branch directly.
AI disclosure (per the README): this change was written with Claude; code and tests were reviewed and verified by a human.
The Python bindings' standalone pretokenization (
pretokenizer(),pretokenized_counts()) is hardwired to the r50k (GPT-2) scheme, while the crate already implements and dispatches every other family internally (cl100k, o200k, qwen2/3.5, olmo3, deepseek_v3, nemotron, kimi). Exposing the scheme selection makes gigatoken usable as a pure pretokenization library — my use case: splitting text exactly the way gpt-oss/GPT-4o (o200k) do, without going through encode.I have this implemented and tested on a branch, but PR creation is currently limited to collaborators, so filing an issue instead:
It is deliberately lean (+128/−30, of which +56 is tests) and adds one optional argument rather than new surface:
pretokenizer(text, pretokenizer="gpt2")andpretokenized_counts(text, separator=None, pretokenizer="gpt2")— same argument name,PretokenizerType::from_nameparsing, andValueErrorasBPETokenizer.from_tiktoken_model(the parse is now a shared helper, so lib.rs shrinks)."gpt2"keeps existing callers byte-identical (covered by a test).FastPretokenizerDispatch: a newPretokenizerType::pretokenize_at(bytes, pos)+FastPretokenizerDispatch::pos()preserve the binding iterator's resume-at-offset pattern, andpretokenizedelegates topretokenize_at(bytes, 0)so no match arms are duplicated.pretokenize_par_bytesgains the scheme parameter; internal train/file-source callers passGPT2explicitly, unchanged in behavior./tail absorption), plus default-unchanged and unknown-name error tests; type stubs updated. Full pytest suite: 1445 passed / 16 skipped (the pre-existing local-data skips); clippy warning count unchanged.Happy to open the PR if you lift the restriction or add me as a collaborator — or feel free to cherry-pick the branch directly.
AI disclosure (per the README): this change was written with Claude; code and tests were reviewed and verified by a human.