Repository navigation
feat(pipeline): Support Metaspace pretokenizer (T5, albert) - #2264
Merged
Merged
Conversation
`memmem::Finder` carries a few hundred bytes of prefilter state on x86_64 (much less on aarch64, which is why this only showed up in CI): a `Literal` stored inline blew `NormalizerWrapper` and `DecoderWrapper` up to 352 bytes through `Replace`, which `clippy::large_enum_variant` rejects.
The Python bindings search with a `&String` pattern — `NormalizedString.replace` and `.split` take a plain `str` — so dropping these impls broke every job that builds the bindings. Reinstating them through `Literal` keeps the public API and still drops the regex engine from the literal path: they used to escape the string and compile a regex on every call. An empty pattern now covers the input by byte length rather than character count, like every other impl (the old count sliced mid-character).
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
SBrandeis
marked this pull request as ready for review
July 29, 2026 16:52
McPatate
approved these changes
Jul 30, 2026
| // Whitespace is thrown away, so cut the text where `WhitespaceSplit` would and write the | ||
| // words back one after the other, each with its own delimiter. | ||
| let mut words = Vec::new(); | ||
| pipeline::PreTokenizer::pre_tokenize(&WhitespaceSplit, input, &mut words)?; |
Member
There was a problem hiding this comment.
Feels weird to call pretokenize here, but I understand the reuse
Base automatically changed from
feat/literal-patterns
to
feat/train_encode_split
July 31, 2026 14:15
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TL;DR
The released Metaspace pretokenizer is in fact a Normalizer (turn whitespaces into the
▁delimiter) + a Split pretokenizerThis PR adds support for this pretokenizer