Skip to content

Commit e0912bd

Browse files
authored
Update tokenizers submodule to 3fc1df7 (pytorch#23028)
### Summary Update `extension/llm/tokenizers` from `1d7ca636` to [`3fc1df71`](meta-pytorch/tokenizers@3fc1df7), the current upstream main commit. This brings in Unicode 17 NFC normalization fixes, reliable linking of the lookahead regex fallback, and the trie-based special-token matcher. [Upstream changes](meta-pytorch/tokenizers@1d7ca63...3fc1df7). ### Test plan Debug build on Apple M1 Pro succeeded. 14/15 CTest executables passed, including `test_normalizer` and `regex_lookahead_link_test`. `test_hf_tokenizer` could not create its temporary fixtures in `/var/tmp` because the local harness denies writes there; an independent write probe confirmed `Operation not permitted`. Upstream macOS, Linux, and Windows CI passed for the pinned commit. ```sh cmake -S extension/llm/tokenizers/test -B /tmp/tokenizer-tests -DCMAKE_BUILD_TYPE=Debug -DCMAKE_POLICY_VERSION_MINIMUM=3.5 cmake --build /tmp/tokenizer-tests --parallel 8 ctest --test-dir /tmp/tokenizer-tests --output-on-failure ``` Authored with OpenAI Codex.
1 parent 9e216c7 commit e0912bd

1 file changed

Lines changed: 1 addition & 1 deletion