Repository navigation
Commit e0912bd
authored
Update tokenizers submodule to 3fc1df7 (pytorch#23028)
### Summary
Update `extension/llm/tokenizers` from `1d7ca636` to
[`3fc1df71`](meta-pytorch/tokenizers@3fc1df7),
the current upstream main commit. This brings in Unicode 17 NFC
normalization fixes, reliable linking of the lookahead regex fallback,
and the trie-based special-token matcher. [Upstream
changes](meta-pytorch/tokenizers@1d7ca63...3fc1df7).
### Test plan
Debug build on Apple M1 Pro succeeded. 14/15 CTest executables passed,
including `test_normalizer` and `regex_lookahead_link_test`.
`test_hf_tokenizer` could not create its temporary fixtures in
`/var/tmp` because the local harness denies writes there; an independent
write probe confirmed `Operation not permitted`. Upstream macOS, Linux,
and Windows CI passed for the pinned commit.
```sh
cmake -S extension/llm/tokenizers/test -B /tmp/tokenizer-tests -DCMAKE_BUILD_TYPE=Debug -DCMAKE_POLICY_VERSION_MINIMUM=3.5
cmake --build /tmp/tokenizer-tests --parallel 8
ctest --test-dir /tmp/tokenizer-tests --output-on-failure
```
Authored with OpenAI Codex.1 parent 9e216c7 commit e0912bd
1 file changed
Lines changed: 1 addition & 1 deletion
Submodule tokenizers updated 21 files
- .github/workflows/pull.yml+5
- BUCK+2-4
- CMakeLists.txt+1
- include/pytorch/tokenizers/bpe_tokenizer_base.h+20-11
- include/pytorch/tokenizers/hf_tokenizer.h+1-1
- include/pytorch/tokenizers/special_token_matcher.h+125
- include/pytorch/tokenizers/tiktoken.h+1-6
- pytorch_tokenizers/__init__.py+1-1
- setup.py+1-1
- src/normalizer.cpp+1-6
- src/regex.cpp+24-4
- src/tiktoken.cpp+13
- test/CMakeLists.txt+28
- test/regex_lookahead_link_test.cpp+23
- test/test_normalizer.cpp+69
- test/test_python_bindings.py+1-1
- test/test_tiktoken.cpp+30-34
- third-party/llama.cpp-unicode/README.md+8-4
- third-party/llama.cpp-unicode/generate_nfc_data.py+95
- third-party/llama.cpp-unicode/include/unicode-nfc-data.h+4.1k
- third-party/llama.cpp-unicode/src/unicode.cpp+77-197
0 commit comments