v1.0.0

Issues

• v1.0.0-rc.2: canonicalize_file rejects ByteLevel add_prefix_space=true supported by 0.23.2

  huggingface/tokenizers#2460 · Filter by author michaelfeil opened on Sep 26, 2026 · v1.0.0

• feat(wasm): publish pipeline

  Unread
  huggingface/tokenizers#2497·SBrandeis opened Oct 8, 2026 · Updated Oct 8, 2026
  1

• Support ByteLevel add_prefix_space during canonicalization

  Unread
  huggingface/tokenizers#2462·ducanhnguyen223 opened Sep 27, 2026 · Updated Oct 13, 2026
  5

• [1.0.0rc2] Possible regressions vs. 0.23.x: Byte atom not found in the vocabulary

  huggingface/tokenizers#2447 · Filter by author xenova opened on Sep 23, 2026 · v1.0.0
  4

  

• feat(python): role_to_token

  Unread
  huggingface/tokenizers#2454·SBrandeis opened Sep 24, 2026 · Updated Oct 8, 2026
  1

• feat(python): composition

  Unread
  huggingface/tokenizers#2414·SBrandeis opened Sep 16, 2026 · Updated Oct 8, 2026·Draft

• feat: experimental C bindings

  Unread
  huggingface/tokenizers#2375·SBrandeis opened Sep 1, 2026 · Updated Oct 8, 2026
  5

• Atomnorm

  target
  Unread
  huggingface/tokenizers#2209·ArthurZucker opened Jul 16, 2026 · Updated Oct 8, 2026
  1

• Draft, manual implem of bitsplit :)

  Unread
  huggingface/tokenizers#2340·ArthurZucker opened Aug 18, 2026 · Updated Oct 8, 2026·Draft
  1

• Don't emit control tokens in untrusted text segments

  huggingface/tokenizers#2498 · Filter by author SBrandeis opened on Oct 8, 2026 · v1.0.0
  3

• [v1] Python bindings are built without a regex backend: regex Split / Replace fail

  huggingface/tokenizers#2511 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] meta-issue: required work for integration with transformers

  huggingface/tokenizers#2528 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Normalized added token at the start of the input gets a stray ▁ with Metaspace

  huggingface/tokenizers#2512 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] A vocabulary containing an empty-string token fails to load

  huggingface/tokenizers#2513 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Added-token ids differ from 0.23 when added_tokens are listed out of id order

  huggingface/tokenizers#2514 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Composition API: build a tokenizer from components, and get/set them

  huggingface/tokenizers#2515 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Serialization: to_str(), save(path), from_str(json)

  huggingface/tokenizers#2516 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Vocabulary lookups

  huggingface/tokenizers#2517 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0
  

• [v1] Added tokens API

  huggingface/tokenizers#2518 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Encoding.tokens, with the original text for unknown pieces

  huggingface/tokenizers#2519 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] DecodeStream

  huggingface/tokenizers#2520 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Offsets and word alignment

  huggingface/tokenizers#2521 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] special_tokens_mask

  huggingface/tokenizers#2522 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Overflowing tokens with stride

  huggingface/tokenizers#2523 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0

• [v1] Decode a list of token strings

  huggingface/tokenizers#2524 · Filter by author SBrandeis opened on Oct 14, 2026 · v1.0.0