All pull requests

Pull requests

• Restore Python DecodeStream for the v1 pipeline

  #2532·starx710 opened Oct 15, 2026·

  ·
  1

• Expose token-string decoding in Rust and Python

  #2531·starx710 opened Oct 15, 2026·

  ·
  1

• Fix WordPiece loading with empty vocabulary tokens

  #2530·snowyukitty opened Oct 14, 2026·

  ·
  1

• Fix Metaspace prefix before normalized added tokens

  #2529·starx710 opened Oct 14, 2026·

  ·
  1

• chore(tk-encode): drop the legacy Encoding

  #2509·SBrandeis opened Oct 14, 2026Collaborator·

  ·
  1

• perf(normalizers): Improve unicode and BERT normalizers

  #2508·SBrandeis opened Oct 13, 2026Collaborator·

  ·

  · v1.0.0
  1

• perf(bindings): speed up Python decode for NumPy int32 and int64 inputs

  #2507·kozistr opened Oct 13, 2026·

  ·

• fix(metaspace): keep internal spaces in the first decoded token

  #2506·aalquwayfili opened Oct 11, 2026·

  ·

• docs: correct canonical Metaspace prepend type

  #2505·j45856021-dev opened Oct 11, 2026·

  ·

• docs: correct v1 Unigram file API status

  #2504·original4422 opened Oct 11, 2026·

  ·
  2

• docs: add Simplified Chinese translation (README.zh.md)

  #2503·JasonYeYuhe opened Oct 10, 2026·

  ·
  1

• perf(bpe trainer): 10–32× faster with 8 threads

  #2501·Yikai-Liao opened Oct 9, 2026·

  ·
  3

• Share the pipeline worker pool across concurrent encode calls

  #2500·ZenAlexa opened Oct 8, 2026·

  ·
  1

  1

• feat(wasm): publish pipeline

  #2497·SBrandeis opened Oct 8, 2026Collaborator·

  ·

  · v1.0.0
  1

• perf(word_cache): implement the shrn trick for bitmask

  #2496·SBrandeis opened Oct 7, 2026Collaborator·

  ·
  1

• chore(security) pin dtolnay/rust-toolchain to v1

  #2495·SBrandeis opened Oct 7, 2026Collaborator·

  ·
  1

• chore(deps): bump the actions group across 1 directory with 5 updates

  dependenciesgithub_actions
  #2494·dependabot[bot] opened Oct 7, 2026Bot·

  ·
  1

• (wip) feat(tk-train): refactor and unify

  #2487·SBrandeis opened Oct 2, 2026Collaborator·

  ·
  1

• fix(added-vocab): keep the original content of normalized added tokens

  #2485·MohammadHijjawi97 opened Oct 1, 2026·

  ·
  5

• fix(bpe): apply end_of_word_suffix to single-character words

  #2484·MohammadHijjawi97 opened Oct 1, 2026·

  ·
  4

• docs: using the crate and its features

  #2481·SBrandeis opened Sep 30, 2026Collaborator·11/11·

  ·
  3

• chore(refactor): move progress bars from tk-encode to tk-train

  #2480·SBrandeis opened Sep 30, 2026Collaborator·10/11·

  ·
  2

• feat(umbrella): expose tk-train through a train feature

  #2479·SBrandeis opened Sep 30, 2026Collaborator·9/11·

  ·
  1

• chore(bindings): use the umbrella tokenizers crate instead of the tk-* sub crates

  #2478·SBrandeis opened Sep 30, 2026Collaborator·8/11·

  ·
  2

• feat(umbrella): rework umbrella crate feature set

  #2477·SBrandeis opened Sep 30, 2026Collaborator·7/11·

  ·
  1