[ZenAlexa]

Summary

Fixes #2499.

Keep concurrent pipeline encodes on the same cached Rayon pool. The current try_lock() path treats a busy cache as a cache miss, constructs another pool, and replaces the shared slot. A burst can therefore create many more workers than the configured pool size, including after warm-up.

Serialize construction and cache lookup within a process-owned cache. Keep fork recovery independent of the inherited mutex: the child callback clears the cache pointer, and a PID check also covers a fork during callback registration. Published caches retain process lifetime; the child leaves the inherited lock and pool untouched. Recheck the configuration generation after construction, release retired pools outside the cache lock, and preserve serial fallback on construction failure.

The shared-pool guarantee applies to a fixed configuration and successful construction. In-flight work can retain a pool from an earlier configuration, and calling threads can participate in encoding.

Resource behavior

The issue reproducer uses GPT-2, 8 configured workers, four 114,688-byte texts per caller, and three fresh processes per cell. Every output ID matches serial encode_into. OS sampling counts distinct tk-encode-* worker TIDs; sampled peaks are lower bounds.

Concurrent callers	Cache state	Baseline worker peaks	This change
8	cold	29, 15, 58	8, 8, 8
8	warm	11, 16, 16	8, 8, 8
16	cold	8, 20, 22	8, 8, 8
16	warm	8, 8, 8	8, 8, 8

The added standalone lifecycle scenario checks exact pool identity under 16 concurrent callers, executes real Rayon jobs, and covers child reconstruction, parent reuse, disabling, re-enabling with an outstanding pool reference, and restoring the default. The same scenario fails on the baseline with concurrent calls constructed distinct shared pools and passes with this change.

Validation

Apple M4 Pro, macOS 26.4, Rust 1.99.0; baseline ff8ae1baabca0707b7fa9f22854652e231a933de.

From tokenizers/, with CARGO_INCREMENTAL=0 CARGO_PROFILE_RELEASE_LTO=off for these behavior checks:

cargo test --locked --release -p tk-encode --features parallelism --test parallelism_lifecycle
cargo test --offline --locked --release -p tk-encode --features parallelism --lib
cargo clippy --offline --locked --release -p tk-encode --features parallelism --all-targets -- -D warnings
cargo check --offline --locked --release -p tk-encode --no-default-features --lib

• Lifecycle scenario passed; existing library regressions: 192 passed, 1 ignored.
• Clippy, the feature-disabled build, formatting, and whitespace checks passed.
• Public encode(...).wait() scenarios preserved complete IDs through configuration changes, fork, cancellation, and subsequent reuse.
• Separate fault-injection scenarios exercised a fork with the inherited cache locked, a second fork generation, configuration changes during construction, and recovery after partial worker creation failed.
• Linux aarch64/musl, Rust 1.99.0: the same release lifecycle scenario passed in three fresh processes; the baseline failed all three with distinct shared pools. The existing library suite passed 192 tests, 1 ignored; the fork and failure scenarios also passed. Execution used a Linux 6.12.76 container limited to 4 CPUs and 2 GiB.
• Cross-target checks for Windows x86-64 MSVC from macOS, Rust 1.98.1: cargo +1.98.1 check --locked --release --target x86_64-pc-windows-msvc -p tk-encode --features parallelism --lib --test parallelism_lifecycle and the feature-disabled library check passed. These are compile-only checks; Windows linking and execution remain unmeasured.

Tokbench

Tokbench 20635869 used the same GPT-2 tokenizer and English, Chinese, and Arabic corpora for both builds. The current tk-* dependencies and EncodeOptions defaults were applied to the pipeline adapter; driver and core timing loops were unchanged. Both builds used optimization level 3, fat LTO, one codegen unit, and no debug information.

Three ABBA blocks produced six measurements per arm; the middle block used --reverse-scaling. Values below are medians, with 8 configured workers:

Native batch throughput	Baseline, MiB/s	This change, MiB/s
English	1358.01	1365.30
Chinese	582.64	580.30
Arabic	870.75	853.40

tokbench --models data/models --corpora data/fixtures --model gpt2 \
  --engine pipeline --reps 3 --padding off \
  --max-threads 8 --scaling-mode native-threads \
  --out native-scaling.json measure scaling

Concurrent latency, input pins, and measurement scope

A separate public-API consumer ran 8 callers, each with 16 distinct approximately 10 KiB documents. Cold runs included pool construction; steady runs used an independent warm-up batch. Makespan included barrier release, owned-input copying, submission, and completion, with output checking outside the timed region. Every measured batch was compared token-by-token with tokenizers 0.23.1.

Makespan	Baseline, µs	This change, µs
English cold	2273.62	1584.31
English steady	1220.25	1341.17
Chinese cold	3033.08	2709.23
Chinese steady	2026.04	2213.46
Arabic cold	2989.42	2546.85
Arabic steady	1964.00	1987.02

The short Arabic single-caller steady case also regressed, from 338.25 to 489.58 µs. A longer 128-document batch measured 1450.98 and 1466.15 µs. Separate worker-sampling runs confirmed extra cold workers on the baseline; their elapsed times are excluded from these tables.

Input pins: openai-community/gpt2@607a30d7 and huggingface/tokbench-corpora@bafa2fb4. The official native-scaling identity check is a separate serial pass; the concurrent consumer checks each actual timed batch.

The measurements establish the worker-resource improvement for these workloads; steady latency and throughput remain mixed on this desktop. Public tokenizer and performance checks cover macOS and GPT-2; the pool lifecycle and library suite also ran on Linux aarch64/musl. Windows runtime, glibc, and the full binding/workspace matrices remain outside the local validation.

Current mainline validation

Merged main fb9429ee into e99dece2 on 2026-10-23. The shared-pool implementation and lifecycle scenario are unchanged. Main's umbrella-crate, feature and binding refactors retain the Python and Node encoding path through tk-encode/parallelism; Wasm keeps its serial configuration.

On macOS arm64 with Rust 1.99.1, the locked Release lifecycle scenario passed, covering concurrent construction, reuse, fork recovery, disabling and re-enabling. The current library suite passed 193 tests with one ignored case; formatting passed.

The Python binding build and the 198-encoding, 16-caller observation reported on 2026-10-19 remain tied to e5d89e92. They were not rerun after the latest binding dependency changes. The earlier benchmarks also retain their recorded revisions; current binding suites and the platform matrix remain for project CI.

Project CI: Please run the repository Rust, Python, Node, and Wasm workflows, including native Windows execution and Linux fork lifecycle coverage. The local lifecycle results and cross-target compilation above are ready for review.

[ZenAlexa]

@SBrandeis, could you approve the pending Rust, Python, Node, and Wasm workflows for the cross-platform checks?

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[SBrandeis]

Hi!
Thanks for your contribution
Sorry for the delay in reviewing, we'll try to get a look into this soon

[ZenAlexa]

Hi! Thanks for your contribution Sorry for the delay in reviewing, we'll try to get a look into this soon

Thank you for your follow-up, whenever you're convenient