[SBrandeis]

TL;DR

Introduce TokenizerSpec backing the Tokenizer pyclass in the Python bindings.

TokenizerSpec mirrors the tokenizer.json config in memory, and compiles a PipelineTokenizer from it.

Benefits:

• Recompile on change. When the pipeline config changes (say, a new normalizer at runtime), we detect it and recompile the PipelineTokenizer on the next encode. Changing padding or truncation never recompiles.
• Cheap copies. Every value sits behind an Arc, and a copy gets a reference to the compiled pipeline. copy.deepcopy no longer goes through __reduce__ and a full re-compilation.
• serialize/deserialize identity writing the tokenizer to disk outputs the same json it was loaded from
• Unified handling of padding and truncation They live in the spec instead of two extra mutexes on the pyclass
• Pickle and repr now use the JSON config

Visuals

	Diagram	What it shows
1		The JSON is the source of truth. The compiled tokenizer is a cache built from it.
2		Edits don't compile. The next encode does, once.
3		Components are shared, like in 0.23.1. Editing one updates every tokenizer using it.
4		Encodes run without the lock or the GIL. Only a recompile makes other encodes wait.
5		Slot → cell → JSON. The first edit after a compile copies the JSON.
6		What each operation copies.

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.