Tokenizer cleanup

README.md

@@ -45,6 +45,7 @@
 > - CoCa's exact-mask API adds `text_valid` after `text` in `encode_text`, `forward`, and `forward_intermediates`. This shifts the older trailing positional arguments (`normalize`, `image_latent`, `image_indices`, and so on); pass those arguments by keyword. For example, replace `model.encode_text(text, False)` with `model.encode_text(text, normalize=False)`.
 > - `CLIPTextCfg.eos_id` no longer defaults to `2` (that value is only correct for XLM-style vocabs). Configs using `pool_type="eos"` must set `eos_id` explicitly, and `get_tokenizer` now validates `eos_id`/`pad_id` against the resolved tokenizer, raising on mismatch instead of pooling/masking silently wrong positions.
 > - `HFTokenizer` no longer fabricates `pad_token_id=0` when the underlying tokenizer has no pad token (id 0 is a real token in most BPE vocabs); variable-text setups fail fast instead. It also forces `padding_side='right'`, which all OpenCLIP pooling/masking assumes.
+> - Tokenizer wrappers now share special-token controls: `encode(..., add_special_tokens=False)` remains body-only by default, while model-facing `tokenizer(...)` defaults to `add_special_tokens=True`. `decode()` / `batch_decode()` default to `skip_special_tokens=False, stop_at_eos=True`; pass `stop_at_eos=False` to inspect tokens after the first EOS. This intentionally changes legacy SimpleTokenizer decode output by hiding post-EOS id-0 fill (`!`) and makes TikToken decode render its reserved control tokens unless `skip_special_tokens=True`.
 > - `MaxPooler` (`hf_pooler_type="max_pooler"`) mask polarity fixed — it previously max-pooled over the padding positions instead of the valid ones.
 > - `CoCa.__init__` no longer takes a `pad_id` argument — `model.pad_id` is derived from the text tower (the id it actually masks with: `text_cfg.pad_id` for native towers, the transformers config pad for HF towers). MaMMUT follows the same pattern. This fixes `coca_roberta-*`, which previously masked with roberta's pad (1) in the tower while the loss ignored 0 — its config now declares `pad_id: 1` and the caption loss no longer trains on padding.
 > - `CoCaTask` builds caption labels masked to `-100`; `CoCaLoss`'s cross-entropy uses `ignore_index=-100`. Its `pad_id` arg is retained for standalone callers passing raw labels (default `0` preserves the old value-based behavior; the task path passes `None`). Validation generative-loss metrics are likewise mask/pad-aware and will report different (correct) values for nonzero-pad models.

src/open_clip/__init__.py

@@ -52,7 +52,7 @@
     download_pretrained,
 )
 from .push_to_hf_hub import push_pretrained_to_hf_hub, push_to_hf_hub
-from .tokenizer import SimpleTokenizer, tokenize, decode
+from .tokenizer import Tokenizer, SimpleTokenizer, tokenize, decode, batch_decode
 from .transform import image_transform, AugmentationCfg
 from .zero_shot_classifier import build_zero_shot_classifier, build_zero_shot_classifier_legacy
 from .zero_shot_metadata import OPENAI_IMAGENET_TEMPLATES, SIMPLE_IMAGENET_TEMPLATES, IMAGENET_CLASSNAMES

src/open_clip/factory.py

@@ -33,7 +33,7 @@
     merge_preprocess_kwargs,
     naflex_eval_transform_v2,
 )
-from .tokenizer import HFTokenizer, SimpleTokenizer, SigLipTokenizer, TikTokenTokenizer, DEFAULT_CONTEXT_LENGTH
+from .tokenizer import HFTokenizer, SimpleTokenizer, SigLipTokenizer, TikTokenTokenizer, Tokenizer, DEFAULT_CONTEXT_LENGTH
 
 HF_HUB_PREFIX = 'hf-hub:'
 _MODEL_CONFIG_PATHS = [Path(__file__).parent / f"model_configs/"]
@@ -835,7 +835,7 @@ def get_tokenizer(
         context_length: Optional[int] = None,
         cache_dir: Optional[str] = None,
         **kwargs, # Additional tokenizer kwargs passed to constructor
-):
+) -> Tokenizer:
     """
     Gets the appropriate tokenizer based on the model identifier schema or name.