Fix int8 quantization (Linear8bitLt) and document save/load - fixes #1012, #921

[joan8627]

Summary

Both #1012 and #921 trace back to the same thing: the int8 tutorial and use-bnb-lineardocs are built aroundbitsandbytes.nn.triton_based_modules SwitchBackLinear), which is no longer installable on a fresh setup:

• triton==2.0.0.post1 (what those kernels were written against) has been from PyPI entirely - every triton version you can actually pip installtoday reorganizedtriton.language.libdeviceaway, which is theAttributeError: module 'triton.language' has no attribute` from error when use try to use int8 operations with OpenCLIP. #1012.
• bitsandbytes>=0.50 dropped triton_based_modules outright (verified by released wheels: present through 0.49.2, gone in 0.50.0).

So this isn't a version-pinning problem, it's a dead code path for anyone from scratch now. This PR moves int8 inference onto bnb.nn.Linear8bitLt(bitsandbytes' actively-maintained LLM.int8() layer, triton dependency at all), fixes two real bugs inreplace_linear/ convert_int8_model_to_inference_mode that surfaced while getting that path working end-to-end, and documents save/load (#921) with a runnable example.

What changed

src/open_clip/utils.py

• replace_linear: the bias copy (bias.data.copy_(...)) silently kept replacement class's own default bias dtype (float32) instead of adopting the source module's dtype. Harmless for the old triton path's typical usage, but a real bug for anyone quantizing a model with any replacement class - the stale fp32 bias upcasts that layer's output and corrupts dtype for everything down crashes the next LayerNorm with a dtype mismatch). Fixed by assigning the bias tensor directly (adopting the source dtype) instead of an in-place .copy_().
• convert_int8_model_to_inference_mode: only handled prepare_for_eval() method (the triton classes). Linear8bitLt quantizes its own weight on .cuda() with no such method, so get_weight_dtype() (transformer.py) fell through to weight.dtype - literally torch.int8 post-quantization - and that dtype leaked into activation casts (x = self.token_embedding(text).to(cast_dtype)), crashing at the first LayerNorm. Added a duck-typed branch (weight.dtype stamps int8_original_dtypethe same way, recovered from the still-unquantized bias (or an explicitdtype` arg, now accepted).

README.md - rewrote the "Int8 Support" section: `example, measured (not estimated) accuracy/speed numbers, save/load guidance, and an explicit note on why the old triton path is dead and not fixable by pinning.

tutorials/int8_tutorial.ipynb - reworked around Linear8bitLt:

• Drops the triton install cell entirely (not needed for this path).
• Replaces the old training-speed benchmark narrative (triton vs. autograd baseline) with an honest latency comparison - int8 comes out ~5.6% slower here, not faster, with an explanation of why (activation quantize/dequantize adds kernel launches that don't pay for themselves at CLIP-ViT-B/32 scale; Linear8bitLt was designed around much larger models).
• New "How much accuracy do we lose?" section: real zero-shot CIFAR-10 eval, fp16 vs. int8.
• New "Save and load a quantized model" section (closes How to persist int8 model? #921), including a round-trip correctness check (not just "it didn't crash") - reloaded model matches the original exactly (0.0 max abs diff on both embeddings).

Measured results (ViT-B-32-quickgelu, openai weights, RTX 3080)

	fp16	int8 (Linear8bitLt, c_fc/c_proj)
CIFAR-10 zero-shot accuracy	88.76%	88.83% (**+0.0
Latency, batch=128	53.9ms	56.9ms (-5.6%, i.e. slower)

Single template/dataset/model - a sanity check, not a comprehensive benchmark, but real and reproducible (see the notebook). The practical benefit of this path is memory (~2x reduction on the quantized layers' weight tensors - only c_fc/c_proj are covered, ~33% block's total linear-layer weight memory; see the TODO in utils.py for attention QKV, which isn't a swappable nn.Linear submodule and is out of scope here), not speed or accuracy.

Out of scope (not attempted here)

• Attention's QKV projection (in_proj_weight) isn't an nn.Linear submodule in open_clip.transformer.Attention - quantizing it needs either a checkpoint-compat-sensitive refactor or a bespoke wrapper around its F.linear call, and touches every vision/text tower in the repo. Left as a follow-up (existing TODO in utils.py).
• No replacement is provided for the training-time SwitchBack speedup path (--use-bnb-linear witchBackLinearGlobal*) - Linear8bitLt is an inference-oriented layer. Flagged as unmaintained in README than silently left broken.

Test plan

• Reproduced error when use try to use int8 operations with OpenCLIP. #1012 in isolation (SwitchBackLinear forward → identical tl.libdevice AttributeError) and confirmed Linear8bitLt runs clean on the same input, with genuine weight quantization (fp32 -> int8 on .cuda(), SCB scale state populated, byte count halved).
• Verified replace_linear + Linear8bitLt numerically tracks an fp32 reference on a toy MLP.
• Full CIFAR-10 zero-shot classification, fp16 vs. int8, via the actual open_clip.utils.replace_linear / convert_int8_model_to_inference_mode APIs (not a workaround).
• Backward-compat: replace_linear/convert_int8_model_to_inference_mode still behave correctly for the existing triton class (construction) and for a no-op plain-nn.Linear case.
• Executed the full tutorial notebook end-to-end via jupyter nbconvert --execute - all cells run clean, including the save/load round-trip correctness check.

[joan8627]

The CI failure (tests/test_coca2.py::test_modern_coca_generate, AttributeError: 'MultimodalGenerationWrapper' object has no attribute ) is unrelated to this PR - this diff never touched generation.pyorcoca_model.py`.

Root cause: transformers' latest release.
requirements-test.txt doesn't pin an upper bound on transformers, so CI installs whatever's latest at run time. Installing latest transformers today pulls transformers==5.14.1, which added an call to self.get_experts_implementation()insideGenerationMixin.generate() (likely part of newer MoE-model support). MultimodalGenerationWrapper only implements nn.Module + GenerationMixin the full transformers.PreTrainedModel), so it doesn't have that Reproduced locally on this branch, unmodified generation.py/ coca_model.py, confirming it's an upstream transformers` version bump 2026-07-17 and now, not anything in this diff.

Timeline:

• PR Tokenizer cleanup #1196 (2026-07-15) passed all 4 test-group CI jobs.
• main's last CI run (2026-07-17, commit a3c2605, which this branch is on) also passed.
• No commits landed on main since, so no CI ran against this codebase until this PR, 13 days later - there's no scheduled/nightly job to dependency drift in between.

Opening a new issue for this.

[joan8627]

Opened #1199.
Tests should pass after that fix.