Gemma3: follow the projection weight dtype at forced-float32 Linear boundaries - #1017
Conversation
…oundaries Full finetuning any Gemma3 model on a GPU without bfloat16 (T4, V100) died in the first forward pass with: RuntimeError: expected mat1 and mat2 to have the same dtype, but got: c10::Half != float gemma3 is in FORCE_FLOAT32, so a float16 request loads float32 weights and UNSLOTH_FORCE_FLOAT32 is set. The forced patches then run the heavy reductions in float32 for fp16 overflow safety and narrow the activation back to a hard-coded float16 at every Linear boundary. That assumed the projection weights were float16, which holds for LoRA and QLoRA, where the base weights stay float16. Full finetuning upcasts the trainable weights to float32, so a float16 activation met a float32 weight and the matmul failed. The crash needs no autocast to reconcile the two dtypes, which is exactly the case on a no-bfloat16 GPU: unsloth reports "Switching to float32 training since model cannot work with float16" and trains with neither fp16 nor bf16 set. Read the boundary dtype off the weights that actually do the multiply instead of hard-coding float16, in Gemma3MLP and in both Gemma3Attention forwards. float16 weights hit an identity branch, so the LoRA and QLoRA path keeps the casts it already had. Verified on gemma-3-270m-it with bfloat16 hidden at import to emulate a T4: full finetuning before: RuntimeError after: trains, loss 1.5520 full finetuning + QAT before: RuntimeError after: trains, loss 1.3969 LoRA 16bit 24.680545806884766 before and after QLoRA 4bit 19.022239685058594 before and after On a bfloat16 GPU, where the forced patches do not engage, full finetuning losses are unchanged as well (1.8550999959309895 and 1.8105351130167644). Adds tests/test_gemma3_forced_float32_boundary_dtype.py.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 5a0d010c3b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| boundary_dtype = _linear_boundary_dtype(self, "gate_proj", "up_proj", "down_proj") | ||
| x = _to_boundary_dtype(x, boundary_dtype) |
There was a problem hiding this comment.
When the normal auto-compiler serializes this patched Gemma3MLP.forward into unsloth_compiled_cache, _linear_boundary_dtype and _to_boundary_dtype become free global names. create_new_function only imports such names from transformers.models.gemma3.modeling_gemma3, but this patch never publishes either helper there, unlike the RMSNorm helpers. The generated forward therefore raises NameError when invoked, breaking compiled Gemma3 training; publish both helpers to the modeling module before installing any forward that references them.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and it would have been a hard failure rather than a degradation. publish_to_modeling_module is already used for the RMSNorm helpers for exactly this reason; these two were not published anywhere, so the serialized forward had two free names the modeling module could not supply.
Fixed in cdf973e: a _publish_boundary_helpers call in all three patches that install a forward referencing them (patch_Gemma3MLP, patch_Gemma3Attention, patch_Gemma3Attention_generic). The guarding test walks the AST rather than checking names, so a fourth patch that uses the helpers without publishing them fails too.
| boundary_dtype = _linear_boundary_dtype(self, "q_proj", "k_proj", "v_proj", "o_proj") | ||
| hidden_states = _to_boundary_dtype(hidden_states, boundary_dtype) |
There was a problem hiding this comment.
When UNSLOTH_FORCE_FLOAT32=0 and Gemma3 is loaded for 4-bit QLoRA with bfloat16 activations, bitsandbytes exposes each packed projection weight as uint8. _linear_boundary_dtype consequently skips all four weights and returns its unconditional torch.float16 fallback, so these lines narrow the bfloat16 hidden states before q/k/v; the previous generic forward passed them through unchanged. That conversion loses bfloat16's exponent range and can introduce infinities on the very Gemma3 path that uses bfloat16 to avoid fp16 overflow, so the generic path should retain the activation or quantizer compute dtype when no floating-point weight is available.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct. patch_Gemma3Attention_generic returns early only when UNSLOTH_FORCE_FLOAT32 == "1", so it is the path that runs when float32 is not forced, which is where a bfloat16 4bit model lands. Every projection is a uint8 blob there, nothing matched, and the hard-coded float16 fallback narrowed activations the unpatched forward passed through untouched.
Fixed in cdf973e: the fallback is now None and _to_boundary_dtype treats that as identity. On the forced-float32 path that is the same value as before, because the activation arriving there is already float16 out of RMSNorm, so those numerics are unchanged.
| return torch.float16 | ||
| pass |
There was a problem hiding this comment.
When a Gemma3 FP8 checkpoint uses Transformers' FP8Linear/FbgemmFp8Linear, the projection's stored weight has torch.float8_e4m3fn dtype, so this condition treats that storage format as the Linear's activation dtype. The new boundary calls then cast the normal bfloat16/fp16 hidden states directly to unscaled float8 before the projection; those FP8 modules instead expect a higher-precision input and perform their own scaled activation quantization, returning in the input dtype. This can lose values before scaling and leaves the patched Q/K normalization and SDPA operating on unsupported float8 outputs, breaking FP8 LoRA/training; only ordinary dense compute dtypes should be inferred directly from weight.dtype.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct. torch.float8_e4m3fn.is_floating_point is True, which is what made the old check accept it:
float8_e4m3fn is_floating_point=True
float8_e5m2 is_floating_point=True
uint8 is_floating_point=False
Fixed in cdf973e by excluding the float8 dtypes, built by getattr lookup so a torch that predates one of them neither raises nor silently stops excluding it. They are skipped rather than treated as terminal, so an ordinary compute dtype further along the projection list still wins. Covered for every float8 dtype the running torch exposes.
…the activation Three defects in the boundary-dtype patch, all found in review. Publish: the auto-compiler serializes these forwards into unsloth_compiled_cache and resolves their free names by importing from the modeling module, so a helper that lives only here raises NameError on the first call. The RMSNorm helpers are published for that reason; these two now are too, from every patch that installs a forward using them. float8: torch.float8_e4m3fn.is_floating_point is True, so an FP8 checkpoint's storage dtype was read as the dtype its Linear wants its activations in. FP8Linear takes bfloat16 or float16 and does its own scaled quantization, so an unscaled cast lost range before the scaling meant to keep it and left SDPA holding a dtype it cannot use. 4bit: the generic path installs when float32 is not forced, where every bitsandbytes projection is a uint8 blob. Nothing matched, and the hard-coded float16 fallback narrowed bfloat16 activations that the unpatched forward passed through untouched. The fallback is now None, which on the forced-float32 path is the same value as before because the activation arriving there is already float16.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: cdf973ec73
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| intermediate_fp16 = intermediate_fp32.to(torch.float16) | ||
| down_proj_out = self.down_proj(intermediate_fp16) | ||
| # Downcast and down_proj. fp16 weights take the same bare cast as before. | ||
| intermediate = _to_boundary_dtype(intermediate_fp32, boundary_dtype) |
There was a problem hiding this comment.
When UNSLOTH_FORCE_FLOAT32=1 with 4-bit QLoRA, every projection weight is packed uint8, so boundary_dtype is None; unlike the fp16 input boundary, however, intermediate_fp32 is the result of the explicitly upcast SwiGLU reduction, and this call now leaves it in float32 instead of performing the previous fp16 downcast. Bitsandbytes Linear4bit preserves its caller's input dtype for the returned result, so down_proj now returns float32 and changes the common QLoRA forward and gradient dtypes; the identical problem occurs for the forced attention output at line 818. Keep None as identity for the generic input path, but fall back to the preexisting fp16 dtype at these forced output boundaries.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and it is a regression this PR introduced. Fixed in 9c6c139.
Mechanism, confirmed against bitsandbytes 0.50.0: Params4bit defaults to quant_storage=torch.uint8, so on 4bit QLoRA every projection weight is a packed uint8 blob, _linear_boundary_dtype skips it and answers None. Linear4bit.forward ends in bnb.matmul_4bit(...).to(inp_dtype), so the result carries the caller's input dtype straight back out. The distinction you drew is the one that matters: at the input boundaries the activation is already fp16 out of RMSNorm, so None as identity is the same value the hard coded fp16 produced; at the two forced output boundaries the value is the deliberately upcast fp32 reduction, so identity is not equivalent.
Fix: a second helper, _to_forced_output_dtype, which falls back to fp16 when no weight answers, used only at down_proj (gemma.py:632) and o_proj (gemma.py:848). The three input boundaries keep _to_boundary_dtype and its None as identity. A separate name rather than a default argument, so the two kinds of boundary stay distinguishable at the call site, and both docstrings now say which one to use where.
Measured on a B200 with UNSLOTH_FORCE_FLOAT32=1, unsloth/gemma-3-270m-it-unsloth-bnb-4bit, 4bit QLoRA, forward plus backward, hooking the real Linear4bit under each LoRA wrapper (18 layers):
| boundary | origin/main | this PR before the fix | after 9c6c139 |
|---|---|---|---|
down_proj in |
fp16 x18 | fp32 x17, fp16 x1 | fp16 x18 |
down_proj out |
fp16 x18 | fp32 x17, fp16 x1 | fp16 x18 |
o_proj in |
fp16 x18 | fp32 x18 | fp16 x18 |
o_proj out |
fp16 x18 | fp32 x18 | fp16 x18 |
gate_proj in |
fp16 x18 | fp16 x18 | fp16 x18 |
q_proj in |
fp16 x18 | fp16 x18 | fp16 x18 |
(the one fp16 row in the down_proj column is the single layer this checkpoint leaves unquantized, whose fp16 weight does answer). After the fix every boundary matches origin/main exactly, and the input boundaries were never affected either way.
The fp32 full finetuning case this PR exists for still works: same probe with full_finetuning=True on unsloth/gemma-3-270m-it, origin/main dies with RuntimeError: expected mat1 and mat2 to have the same dtype, but got: c10::Half != float, and with the fix it completes with every boundary in fp32.
Regression tests added in tests/test_gemma3_forced_float32_boundary_dtype.py: the real patched Gemma3MLP and Gemma3Attention forwards driven with a Linear4bit stand-in (uint8 weight, result cast back to the caller's input dtype), plus a direct assertion that the two helpers differ on None. 16 passed. Reverting just the two call sites turns the two new behavioural tests red on the exact dtype (assert [torch.float32] == [torch.float16]), restoring them turns them green.
|
Validated on hardware after the three fixes, not only in unit tests.
That notebook is the one that surfaced the original bug in a sweep, so it is the right thing to hold this against. |
_linear_boundary_dtype skips a non floating point weight, so on 4bit QLoRA every projection is a packed uint8 blob and the boundary dtype is None. At the input boundaries None as identity is equivalent to the old hard coded fp16, because the activation arriving from RMSNorm is already fp16. At the two forced output boundaries it is not: the value there is the deliberately upcast float32 SwiGLU and attention reduction, and bitsandbytes Linear4bit returns its caller's input dtype, so down_proj and o_proj started handing back float32 and changed the forward and gradient dtypes of the common QLoRA path. Add _to_forced_output_dtype, which defaults to fp16 when no weight answers, and use it at those two call sites only. A separate name rather than a default argument keeps the two kinds of boundary distinguishable where they are used.
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9c6c1395da
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if dtype is None or not dtype.is_floating_point: continue | ||
| if dtype in _STORAGE_ONLY_FLOAT_DTYPES: continue | ||
| return dtype |
There was a problem hiding this comment.
When a 4-bit Gemma model uses bnb_4bit_quant_storage="bfloat16", "float16", or "float32"—valid settings propagated in vllm_utils.py:1356-1362 and 1388-1392—the packed Params4bit.weight passes these checks even though its dtype describes storage, not the activation compute dtype. The forced and generic forwards consequently cast activations and forced outputs to that storage dtype; the patched Linear4bit then returns the caller dtype (temporary_patches/bitsandbytes.py:86-100), changing QLoRA forward/gradient dtypes and numerics despite the weight still being quantized. Skip weights carrying quant_state regardless of whether their storage dtype is floating-point.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Correct, and fixed in 165c826.
Mechanism. _linear_boundary_dtype's two filters were dtype.is_floating_point and membership in _STORAGE_ONLY_FLOAT_DTYPES, and that set holds only the five float8 variants. bnb_4bit_quant_storage defaults to uint8, which fails the first filter, so the default path was already skipped and that is why this went unnoticed. Verified on bitsandbytes 0.50.0 / torch 2.9.1, B200:
| quant_storage | weight.dtype | is_floating_point | quant_state |
|---|---|---|---|
| uint8 | uint8 | False | not None |
| bfloat16 | bfloat16 | True | not None |
| float16 | float16 | True | not None |
| float32 | float32 | True | not None |
So a float storage dtype passes both filters while the tensor is still packed 4bit, and quant_state is the only reliable discriminator. Linear4bit dequantizes to its own compute_dtype and returns the caller's input dtype (temporary_patches/bitsandbytes.py, matmul_4bit(...).to(inp_dtype)), so answering with the storage dtype propagates into both the input boundaries and the two forced output boundaries.
Liveness: prospective, but reachable through public config. Not a live break on anything shipped. Every Unsloth Gemma3 4-bit checkpoint I checked (gemma-3-270m-it-unsloth-bnb-4bit, gemma-3-4b-it-unsloth-bnb-4bit, gemma-3-12b-it-bnb-4bit) has bnb_4bit_quant_storage: uint8, and that is also the BitsAndBytesConfig default. It is reached by a user-supplied config, which is not exotic: FSDP can only shard float dtypes, so the FSDP-QLoRA guidance is explicitly to set bnb_4bit_quant_storage=torch.bfloat16, and vllm_utils.py:1362,1392, moe_utils_bnb4bit.py:229,237 and moe_bnb.py:321 all plumb the configured value straight through. Worth noting FastModel itself cannot reach it for a model with a prequantized variant, because the loader rewrites the repo name to -unsloth-bnb-4bit, whose weights ship already packed as uint8; it is reached by quantizing on the fly through an explicit BitsAndBytesConfig.
Fix. quant_state is tested ahead of the dtype checks, read with getattr(..., "quant_state", None) so a plain nn.Parameter or a non-bitsandbytes backend is untouched. The float8 set stays, since it covers a different case: a genuinely unquantized float8 weight, which carries no quant_state. The docstring sentence asserting that bitsandbytes stores a 4bit weight as a uint8 blob is corrected, since that is only the default.
Census. Grepped _linear_boundary_dtype, quant_state, _STORAGE_ONLY_FLOAT_DTYPES and quant_storage across the zoo and unsloth source trees, all local worktrees, unsloth_compiled_cache/ and installed site-packages. Every other place that has to tell "quantized" apart from "compute dtype" already keys off quant_state:
temporary_patches/bitsandbytes.py:69patchedLinear4bit.forwardbranches on ittemporary_patches/gemma4_float32.py:69_unsloth_gemma4_ple_cast_inputreturns early on it, ahead of its dtype testscompiler.py:1453the serialized twin of that helper does the sametemporary_patches/moe_grouped_modulelist.py:175_lin_compute_dtypeusescompute_dtype/quant_state.dtype, neverweight.dtype, forParams4bittemporary_patches/moe_utils_bnb4bit.py:441guards on the quant_state dtype
_linear_boundary_dtype was the sole outlier, so this makes Gemma3 consistent with the Gemma4 PLE helper rather than introducing a new rule. The only other hit is the generated unsloth_compiled_cache/unsloth_compiled_module_gemma3.py, which calls the helper rather than defining it and resolves it by import, so it picks the fix up automatically.
Evidence. unsloth/gemma-3-270m-it quantized on the fly with bnb_4bit_quant_storage=bfloat16 under UNSLOTH_FORCE_FLOAT32=1, B200, real Linear4bit boundaries hooked. All 36 modules report weight.dtype=bfloat16 with quant_state set.
| boundary | before | after |
|---|---|---|
| q_proj.in / .out | bfloat16 | float16 |
| o_proj.in / .out | bfloat16 | float16 |
| gate_proj.in / .out | bfloat16 | float16 |
| down_proj.in / .out | bfloat16 | float16 |
Helper answer goes from self_attn:bfloat16 x18, mlp:bfloat16 x18 to None everywhere, and the loss moves (2.8917 -> 2.9123), so the numerics really were being changed. The two cases already proven on this PR are unregressed: default-uint8 QLoRA still answers None at every boundary, and full_finetuning=True still answers float32 at all 36.
Tests. Added to tests/test_gemma3_forced_float32_boundary_dtype.py, driving the real helper: uint8 storage skipped, bfloat16/float16/float32 storage with quant_state skipped (the regression test), an unquantized bfloat16/float16/float32 weight with no quant_state still answered (the discriminator that keeps this from becoming a blanket "skip all bfloat16"), a bare nn.Parameter still answered, and float8 still skipped. The storage cases run twice, once against a tensor carrying a quant_state and once against a real Params4bit on GPU. 27 passed.
Sabotage-checked: reverting only the quant_state line takes the suite to 6 failed, 21 passed, exit 1, and the six are exactly the float-storage cases, both stand-in and real bitsandbytes; the uint8 cases and every unquantized-weight case stay green. Restoring gives 27 passed, exit 0.
bnb_4bit_quant_storage defaults to uint8, which is not floating point, so _linear_boundary_dtype already skipped a packed 4bit weight on the default path. It is a public knob though: FSDP can only shard float dtypes, so FSDP-QLoRA setups set it to bfloat16, and vllm_utils and the bnb MoE loaders plumb the configured value through. Params4bit.weight.dtype is then bfloat16 while the tensor is still packed 4bit, and the floating-point test reads a storage container as the activation dtype. Linear4bit dequantizes to its own compute dtype and returns the caller's input dtype, so answering there casts activations and forced outputs to the storage dtype and changes QLoRA forward and gradient numerics. Measured on gemma-3-270m-it quantized with quant_storage=bfloat16 under UNSLOTH_FORCE_FLOAT32, all eight q/o/gate/down boundaries carried bfloat16 in and out instead of float16, and the loss moved. Test quant_state ahead of the dtype checks. The float8 set stays: it covers a genuinely unquantized float8 weight, which carries no quant_state. Read with getattr so a plain nn.Parameter or a non-bitsandbytes backend is unaffected.
|
@codex review |
1 similar comment
|
@codex review |
|
Codex Review: Didn't find any major issues. Can't wait for the next one! Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
What broke
Full finetuning any Gemma3 model on a GPU without bfloat16 (T4, V100) died in the first forward pass:
gemma3is inFORCE_FLOAT32, so a float16 request loads float32 weights and setsUNSLOTH_FORCE_FLOAT32=1. The forced patches (patch_Gemma3RMSNorm,patch_Gemma3MLP,patch_Gemma3Attention) run the heavy reductions in float32 for fp16 overflow safety, then narrow the activation back to a hard-codedtorch.float16at every Linear boundary.That hard-coded float16 assumes the projection weights are float16. They are for LoRA and QLoRA, where the base weights stay float16. Full finetuning upcasts the trainable weights to float32, so a float16 activation met a float32 weight.
Nothing reconciled the two dtypes because there was no autocast to do it. That is precisely the situation on a no-bfloat16 GPU: unsloth reports
Switching to float32 training since model cannot work with float16and trains with neitherfp16norbf16set.Two things worth flagging, because the original report had them the other way round:
Gemma3_(270M)_Phone_Deployment.ipynb, which usesqat_scheme = "int4", but plainfull_finetuning = Truewith noqat_schemefails identically. torchao's fake quantizer restores the weight dtype (fq.view(w.shape).to(w.dtype)in both 0.15.0 and 0.17.0), soFakeQuantizedLinearis not what makes the weight float32. The whole model is float32.input_layernormreceives float32 and emits float16, because_gemma3_rms_norm_float32ends in.to(torch.float16). The projection is simply the first Linear that float16 reaches.What happens after
The boundary dtype is read off the weights that actually do the multiply, rather than hard-coded, in
Gemma3MLPand in bothGemma3Attentionforwards. Two small helpers:_linear_boundary_dtype(module, *attr_names)returns the dtype of the first floating-point projection weight. A bitsandbytes 4bit base weight is auint8blob and is not floating point, so it is skipped and that path keeps the float16 default it already used._to_boundary_dtype(x, dtype)returnsxunchanged when the dtype already matches.RMSNorm is deliberately left alone. Its own weight is float32 even under LoRA, because
UNSLOTH_HIGH_PRECISION_LAYERNORM=1upcasts the norms, so deriving the boundary from the norm weight would upcast the common path too. Deriving it from the projections is what distinguishes the two regimes.Blast radius
float16 weights hit the identity branch of
_to_boundary_dtypeon every call, so LoRA and QLoRA keep exactly the casts they had and add one dtype comparison per boundary. Measured ongemma-3-270m-it, bfloat16 hidden at import to emulate a T4:RuntimeError: Half != floatqat_scheme="int4", no bf16RuntimeError: Half != floatOn a bfloat16 GPU, where the forced patches do not engage, full finetuning is unchanged as well: 1.8550999959309895 without QAT and 1.8105351130167644 with it, identical before and after. The LoRA numbers are bitwise stable across reruns, so the equality is meaningful rather than noise.
No change to the AMD, Intel or MLX paths, and no device specific calls: only
tensor.dtypeandtensor.to(dtype).Test evidence
tests/test_gemma3_forced_float32_boundary_dtype.py, 6 tests, CPU only, no network:Full suite,
pytest tests/ -q --ignore=tests/test_upstream_pinned_symbols_trl_vllm.py(that file has a hard-coded absolute path and raisesPermissionErrorat collection onmain):Every test is behavioural. None of them match against the source text of
gemma.py.Each test was proved discriminating by reverting the matching source arm and confirming that specific test fails:
_to_boundary_dtypealways fp16_linear_boundary_dtypealways fp16_linear_boundary_dtypeunit_to_boundary_dtypeunitC, D and E show each of the three wired call sites is pinned by exactly one distinct test. Under sabotage C the failure is the original error,
expected m1 and m2 to have the same dtype, but got: c10::Half != float(CPU wording; CUDA saysmat1 and mat2).The bitwise test is a regression guard for the common path rather than a test of the fix, so it passes with and without the fix by design. It is pinned by sabotage F, which drops the float32 reduction.
Relationship to #706
#706 targets the same crash. It is 367 commits behind
mainand no longer applies:mainhas since moved the RMSNorm into the module-level compiled_gemma3_rms_norm_float32, and merging conflicts.It also derives the RMSNorm boundary from that norm's own weight. Because
UNSLOTH_HIGH_PRECISION_LAYERNORM=1makes the norms float32 even under LoRA, that upcasts the LoRA path too. On its own branch it changes the LoRA forward from 24.680545806884766 to 24.725526809692383 and QLoRA from 19.022239685058594 to 19.043985366821290, which its description states is bit-identical. This PR keeps those two numbers exactly. Happy to close #706 in favour of this, or to fold anything wanted from it, including itsGemma3MultiModalProjectorpatch, which is not carried here.Same assumption elsewhere, not fixed here
Kept scoped to Gemma3, but the identical hard-coded float16 boundary exists in:
temporary_patches/qwen3_moe_float32.pylines 121 and 264-265 (attn_output.to(torch.float16)straight intoself.o_proj)temporary_patches/gemma4_float32.pylines 301, 313 and 495-496temporary_patches/misc.pylines 1643 and 1652#978 addresses the Qwen3.5 equivalent.