Repository navigation
Conversation
Summary: On GPUs with `shader-f16`, Llama 3.2 1B's Q/K/V projections are fused by `QkvBk64`. When the live M doesn't use the fused kernel (any M other than 128, 508 or 512, including decode), the fusion's resize hook checks that the separate projections restored their dispatches. It required every dispatch in each member's range to be live. With a dynamic M each `linear_q4gsw` member records a decode and a prefill route, and `select_dispatch_route` keeps the inactive route at a zero grid, so the check threw on every unfused call: "linear_q4gsw_bk64_qkv(resize): separate projection dispatch was not restored before the QKV resize hook ran". This breaks Llama 3.2 1B chat in the ExecuTorch WebGPU Playground on executorch.ai on Apple GPUs (reported on Metal 3). The check now requires one live dispatch per member, which still catches a member whose hook didn't run. CI runs on SwiftShader, which has no `shader-f16`, so the fused path and `DynamicShape.QkvBk64LiveRoutesProfile` (M = 128, 1, 128) don't run there. Differential Revision: D126153972
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24033
Note: Links to docs will display an error until the docs builds have been completed. ✅ No FailuresAs of commit 5ae9624 with merge base 20675ac ( This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
|
@psiddh has exported this pull request. If you are a Meta employee, you can view the originating Diff in D126153972. |
This PR needs a
|
Summary:
On GPUs with
shader-f16, Llama 3.2 1B's Q/K/V projections are fused byQkvBk64. When the live M doesn't use the fused kernel (any M other than 128, 508 or 512, including decode), the fusion's resize hook checks that the separate projections restored their dispatches. It required every dispatch in each member's range to be live. With a dynamic M eachlinear_q4gswmember records a decode and a prefill route, andselect_dispatch_routekeeps the inactive route at a zero grid, so the check threw on every unfused call: "linear_q4gsw_bk64_qkv(resize): separate projection dispatch was not restored before the QKV resize hook ran".This breaks Llama 3.2 1B chat in the ExecuTorch WebGPU Playground on executorch.ai on Apple GPUs (reported on Metal 3). The check now requires one live dispatch per member, which still catches a member whose hook didn't run.
CI runs on SwiftShader, which has no
shader-f16, so the fused path andDynamicShape.QkvBk64LiveRoutesProfile(M = 128, 1, 128) don't run there.Differential Revision: D126153972