Skip to content

Commit c3bce5d

Browse files
authored
Add fused rms norm pass (pytorch#22849)
Summary: FuseRMSNormPass folds the HuggingFace RMSNorm decomposition into a single aten.rms_norm. HF *RMSNorm lowers to `weight * (x * rsqrt(mean(x**2, -1, keepdim=True) + eps))`. The pass matches that chain (a mul over rsqrt / add / mean / pow), reads eps from the lifted constant added to the variance, and rewrites the whole ~6-op sequence to `aten.rms_norm(x, [D], weight, eps)` -- torch's F.rms_norm -- so the native runtime executes RMSNorm as one op. The matcher also sees through the optional float32 up-cast HF wraps around the variance computation and the matching down-cast after the normalize. When the up-cast is present it must be shared by both the variance and the scale path; otherwise the chain is not a RMSNorm apply and is left alone. Note that the fused aten.rms_norm computes in the input dtype, so for bf16/fp16 models this folds away HF's fp32 variance accumulation. Every intermediate node must be single-use so the decomposition DCEs away after the rewrite. The pass is added to get_default_passes(), so it runs pre-partition on the edge-dialect graph, and aten.rms_norm.default is added to the partitioner's _SUPPORTED_NON_CORE_OPS so it is both claimed by the delegate and preserved (not decomposed back) via ops_to_not_decompose. Reviewed By: digantdesai Differential Revision: D111753289 Pull Request resolved: pytorch#22849
1 parent cfc96cc commit c3bce5d

6 files changed

Lines changed: 1873 additions & 0 deletions

File tree

‎backends/native/BUCK‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -33,6 +33,7 @@ fbcode_target(
3333
"//caffe2:torch",
3434
"//executorch/backends/transforms:collapse_view_copy",
3535
"//executorch/backends/transforms:fuse_gqa_with_sdpa",
36+
"//executorch/backends/transforms:fuse_rms_norm",
3637
"//executorch/backends/transforms:normalize_sdpa_input_rank",
3738
"//executorch/exir:lib",
3839
"//executorch/exir/_serialize:lib",

‎backends/native/partitioner.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -38,6 +38,7 @@
3838
torch.ops.aten.linear.default,
3939
torch.ops.aten.addmm.default,
4040
torch.ops.aten.scaled_dot_product_attention.default,
41+
torch.ops.aten.rms_norm.default,
4142
]
4243

4344
# Maps a control-flow higher-order op to the arg indices of its branch submodule

‎backends/native/passes/__init__.py‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,7 @@
4040

4141
from executorch.backends.transforms.collapse_view_copy import CollapseViewCopyPass
4242
from executorch.backends.transforms.fuse_gqa_with_sdpa import FuseGQAWithSDPAPass
43+
from executorch.backends.transforms.fuse_rms_norm import FuseRMSNormPass
4344
from executorch.backends.transforms.normalize_sdpa_input_rank import (
4445
NormalizeSDPAInputRankPass,
4546
)
@@ -52,6 +53,7 @@
5253
"BACKEND_INPLACE_OPS",
5354
"CollapseViewCopyPass",
5455
"FuseGQAWithSDPAPass",
56+
"FuseRMSNormPass",
5557
"get_default_passes",
5658
"NativeReinplacePass",
5759
"NormalizeSDPAInputRankPass",
@@ -69,6 +71,7 @@ def get_default_passes() -> List[Union[ExportPass, ExportedProgramPassBase]]:
6971
return [
7072
FuseGQAWithSDPAPass(),
7173
NormalizeSDPAInputRankPass(),
74+
FuseRMSNormPass(fold_dtype_casts=True, allow_lossy_weight_casts=True),
7275
CollapseViewCopyPass(),
7376
CSEPass(),
7477
NativeReinplacePass(),

0 commit comments

Comments
 (0)