Skip to content

Commit 6633d47

Browse files
committed
docs(muse-glimmer): explain weightless QK norms
Signed-off-by: yaoyu-33 <yaoyu.094@gmail.com>
1 parent f1a9bd8 commit 6633d47

1 file changed

Lines changed: 2 additions & 0 deletions

File tree

‎src/megatron/bridge/models/muse_glimmer/modeling_muse_glimmer.py‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -62,6 +62,8 @@ class MuseGlimmerSelfAttention(SelfAttention):
6262

6363
def __init__(self, config: TransformerConfig, *args: Any, **kwargs: Any) -> None:
6464
super().__init__(config, *args, **kwargs)
65+
# TE RMSNorm always registers an affine weight, while Muse Q/K norms are
66+
# parameter-free; use native RMSNorm to preserve the HF checkpoint contract.
6567
self.q_layernorm = nn.RMSNorm(
6668
self.hidden_size_per_attention_head,
6769
eps=config.layernorm_epsilon,

0 commit comments

Comments
 (0)