Skip to content

Commit fb8404c

Browse files
eagle705wplf
andauthored
fix: use HF-style attn mask path for qwen3-vl RoPE in BSHD training (#2582)
Signed-off-by: joosungy <joosungy@nvidia.com> Co-authored-by: Li Jinliang <jinliangl@nvidia.com>
1 parent 708a759 commit fb8404c

1 file changed

Lines changed: 4 additions & 1 deletion

File tree

  • src/megatron/bridge/models/qwen_vl/modelling_qwen3_vl

‎src/megatron/bridge/models/qwen_vl/modelling_qwen3_vl/model.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -487,6 +487,9 @@ def forward(
487487

488488
if position_ids is None:
489489
# BSHD
490+
# Megatron uses 4D bool masks ([B|1,1,S,S], True=masked); HF uses 2D keep masks ([B,S], 1=keep)
491+
# For simplicity, we set hf_attention_mask to None.
492+
hf_attention_mask = None
490493
position_ids, _ = get_rope_index(
491494
self.config.spatial_merge_size,
492495
self.image_token_id,
@@ -495,7 +498,7 @@ def forward(
495498
input_ids,
496499
image_grid_thw=image_grid_thw,
497500
video_grid_thw=video_grid_thw,
498-
attention_mask=attention_mask,
501+
attention_mask=hf_attention_mask,
499502
) # [3*b*s]
500503
if packed_seq_params is not None:
501504
# convert position_ids to THD format

0 commit comments

Comments
 (0)