Skip to content

Commit 9a19a74

Browse files
committed
[misc] say why the collective entry was dropped, not just that it was
The replaced set's _c10d_functional::reduce_scatter_tensor is correct where it came from: torchtitan runs Megatron-style sequence parallelism, which reduce-scatters inside the forward, and compiles, which is what puts collectives in the _c10d_functional namespace. FastVideo redistributes with Ulysses all-to-all and cannot compile the training path, so neither condition holds. Verified against FSDP2 with the trainer's wrapping order: re-adding both collective names leaves every op's MUST_SAVE/PREFER_RECOMPUTE decision unchanged and the loss bitwise identical, so the removal is a no-op.
1 parent 695c716 commit 9a19a74

1 file changed

Lines changed: 8 additions & 5 deletions

File tree

fastvideo/train/utils/activation_checkpoint.py

Lines changed: 8 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -53,11 +53,14 @@ class CheckpointType(str, Enum):
5353
"fastvideo_kernel::block_sparse_attn_triton",
5454
}
5555

56-
# The replaced set also named _c10d_functional::reduce_scatter_tensor, which
57-
# never fires: under FSDP2 the reduce-scatter runs in the post-backward hook,
58-
# outside the region. What does appear inside it is the parameter all-gather
59-
# (c10d::_allgather_base_), and retaining that would hold every checkpointed
60-
# block's unsharded weights at once, so it is deliberately not listed.
56+
# No collective is listed. The replaced set named
57+
# _c10d_functional::reduce_scatter_tensor, which is correct in torchtitan, where
58+
# Megatron-style sequence parallelism reduce-scatters inside the forward. Ulysses
59+
# redistributes with all-to-all instead, so FastVideo has no forward
60+
# reduce-scatter to retain; FSDP2's runs in the post-backward hook, outside the
61+
# region. The one collective that does appear inside a block is the parameter
62+
# all-gather, and retaining that would keep every checkpointed block's unsharded
63+
# weights resident at once, which is what FSDP exists to avoid.
6164

6265
# VMoBA and the FA3 training path go through torch.autograd.Function rather than
6366
# the dispatcher, so no policy can reach them; they get full recomputation.

0 commit comments

Comments
 (0)