Merge remote-tracking branch 'origin/main' into svi

# Conflicts:
#	fastvideo/pipelines/stages/__init__.py

Author: H1yori233

.agents/STATUS.md

@@ -0,0 +1,73 @@
+---
+date: 2026-05-07
+experiment: PR #1280 (daVinci-MagiHuman port), distill DiT parity bring-up
+category: porting
+severity: important
+---
+
+# Conversion `--cast-bf16` Needs an FP32-Keep Suffix Allowlist
+
+## What Happened
+
+`scripts/checkpoint_conversion/convert_magi_human_to_diffusers.py --cast-bf16`
+produced a converted distill DiT checkpoint that loaded cleanly, ran end-to-
+end, and emitted reasonable output — but `test_magi_human_distill_parity`
+showed `diff_mean=0.114` against the upstream reference. The base DiT was
+bit-exact with the same conversion script. Only the distill variant
+regressed.
+
+The error was small enough that visual quality looked normal, but large
+enough to fail bit-exact parity. The MagiHuman base + distill DiTs share
+most of their architecture, so a difference that affected only distill was
+counterintuitive.
+
+## Root Cause
+
+`--cast-bf16` was downcasting **all** fp32 tensors to bf16 indiscriminately.
+The base checkpoint and the FastVideo `final_linear` / adapter modules
+require eight specific tensors to remain in fp32:
+
+- LayerNorm `gamma` / `beta` weights for the final residual exit
+- Adapter projection biases
+- A handful of scale parameters in the output projection chain
+
+These tensors participate in chains where bf16 precision causes accumulation
+error large enough to drift the parity check. The base DiT happened to not
+hit those specific chains in the path the test exercised (different
+attention mask shape, different audio interleave); the distill variant did.
+
+## Fix / Workaround
+
+Added `_FP32_KEEP_SUFFIXES` allowlist to
+`convert_magi_human_to_diffusers.py` (commit `829f70d3`) and gated `--cast-
+bf16` on it. Tensors whose state-dict key ends with any allowlisted suffix
+keep their original fp32 dtype regardless of the flag.
+
+Distill DiT parity went from `diff_mean=0.114` (silently wrong) to bit-exact
+in one commit.
+
+## Prevention
+
+1. **Treat `--cast-bf16` as opinionated, not blanket.** Any conversion
+   script that supports a global dtype downcast flag MUST own an explicit
+   allowlist of fp32-keep tensors, documented at the top of the file.
+
+2. **The `add-model-conversion` skill** should enforce two checks for any
+   converter that ships a `--cast-bf16`-style flag:
+   - Run the parity test for **every** variant of the model (base, distill,
+     SR, etc.), not just the headline variant. Different variants exercise
+     different code paths.
+   - Diff the converted checkpoint's dtype map against the upstream
+     reference and assert the allowlist covers every fp32 tensor in the
+     reference.
+
+3. **For MagiHuman specifically**: if you add or rename DiT modules that
+   touch `final_linear`, the adapter, or any LayerNorm in the residual exit
+   path, **check that any fp32-required tensors are covered by
+   `_FP32_KEEP_SUFFIXES`** in the conversion script and re-run
+   `test_magi_human_distill_parity` (it's the canary).
+
+4. The lesson generalizes beyond MagiHuman: any DiT that uses bf16 mixed
+   precision but keeps specific tensors in fp32 (a common pattern with
+   flash-attn-style backends) needs this allowlist for any conversion that
+   downcasts.

.agents/lessons/2026-05-07_conversion-cast-bf16-suffix-allowlist.md

@@ -0,0 +1,77 @@
+---
+date: 2026-05-07
+experiment: PR #1280 (daVinci-MagiHuman port), DiT parity bring-up
+category: porting
+severity: important
+---
+
+# DiT Dtype Boundary Alignment with Flash-Attn-Style Backends
+
+## What Happened
+
+DiT bit-exact parity for daVinci-MagiHuman against the upstream reference
+sat at `diff_max=0.5` after the architecture port was complete and weight
+loading was correct. The error grew with depth (later layers diverged more
+than earlier ones), suggesting an accumulating numerical drift rather than
+a structural mismatch. None of the obvious culprits (RoPE, GQA expansion,
+attention mask handling) accounted for the pattern.
+
+## Root Cause
+
+Four cumulative dtype-boundary mismatches, each individually small but
+together pushing parity from `diff_max=0.5` to bit-exact (`diff_max=0.0`):
+
+1. **SDPA inputs were not cast to bf16.** Upstream's `flash_attn_with_cp`
+   internally casts Q/K/V to bf16 at `dit_module.py:508` before the kernel.
+   FastVideo was passing fp32 tensors through, getting numerically different
+   intermediates even though the kernel accepts both.
+
+2. **Post-attention output was kept in bf16 across the per-head gating
+   multiply.** Upstream upcasts to fp32 before the gating, FastVideo did the
+   gate in bf16 then upcast.
+
+3. **A residual-stream cast at the block boundary.** FastVideo had a
+   `.to(bf16)` then `.to(fp32)` at the start of each block. Upstream keeps
+   the residual stream **continuously in fp32** across all 40 layers; only
+   the inputs to specific kernels are temporarily downcast.
+
+4. **Parity test scheduler used a double-shift.** A separate per-block fix
+   (Wave 11 production migration) — single-shift schedule is what upstream
+   uses; the parity test was double-shifting.
+
+## Fix / Workaround
+
+Four cumulative changes in `fastvideo/models/dits/magi_human.py` (commit
+`3a4816cb`), each with a comment at the call site explaining the upstream
+parity rationale:
+
+- Cast SDPA inputs to bf16 right before the attention call.
+- Upcast attention output to fp32 before the per-head gate multiply.
+- Drop the residual-stream `.to(bf16)`/`.to(fp32)` wrapper at the block
+  boundary; let the residual stay fp32 throughout.
+- Single-shift schedule in the parity test fixture (matches upstream Wave 11).
+
+## Prevention
+
+1. **For any DiT port with a flash-attn-style backend**, treat the dtype of
+   the residual stream as a load-bearing invariant, not a performance knob.
+   Document it in the model's per-pipeline AGENTS.md. MagiHuman's invariant:
+   *residual stream stays fp32 across all blocks; only kernel inputs are
+   temporarily bf16*.
+
+2. **Use layer-by-layer activation hooks** when DiT parity is close-but-not-
+   bit-exact and the gap grows with depth. The
+   `fastvideo/hooks/activation_trace.py` infra exists exactly for this case
+   (`add-model-trace` skill). In MagiHuman's case it would have localized the
+   first divergence point in one pass.
+
+3. **The `add-model-port-dit` skill** should explicitly call out:
+   - SDPA input dtype must match the upstream kernel's internal cast.
+   - Post-attention upcast happens **before** any per-head gate, not after.
+   - Residual stream dtype across block boundaries is a parity invariant.
+   These rules apply to any DiT port whose upstream uses a flash-attn-style
+   backend (`flash_attn_with_cp`, `flex_flash_attn_func`, etc.).
+
+4. **Add an "intermediate-layer parity" test** for new DiT ports — comparing
+   activations at layer 5, 10, 20, 30 — not just the final output. A growing-
+   with-depth pattern is otherwise indistinguishable from "almost right".

.agents/lessons/2026-05-07_dit-dtype-boundary-with-flash-attn.md

@@ -0,0 +1,69 @@
+---
+date: 2026-05-07
+experiment: PR #1280 (daVinci-MagiHuman port), Wave 14
+category: porting
+severity: critical
+---
+
+# Silent Channel-Major Token-Packing Bugs
+
+## What Happened
+
+While porting daVinci-MagiHuman (`fastvideo/pipelines/basic/magi_human/`),
+the pipeline-parity test passed bit-exactly but the E2E user-visible output
+was **pure static noise**. Latent tensors compared identically against the
+upstream reference at every checkpointed boundary, yet decoded videos showed
+no recognizable content. The discrepancy reproduced on every variant
+(base / distill / SR-540p / SR-1080p) with the same noise profile.
+
+## Root Cause
+
+Video tokens were being packed **spatial-major** instead of **channel-major**:
+
+```python
+# What we had (spatial-major, WRONG)
+einops.rearrange(x, "b c (T pT) (H pH) (W pW) -> b (T H W) (pT pH pW C)", ...)
+
+# What upstream's UnfoldNd produces (channel-major, CORRECT)
+einops.rearrange(x, "b c (T pT) (H pH) (W pW) -> b (T H W) (C pT pH pW)", ...)
+```
+
+A single-character einops reorder. The pipeline-parity test used FastVideo's
+own packer on **both** sides of the comparison, so the bug was invisible there
+— both sides agreed on the wrong layout. The DiT consumed those tokens
+without complaint because the channel dimension only matters at decode time,
+when the VAE's first conv expects channel-major input. By that point the test
+boundary was already passed.
+
+The bug was load-bearing for any token-packed format that downstream feeds
+into a `UnfoldNd`-shaped consumer. Wave 14 of the port took multiple bug-hunt
+iterations and an Oracle consultation to localize.
+
+## Fix / Workaround
+
+Single-character einops change in `stages/latent_preparation.py:_img2tokens`
+(commit `6d190693` of the original PR). After the fix, all four variants
+produced expected E2E output and the pipeline-parity tests still passed
+because both sides of the parity check are now correct.
+
+## Prevention
+
+1. **Never use the FastVideo-side packer on both sides of a parity test.**
+   At least one parity boundary must compare against an upstream tensor
+   produced by the upstream packer. For MagiHuman this means a separate
+   `_img2tokens` parity test that feeds upstream `UnfoldNd` output as the
+   reference, not FastVideo's reformatted equivalent.
+
+2. **Add an E2E hash check** alongside latent-parity. The mp4 SHA was the
+   first signal that something was wrong; if it had been part of the standard
+   parity battery, the bug would have surfaced in Wave 1, not Wave 14. See
+   `fastvideo/tests/ssim/test_magi_human_similarity.py` for the CI version.
+
+3. **For any new model port that involves explicit tensor reshaping into
+   tokens**, document the expected packing order (`(C pT pH pW)` vs
+   `(pT pH pW C)`) at the call site and assert the layout matches the
+   downstream consumer's expectation.
+
+4. The `add-model-port-dit` skill's parity gate should require an E2E hash
+   check for any DiT that does video token packing, not just latent
+   bit-exactness.

.agents/lessons/2026-05-07_silent-channel-major-packing-bugs.md

@@ -31,7 +31,7 @@ FastVideo-WorldModel/
 │   │   │   └── distribution_matching/  # DMD2Method, SelfForcingMethod
 │   │   ├── models/            #   Per-role model wrappers (ModelBase, CausalModelBase)
 │   │   │   ├── wan/           #     WanModel, WanCausalModel
-│   │   │   └── matrixgame/    #     MatrixGameModel, MatrixGameCausalModel
+│   │   │   └── matrixgame2/   #     MatrixGame2Model, MatrixGame2CausalModel
 │   │   ├── callbacks/         #   Composable hooks (grad_clip, ema, validation)
 │   │   └── utils/             #   Config, builder, checkpoint, optimizer, tracking
 │   ├── training/              # Legacy training infrastructure (being phased out)
@@ -44,7 +44,7 @@ FastVideo-WorldModel/
 │   │   ├── wan_distillation_pipeline.py # Wan distillation
 │   │   ├── self_forcing_distillation_pipeline.py # Self-forcing distill
 │   │   ├── ltx2_training_pipeline.py   # LTX-2 training
-│   │   └── matrixgame_training_pipeline.py # MatrixGame training
+│   │   └── matrixgame2_training_pipeline.py # Matrix-Game 2.0 training
 │   ├── attention/             # Attention backends
 │   ├── distributed/           # Sequence/tensor parallel utilities
 │   ├── layers/                # Tensor-parallel layers
@@ -95,7 +95,7 @@ FastVideo-WorldModel/
 | Wan distillation (DMD) | `fastvideo/training/wan_distillation_pipeline.py` | `torchrun --nproc_per_node N` |
 | Self-forcing distill | `fastvideo/training/wan_self_forcing_distillation_pipeline.py` | `torchrun --nproc_per_node N` |
 | LTX-2 finetune | `fastvideo/training/ltx2_training_pipeline.py` | `torchrun --nproc_per_node N` |
-| MatrixGame | `fastvideo/training/matrixgame_training_pipeline.py` | `torchrun --nproc_per_node N` |
+| Matrix-Game 2.0 | `fastvideo/training/matrixgame2_training_pipeline.py` | `torchrun --nproc_per_node N` |
 
 ## W&B Integration