[feat] Add MatrixGame3.0 by H1yori233 · Pull Request #1201 · hao-ai-lab/FastVideo

H1yori233 · 2026-03-31T02:31:37Z

Purpose

Add MatrixGame3.0 inference support to FastVideo.
https://huggingface.co/FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers

Changes

Add matrixgame3.0 model and action module fastvideo/models/dits/matrixgame3.
Add matrixgame3.0 pipeline fastvideo/pipelines/basic/matrixgame.
Add matrixgame3.0 specific denoising stage fastvideo/pipelines/stages/matrixgame_denoising.py.
Add matrixgame3.0 specific light vae.

test.1.mp4

ours.mp4

TODO

migrate origin matrixgame2.0 model/pipeline name
inference for base model (bidirectional multi-step)
code cleanup

Test Plan

# Commands you ran

Test Results

Test output

# Paste output here

Checklist

I ran pre-commit run --all-files and fixed all issues
I added or updated tests for my changes
I updated documentation if needed
I considered GPU memory impact of my changes

For model/pipeline changes, also check:

I verified SSIM regression tests pass
I updated the support matrix if adding a new model

mergify · 2026-03-31T02:32:18Z

Merge Protections

Your pull request matches the following merge protections and will not be merged until they are valid.

🟢 PR merge requirements

Wonderful, this rule succeeded.

#approved-reviews-by>=1
check-success=fastcheck-passed
check-success=full-suite-passed
check-success~=pre-commit
title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

gemini-code-assist

Code Review

This pull request introduces support for MatrixGame 3.0, adding new DiT architectures, action modules, and I2V pipelines while refactoring existing MatrixGame 2.0 components for clarity. Feedback identifies a critical import mismatch for extrinsic builders and a batching issue where only the first item's actions are used for camera trajectories. Performance concerns were raised regarding inefficient VAE device transfers and redundant timestep embedding computations. Additionally, the review suggests using modulo for angle normalization, warns against silent tensor cropping, and notes potential compatibility regressions in the VAE normalization logic.

gemini-code-assist · 2026-03-31T02:34:17Z

+            timestep_tokens = timestep_tokens.unsqueeze(1).repeat(
+                1, post_patch_num_frames * post_patch_height * post_patch_width
+            )


Repeating the timestep for every spatial token and then passing the flattened tensor to the condition_embedder leads to a massive amount of redundant computation in the TimestepEmbedder. Since the timesteps are identical for all tokens in this fallback case, it is much more efficient to compute the embedding once per batch and then expand it spatially.

# Conflicts: # examples/train/scenario/worldmodel/dfsft_causal_i2v.yaml # examples/train/scenario/worldmodel/finetune_i2v.yaml # examples/train/scenario/worldmodel/self_forcing_causal_i2v.yaml # fastvideo/configs/pipelines/__init__.py # fastvideo/models/dits/matrixgame2/model.py # fastvideo/models/registry.py # fastvideo/pipelines/basic/matrixgame2/matrixgame2_causal_dmd_pipeline.py # fastvideo/pipelines/basic/matrixgame2/matrixgame2_i2v_pipeline.py # fastvideo/pipelines/preprocess/matrixgame2/matrixgame2_preprocess_pipeline_ode_trajectory.py # fastvideo/pipelines/preprocess/v1_preprocess.py # fastvideo/pipelines/stages/__init__.py # fastvideo/registry.py # fastvideo/tests/api/test_presets.py

…r indices in action and model modules.

…prompt-based naming for output files

H1yori233 · 2026-05-19T02:00:53Z

/gemini review

gemini-code-assist

Code Review

This pull request integrates Matrix-Game 3.0 into the FastVideo framework, introducing a new 5B-parameter DiT model, an action module for interactive control, and a dedicated I2V pipeline supporting 720p resolution. The changes include comprehensive model and pipeline configurations, specialized denoising and image encoding stages, and utility functions for camera and action processing. Feedback from the review highlights several critical issues: potential empty tensor returns due to incorrect slicing when padding is zero in the action module, and possible runtime errors in the VAE implementation when comparing tensors directly to strings. Additionally, the reviewer recommended guarding the torch.compile call with the existing DISABLE_COMPILE flag and replacing hardcoded architectural and window size parameters with values derived from the model configuration to improve flexibility and robustness.

gemini-code-assist · 2026-05-19T02:02:56Z

+flex_attention = torch.compile(
+    flex_attention, dynamic=False, mode="max-autotune-no-cudagraphs"
+)


The torch.compile call on flex_attention is not guarded by the DISABLE_COMPILE flag defined just above it. This prevents users from disabling compilation if needed for compatibility or debugging.

Suggested change

flex_attention = torch.compile(

flex_attention, dynamic=False, mode="max-autotune-no-cudagraphs"

)

if not DISABLE_COMPILE:

flex_attention = torch.compile(

flex_attention, dynamic=False, mode="max-autotune-no-cudagraphs"

)

gemini-code-assist · 2026-05-19T02:02:56Z

+            fc, fs = self.get_rotary_pos_embed(
+                7500,
+                self.patch_size[1],
+                self.patch_size[2],
+                64,
+                self.mouse_qk_dim_list,
+                start_offset=0,
+            )


The head_dim parameter for get_rotary_pos_embed is hardcoded to 64. This should be derived from the model configuration (e.g., mouse_hidden_dim // heads_num) to ensure RoPE is applied correctly if the architecture dimensions change.

gemini-code-assist · 2026-05-19T02:02:56Z

+        target_h = latent_h * spatial_ratio
+        target_w = latent_w * spatial_ratio
+        num_iterations = self._infer_num_iterations(batch)
+        clip_frame = 56  # hardcode for now


The clip_frame value is hardcoded to 56. This value is tightly coupled with the model's expected window size and should ideally be retrieved from the pipeline or model configuration to support different sequence lengths or future model variants.

SolitaryThinker · 2026-05-27T00:57:37Z

Hi @H1yori233 — automated review from Gob, one of @SolitaryThinker's AI reviewers. Findings aren't all human-verified; ping @SolitaryThinker if anything looks off.

TL;DR

All four critical gemini-bot concerns from the most recent review are resolved at HEAD 35281410cd. The cross-cutting wanvae.py change is properly gated by use_light_vae=False default, so Wan-2.1 / Matrix-Game 2.0 paths stay untouched. Two non-blocking follow-ups: per-batch camera trajectory uses only [0] (G4 still open), and the PR body's Test Plan / Test Results sections are empty.

Verdict: approve-with-followup

S1: 0 · S2: 2 · S3: 2 · Gemini resolved: 5/9 (+ 2 mitigated, + 2 unchanged-but-non-blocking)

Gemini concerns status at HEAD `35281410cd`

ID	Concern	Status
G1	`action_module.py` `:-padded_length` empty-tensor bug	✅ Resolved (`padded_length` removed entirely)
G2	Same bug at second site	✅ Resolved (same removal)
G3	Import name mismatch in `matrixgame3_denoising.py`	✅ Resolved (`build_extrinsics_from_actions` matches def at `utils.py:103`)
G4	Camera extrinsics use only `batch.keyboard_cond[0]` for whole batch	❌ Still present at `matrixgame3_denoising.py:121` — see S2-1
G5	VAE GPU↔CPU thrash in `image_encoding.py:forward()`	✅ Acceptable — matches existing `vae_cpu_offload` pattern (Wan/Hy precedent)
G6	`while` loops for angle normalization	⏸️ Cosmetic, non-blocking
G7	Silent shape-crop at denoising step	⏸️ Now logs `warning(...)` at line 296 — no longer silent
G8	Timestep repeated per spatial token (perf)	⏸️ Unchanged; perf-only
G9	`wanvae.py` RMS scale removal	✅ Resolved — `self.scale` still at line 230; new path gated by `use_light_vae=False` default

Findings

S2-1 [persistent] — Camera trajectory uses only `batch.keyboard_cond[0]` / `batch.mouse_cond[0]`

File: fastvideo/pipelines/stages/matrixgame3_denoising.py:121

extrinsics_all = build_extrinsics_from_actions(batch.keyboard_cond[0], batch.mouse_cond[0]).to(device)

For batch_size > 1, every batch item silently inherits the trajectory of item 0. Current MG3 presets are single-sample so this never triggers in shipped paths, but anyone batching multi-prompt I2V will get incorrect trajectories with no error.

Suggested fix — either:

Loop per-batch-item and stack into a batched extrinsics_all, propagating the batch dim through subsequent slicing, OR
Assert batch.keyboard_cond.shape[0] == 1 at stage entry with a clear error documenting MG3's current batch-size=1 contract.

Option (2) is the cheap defensive fix and matches the actual user-facing surface.

S2-2 [persistent] — PR body Test Plan + Test Results are empty placeholders

PR checklist is all ✅ but the Test Plan and Test Results sections have no SSIM score, no CI/Modal run link, no example output path. SSIM test file exists and looks correct, but reviewers can't independently verify it has run. Recommend pasting the actual SSIM score (or a Buildkite/Modal link) before merge.

What's good

Heavy port shipped end-to-end after Matrix-Game 2.0 ([feat] Add Matrix-Game 2.0 #938) — author has track record on this exact surface.
Cross-cutting wanvae.py change is gated by use_light_vae=False default; Wan-2.1 / MG2.0 callers are bit-identical. vision_utils.resize "crop" mode is a new elif (additive). ForwardBatch.num_iterations / use_base_model are optional with safe defaults. This is exactly the right pattern for a shared-file modification.
Registry / preset / SSIM test / example / support_matrix.md all use the same HF ID FastVideo/Matrix-Game-3.0-Base-Distilled-Diffusers.
All 28 commits authored by @H1yori233, no AI co-author trailers.

— Gob (@SolitaryThinker's AI reviewer). Full review archived locally.

The matrixgame3_denoising stage derives camera/mouse extrinsics from batch.keyboard_cond[0] / batch.mouse_cond[0], which silently uses item-0's trajectory for the whole batch when batch_size > 1. Add a loud assertion documenting MG3's current batch-size=1 contract so future callers fail fast rather than getting incorrect trajectories. Addresses S2-1 from PR hao-ai-lab#1201 gob review (hao-ai-lab#1201 (comment)). Option 2 (assertion guard) chosen over option 1 (loop+stack) per maintainer direction; if batched action streams are eventually needed, build_extrinsics_from_actions can be made per-batch-item then.

Lowers MATRIXGAME3_PARAMS height/width from 720x1280 to 352x640 in the SSIM regression test (matches MatrixGame 2.0 convention from the MatrixGame2SamplingParam defaults). MATRIXGAME3_FULL_QUALITY_PARAMS remains 720x1280 — it derives from SamplingParam.from_pretrained and covers the full-quality tier separately. Compute reduction: ~4x vs 720p (352*640 / 720*1280 ≈ 0.24). Brings the SSIM test within typical L40S CI runtime budget so a HF reference can be seeded against this resolution. Followup to PR hao-ai-lab#1201 review: hao-ai-lab#1201 (comment)

Two related changes to fastvideo/tests/ssim/test_matrixgame3_similarity.py: 1. Lower MATRIXGAME3_PARAMS height/width from 720x1280 to 352x640 (matches MatrixGame 2.0 convention from MatrixGame2SamplingParam defaults). MATRIXGAME3_FULL_QUALITY_PARAMS remains 720x1280 — it derives from SamplingParam.from_pretrained and covers the full-quality tier separately. Compute reduction: ~4x vs 720p (352*640 / 720*1280 ≈ 0.24). 2. Add fallback_device_prefix='L40S' to resolve_device_reference_folder so non-tabled GPUs (B200, etc) fall back to L40S references with a clean warning instead of TypeError-crashing at build_generated_output_dir. Matches the precedent in fastvideo/tests/ssim/test_sd35_similarity.py. Together: brings the SSIM test within L40S CI runtime budget AND makes it runnable locally on B200 dev boxes before the HF reference is seeded. Followup to PR hao-ai-lab#1201 review: hao-ai-lab#1201 (comment)

SolitaryThinker · 2026-05-27T09:11:16Z

/merge

H1yori233 · 2026-05-27T20:53:08Z

/test vsa

H1yori233 and others added 7 commits March 27, 2026 22:31

add mg3 model

74c0c72

light vae for mg3

202a81b

cleanup

4dd54bf

fix action

c383441

some check in denoising stage

0cf1c10

update model

941e843

Merge remote-tracking branch 'origin/main' into feat/kaiqin/add-mg-3

ece6fab

mergify Bot added type: feat New feature or capability scope: training Training pipeline, methods, configs scope: inference Inference pipeline, serving, CLI scope: infra CI, tests, Docker, build scope: model Model architecture (DiTs, encoders, VAEs) labels Mar 31, 2026

gemini-code-assist Bot reviewed Mar 31, 2026

View reviewed changes

cleanup and precommit

1ae0efe

This comment was marked as resolved.

Sign in to view

mergify Bot added the needs-rebase PR has merge conflicts label Apr 5, 2026

H1yori233 added 2 commits May 14, 2026 09:06

Merge remote-tracking branch 'origin/main' into feat/kaiqin/add-mg-3

9146d1e

rename matrixgame2

99be70d

mergify Bot added scope: data Data preprocessing, datasets scope: docs Documentation and removed needs-rebase PR has merge conflicts labels May 14, 2026

H1yori233 added 8 commits May 15, 2026 07:34

inference script

5a040f5

cleanup

b92d9de

align light vae

1c26c2b

some fix

1715b18

update doc

5b3be80

grid_sizes use tuple instead of tensor

d209f16

removed unnecessary device transfers and optimized tensor creation fo…

a067cf2

…r indices in action and model modules.

H1yori233 added 3 commits May 19, 2026 01:32

Improved tensor creation by ensuring proper generator type and added …

956e85a

…prompt-based naming for output files

Refactor video file handling in SSIM test

4f8b730

some fix

d2e2041

gemini-code-assist Bot reviewed May 19, 2026

View reviewed changes

H1yori233 marked this pull request as ready for review May 19, 2026 02:23

some fix

d296ccf

H1yori233 force-pushed the feat/kaiqin/add-mg-3 branch from 0453dd6 to d296ccf Compare May 19, 2026 02:25

H1yori233 added 6 commits May 19, 2026 04:45

remove kvcache dead code

a1de2d7

remove deadcode

fd35b66

remove onboarding

96505d1

configuration and inference parameters

853db84

some fix

df40e2c

fix name

3528141

SolitaryThinker mentioned this pull request May 27, 2026

[docs]: PR #1201 (MG3 port) — Test Plan + Test Results sections empty #1406

Closed

SolitaryThinker approved these changes May 27, 2026

View reviewed changes

SolitaryThinker force-pushed the feat/kaiqin/add-mg-3 branch from dd27265 to cd6cd98 Compare May 27, 2026 08:23

SolitaryThinker force-pushed the feat/kaiqin/add-mg-3 branch from cd6cd98 to d0f9d12 Compare May 27, 2026 08:24

H1yori233 mentioned this pull request May 27, 2026

[tracker] Open add-model PRs #1407

Open

github-actions Bot added the ready PR is ready to merge label May 27, 2026

mergify Bot merged commit afdb6fb into hao-ai-lab:main May 27, 2026
24 of 25 checks passed

H1yori233 mentioned this pull request May 31, 2026

[RFC]: MG3 SPEED #1418

Closed

1 task

H1yori233 deleted the feat/kaiqin/add-mg-3 branch June 2, 2026 00:02

Uh oh!

Conversation

H1yori233 commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Changes

TODO

Test Plan

Test Results

Checklist

Uh oh!

mergify Bot commented Mar 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🟢 PR merge requirements

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot Mar 31, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

This comment was marked as resolved.

H1yori233 commented May 19, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot May 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 19, 2026

Choose a reason for hiding this comment

Uh oh!

gemini-code-assist Bot May 19, 2026

Choose a reason for hiding this comment

Uh oh!

SolitaryThinker commented May 27, 2026

TL;DR

Verdict: approve-with-followup

Gemini concerns status at HEAD 35281410cd

Findings

S2-1 [persistent] — Camera trajectory uses only batch.keyboard_cond[0] / batch.mouse_cond[0]

S2-2 [persistent] — PR body Test Plan + Test Results are empty placeholders

What's good

Uh oh!

SolitaryThinker commented May 27, 2026

Uh oh!

H1yori233 commented May 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

H1yori233 commented Mar 31, 2026 •

edited

Loading

mergify Bot commented Mar 31, 2026 •

edited

Loading

Gemini concerns status at HEAD `35281410cd`

S2-1 [persistent] — Camera trajectory uses only `batch.keyboard_cond[0]` / `batch.mouse_cond[0]`