[feat]: add MiniMax H3 Ref2VA and LoRA training support

[Yanghao-Wang]

Purpose

Add MiniMax H3 Ref2VA training support and enable LoRA training for both T2VA and Ref2VA in the modular trainer.

Exact reviewed head: 2495dfea005bdb7d5ffe163049a233e50e47f425

Changes

• Add the Ref2VA manifest contract, dataset, transactional preprocessing pipeline, model plugin, validation callback, and full/LoRA training configs.
• Insert LoRA modules on the meta model before FSDP so base weights and adapters share the same DTensor ownership, gradient synchronization, and DCP topology.
• Export H3 training checkpoints into the physical transformer/ or transformer_ref/ component using canonical Diffusers safetensors naming and 5 GB sharding, while preserving the legacy filename for models that have not opted in.
• Canonicalize activation-checkpoint wrapper paths while merging LoRA adapters for export; strict native reload and forward equivalence are covered by tests.
• Replace the earlier sample media with a repository-owned synthetic geometric video, image, and tone-audio fixture, including a generator, provenance metadata, and recorded hashes. The superseded media is absent from every commit reachable from this head; the original contributor commits retain their authorship.
• Drain staged CPU checkpoint tensors as the FSDP device state is materialized. This removes the simultaneous full CPU + CUDA checkpoint copies that caused the real H3 initialization gate to exceed GB10 unified memory.

Validation

All results below are from the exact reviewed head unless noted otherwise.

• Exact CI unit command (.buildkite/scripts/unit_test.sh): 964 passed, 7 skipped.
• Focused H3 training/export/FSDP/preprocessing/VAE contracts: 32 passed.
• Loader and mixed-dtype coverage: 7 passed, 1 expected GPU-only skip.
• Pinned Diffusers reference at abc5e9bf71fd38f53cd471bc3acaa84bc5ecbfdc: 35 passed with no parity skips.
• Synthetic fixture regeneration check: all three recorded SHA-256 hashes matched.
• Pre-commit passed on all PR paths before the final loader-memory delta. The final five paths are deliberately excluded from formatter/type/spelling hooks by repository policy; the filename hook passed.

The opt-in real-checkpoint gate passed on one 121 GiB NVIDIA GB10:

• strict load of all 14 transformer_ref safetensors shards;
• 33.30B base parameters;
• 312 LoRA wrappers and 624 trainable DTensor adapter parameters;
• 172,949,504 trainable adapter elements with the same FSDP placement as their base weights;
• no activation-checkpoint wrappers or per-layer CPU snapshots;
• 1 passed in 279.56 seconds, with 62.08 GiB peak CUDA allocation and 62.50 GiB peak reservation.

Evidence boundary

The real-checkpoint gate validates strict loading and LoRA/FSDP ownership only. It does not run a forward pass, dataloader, optimizer step, full training job, export, reload, or inference. The CPU contracts cover those shapes and transformations, but this PR does not claim a completed full-checkpoint training/export/inference run.

Full H3 export deliberately gathers an additional roughly 62 GiB state on rank 0 while the live model exists. A 121 GiB unified-memory GB10 is therefore not a validated full-export target; use a larger-memory host for that gate.

[mergify]

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

	Protection	Waiting on
🔴	PR merge requirements	👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

• #approved-reviews-by>=1
• check-success=full-suite-passed

This rule is failing.

• #approved-reviews-by>=1
• check-success=full-suite-passed
• check-success=fastcheck-passed
• check-success~=pre-commit
• title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

[AngadSethi]

@Yanghao-Wang, are you also planning to add support for FL2VA training?

[Yanghao-Wang]

@Yanghao-Wang, are you also planning to add support for FL2VA training?

Thanks for asking. FL2VA training is not included in the scope of this PR, which focuses on Ref2VA and LoRA support for T2VA/Ref2VA. It can be added in a follow-up PR. ^-^

[AngadSethi]

Got it, thanks!

[Yanghao-Wang]

@SolitaryThinker Hi maintainers, fastcheck is blocked by 8 unit/contract failures in Attn-QAT, schema inventory, CI collection, and FA4 policy tests.

I verified that the failing tests and corresponding source files are byte-identical to the base commit and are outside this PR’s changes. This PR appears to have triggered a broader test lane that exposed existing inconsistencies.

Could you please confirm and advise whether fastcheck can be overridden for this PR or if I should wait for an upstream fix?

The log is here.

[SolitaryThinker]

/merge

[SolitaryThinker]

/test full

[SolitaryThinker]

/test fastcheck

[SolitaryThinker]

/test full

[mergify]

This PR has merge conflicts with the base branch. Please rebase:

git fetch origin main
git rebase origin/main
# Resolve any conflicts, then:
git push --force-with-lease

[Yanghao-Wang]

Hi @SolitaryThinker, I noticed that this PR now needs a rebase after the recent changes to main. Since the current head includes your follow-up commits and you previously updated the branch directly, I wanted to coordinate before making any changes. Are you planning to handle the rebase, or would you prefer me to take care of it? I’m happy to do so — I just want to avoid conflicting pushes. Thanks!

[Yanghao-Wang]

Hi @SolitaryThinker, since the branch has remained unchanged and main has continued to move, I’m going to proceed with rebasing #1757 onto the current main to unblock it. I’ll preserve your follow-up commits and integrate the #1771 inference loader path with the training pre-FSDP path. I’ll use --force-with-lease to avoid overwriting any concurrent update. Please let me know if you are already working on the branch.

[Yanghao-Wang]

Quick update: I’ve completed the rebase onto the current main and pushed it with --force-with-lease. The PR is now mergeable, and fastcheck passed. The Trigger Merge Gate stopped at “Cancel previous Buildkite builds” before the merge tests were launched. It looks like the trigger may need a maintainer rerun when convenient. Thanks!

[Yanghao-Wang]

Hi @SolitaryThinker, just following up on the rebased head. Fastcheck is green, and the PR is mergeable, but the merge gate has not been retriggered yet. Is there anything else you’d like me to address before it can proceed? Thanks!