[perf]: Add guarded MiniMax-H3 Sol Engine inference optimizations

[H1yori233]

Summary

This rebases the unique MiniMax-H3 Sol Engine work onto current main and narrows it to two default-off inference optimizations:

• precompute fixed-schedule AdaLN projections and reuse exact per-step tables;
• use packed sequence-parallel QKV relayout around a direct all-to-all path.

The rebase preserves the already-merged per-instance compile policy, packed-varlen FlashAttention 4 path, compile-safe H3 fusion custom ops, and generic fused Ulysses implementation from main. Superseded fusion rewrites from the old branch were dropped.

Runtime contracts

• AdaLN precompute rejects layerwise DiT offload, FSDP inference, and incompatible checkpoint/schedule reuse before mutating the transformer.
• Schedule construction, latent transfer, trajectory projection, and denoising all return a full-CPU-offloaded transformer to CPU on failure.
• AdaLN tables are installed atomically. The cursor is device-following and nonpersistent; the irreversible post-precompute module is explicitly not a reloadable checkpoint surface.
• Packed sequence parallelism is active only for no-grad inference with SP world size greater than one, batch size one, dense FlashAttention, and the supported H3 layout. Grad-enabled execution uses the existing autograd-aware Ulysses path.
• SP=1 is an identity path and does not require a process group or force FlashAttention.
• Direct collectives validate live group, backend, world size, tensor/group device, contiguity, and sequence metadata. Q/K/V relayout validates matching shape, dtype, device, CUDA residency, and positive world size.

Both flags and their lifecycle, memory, backend, and topology constraints are documented in the inference optimization guide.

CI ownership

• The existing transformer fastcheck lane runs a real world-4 Gloo contract. Its oracle depends on Q, K, and V and proves rank ordering, semantic trimming, and both collectives without pretending to be CUDA evidence.
• The existing four-GPU SSIM lane runs the same test first in strict mode. It fails unless four CUDA devices, NCCL, and the production Triton relayout route execute and emit mode=cuda-production.
• The nested launcher preserves the runner-assigned MASTER_PORT. No new CI lane or runner identity is introduced.

Exact-head validation

Reviewed head: 7299c643e466e007b0c46e33efb958c1dd14d72b

• Canonical unit lane: 952 passed, 7 skipped.
• Focused H3 fusion, compile, routing, AdaLN, relayout, distributed, planner, and CI-contract suite: 81 passed; the final latent-transfer cleanup case also passed after root review.
• Real world-4 Gloo contract: passed with mode=gloo-contract.
• Local one-GPU GB10: exact CUDA relayout/custom-op compile coverage passed.
• Project pre-commit hooks, mypy, shell syntax, and git diff --check: passed.
• Contributor attribution is preserved; all commits are trailer-free.

The local machine exposes one GPU, so strict CUDA world-4 cannot run locally. The requested full suite must provide that release gate before approval. Full H3 checkpoint media/SSIM parity was not run locally; the CI quality lane remains authoritative.

The earlier 4x GB200 timing on the historical branch was one post-warmup dense-FA4 DiT-stage sample collected before this rebase. It is retained only as scoped motivation, not as an exact-head performance claim.

[mergify]

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

	Protection	Waiting on
🔴	PR merge requirements	👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

• #approved-reviews-by>=1
• check-success=full-suite-passed

This rule is failing.

• #approved-reviews-by>=1
• check-success=full-suite-passed
• check-success=fastcheck-passed
• check-success~=pre-commit
• title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

[mergify]

Pre-commit checks failed

Hi @H1yori233, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

• yapf: yapf -i <file> (formatting)
• ruff: ruff check --fix <file> (linting)
• codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

[mergify]

This PR has merge conflicts with the base branch. Please rebase:

git fetch origin main
git rebase origin/main
# Resolve any conflicts, then:
git push --force-with-lease

[SolitaryThinker]

Thanks for the optimization work. I found two correctness blockers and one evidence gap that should be resolved before merge.

1. FASTVIDEO_MINIMAX_H3_ADALN_PRECOMPUTE=1 is incompatible with the default dit_layerwise_offload=True. The offload hook replaces each block's parameters with zero-sized placeholders and materializes them only in that block's pre_forward; prepare_adaln_trajectory() calls block.adaln_proj directly outside the hook. The current guard rejects FSDP but not layerwise offload, so the advertised opt-in fails under the default inference configuration. Please either support materialization safely or reject the combination during pipeline initialization with an actionable message. I validated the early-rejection approach against default layerwise offload, FSDP, rank-reduced checkpoints, schedule reuse, and the default-off path (42 focused tests passed).

2. The branch now conflicts with main after #1741 in fastvideo/attention/layer.py. The resolution needs to retain #1741's deferred, per-attention-instance compile boundary; restoring the earlier class-definition-time wrapper would regress regional compile. Please rebase and rerun the exact-head dense compile/SP tests.

3. The packed-SP collective has local kernel tests but no real world-4 collective parity/failure gate, and the route is not guarded by torch.is_grad_enabled(). Its custom collective has no registered backward, so enabling the environment flag can intercept a training forward even though the feature is described as inference-only. Please add a no-grad guard (or backward) and a real multi-rank parity test covering both directions and uneven local sequence padding.

Please also narrow or qualify “lossless”: the fusion test compares compiled fused output with the same eager fused operation, not with the unfused full pipeline. Existing fixed-seed Preview F4 evidence is deterministic but materially non-parity versus the eager/unfused route, so that serving profile remains report-only.

For integration provenance, this PR is useful but does not by itself reproduce the reported Preview SP-4 latency: sparse VSA cannot enable FASTVIDEO_MINIMAX_H3_PACKED_SP, and the 6.9-second route additionally needs temporal-parallel VAE decode (#1744). AdaLN precompute and packed SP were both disabled in that measurement.

The requested H3-specific relayout placement is implemented and the corresponding thread is resolved on the rebased head.

[SolitaryThinker]

/test full

[SolitaryThinker]

/merge

[SolitaryThinker]

/merge

[SolitaryThinker]

/test full

[SolitaryThinker]

/test full

[mergify]

This PR has merge conflicts with the base branch. Please rebase:

git fetch origin main
git rebase origin/main
# Resolve any conflicts, then:
git push --force-with-lease