[feat] Add MiniMax H3 MLX T2VA inference

[aryan5v]

Purpose

Add MiniMax H3 Preview text-to-video-with-audio inference to FastVideo's existing Apple Silicon MLX runtime.

The initial scope is intentionally narrow: T2VA baseline generation plus temporal --fast. FL2VA, Ref2VA, spatial fast mode, two-pass refinement, VSA, and VideoGenerator registry dispatch remain follow-up work.

Changes

• Add a native MLX MiniMax H3 DiT with the upstream packed audio/video layout, dual schedulers, AdaLN step caching, and affine INT8, INT6, and INT4 checkpoints.
• Stream the Qwen3-VL text conditioner by embedding row and decoder layer so the released encoder does not become fully resident.
• Add native MLX video and audio VAE loaders and decoders, including tiled video decode and stereo 32 kHz audio.
• Add a phased T2VA pipeline that loads one heavyweight component at a time and muxes H.264 video with AAC audio.
• Add temporal fast mode that denoises fewer video rows, preserves full-duration audio, and reconstructs the requested frame count with the existing MLX RIFE backend.
• Add a local checkpoint converter, runnable example, Apple Silicon setup guide, support-matrix entry, and parity coverage.

Test Plan

pytest -q \
  fastvideo/tests/mlx/test_mlx_minimax_h3_parity.py \
  fastvideo/tests/mlx/test_mlx_minimax_h3_conditioner.py \
  fastvideo/tests/mlx/test_mlx_minimax_h3_fast_mode.py \
  tests/local_tests/minimax_h3/test_mlx_video_vae_parity.py \
  tests/local_tests/minimax_h3/test_mlx_audio_vae_parity.py

pre-commit run --files \
  fastvideo/mlx_runtime/__init__.py \
  fastvideo/mlx_runtime/fastwan.py \
  fastvideo/mlx_runtime/rife_interp.py \
  fastvideo/mlx_runtime/minimax_h3.py \
  fastvideo/mlx_runtime/minimax_h3_audio_vae.py \
  fastvideo/mlx_runtime/minimax_h3_video_vae.py \
  fastvideo/mlx_runtime/minimax_h3_conditioner.py \
  fastvideo/mlx_runtime/minimax_h3_pipeline.py \
  examples/inference/basic/mlx_fasth3.py \
  docs/getting_started/installation/mps.md \
  docs/inference/support_matrix.md \
  examples/inference/basic/README.md

python -m compileall -q \
  fastvideo/mlx_runtime \
  examples/inference/basic/mlx_fasth3.py \
  scripts/checkpoint_conversion/convert_minimax_h3_mlx.py

Test Results

Validated locally on an Apple M4 Max with 36 GB unified memory.

Run	Configuration	Output	Phase timings	Peak MLX memory
Baseline T2VA	INT6, 832x480, 124 frames, 24 fps, 4 denoise steps, seed 2026	5.17 s H.264 + stereo AAC 32 kHz; Whisper transcript: "Fast H3 is amazing."	condition 77.26 s; denoise 383.73 s; video decode 103.32 s; audio decode 0.67 s; mux 0.39 s; total 565.37 s	condition 1.93 GiB; denoise 19.63 GiB; video decode 11.03 GiB; audio decode 2.64 GiB
Temporal fast T2VA	INT6, 1280x720, 124 frames, 24 fps, 4 denoise steps, seed 2027, --fast	5.17 s H.264 + stereo AAC 32 kHz; Whisper transcript: "Fast mode and Fast H3 is even faster."	condition 85.98 s; denoise 703.12 s; video decode 142.29 s; RIFE 12.59 s; audio decode 0.78 s; mux 0.55 s; total 945.31 s	condition 1.93 GiB; denoise 21.10 GiB; video decode 12.09 GiB; RIFE 1.53 GiB; audio decode 2.48 GiB

The two runs use different resolutions, so their wall times are not a same-resolution speed comparison. Temporal fast mode reduced the 124-frame request from 37 target video latent frames to 22 denoised video latent frames while retaining all 207 audio latent frames.

Focused test output

44 passed, 21 warnings in 13.13s

yapf........................................Passed
ruff (legacy alias)........................Passed
codespell...................................Passed
PyMarkdown.................................Passed
mypy........................................Passed
Check for spaces in all filenames..........Passed
Suggestion.................................Passed

SSIM: Not run. There is no MiniMax H3 MLX reference artifact in the SSIM suite yet. The PR instead includes independent PyTorch/MLX DiT, scheduler, packing, conditioner, and real-weight bounded VAE parity gates plus two end-to-end media runs.

Checklist

• I ran pre-commit on every changed file covered by the repository hooks and fixed all issues
• I added or updated tests for my changes
• I updated documentation if needed
• I considered GPU and unified-memory impact of my changes

For model/pipeline changes, also check:

• I verified SSIM regression tests pass (not available for this new MLX path)
• I updated the support matrix if adding a new model

[mergify]

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

	Protection	Waiting on
🔴	PR merge requirements	👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

• #approved-reviews-by>=1
• check-success=full-suite-passed

This rule is failing.

• #approved-reviews-by>=1
• check-success=full-suite-passed
• check-success=fastcheck-passed
• check-success~=pre-commit
• title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

[mergify]

Pre-commit checks failed

Hi @aryan5v, the pre-commit checks have failed. To fix them locally:

# Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install

# Run all checks and auto-fix what's possible
pre-commit run --all-files

Common fixes:

• yapf: yapf -i <file> (formatting)
• ruff: ruff check --fix <file> (linting)
• codespell: codespell --write-changes <file> (spelling)

After fixing, commit and push the changes. The checks will re-run automatically.

For future commits, pre-commit will run automatically on changed files before each commit.

[SolitaryThinker]

Review — MiniMax H3 MLX T2VA inference

Overview

Adds a native Apple Silicon (MLX) text-to-video-with-audio path for MiniMax H3: a packed joint audio/video DiT with dual rectified-flow schedulers and an AdaLN precompute cache, a layer-streamed Qwen3-VL conditioner, native video/audio VAE decoders, a phased pipeline (condition → denoise → decode → mux), temporal --fast mode via RIFE, a checkpoint converter (INT8/INT6/INT4), docs, and parity tests against the upstream PyTorch reference. Scope is deliberately narrow (T2VA only) and honestly documented.

Overall: high quality. The memory engineering is thoughtful and explained in place (per-block mx.eval with the observed OOM rationale, streamed shard loading, one-heavyweight-component-at-a-time phasing), formats are versioned (H3_FORMAT_VERSION, prompt-cache version), the mux writes atomically via tmp+rename, quantization refuses to silently fall back, and the parity-test strategy (tiny random-weight models vs. the torch reference, plus real-weight bounded segments) carries the numerical-correctness burden well. Note: this review is static — I did not run the MLX/Metal paths.

Issues

Correctness / UX

• --steps ≠ 4 crashes with a raw KeyError on every converted checkpoint. The converter always builds the AdaLN cache and drops the projection weights (convert_minimax_h3_mlx.py passes adaln_cache_timesteps; _flatten_h3_weights skips the None entries, so saved checkpoints have no adaln_proj.linear.*). When the pipeline sees a ladder mismatch it calls dit.precompute_adaln(union, drop_weights=True) (minimax_h3_pipeline.py denoise()), which hits block["adaln_proj.linear.weight"] in _adaln_tables → KeyError: 'adaln_proj.linear.weight' — after ~80 s of conditioning and a full DiT load. Since the example advertises --steps, this deserves a first-class error ("checkpoint exported with a fixed 4-step AdaLN ladder; use --steps 4 or re-export") raised before any heavy work, or ideally in parse_args/pipeline preflight by reading the checkpoint manifest.
• Late-failure preflight gaps. ffmpeg availability is checked only in mux() — i.e., after 10–15 minutes of generation. Similarly, --fast with the default fast_sharpen=0.6 imports cv2 only after denoise + decode + RIFE (_sharpen_frames). _validate_inputs_before_loading() already validates model files up front; extend that pattern: check ffmpeg at generate() entry, and cv2/RIFE weights when fast is requested. Also worth confirming opencv is a declared dependency of the MLX extra.
• mlx_h3_audio_vae_from_dir silently ignores its storage_dtype parameter — it's accepted but never forwarded to mlx_h3_audio_vae_from_file. The pipeline doesn't pass it so there's no behavior bug today, but a caller passing bf16 gets fp32 with no warning. Honor it or remove it.
• MiniMaxH3StepCache.positions() can raise IndexError instead of its intended ValueError: np.searchsorted returns len(timesteps) for a value above the cached max, so self.timesteps[positions] goes out of bounds before the friendly mismatch message fires. Clamp or pre-check.

Robustness (minor)

• Prompt-cache writes aren't atomic (encode_prompt): a crash mid-np.savez leaves a corrupt .npz that every future run will try to load (and fail on) since the key is content-hashed. The mux already does tmp+replace correctly — same pattern would fit here. Relatedly, an ffmpeg failure leaves .tmp.mp4/.tmp.wav behind.
• metal_wired_limit_gib=30.0 is a fixed default regardless of device memory; consider deriving from mx.metal.device_info() (nit — docs already gate this at the 36 GB tier).

Performance (nits, non-blocking)

• _reflect_pad_axis (minimax_h3_video_vae.py) concatenates one single-index mx.take per output index — including the entire unpadded body — so it's O(H) kernel launches per pad. Decode doesn't hit it (encoder-only), but it will hurt the future Ref2VA/encode path; concatenating [left_reflection, x, right_reflection] in 3 pieces (like the audio module's _replicate_pad) avoids it.
• _ShardIndex.get/get_row re-open and re-parse the safetensors JSON header on every call — once per token for embeddings and once per weight per layer. Caching the parsed header per shard path is a one-liner and should trim the 77–86 s conditioning phase.

Dead / vestigial code

• mlx_h3_bf16_forward_streamed_from_diffusers_safetensors (~150 lines) is not called by any test or the pipeline — presumably the manual real-weight validation harness. Either wire a (skippable) test to it or move it out of the production module.
• Also unused: resolve_canvas_size, the compile param / _enable_compile / _compiled_forward machinery (never read in either forward), ACTIVATION_KERNEL_SIZE and the activation_ratio property (audio VAE), _ShardIndex.has/keys_with_prefix, the import mlx.core as _mx # noqa line in mlx_h3_audio_vae_from_dir, the token_pad() helper that ignores its argument and returns 0 (test_mlx_video_vae_parity.py), and apparently-unused imports in the parity test (AUDIO_TIMESTEP, VIDEO_TIMESTEP, build_schedulers).
• Hardcoded model dims in the pipeline — noise shapes (rows, 96) / (rows, 32), decode channels 24, //16 spatial ratio — are all derivable from the DiT/VAE configs already loaded. Fine for a fixed model, but one config change away from silent shape drift; at minimum assert against dit.patch_dim/vae.latent_channels.
• Style nit: the double-conditional in _validate_inputs_before_loading (if not any(...) if vae_dir.exists() else True:) is a precedence puzzle; if not (vae_dir.exists() and any(...)) reads instantly.

Test coverage

Strong where it matters most (DiT forward, scheduler, packing, AdaLN cache-vs-faithful, INT8 SNR, VAE primitives + real-weight bounded segments, RIFE frame-count math, geometry contracts). Gaps worth closing:

• No save_mlx_h3_checkpoint → load_mlx_h3_checkpoint round-trip test. This format is the shipping artifact (quantized rebuild, manifest versioning, AdaLN-cache persistence), it's pure logic that runs on the tiny fixture in CI, and a regression here bricks every user's converted checkpoint. A tiny round-trip (save quantized DiT + cache → load → compare forward_with_cache output) would also have surfaced the --steps ≠ 4 KeyError.
• No test exercises denoise()/generate() orchestration (understandable given hardware needs, but the steps-mismatch path above shows the cost).
• tests/local_tests/ real-weight cases gate on a local snapshot — good that they self-skip; note they'll be perpetually skipped in CI, so the FP32 acceptance claims rest on local runs.

Security

No concerns: safetensors only (no pickle), np.load without allow_pickle, subprocess.run with list args and no shell, AutoTokenizer.from_pretrained on local dirs without trust_remote_code. The hand-rolled safetensors header parser trusts header_len from the file, but inputs are user-supplied local checkpoints — acceptable.

Conventions

Matches the repo: SPDX headers, init_logger, yapf/ruff/mypy per the pre-commit run, pytest.importorskip gating, and fastvideo.mlx_runtime stays importable without MLX (the pipeline lazily imports the module-level-mx VAE/conditioner modules — works, though the inconsistency with minimax_h3.py's function-level import mlx.core blocks is worth a follow-up cleanup). Docs/support-matrix updates and the MiniMax license note are appreciated.

Verdict

Approve with minor changes. Nothing here is architecturally wrong and the numerics are well-gated. Before merge: (1) a clear, early error for --steps ≠ 4 on AdaLN-dropped checkpoints, (2) fail-fast preflight for ffmpeg/cv2 in generate(), (3) fix or drop the ignored storage_dtype, and (4) a checkpoint save/load round-trip test. The rest (atomic cache writes, dead-code trim, header caching, reflect-pad perf) can be follow-ups.

---

🤖 Review generated with Claude Code

[aryan5v]

Addressed the requested pre-merge changes in efa55ed:

• Reject an incompatible fixed AdaLN step ladder before prompt conditioning or DiT loading, with a clear re-export message.
• Check ffmpeg, OpenCV when sharpening is enabled, and RIFE weights before generation starts.
• Reject non-FP32 audio VAE storage instead of silently ignoring storage_dtype.
• Add an INT8 checkpoint save/load round-trip that verifies persisted AdaLN cache inference.
• Return the intended ValueError for an out-of-range cached timestep.

Validation:

39 passed, 20 warnings in 12.84s
pre-commit: passed
compileall: passed
git diff --check: passed

The non-blocking cleanup and broader follow-up suggestions remain outside this PR's narrow T2VA scope.

[aryan5v]

Pushed the remaining cleanup and performance review items in a8ef3c5:

• Made prompt-cache writes atomic and ensured failed muxes remove temporary media files.
• Derived the default wired-memory limit from physical memory while retaining the tested 30 GiB cap.
• Replaced per-element reflect padding with three-piece concatenation.
• Removed the unused streamed BF16 harness, canvas resolver, compile flags, conditioner helpers, audio constants/property, and test stubs/imports.
• Derived denoise and decode dimensions from the DiT/VAE configs and added mismatch checks.
• Validated decoder weights before constructing the audio VAE and made block/cache pairing strict.
• Reused checkpoint filename constants in the converter and simplified the remaining test imports.

Validation:

44 passed, 21 warnings in 13.13s
pre-commit: passed, including mypy
compileall: passed
git diff --check: passed