Skip to content

[feat]: add opt-in CUDA TAEH3 preview decode for FastH3 - #1795

Merged
SolitaryThinker merged 1 commit into
hao-ai-lab:mainfrom
aryan5v:aryan/h3-cuda-taeh3-preview
Sep 1, 2026
Merged

[feat]: add opt-in CUDA TAEH3 preview decode for FastH3#1795
SolitaryThinker merged 1 commit into
hao-ai-lab:mainfrom
aryan5v:aryan/h3-cuda-taeh3-preview

Conversation

@aryan5v

@aryan5v aryan5v commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Adds an opt-in CUDA/CPU TAEH3 preview decoder (--video-decode-backend taeh3) so FastH3 T2VA never materializes the 9.7 GiB video VAE.
  • Sequential start (already on main via [perf] Sequential MiniMax H3 start with GPU-direct DiT load #1793) still encodes with Qwen, releases it, then loads DiT + audio VAE. TAEH3 T2VA omits the video VAE from that second load. FL2VA/Ref2VA still require --video-decode-backend h3-vae.
  • Unpatchify reads DiT patch_size from pipeline config / checkpoint config.json, so decode does not need a live transformer just for three integers.
  • Pinned TAEH3 weights: SHA-256 4fd022bfcab08772fe0536b17ea1a3bbb5625be11e397868d1c5d891863d4c13 (madebyollin/taehv @ 62f7591). Reconstruction is approximate, not lossless.

This is independent of #1761. It targets main (which already has sequential start #1793 and MLX TAEH3 #1794). Do not stack it on lazy-module-load.

Spark2 GB10 alpine 768×1344×124, VSA-DataFree, seed 2026: TAEH3 decode 2.4 s vs full VAE 68 s; one T2VA generation 224 s end-to-end.

Test plan

  • pytest TAEH3 CPU contracts, sequential-start skip-VAE, VAE streaming, FastH3 example (25 passed)
  • Spark GB10 TAEH3 T2VA (earlier session): 2.4 s decode, 224 s e2e
  • Reviewer: python examples/inference/basic/basic_fasth3.py --num-gpus 1 --video-decode-backend taeh3 --no-warmup --repeats 1 ...

Skip the 9.7 GiB video VAE on T2VA so GB10 can reconstruct alpine 768 in a couple of seconds instead of a full VAE pass. Independent of lazy-module-load; uses sequential start already on main.
@mergify mergify Bot added type: feat New feature or capability scope: inference Inference pipeline, serving, CLI scope: infra CI, tests, Docker, build scope: docs Documentation scope: model Model architecture (DiTs, encoders, VAEs) labels Aug 31, 2026
@mergify

mergify Bot commented Aug 31, 2026

Copy link
Copy Markdown
Contributor

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

Protection Waiting on
🔴 PR merge requirements 👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

  • #approved-reviews-by>=1
  • check-success=full-suite-passed
This rule is failing.
  • #approved-reviews-by>=1
  • check-success=full-suite-passed
  • check-success=fastcheck-passed
  • check-success~=pre-commit
  • title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

@SolitaryThinker SolitaryThinker left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found three issues that should be addressed before merge:

  • [P2] Preserve latent-output semantics. In fastvideo/pipelines/basic/minimax_h3/stages/minimax_h3_decoding.py, the TAEH3 branch returns normalized diffusion latents for output_type="latent", while the existing H3-VAE branch calls denormalize_latents() first. MiniMax H3's published latent mean/std are non-identity, so the meaning of latent output currently changes solely with the decoder selection even though no decoding occurs. Please use the pipeline VAE config to apply the same denormalization without loading the full VAE weights.

  • [P2] Avoid decoding the complete video on every SP rank. _decode_participation() lets every rank continue when vae_parallel_decode=True, but the TAEH3 branch performs no distributed sharding or collectives; every rank runs the full decoder and allocates the full output. basic_fasth3.py enables parallel VAE decode by default, so the documented multi-GPU path hits this. Please select serial output-rank participation for TAEH3 or implement actual sharding.

  • [P2] Add committed upstream numerical parity. test_taeh3_chunk_sizes_agree_on_cpu compares two execution paths within the new implementation, not either path against the pinned TAEH3 reference. This PR does not satisfy the add-model component gate because a non-skip official-reference comparison is missing; the risk is undetected drift in temporal memory or per-chunk frame trimming. I independently compared the current implementation against the pinned reference on CPU and observed exact agreement for a full chunk and max error around 4.3e-6 for a split-chunk run, but that evidence should be captured in a committed parity test.

Fastcheck and pre-commit are green. Full Suite and GPU media-quality verification are still pending.

@SolitaryThinker
SolitaryThinker merged commit cf6a00b into hao-ai-lab:main Sep 1, 2026
17 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

scope: docs Documentation scope: inference Inference pipeline, serving, CLI scope: infra CI, tests, Docker, build scope: model Model architecture (DiTs, encoders, VAEs) type: feat New feature or capability

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants