[feat]: add opt-in CUDA TAEH3 preview decode for FastH3 - #1795
Conversation
Skip the 9.7 GiB video VAE on T2VA so GB10 can reconstruct alpine 768 in a couple of seconds instead of a full VAE pass. Independent of lazy-module-load; uses sequential start already on main.
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI
🔴 PR merge requirementsWaiting for
This rule is failing.
|
SolitaryThinker
left a comment
There was a problem hiding this comment.
I found three issues that should be addressed before merge:
-
[P2] Preserve latent-output semantics. In
fastvideo/pipelines/basic/minimax_h3/stages/minimax_h3_decoding.py, the TAEH3 branch returns normalized diffusion latents foroutput_type="latent", while the existing H3-VAE branch callsdenormalize_latents()first. MiniMax H3's published latent mean/std are non-identity, so the meaning of latent output currently changes solely with the decoder selection even though no decoding occurs. Please use the pipeline VAE config to apply the same denormalization without loading the full VAE weights. -
[P2] Avoid decoding the complete video on every SP rank.
_decode_participation()lets every rank continue whenvae_parallel_decode=True, but the TAEH3 branch performs no distributed sharding or collectives; every rank runs the full decoder and allocates the full output.basic_fasth3.pyenables parallel VAE decode by default, so the documented multi-GPU path hits this. Please select serial output-rank participation for TAEH3 or implement actual sharding. -
[P2] Add committed upstream numerical parity.
test_taeh3_chunk_sizes_agree_on_cpucompares two execution paths within the new implementation, not either path against the pinned TAEH3 reference. This PR does not satisfy the add-model component gate because a non-skip official-reference comparison is missing; the risk is undetected drift in temporal memory or per-chunk frame trimming. I independently compared the current implementation against the pinned reference on CPU and observed exact agreement for a full chunk and max error around 4.3e-6 for a split-chunk run, but that evidence should be captured in a committed parity test.
Fastcheck and pre-commit are green. Full Suite and GPU media-quality verification are still pending.
Summary
--video-decode-backend taeh3) so FastH3 T2VA never materializes the 9.7 GiB video VAE.--video-decode-backend h3-vae.patch_sizefrom pipeline config / checkpointconfig.json, so decode does not need a live transformer just for three integers.4fd022bfcab08772fe0536b17ea1a3bbb5625be11e397868d1c5d891863d4c13(madebyollin/taehv @ 62f7591). Reconstruction is approximate, not lossless.This is independent of #1761. It targets
main(which already has sequential start #1793 and MLX TAEH3 #1794). Do not stack it onlazy-module-load.Spark2 GB10 alpine 768×1344×124, VSA-DataFree, seed 2026: TAEH3 decode 2.4 s vs full VAE 68 s; one T2VA generation 224 s end-to-end.
Test plan
pytestTAEH3 CPU contracts, sequential-start skip-VAE, VAE streaming, FastH3 example (25 passed)python examples/inference/basic/basic_fasth3.py --num-gpus 1 --video-decode-backend taeh3 --no-warmup --repeats 1 ...