[new-model] Add native Helios-Distilled T2V pipeline by FlappyBob · Pull Request #1670 · hao-ai-lab/FastVideo

FlappyBob · 2026-07-31T21:45:36Z

Purpose

Add native, runnable FastVideo support for BestWishYsh/Helios-Distilled
text-to-video generation. This expands the original transformer-only PR with
the scheduler, stage-composed pipeline, registry/preset wiring, typed inference
example, component parity, pipeline parity, and real-video evidence requested
during review.

Supported Scope

Helios-Distilled T2V;
official Diffusers-layout checkpoint at revision
1999182614cb08d3bdcc46b9827504af2914b87b;
native Helios transformer and DMD scheduler;
native FastVideo pipeline using the existing Wan VAE, UMT5 encoder, and
tokenizer loader after exact-asset parity.

Base/Mid checkpoints, transformer_ode, training, ODE workflows, I2V, and
conditioned generation are outside this PR.

Changes

add HeliosConfig and native HeliosTransformer3DModel with direct
1101/1101 checkpoint-key compatibility;
add native HeliosDMDScheduler and scheduler registry resolution;
add HeliosPipelineConfig, helios_distilled_t2v preset, exact model
detection, and HeliosPyramidPipeline entry-class resolution;
compose validation, shared text encoding/conditioning, Helios-specific
autoregressive pyramid denoising, and chunk decoding stages;
expose Helios sampling controls through SamplingParam, typed
SamplingConfig, CLI parsing, schema inventory, and ForwardBatch;
add a typed public inference example at
examples/inference/basic/basic_helios_distilled_t2v.py;
add component, stage, smoke, end-to-end latent, and real-video integrity
tests plus reviewer setup/state records.

The worker moves final decoded/latent outputs to CPU before multiprocessing
return. This prevents CUDA-IPC allocation OOM after high-memory inference and
is protected by a regression test.

Pinned References

architecture: PKU-YuanGroup/Helios@8f2a2faab3298c8a7630a2c73aea37c01b5bab01;
executable parity reference: Diffusers 0.39.0;
checkpoint: BestWishYsh/Helios-Distilled@1999182614cb08d3bdcc46b9827504af2914b87b.

Test Results

Components

Transformer: 24 passed, 0 skipped
  strict load: 1101/1101
  full BF16 normal abs-mean drift: 0.0136%
  full BF16 pyramid abs-mean drift: 0.0305%
  tiny normal/pyramid: exact
  FlashAttention-vs-SDPA: exact
  SP=2 max/mean diff: 0.00000036 / 0.00000005

VAE + UMT5/tokenizer + scheduler: 17 passed, 0 skipped
  VAE decode max/mean diff: 0 / 0
  UMT5 FP32 max/mean diff: 0.00000125 / 0.00000010
  UMT5 BF16 max/mean diff: 0.02343750 / 0.00160135
  scheduler stages/steps: bit-exact

Pipeline

math/stage/smoke: 29 passed
end-to-end latent parity: 1 passed in 82.85 s
shape: [1, 16, 9, 16, 24]
cosine: 0.97970295
abs-mean drift: 0.3763%
mean absolute diff: 0.17001711
RMSE: 0.24650776

Public Example

H.264, 640x384, 24 FPS, 33 frames, 1.375 s
generation: 29.43 s
full ffmpeg decode: passed
container/non-black-frame gate: passed
SHA-256: 2e6d5099715256b16b865cb300479b328b2835f9f8792cb5427a98c17eb0b038

Static Checks

pre-commit run --all-files: all hooks passed

CI Base Alignment

The branch is rebased onto upstream/main@a159b63c. The current shared
.buildkite/scripts/unit_test.sh passes all 1047 collected tests locally in
28.70 seconds. pre-commit run --all-files also passes.

Quality Regression

Local real-video integrity and non-black-frame checks pass. Publishing a CI
SSIM reference is deferred because reference upload requires separate approval;
this PR does not claim CI SSIM passed.

Checklist

gemini-code-assist · 2026-07-31T21:45:39Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

github-actions

Welcome to FastVideo! Thanks for your first pull request.

How our CI works:

PRs run a two-tier CI system:

Pre-commit — formatting (yapf), linting (ruff), type checking (mypy). Runs immediately on every PR.
Fastcheck — core GPU tests (encoders, VAEs, transformers, kernels, unit tests). Runs automatically via Buildkite on relevant file changes (~10-15 min).
Full Suite — integration tests, training pipelines, SSIM regression. Runs only when a reviewer adds the ready label.

Before your PR is reviewed:

pre-commit run --all-files passes locally
You've added or updated tests for your changes
The PR description explains what and why

If pre-commit fails, a bot comment will explain how to fix it. Fastcheck and Full Suite results appear in the Checks section below.

Useful links:

mergify · 2026-07-31T21:46:17Z

Merge Protections

🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI

	Protection	Waiting on
🔴	PR merge requirements	👀 reviews and 🤖 CI

🔴 PR merge requirements

Waiting for

#approved-reviews-by>=1
check-success=full-suite-passed

This rule is failing.

#approved-reviews-by>=1
check-success=full-suite-passed
check-success=fastcheck-passed
check-success~=pre-commit
title~=(?i)^\[(feat|feature|bugfix|fix|refactor|perf|ci|doc|docs|misc|chore|kernel|new.?model|skill|skills|infra)\]

gemini-code-assist · 2026-08-03T00:27:52Z

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

mergify · 2026-08-04T21:56:43Z

This PR has merge conflicts with the base branch. Please rebase:

git fetch origin main
git rebase origin/main
# Resolve any conflicts, then:
git push --force-with-lease

mergify · 2026-08-10T00:32:23Z

This PR has merge conflicts with the base branch. Please rebase:

git fetch origin main
git rebase origin/main
# Resolve any conflicts, then:
git push --force-with-lease

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 099c09867c

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-08-29T21:52:44Z

+
+        temporal_scale = fastvideo_args.pipeline_config.vae_config.arch_config.scale_factor_temporal
+        generated_frames = get_generated_pixel_frames(frames.shape[2], temporal_scale)
+        batch.output = frames[:, :, :generated_frames].detach().to(dtype=torch.float32, device="cpu")


Trim decoded chunks to the requested frame count

For the default preset, num_frames=240 creates eight 33-frame chunks, but this calculation retains 261 frames (get_generated_pixel_frames(264, 4)) rather than the requested 240. The same issue is more pronounced for small non-chunk-aligned requests—for example, requesting 34 frames returns 65—so saved-video duration and API metadata no longer match the caller's request; crop to batch.num_frames or reject unsupported frame counts.

Useful? React with 👍 / 👎.

Fixed in 3238ae7. The chunk decoder now caps decoded output at batch.num_frames, and the regression test requests 34 frames from two chunks and asserts an exact 34-frame result.

chatgpt-codex-connector · 2026-08-29T21:52:44Z

+        raise ValueError("Helios UMT5 produced NaN hidden states")
+    trimmed = [hidden[:length] for hidden, length in zip(hidden_states, sequence_lengths, strict=True)]
+    return torch.stack(
+        [torch.cat([hidden, hidden.new_zeros(512 - hidden.shape[0], hidden.shape[1])]) for hidden in trimmed],


Cap Helios text length before padding

When a caller uses the public max_sequence_length override above 512 and supplies a prompt longer than 512 tokens, hidden.shape[0] exceeds 512 and hidden.new_zeros(512 - hidden.shape[0], ...) receives a negative dimension, aborting generation. Validate or clamp the override to 512 before encoding, or truncate here before applying fixed-length padding.

Useful? React with 👍 / 👎.

Fixed in 3238ae7. Helios text postprocessing now truncates valid tokens to the fixed 512-token model context before padding, with a 520-token regression test.

github-actions Bot reviewed Jul 31, 2026

View reviewed changes

mergify Bot added type: new-model New model support scope: model Model architecture (DiTs, encoders, VAEs) labels Jul 31, 2026

FlappyBob force-pushed the add-helios-transformer-draft branch from e7fc69b to 20814fa Compare July 31, 2026 22:05

FlappyBob marked this pull request as ready for review August 3, 2026 00:27

mergify Bot added the needs-rebase PR has merge conflicts label Aug 4, 2026

FlappyBob force-pushed the add-helios-transformer-draft branch from ab0eb5b to 7c494fc Compare August 9, 2026 00:43

mergify Bot removed the needs-rebase PR has merge conflicts label Aug 9, 2026

mergify Bot added the needs-rebase PR has merge conflicts label Aug 10, 2026

FlappyBob force-pushed the add-helios-transformer-draft branch from 7c494fc to 996b996 Compare August 26, 2026 20:13

FlappyBob changed the title ~~[new-model] Add native Helios transformer~~ [new-model] Add native Helios-Distilled T2V pipeline Aug 26, 2026

FlappyBob marked this pull request as draft August 26, 2026 20:13

mergify Bot added scope: inference Inference pipeline, serving, CLI scope: infra CI, tests, Docker, build scope: docs Documentation and removed needs-rebase PR has merge conflicts labels Aug 26, 2026

FlappyBob added 9 commits August 29, 2026 15:54

[new-model] Add native Helios transformer

3ac2ad3

[new-model] Harden Helios history validation

5fd4ed9

[test] Update Helios attention backend scope

3200869

[feat] Add Helios sampling contract

1cb2765

[new-model] Add native Helios DMD scheduler

9dce388

[test] Verify Helios reused components

526f9ec

[new-model] Add Helios pyramid sampling primitives

c24d3f1

[new-model] Add Helios autoregressive pyramid stages

8867993

[new-model] Register Helios-Distilled T2V pipeline

0383896

FlappyBob added 3 commits August 29, 2026 15:54

[test] Add Helios end-to-end latent parity

70007fc

[misc] Add Helios-Distilled inference example

f610f08

[test] Record Helios port verification

099c098

FlappyBob force-pushed the add-helios-transformer-draft branch from 996b996 to 099c098 Compare August 29, 2026 21:30

FlappyBob marked this pull request as ready for review August 29, 2026 21:46

chatgpt-codex-connector Bot reviewed Aug 29, 2026

View reviewed changes

fix: handle Helios output boundary cases

3238ae7

H1yori233 self-assigned this Sep 2, 2026

Navigation Menu

Uh oh!

[new-model] Add native Helios-Distilled T2V pipeline - #1670

[new-model] Add native Helios-Distilled T2V pipeline#1670
FlappyBob wants to merge 13 commits into
hao-ai-lab:mainfrom
FlappyBob:add-helios-transformer-draft

FlappyBob commented Jul 31, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Jul 31, 2026

Uh oh!

github-actions Bot left a comment

Uh oh!

mergify Bot commented Jul 31, 2026 •

edited

Loading

Uh oh!

gemini-code-assist Bot commented Aug 3, 2026

Uh oh!

mergify Bot commented Aug 4, 2026

Uh oh!

mergify Bot commented Aug 10, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Uh oh!

chatgpt-codex-connector Bot Aug 29, 2026

Uh oh!

FlappyBob Aug 29, 2026

Uh oh!

chatgpt-codex-connector Bot Aug 29, 2026

Uh oh!

FlappyBob Aug 29, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

FlappyBob commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Purpose

Supported Scope

Changes

Pinned References

Test Results

Components

Pipeline

Public Example

Static Checks

CI Base Alignment

Quality Regression

Checklist

Uh oh!

gemini-code-assist Bot commented Jul 31, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

mergify Bot commented Jul 31, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Merge Protections

🔴 PR merge requirements

Uh oh!

gemini-code-assist Bot commented Aug 3, 2026

Uh oh!

mergify Bot commented Aug 4, 2026

Uh oh!

mergify Bot commented Aug 10, 2026

Uh oh!

chatgpt-codex-connector Bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector Bot Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

FlappyBob Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

chatgpt-codex-connector Bot Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

FlappyBob Aug 29, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

FlappyBob commented Jul 31, 2026 •

edited

Loading

mergify Bot commented Jul 31, 2026 •

edited

Loading