Skip to content

Commit 996b996

Browse files
FlappyBobFlappyBob
committed
[test] Record Helios port verification
1 parent f941ad3 commit 996b996

2 files changed

Lines changed: 286 additions & 0 deletions

File tree

‎tests/local_tests/helios/PORT_STATUS.md‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,110 @@
1+
# Helios Port Status
2+
3+
## Summary
4+
5+
- model_family: `helios`
6+
- workload_types: `T2V`
7+
- official_ref: Diffusers `0.39.0` `HeliosPyramidPipeline`; `PKU-YuanGroup/Helios@8f2a2faa`
8+
- official_ref_dir: `../Helios`
9+
- hf_weights_path: `BestWishYsh/Helios-Distilled`
10+
- local_weights_dir: `official_weights/helios`
11+
- source_layout: `diffusers`
12+
- local_tests_readme: `tests/local_tests/helios/README.md`
13+
14+
## Current Phase
15+
16+
- phase: `final_verification`
17+
- status: `complete`
18+
- owner: `orchestrator`
19+
- last_updated: `2026-08-26`
20+
21+
## Component Matrix
22+
23+
| Component | Type | Reuse/Port | Official Definition | Official Instantiation | FastVideo Target | Prototype | Conversion | Parity | Open Issues |
24+
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
25+
| transformer | dit | ported | Diffusers `HeliosTransformer3DModel` | Exact HF transformer config; short/mid/long histories and frame indices | native Helios DiT/config | complete | identity 1101/1101 | non_skip_pass | none |
26+
| scheduler | scheduler | ported | Diffusers `HeliosDMDScheduler` | Three stages, dynamic shift, stage-local re-noising | native Helios DMD scheduler | complete | not_needed | non_skip_pass, bit-exact | none |
27+
| vae | vae | reused | Diffusers `AutoencoderKLWan` | Exact Helios VAE config/weights | native Wan VAE | complete | not_needed | non_skip_pass, decode exact | none |
28+
| text_encoder | encoder | reused | Transformers `UMT5EncoderModel` | UMT5-XXL, max length 512 | native UMT5 | complete | not_needed | non_skip_pass FP32/BF16 | none |
29+
| tokenizer | encoder | passthrough | `T5TokenizerFast` | Exact checkpoint assets | production tokenizer loader | complete | not_needed | exact IDs/masks | none |
30+
| pipeline | pipeline | ported | Diffusers `HeliosPyramidPipeline` | 9-frame AR chunks, `[16,2,1]` history, 3 spatial stages | `HeliosPyramidPipeline` | complete | not_needed | non_skip_pass | T2V Distilled only |
31+
32+
## Conversion State
33+
34+
- conversion_script: `not_needed`
35+
- converted_weights_dir: `not_needed`
36+
- source_layout: `diffusers`
37+
- strict_load_status: transformer 1101/1101; VAE strict; UMT5 all parameters loaded
38+
- passthrough_components: tokenizer assets
39+
- retry_history: none
40+
41+
## Parity Commands
42+
43+
| Scope | Command | Last Result | Notes |
44+
| --- | --- | --- | --- |
45+
| transformer | `pytest tests/local_tests/transformers/test_helios_transformer_parity.py -v -s` | current PR component evidence: non-skip PASS | strict load, tiny/full, SP=2, FA2/SDPA |
46+
| scheduler | `pytest tests/local_tests/schedulers/test_helios_dmd_scheduler_parity.py -v -s` | 10 passed | registry plus bit-exact schedules/steps |
47+
| VAE | `pytest tests/local_tests/vaes/test_helios_vae_parity.py -v -s` | 2 passed | decode diff max/mean 0/0 |
48+
| UMT5/tokenizer | `pytest tests/local_tests/encoders/test_helios_umt5_parity.py -v -s` | 5 passed | exact tokenizer; FP32/BF16 encoder parity |
49+
| pipeline math/stage/smoke | three `test_helios_pipeline_*` files | 29 passed | includes CUDA block noise and CPU-output regression |
50+
| pipeline parity | `pytest tests/local_tests/pipelines/test_helios_pipeline_parity.py -v -s` | 1 passed in 130.11 s | cosine 0.979703, MAE 0.170017, RMSE 0.246508, drift 0.376% |
51+
| typed API regression | smoke + parser/compat/config tests | 42 passed | registry, preset, class resolution, typed/CLI fields |
52+
| typed example | `python examples/inference/basic/basic_helios_distilled_t2v.py` | PASS, generation 29.77 s | H.264 640×384, 33 frames, full decode |
53+
| quality/container | `HELIOS_QUALITY_CANDIDATE=... pytest test_helios_quality_regression.py` | 1 passed, 1 skipped | reference comparison deferred pending upload approval |
54+
55+
## Open Questions
56+
57+
| ID | Question | Owner | Needed By Phase | Status | Resolution |
58+
| --- | --- | --- | --- | --- | --- |
59+
| Q001 | Is `transformer_ode` required for Distilled T2V? | orchestrator | Phase 1 | resolved | No model index or official T2V call loads it; excluded. |
60+
| Q002 | Can the native Wan VAE be reused? | component:vae | Phase 3 | resolved | Exact config/weight strict load and decode parity are bit-exact. |
61+
| Q003 | Can native UMT5/tokenizer be reused? | component:encoder | Phase 3 | resolved | Token IDs/masks exact; FP32 and BF16 output parity pass. |
62+
| Q004 | Where do Helios DMD runtime fields live? | component:scheduler | Phase 1 | resolved | Scheduler owns sigma/timestep math; public per-call knobs live in SamplingParam/preset. |
63+
64+
## Issues And Blockers
65+
66+
| ID | Phase | Component | Severity | Issue | Evidence | Owner | Status | Resolution |
67+
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
68+
| I001 | prep | environment | medium | No project venv initially. | system Python lacked ML deps | prep | resolved | Python 3.12 project venv installed. |
69+
| I002 | prep | repository | low | Early branch was behind main. | rev-list showed upstream commits | prep | resolved | Final integration branch rebased on `b2062556`. |
70+
| I003 | prep | weights | low | Snapshot includes unused 58 GB ODE transformer. | model indexes omit `transformer_ode` | orchestrator | resolved | Pinned inference assets downloaded without ODE directory. |
71+
| I004 | parity | tokenizer | low | Direct loader import had a repository-root import cycle in the older workspace. | collection traceback | orchestrator | resolved_for_test | Production loader runs in a clean subprocess. |
72+
| I005 | official_reference | CUDA accounting | low | Peak reset initially used an invalid device after offload hooks. | runtime error before inference | orchestrator | resolved | Set current device before hooks; parity completes. |
73+
| I006 | transformer | RoPE buffers | medium | Meta construction left non-persistent RoPE buffers on meta. | meta/cuda einsum mismatch | component:transformer | resolved | Loader materializes exact buffers. |
74+
| I007 | encoder | BF16 drift | low | Fused-QKV and three-GEMM paths differ slightly. | max 0.0234375, mean 0.00160135 | component:encoder | resolved | FP32 proves math; separate bounded BF16 gate passes. |
75+
| I008 | pipeline | history geometry | high | Coarse-stage history RoPE used current-grid positions. | 294 history tokens versus 72 positions | component:transformer | resolved | Short-grid positions are center-downsampled for mid/long histories. |
76+
| I009 | pipeline | guidance cross-attention | medium | History and current queries were combined before masking. | activation trace diverged | component:transformer | resolved | Current/history split occurs before cross-attention. |
77+
| I010 | tests | repository package root | low | A root `__init__.py` makes invalid worktree basenames fail mypy. | `helios-pr... is not a valid Python package name` | orchestrator | resolved | Worktree basename changed to `helios_pr1670_full_pipeline`. |
78+
| I011 | registry | variant safety | high | Broad Helios detection would route Base/Mid to Distilled. | negative metadata probes | orchestrator | resolved | Require pipeline class, `is_distilled=true`, and Helios DMD scheduler. |
79+
| I012 | quality | tiny smoke | medium | 128×192 is not a visual-quality target. | poor-detail local smoke | orchestrator | resolved_as_scope | Public example and integrity gate use 384×640. |
80+
| I013 | distributed | nightly coverage | medium | SP/FSDP/repeated-run coverage is not a committed CI lane. | earlier local SP/FSDP smoke passed | orchestrator | open_nightly | Keep as follow-up; no absent runner is referenced by this PR. |
81+
| I014 | production_validation | multiprocessing output | high | Returning GPU output through CUDA IPC OOMed while the 14.31B DiT remained resident. | v1 completed stages then failed in `_new_shared_cuda` | pipeline | resolved | Decode/latent outputs move to CPU inside worker; regression is GREEN and typed example v2 passes. |
82+
| I015 | final_verification | upstream main | high | Integration base `6388db81` had eight unrelated unit-lane failures. | 909 passed and 8 failed before upstream CI/schema fixes. | upstream | resolved | Rebased onto `b2062556`; the current shared unit script passes all 964 tests. |
83+
84+
## Escape Hatches
85+
86+
No escape hatch is open. CI SSIM reference upload remains deliberately
87+
unauthorized and is recorded as a quality-regression deferral, not a blocker to
88+
local pipeline parity.
89+
90+
## Decisions
91+
92+
| Date | Decision | Rationale | Impact |
93+
| --- | --- | --- | --- |
94+
| 2026-07-11 | First public scope is Helios-Distilled T2V. | Keep a reviewable, verifiable variant. | Base/Mid, ODE, training and conditioned modes remain out of scope. |
95+
| 2026-07-11 | Use Diffusers 0.39.0 as executable parity reference. | It matches the published Diffusers-layout checkpoint. | No conflicting Helios research requirements are installed. |
96+
| 2026-07-11 | Preserve 1101 transformer keys directly. | Official and native key surfaces match. | No conversion script. |
97+
| 2026-07-11 | Reuse Wan VAE and UMT5 only after exact-asset parity. | Architecture resemblance is insufficient. | Both reused components have non-skip evidence. |
98+
| 2026-08-25 | Use a dedicated Helios pyramid stage inside `ComposedPipelineBase`. | Generic denoising cannot express AR history, three spatial levels and stage-local DMD. | One FastVideo architecture, model-specific stages only where required. |
99+
| 2026-08-25 | Keep zero-init call fields but do not apply zero-star math for the pinned Distilled checkpoint. | Its model index declares `is_cfg_zero_star=false`; Diffusers also takes the standard CFG branch. | Signature stays compatible without claiming an inactive feature changes output. |
100+
| 2026-08-25 | Move worker output to CPU before multiprocessing return. | Prevent CUDA IPC allocation after high-memory inference. | Public typed example is robust on 48 GB cards. |
101+
102+
## Handoff Notes
103+
104+
- Required components and pipeline parity are green on the integration branch.
105+
- `quality_regression=deferred_with_reason`: local real-video integrity passes;
106+
publishing a CI reference needs separate approval.
107+
- Repository-wide pre-commit is green. The current shared unit script passes
108+
all 964 collected tests; `I015` is resolved by the `b2062556` rebase.
109+
- No weights, generated media, reference clone, private report, token, push, or
110+
PR mutation is part of this state file.

‎tests/local_tests/helios/README.md‎

Lines changed: 176 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,176 @@
1+
# Helios Local Verification
2+
3+
Reviewer-facing setup and verification record for the native
4+
`BestWishYsh/Helios-Distilled` text-to-video port. These tests compare FastVideo
5+
against the exact Diffusers Helios implementation and require local assets for
6+
non-skip author verification.
7+
8+
Port state and resolved issues live in
9+
`tests/local_tests/helios/PORT_STATUS.md`.
10+
11+
## Pinned Sources
12+
13+
| Field | Value |
14+
| --- | --- |
15+
| Model family | `helios` |
16+
| Public scope | Helios-Distilled T2V |
17+
| Architecture source | `PKU-YuanGroup/Helios@8f2a2faab3298c8a7630a2c73aea37c01b5bab01` |
18+
| Executable parity reference | Diffusers `0.39.0` `HeliosPyramidPipeline` |
19+
| HF checkpoint | `BestWishYsh/Helios-Distilled` |
20+
| HF revision | `1999182614cb08d3bdcc46b9827504af2914b87b` |
21+
| Local assets | `official_weights/helios` |
22+
| Source layout | Diffusers; no conversion required |
23+
| Integration base | `upstream/main@b2062556a9bc100a4edd7636cda47078efc41389` |
24+
25+
The official T2V model index loads one `transformer`; `transformer_ode` is not
26+
declared or used and remains outside this PR. Base/Mid, training, ODE, I2V, and
27+
other variants are not supported by this first pipeline.
28+
29+
## Environment And Assets
30+
31+
Run from the FastVideo repository root:
32+
33+
```bash
34+
uv venv --python 3.12 --seed
35+
UV_TORCH_BACKEND=cu130 uv pip install -e ".[dev]"
36+
37+
.venv/bin/python -c "from diffusers import AutoencoderKLWan, HeliosDMDScheduler, HeliosPyramidPipeline, HeliosTransformer3DModel; from transformers import AutoTokenizer, UMT5EncoderModel; print('imports ok')"
38+
```
39+
40+
The verified environment uses Python 3.12.3, PyTorch `2.12.0+cu130`, CUDA 13,
41+
Diffusers 0.39.0, Transformers 5.13.1, and `flash-attn==2.8.1`.
42+
43+
Download the pinned inference snapshot without the unused ODE transformer:
44+
45+
```bash
46+
.venv/bin/python \
47+
.agents/skills/add-model-01-prep/scripts/download_hf_weights.py \
48+
BestWishYsh/Helios-Distilled \
49+
official_weights/helios \
50+
--revision 1999182614cb08d3bdcc46b9827504af2914b87b \
51+
--ignore-pattern "transformer_ode/*"
52+
```
53+
54+
Use only `HF_TOKEN`, `HUGGINGFACE_HUB_TOKEN`, or `HF_API_KEY` when auth is
55+
needed. Never put a token value in this file or a command log.
56+
57+
## Component Matrix
58+
59+
| Component | FastVideo target | Test | Current author result |
60+
| --- | --- | --- | --- |
61+
| Transformer | `fastvideo/models/dits/helios.py` | `tests/local_tests/transformers/test_helios_transformer_parity.py` | Non-skip PASS: 1101/1101 strict load, tiny exact normal/pyramid, full BF16 normal/pyramid, SP=2, FlashAttention-vs-SDPA |
62+
| Scheduler | `fastvideo/models/schedulers/scheduling_helios_dmd.py` | `tests/local_tests/schedulers/test_helios_dmd_scheduler_parity.py` | 10 passed: registry resolution plus bit-exact three-stage/amplify/step parity |
63+
| VAE | Existing native Wan VAE | `tests/local_tests/vaes/test_helios_vae_parity.py` | Non-skip PASS; decode max/mean diff `0/0` |
64+
| UMT5 | Existing native UMT5 | `tests/local_tests/encoders/test_helios_umt5_parity.py` | FP32 max/mean `1.25e-6/1.0e-7`; BF16 `0.0234375/0.00160135` |
65+
| Tokenizer | Production tokenizer loader | `tests/local_tests/encoders/test_helios_umt5_parity.py` | Exact IDs and attention masks |
66+
| Pipeline | `fastvideo/pipelines/basic/helios/` | `tests/local_tests/pipelines/test_helios_pipeline_parity.py` | Non-skip final-latent PASS |
67+
68+
## Mandatory Commands
69+
70+
Run component gates before pipeline gates:
71+
72+
```bash
73+
PYTHONPATH="$PWD" .venv/bin/pytest \
74+
tests/local_tests/transformers/test_helios_transformer_parity.py -v -s
75+
76+
PYTHONPATH="$PWD" .venv/bin/pytest \
77+
tests/local_tests/vaes/test_helios_vae_parity.py -v -s
78+
79+
PYTHONPATH="$PWD" .venv/bin/pytest \
80+
tests/local_tests/encoders/test_helios_umt5_parity.py -v -s
81+
82+
PYTHONPATH="$PWD" .venv/bin/pytest \
83+
tests/local_tests/schedulers/test_helios_dmd_scheduler_parity.py -v -s
84+
85+
PYTHONPATH="$PWD" .venv/bin/pytest \
86+
tests/local_tests/pipelines/test_helios_pipeline_math.py \
87+
tests/local_tests/pipelines/test_helios_pipeline_stages.py \
88+
tests/local_tests/pipelines/test_helios_pipeline_smoke.py -v -s
89+
90+
DISABLE_SP=1 PYTHONPATH="$PWD" .venv/bin/pytest \
91+
tests/local_tests/pipelines/test_helios_pipeline_parity.py -v -s
92+
```
93+
94+
## End-To-End Latent Parity
95+
96+
Both implementations load the same checkpoint from scratch and use CPU seed
97+
42, 128×192, 33 pixel frames, guidance 1.0, three `[2,2,2]` DMD stages,
98+
history `[16,2,1]`, and nine latent frames per chunk.
99+
100+
```text
101+
shape [1, 16, 9, 16, 24]
102+
official abs mean 0.95911187
103+
FastVideo abs mean 0.96272093
104+
abs-mean relative drift 0.3763%
105+
max absolute diff 1.89453125
106+
mean absolute diff 0.17001711
107+
RMSE 0.24650776
108+
cosine 0.97970295
109+
result PASS
110+
```
111+
112+
Fixed gates are shape equality, cosine ≥0.95, abs-mean drift ≤5%, mean
113+
absolute error ≤0.30, and RMSE ≤0.40. The managed current-head run completed in
114+
130.11 seconds with one passed test and no skip.
115+
116+
## Typed Public Example
117+
118+
```bash
119+
HELIOS_MODEL_PATH=official_weights/helios \
120+
HELIOS_OUTPUT_PATH=outputs/helios/helios_distilled_t2v_384x640_33f.mp4 \
121+
PYTHONPATH="$PWD" .venv/bin/python \
122+
examples/inference/basic/basic_helios_distilled_t2v.py
123+
```
124+
125+
The example uses `VideoGenerator.from_config`, `GeneratorConfig`,
126+
`GenerationRequest`, and typed sampling/output configs. The current-head run
127+
completed generation in 29.77 seconds and produced:
128+
129+
```text
130+
codec H.264
131+
resolution 640x384
132+
fps 24
133+
frames 33
134+
duration 1.375 s
135+
SHA-256 2e6d5099715256b16b865cb300479b328b2835f9f8792cb5427a98c17eb0b038
136+
```
137+
138+
Full ffmpeg decode and the non-black-frame/container quality gate pass. The
139+
contact sheet contains the requested fish, coral, and underwater scene.
140+
141+
Run the committed local quality gate with:
142+
143+
```bash
144+
HELIOS_QUALITY_CANDIDATE=outputs/helios/helios_distilled_t2v_384x640_33f.mp4 \
145+
PYTHONPATH="$PWD" .venv/bin/pytest \
146+
tests/local_tests/helios/test_helios_quality_regression.py -v -s
147+
```
148+
149+
Expected author result is one passed integrity test and one skipped reference
150+
comparison. A CI SSIM reference is deliberately `deferred_with_reason`: its
151+
publication requires separate upload approval, which this PR does not have.
152+
153+
## Review Notes
154+
155+
- Production code does not import Diffusers or Transformers model classes;
156+
those imports exist only in parity tests. The existing tokenizer boundary is
157+
reused.
158+
- The dedicated pyramid stage is required by the official nine-frame
159+
autoregressive chunks, three spatial levels, history geometry, block-noise
160+
covariance, stage-local DMD schedules, and chunk decode contract.
161+
- `use_zero_init` and `zero_steps` remain in the public call surface for
162+
Diffusers signature compatibility. The pinned Distilled model declares
163+
`is_cfg_zero_star=false`, so both official and FastVideo standard-CFG paths
164+
intentionally ignore zero-star initialization for this checkpoint.
165+
- Final decoded and latent outputs are moved to CPU inside the worker before
166+
multiprocessing return. A regression test prevents CUDA-IPC OOM from
167+
returning a GPU tensor while the 14.31B transformer remains resident.
168+
- Generated media, tensor artifacts, weights, reference clones, and private
169+
reports remain ignored and must not be committed.
170+
171+
## Latest-Main Unit-Lane Baseline
172+
173+
The branch is rebased onto `b2062556`, which includes the upstream CI/schema
174+
repairs that followed the earlier `6388db81` baseline. The current
175+
`.buildkite/scripts/unit_test.sh` completes with `964 passed` and 21 warnings in
176+
32.04 seconds. `pre-commit run --all-files` also passes completely.

0 commit comments

Comments
 (0)