[bugfix]: condition HunyuanVideo 1.5 i2v on the reference image - #1693
[bugfix]: condition HunyuanVideo 1.5 i2v on the reference image#1693KyleNeverGivesUp wants to merge 2 commits into
Conversation
mergify
Bot
added
type: bugfix
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews and 🤖 CI
🔴 PR merge requirementsWaiting for
This rule is failing.
|
Pre-commit checks failedHi @KyleNeverGivesUp, the pre-commit checks have failed. To fix them locally: # Install pre-commit if you haven't already
uv pip install pre-commit
pre-commit install
# Run all checks and auto-fix what's possible
pre-commit run --all-filesCommon fixes:
After fixing, commit and push the changes. The checks will re-run automatically. For future commits, |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 05402cb4d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| # The i2v checkpoints ship a SigLIP vision tower; declaring it here is what | ||
| # makes the loader build one. | ||
| image_encoder_config: EncoderConfig = field(default_factory=SiglipVisionConfig) |
There was a problem hiding this comment.
Both I2V config subclasses inherit Hunyuan15T2V480PConfig.__post_init__, which sets vae_config.load_encoder = False, so VAELoader constructs AutoencoderKLHunyuanVideo15 without its encoder. For every reference-image request, the newly wired Hy15ImageEncodingStage then calls self.vae.encode(...), whose _encode accesses the missing encoder and fails before denoising; override the I2V initialization so the VAE loads both encoder and decoder.
Useful? React with 👍 / 👎.
| if fastvideo_args.vae_cpu_offload: | ||
| self.vae.to("cpu") | ||
|
|
||
| cond_latents = cond_latents.repeat(1, 1, latent_temporal, 1, 1) |
There was a problem hiding this comment.
When num_videos_per_prompt > 1 or a prompt list produces a latent batch larger than one, raw_latent_shape[0] reflects that larger batch but the single reference image produces one conditioning latent, and this repeat only expands time. Consequently batch.image_latent remains batch size 1 and DenoisingStage cannot concatenate it with the multi-sample latent tensor; expand the conditioning and mask to the requested latent batch size.
Useful? React with 👍 / 👎.
|
The failing pre-commit check here is not from this PR. Fixed separately in #1700 to keep this diff clean. Once that lands I will rebase and the check should go green. |
05402cb to
08d8b5a
Compare
|
@SolitaryThinker rebased onto main now that #1700 and #1702 landed, pre-commit |
08d8b5a to
805d487
Compare
|
Rebased onto current main, now zero commits behind. The earlier failure was Ran the full |
805d487 to
31c1cec
Compare
Fixes hao-ai-lab#1692. Passing image_path to HunyuanVideo15ImageToVideoPipeline raised StageVerificationError before denoising. Hy15ImageEncodingStage only wrote image_embeds on the branch where pil_image is None, so with an image the field kept its ForwardBatch default of [] and the inherited output check rejected it. Nothing encoded the image either way: the SiglipVisionModel and SiglipImageProcessor the i2v checkpoints ship were not in the pipeline's required modules, and the stage was constructed with image_encoder=None. Declare the SigLIP pair on the two i2v pipeline configs, load it in the i2v pipeline, and give the stage the path it lacked: SigLIP hidden states into image_embeds, VAE latents of the reference image into the first frame of the conditioning stream, and a mask channel set to 1 on that frame. Channel order is the part that is easy to get wrong, because it is invisible until an image is supplied. DenoisingStage appends video_latent and a zero pad separately, which yields [latents 32][mask 1][conditioning 32], but the model expects [latents 32][conditioning 32][mask 1]. For t2v both trailing blocks are zero, so the two orders produce the same tensor and the discrepancy never surfaces. The i2v path therefore populates image_latent as one 33 channel block and leaves video_latent unset, so denoising takes the branch that appends it whole. Reference for the expected layout is the diffusers HunyuanVideo15ImageToVideoPipeline, which builds cat([latents, cond_latents_concat, mask_concat], dim=1); HYWorldImageEncodingStage already assembles image_latent the same way. t2v and the two super-resolution pipelines are untouched. They never carry a pil_image, so they keep the existing branch, including video_latent, which SRDenoisingStage reads directly. Adds CPU tests over both paths: the t2v zeros, the SigLIP embeddings being non-zero so the transformer's is_t2v check turns off, the conditioning and mask landing in that order with the mask on frame 0 only, the scaling factor being applied, video_latent staying unset, and a clear error when a pipeline without the encoder is handed an image. fastvideo/tests/stages/test_hy15_image_encoding_stage.py: 7 passed Not verified here: an end to end i2v run on a GPU. Doing that next and will post the before and after in the issue.
31c1cec to
ea0ad12
Compare
… latent batch Both I2V configs inherited Hunyuan15T2V480PConfig.__post_init__, which sets vae_config.load_encoder = False. The i2v image-encoding stage calls self.vae.encode, and AutoencoderKLHunyuanVideo15 builds its encoder only when load_encoder is set, so every real checkpoint raised AttributeError before denoising. The shipped test hid this behind a fake VAE. Both I2V configs now enable the encoder after super().__post_init__ and check_pipeline_config rejects a config that turns it back off. The stage also assumed a batch of one. SigLIP embeddings, the VAE conditioning latents, and the mask now expand to the latent batch, and an incompatible batch raises instead of broadcasting wrong. The generic denoising stage still asserts batch size one, so this fixes the stage contract, not end-to-end batched generation. The encode path gained two guards it could not reach before this fix made it live. Tiling now follows pipeline_config.vae_tiling like every other VAE encoding stage in the file, since the decoding stage enables tiling on the shared VAE and nothing turns it off, which made the first and later generations encode the same reference differently. The reference image is aligned to the VAE's spatial compression ratio of 16 rather than a hardcoded 8, which let a height like 552 crash inside the encoder reshape. The test builds the real AutoencoderKLHunyuanVideo15 from the real I2V config with a lightweight encoder and decoder, so the load_encoder wiring is what is under test. Reverting the production half fails ten test ids, five of them with the original AttributeError.
1 participant
Purpose
HunyuanVideo15ImageToVideoPipelinecannot generate from a reference image.Passing
image_pathraisesStageVerificationErrorbefore denoising, and evenpast that point the image has no route into the transformer.
Fixes #1692
Changes
The i2v checkpoints ship a SigLIP pair that FastVideo never asked for. Their
model_index.jsondeclares:Neither was in
_required_config_modules, soHy15ImageEncodingStagewasconstructed with
image_encoder=None. The stage only wroteimage_embedsonthe
pil_image is Nonebranch, so supplying an image left it at itsForwardBatchdefault of[]and the inherited output check rejected it.Three changes:
Hunyuan15I2V480PStepDistilledConfigandHunyuan15I2V720PConfigdeclareimage_encoder_config = SiglipVisionConfig, which is what makes the loaderbuild one. FastVideo already has the SigLIP implementation
(
fastvideo/models/encoders/siglip.py, registered atfastvideo/models/registry.py:106), so no new model code.image_encoderandfeature_extractorto its requiredmodules and passes them, plus the VAE, into the stage.
Hy15ImageEncodingStagegains the i2v path: SigLIP hidden states intoimage_embeds, VAE latents of the reference image into the first frame ofthe conditioning stream, and a mask channel set to 1 on that frame.
Channel order
This is the part that is invisible until an image is actually supplied.
DenoisingStageappendsvideo_latentand a zero pad separately(
denoising.py:405-414), producingbut the model expects
For t2v both trailing blocks are zero, so the two orders yield an identical
tensor and the discrepancy never surfaces. The i2v path therefore populates
image_latentas one 33 channel block and leavesvideo_latentunset, sodenoising takes the
elif batch.image_latent is not Nonebranch that appends itwhole.
The expected layout is not a guess. The diffusers reference implementation,
diffusers/pipelines/hunyuan_video1_5/pipeline_hunyuan_video1_5_image2video.py,builds
cat([latents, cond_latents_concat, mask_concat], dim=1), appliesscaling_factorto the image latents, repeats them over the temporal dimensionand zeroes every frame after the first, and sets the mask to 1 on frame 0.
HYWorldImageEncodingStagein this repo already assemblesimage_latentthesame way.
Blast radius
t2v and the two super-resolution pipelines share this stage but never carry a
pil_image, so they keep the existing branch untouched, includingvideo_latent, whichSRDenoisingStagereads directly(
sr_denoising.py:168). A pipeline built without the encoder that is handed animage now raises a message naming the missing pieces rather than failing an
opaque field check.
Test Plan
pytest fastvideo/tests/stages/test_hy15_image_encoding_stage.py -q pre-commit run --files fastvideo/pipelines/stages/image_encoding.py \ fastvideo/pipelines/basic/hunyuan15/hunyuan15_i2v_pipeline.py \ fastvideo/configs/pipelines/hunyuan15.py \ fastvideo/tests/stages/test_hy15_image_encoding_stage.pyCPU only, no checkpoint needed. The tests fake SigLIP and the VAE, since what
needs guarding is the layout rather than the numerics.
Covered:
image_embeds, which is what keeps the transformer'sis_t2vcheck true, and still populatesvideo_latentfor the SR stagesimage_embedsare non-zero, which is what turns that check offon frame 0 only and the conditioning zero after frame 0
scaling_factoris appliedvideo_latentstays unset on the i2v path, so denoising cannot take thebranch that would swap the blocks
Test Results
Test output
Before, from #1692:
Checklist
Memory: the i2v pipelines now load a SigLIP vision tower they previously did
not. It honours
image_encoder_cpu_offloadand is moved back to CPU after theencode, same as the other image encoding stages.
Not verified here: an end to end i2v run on a GPU. I am doing that next and
will post before and after in #1692. Marking this ready rather than draft since
the change is self contained and the layout question is settled against the
diffusers reference, but happy to hold it until that lands if you would rather
see the run first.
There is also no i2v example script for HY1.5 and no HY1.5 entry under
fastvideo/tests/ssim/, which is why this went unnoticed. Worth noting that anSSIM reference alone would not catch a regression here once the crash is gone,
since a t2v fallback still produces a plausible video. Happy to add an example
script in a follow up.