The model trained in Stage 1 exhibits jitter during inference.

[todochenxi]

Hello, thank you for your excellent work. I am attempting to retrain a new model, but when testing the video generation using the four models from Stage 1 (audioproj, audio_modules, motion_modules have been reset to zero, epoch=17000), I encounter significant jitter issues. If possible, could you provide any insight into what might be causing this problem

[todochenxi]

18000.mp4

just like this,

[cuijh26]

Because stage 1 couldn't guarantee the temporal consistency, resulting in the jitter issue. You train the stage2, maybe this problem could be solved.

[todochenxi]

I have tested with the same settings using the checkpoint provided by hallo2, and it works very stably. What could be the reason for this?

[yuchenli-sony]

jitter issue is exist because of temporal attention @todochenxi if you check the stage 1 of hallo2, there is no train for temporal which is mainly align with audio attention