|
| 1 | +# Fast mode (RIFE) — Apple Silicon |
| 2 | + |
| 3 | +`--fast` makes local generation ~2.7× faster by **generating fewer frames and |
| 4 | +interpolating the rest** with an Apple-Silicon-native RIFE model, instead of |
| 5 | +denoising every frame. Video-diffusion denoise is dominated by self-attention, |
| 6 | +which is O(tokens²); halving the frames cuts the token count ~2× and the denoise |
| 7 | +compute ~3.7×, so the wall-clock drops far more than 2×. RIFE (which estimates |
| 8 | +its own optical flow — no motion vectors needed) fills the dropped frames back |
| 9 | +in for ~1.4 s, and a light unsharp pass counters its softening. |
| 10 | + |
| 11 | +Measured on the 1.3B INT8 QAD model (fox, 480×832×81, M4): generate 41 + RIFE→81 |
| 12 | +runs in ~35 s of denoise vs ~90 s full, at reconstruction MS-SSIM **0.97**. |
| 13 | +Reproduce with `python -m fastvideo.benchmarks.eval_metalfx_rife --mode int8`. |
| 14 | + |
| 15 | +> **Note:** Apple's *MetalFX* frame interpolation is **not** usable here — it |
| 16 | +> requires game-engine motion vectors + depth, which diffusion output lacks. We |
| 17 | +> use the video-native **`rife-mlx`** model instead (Metal-backed, torch-free). |
| 18 | +
|
| 19 | +## Install |
| 20 | + |
| 21 | +```bash |
| 22 | +uv pip install -e ".[mlx]" # RIFE ships vendored; this only needs MLX |
| 23 | +``` |
| 24 | + |
| 25 | +## Use |
| 26 | + |
| 27 | +```bash |
| 28 | +python examples/inference/basic/mlx_wan_prompt_to_video.py \ |
| 29 | + --mlx-checkpoint <FastWan2.1-T2V-1.3B-INT8-QAD> \ |
| 30 | + --prompt "A red fox trotting through a snowy pine forest at golden hour, cinematic" \ |
| 31 | + --num-frames 81 --fast \ |
| 32 | + --output-path video_samples/fox_fast.mp4 |
| 33 | +``` |
| 34 | + |
| 35 | +`--num-frames` stays the *target* length; fast mode generates the smallest |
| 36 | +VAE-aligned keyframe count that RIFE can interpolate to that target. |
| 37 | + |
| 38 | +| Flag | Default | Meaning | |
| 39 | +|---|---|---| |
| 40 | +| `--fast` / `--no-fast` | off | enable fast mode | |
| 41 | +| `--fast-factor` | 2 | generate 1/factor of the frames (2 = half) | |
| 42 | +| `--fast-sharpen` | 0.6 | light unsharp strength to counter RIFE softness (0 disables) | |
| 43 | + |
| 44 | +Fast mode composes with everything else (`--mlx-quantization int8`, |
| 45 | +`--mlx-compile`, TAEHV vs `--decode-backend wan-vae`). Keep `--fast-factor` at 2 |
| 46 | +for quality — larger temporal gaps are where RIFE starts inventing motion. |
| 47 | + |
| 48 | +## Spatial fast mode (`--fast-spatial`) |
| 49 | + |
| 50 | +The spatial twin of `--fast`: instead of dropping frames, drop pixels. Denoise |
| 51 | +*and decode* at `height/width // fast-spatial-scale`, then resample the decoded |
| 52 | +frames up to the requested size. Self-attention is O(tokens²), so halving each |
| 53 | +spatial axis cuts the token count 4× and the denoise time far more than that — |
| 54 | +measured on the 1.3B INT8 QAD model at 480×832×81, M4 Max: **86.1 s → 10.3 s** |
| 55 | +of denoise. It composes with `--fast`; both together run the same clip in |
| 56 | +**4.5 s** of denoise. |
| 57 | + |
| 58 | +```bash |
| 59 | +python examples/inference/basic/mlx_wan_prompt_to_video.py \ |
| 60 | + --prompt "A red fox trotting through a snowy pine forest at golden hour, cinematic" \ |
| 61 | + --height 480 --width 832 --num-frames 81 --fast-spatial \ |
| 62 | + --output-path video_samples/fox_fast_spatial.mp4 |
| 63 | +``` |
| 64 | + |
| 65 | +| Flag | Default | Meaning | |
| 66 | +|---|---|---| |
| 67 | +| `--fast-spatial` / `--no-fast-spatial` | off | enable spatial fast mode | |
| 68 | +| `--fast-spatial-scale` | 2 | denoise at 1/scale of each spatial axis | |
| 69 | +| `--fast-spatial-upsample-mode` | `lanczos` | pixel interpolation kernel (`lanczos`, `cubic`, `bilinear`, `nearest`) | |
| 70 | +| `--fast-spatial-sharpen` | 0.4 | light unsharp strength to counter resampling softness (0 disables) | |
| 71 | + |
| 72 | +### The upsample must happen in pixel space |
| 73 | + |
| 74 | +This is the one thing to get right. The obvious implementation — bilinearly |
| 75 | +upsample the finished latents and decode at the target size — **does not work**, |
| 76 | +and produces a distinctive failure: correct composition and silhouette under a |
| 77 | +smeared, hazy veil, with ringing along strong edges. |
| 78 | + |
| 79 | +A Wan latent cell is a *learned code* for an 8×8 (Wan2.1) or 16×16 (Wan2.2) |
| 80 | +pixel block, not a low-pass sample of the image. The average of two adjacent |
| 81 | +codes is not the code of the averaged blocks; it is a vector the decoder was |
| 82 | +never trained on. Measured on Wan2.1-1.3B at 480×832, a 2× bilinear latent |
| 83 | +upsample destroys **62%** of the latent's high-frequency energy while leaving |
| 84 | +its overall magnitude intact — exactly the signature of that veil. At Wan2.2-5B |
| 85 | +the same operation degrades to black or noise. |
| 86 | + |
| 87 | +Decoded RGB frames have no such problem: an image *is* a sampled 2-D signal, so |
| 88 | +Lanczos interpolation is the operation it was defined for. The result is soft — |
| 89 | +it carries stage-1's real detail budget and no more — but clean and coherent. |
| 90 | + |
| 91 | +`--refine` gets away with a latent-space upsample only because a second DMD pass |
| 92 | +re-denoises the hand-off; spatial fast mode passes the latent straight to the |
| 93 | +decoder, so it cannot. |
| 94 | + |
| 95 | +## Refine (`--refine`) stage-2 timesteps |
| 96 | + |
| 97 | +`--refine` hands stage 1 to stage 2 as `(1 - sigma) * upsampled + sigma * noise`, |
| 98 | +where `sigma` comes from the *first* stage-2 timestep. FastWan's DMD grid opens |
| 99 | +at `t=1000`, which is `sigma == 1` exactly — so a stage-2 grid that starts there |
| 100 | +weights the stage-1 result at zero and refine silently degrades into a plain |
| 101 | +full-resolution run at twice the cost. |
| 102 | + |
| 103 | +Left unset, `--refine-dmd-denoising-steps` now derives the stage-2 grid from the |
| 104 | +stage-1 one with leading full-noise steps dropped (`1000,757,522` → `757,522`). |
| 105 | +That keeps the pass on timesteps the distilled student was trained on while |
| 106 | +letting stage-1 structure through: hand-off `sigma = 0.757`, stage-1 weight |
| 107 | +`0.243`. Passing a grid that starts at full noise is now an error rather than a |
| 108 | +silently wasted pass. |
| 109 | + |
| 110 | +The run prints the resolved hand-off so it is visible: |
| 111 | + |
| 112 | +``` |
| 113 | +[refine] stage-2 hand-off sigma=0.7568 (stage-1 weight 0.2432) |
| 114 | +``` |
| 115 | + |
| 116 | +There is a trade-off in choosing that grid. Later start = more of the draft |
| 117 | +survives, but fewer stage-2 steps. On Wan2.1 the default `757,522` gives weight |
| 118 | +0.243 with two steps; `--refine-dmd-denoising-steps 522` gives weight 0.478 with |
| 119 | +one. `--refine-sigma` decouples the noise level from the timestep entirely — it |
| 120 | +logs a warning, because the DiT is then told a timestep that does not match the |
| 121 | +noise it receives. |
| 122 | + |
| 123 | +**Wan2.2-5B has a lower ceiling.** Its warped schedule maps `1000,757,522` to |
| 124 | +sigmas `1.000, 0.940, 0.845`, so the best available stage-1 weight is **0.060** |
| 125 | +(vs 0.243 at 1.3B). Un-warped (`--no-warp`) the same grid gives `1.000, 0.757, |
| 126 | +0.522` and a weight of 0.243 — but warping is what matches the FastVideo |
| 127 | +sampling schedule, so turning it off changes the timesteps the distilled student |
| 128 | +sees. Which is better at 5B is unresolved and needs a run on real 5B weights. |
0 commit comments