Adds fastvideo serve support for FastMetal (Wan on Mac/MLX) — all three sizes: 1.3B, 14B, and 5B. - #1802
Adds fastvideo serve support for FastMetal (Wan on Mac/MLX) — all three sizes: 1.3B, 14B, and 5B.#1802Ishxn20 wants to merge 3 commits into
Conversation
mergify
Bot
added
scope: inference
|
Merge Protections🔴 1 of 1 protections blocking · waiting on 👀 reviews, 🤖 CI and 🙋 you
🔴 PR merge requirementsWaiting for
This rule is failing.
|
There was a problem hiding this comment.
🟡 Changes recommended
There are correctness issues in the MLX request validation path (task handling) and the PR also introduces a new serving/runtime surface area that still needs real-hardware validation before it’s safe to merge.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR extends FastVideo’s OpenAI-compatible serving stack to support a non-CUDA runtime (MLX) by allowing the server to plug in an alternate generator implementation and runtime-specific request validation, then adds a native MLX “FastMetal Wan” server/pipeline for 1.3B, 14B, and 5B.
Changes:
- Add a pluggable
generator_factory+ per-runtimevideo_request_validatorto the shared OpenAI server app/engine so non-CUDA backends can be served. - Introduce MLX Wan pipelines (Wan2.1 for 1.3B/14B and Wan2.2-TI2V for 5B) plus a dedicated MLX Wan server entrypoint.
- Add tests and example YAML configs covering config parsing, validation, and generator dispatch for all three model sizes.
File summaries
| File | Description |
|---|---|
| fastvideo/entrypoints/openai/api_server.py | Adds runtime/generator factory hooks and conditionally disables image routes for MLX runtime. |
| fastvideo/entrypoints/openai/serving_engine.py | Generalizes generator shape via ServingGenerator and adds runtime-specific request validation hook. |
| fastvideo/entrypoints/openai/state.py | Updates global generator typing to the new serving generator protocol. |
| fastvideo/entrypoints/openai/video_api.py | Runs runtime-specific request validation before CUDA-oriented model/LoRA validation. |
| fastvideo/entrypoints/openai/mlx_wan_server.py | New MLX Wan server entrypoint, config parsing, request allowlist validation, and MLX generator implementation. |
| fastvideo/mlx_runtime/wan_pipeline.py | New MLX Wan2.1 and Wan2.2-TI2V pipelines and shared prompt/rope helpers for repeated server calls. |
| fastvideo/tests/entrypoints/test_mlx_wan_server.py | Tests MLX Wan server config parsing, allowlist validation, and generator dispatch/routing. |
| fastvideo/tests/mlx/test_mlx_wan_pipeline.py | Tests MLX Wan pipeline constructors’ filesystem + checkpoint-shape validation. |
| examples/serving/mlx_wan21_1_3b.yaml | Example serving config for FastMetal Wan2.1 1.3B. |
| examples/serving/mlx_wan21_14b.yaml | Example serving config for FastMetal Wan2.1 14B. |
| examples/serving/mlx_wan22_5b.yaml | Example serving config for FastMetal Wan2.2-TI2V 5B. |
Review details
- Files reviewed: 11/11 changed files
- Comments generated: 3
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| if request.task not in (None, "t2v"): | ||
| raise ValueError("Wan MLX serving supports task=t2v only.") |
| try: | ||
| manifest = json.loads(manifest_path.read_text()) | ||
| except (json.JSONDecodeError, OSError): | ||
| return None | ||
| config = manifest.get("config", manifest) | ||
| channels = config.get("in_channels") | ||
| return int(channels) if channels is not None else None |
| def get_generator() -> ServingGenerator: | ||
| """Return the global VideoGenerator instance (set during startup).""" | ||
| assert _generator is not None, "Server not initialized — generator is None" | ||
| return _generator |
61bf7c4 to
56ef624
Compare
|
Rebased onto main to pick up #1798’s shared MLX serving infrastructure; no content changes (range-diff clean). |
| "height", | ||
| "fps", | ||
| "num_frames", | ||
| "seconds", |
There was a problem hiding this comment.
[P1] Align or reject seconds before admitting the job. The shared adapter turns an explicit seconds into num_frames = seconds * fps; with both shipped defaults (16 and 24 fps), that is always 0 mod 4, while plan_refine_resolutions requires Wan frames to be 1 mod 4. A normal OpenAI-style request therefore gets queued and fails only inside generation. Please validate the merged request shape synchronously and either map duration to the nearest valid frame grid (for example seconds * fps + 1) or do not advertise seconds for this runtime.
| tokenizer = AutoTokenizer.from_pretrained(model_root / "tokenizer", local_files_only=True) | ||
| text_encoder = UMT5EncoderModel.from_pretrained( | ||
| model_root / "text_encoder", | ||
| torch_dtype=torch.bfloat16, |
There was a problem hiding this comment.
[P2] Keep the 5B prompt-encoder path on the recipe-validated dtype/device, or prove the new path on hardware. mlx_wan22_generate.py deliberately encodes 5B prompts in FP16 on CPU, but this shared helper forces BF16 on MPS and only casts the already-rounded embeddings back to FP16 afterward. That is not the same math (BF16 loses three mantissa bits), and this PR has no real-Mac generation/parity run to show the 5B output remains valid. Please parameterize dtype/device by model family and retain the 5B FP16 path, with an actual FastMetal-5B smoke/parity result.
SolitaryThinker
left a comment
SolitaryThinker
left a comment
There was a problem hiding this comment.
Requesting changes for the two inline blockers and the still-open task-validation mismatch:
- An explicit seconds request is admitted but the shared adapter produces a frame count that violates the Wan 1-mod-4 temporal grid under both shipped FPS defaults.
- The 5B server changes the maintained FP16/CPU prompt-encoder path to BF16/MPS without real-hardware parity evidence.
- validate_wan_video_request accepts task=t2v, but the shared request adapter rejects every non-None task for non-MiniMax models, as the existing inline review notes.
Please also address the malformed-manifest validation comment, run at least one real Apple-Silicon generation for each distinct pipeline (Wan2.1 and Wan2.2), and prefix the PR title with [feat] so merge protection can pass. Changed-file pre-commit is green on the rebased head; the focused pytest collection is not runnable on this Linux host because package import initializes Triton without an active GPU driver.
The server couldn't run non-Nvidia models at all. Added the ability to plug in a different generator, so MLX (and anything else non-CUDA) can be served through the same system CUDA already uses. This part re-derives the same capability [feat] Add an H3 server cookbook and prompt playground #1798 (H3 MLX serving) already builds — it touches the same 4 files, so whichever PR merges second will need a quick rebase.
Built the actual Mac pipeline for Wan. It didn't exist before — the only Mac version of Wan was a script that ran once and exited, with no way to keep it loaded and serve requests. 1.3B and 14B share one pipeline (same underlying model); 5B is a different, newer architecture, so it gets its own.
Test plan