[Bug] Prompt completely ignored in MLX runtime (Apple Silicon) - identical outputs across different prompts, confirmed on QAD-1.3B and FP8-1.3B

[Carlwang1026]

Bug Report: Prompt Ignored in MLX Runtime (Apple Silicon) - All Prompts Generate Identical/Unrelated Videos

Summary

When using FastVideo's MLX runtime on Apple Silicon (M4 Mac mini, 16GB), the text prompt is completely ignored during generation. Different prompts produce identical videos (same seed), and the generated content is unrelated to the prompt. This has been confirmed across two different model variants (FastWan-QAD-1.3B and FastWan-QAD-FP8-1.3B), ruling out model-specific corruption.

Environment

Hardware:

• Mac mini, Apple M4, 16GB unified memory
• macOS 26.6.2

Software:

• Python 3.12
• torch 2.12.0
• mlx 0.32.1
• mlx-metal 0.32.1
• fastvideo (GitHub main, 2026-08-21)
• diffusers 0.33.0.dev0
• transformers 5.0.0

Models tested (both fail):

• FastVideo/FastWan-QAD-1.3B (27GB, standard)
• FastVideo/FastWan-QAD-FP8-1.3B (32GB, FP8 quantized)

Steps to Reproduce

python examples/inference/basic/mlx_wan_prompt_to_video.py \
  --model-root <model_dir> \
  --prompt "<ANY_PROMPT>" \
  --height 480 --width 480 \
  --num-frames 25 --fps 8 \
  --seed 42 \
  --output out.mp4

Test 1: Different Prompts, Same Seed → IDENTICAL output

Prompt	Seed	Output MD5
"A cute cat playing with a ball of yarn in a sunny garden"	42	c2cd2fad32949aae669ca0d346c12ec9
"A golden retriever dog running on a beach with waves"	42	c2cd2fad32949aae669ca0d346c12ec9

Both prompts produce the EXACT SAME video (bit-identical MD5). The video shows a female tennis player — unrelated to cat or dog.

Test 2: Different Seeds, Same Prompt → output varies (seed works)

Prompt	Seed	Output MD5	VLM-analyzed content
"A cute cat..."	100	543388aae4049c321c3d3f5206a194ae	Female figure, outdoor
"A cute cat..."	200	216a7721e3fb5fc49cee99dd1d85727b	Female figure, motion
"A cute cat..."	300	2e7f2e741e037edab488daa3701df264	Fluorescent parrot

✅ Random seed DOES affect output. ❌ Prompt NEVER affects output.

Test 3: FP8 model also ignores prompt

Model	Prompt	Seed	Output MD5	Content
FastWan-QAD-FP8-1.3B	"A cute cat playing with a ball of yarn in a sunny garden"	42	5d290df28f4a5d1d6141366b28777274	Male figure covering face

FP8 model generates different content than standard model (weights loaded correctly), but still completely ignores the prompt.

What Was Ruled Out

1. ❌ Model weight corruption — Two independently downloaded model variants (27GB + 32GB) both fail identically. Transformer weights verified: 825 tensors, correct shapes.
2. ❌ Cache issue — Cleared MLX cache and ~/.cache/fastvideo/prompt_embeds/, re-ran: identical MD5 output.
3. ❌ Random seed broken — Different seeds produce different videos, so RNG works.
4. ❌ Obvious code path break — encoder_hidden_states is correctly created (line 839) and passed to DiT (line 924); _forward in fastvideo/mlx_runtime/fastwan.py correctly calls self.condition(timestep, encoder_hidden_states) and passes embeddings to every block.

Likely Root Cause

The bug is in the MLX runtime's handling of cross-attention (attn2) — the prompt embeddings are being silently ignored inside the DiT transformer blocks. Possible sub-causes:

1. Weight mapping issue in mlx_dit_from_diffusers_safetensors / _WAN_BLOCK_KEY_MAP — attn2 (cross-attention) weights may not be correctly mapped/loaded, causing cross-attention to compute with wrong or zeroed context.
2. mx.compile graph caching — compiled graph may not properly capture encoder_hidden_states as an input dependency.
3. Attention backend bug — MLX attention implementation may ignore the context argument.

Request

This is a critical bug — the MLX runtime is unusable for text-to-video on Apple Silicon since prompts have zero effect. Please investigate:

1. Verify cross-attention (attn2) weight loading in the MLX DiT loader
2. Test with mx.compile disabled to isolate graph caching issues
3. Compare against the working NVIDIA path (examples/inference/optimizations/FastWan_QAD_TAEHV.py)

I can provide full logs, prompt embedding cache files, and any additional diagnostics.

---

Reporter: Carl Wang (ManLing Tech)
Date: 2026-08-21 18:35 GMT+8
Tested on: Mac mini M4 16GB, macOS 26.6.2

[github-actions]

Welcome to FastVideo! Thanks for opening your first issue.

To help us investigate, please include:

• FastVideo version: pip show fastvideo
• GPU: nvidia-smi output (GPU model, driver, CUDA version)
• Python version: python --version
• OS: e.g., Ubuntu 22.04

If this is a bug, a minimal reproduction script helps us fix it faster.

Useful links:

• Documentation
• Contributing Guide
• Slack

[aryan5v]

hey, thanks for the detailed write-up. you accidentally pointed the MLX example at the CUDA FastWan-QAD models (FastWan-QAD-1.3B / FastWan-QAD-FP8-1.3B). those are the NVIDIA NVFP4/FP8 checkpoints. the Mac runtime wants FastMetal-QAD, the INT8 student we actually trained for Metal.

that's why the seed still changed the video and the prompt did nothing. different weights, wrong precision.

on a 16 GB M4 mini:

hf download FastVideo/FastMetal-1.3B-QAD --local-dir ./FastMetal-1.3B-QAD

python examples/inference/basic/mlx_wan_prompt_to_video.py \
  --model-root ./FastMetal-1.3B-QAD \
  --mlx-checkpoint ./FastMetal-1.3B-QAD \
  --height 480 --width 832 --num-frames 81 \
  --prompt "A bird's-eye view of a misty forest valley at dawn."

I'm also opening a PR that refuses those CUDA checkpoints on the MLX path so this is harder to hit again. give FastMetal a retry and let me know how it goes.

[aryan5v]

update: we actually ran this locally. same seed, your two prompts (cat vs dog). FastMetal-1.3B-QAD followed the prompt — two different videos. also loaded FastWan-QAD-1.3B through the MLX path the same way you did (no --mlx-checkpoint, Diffusers + INT8). still two different videos, not the same md5. couldn't recreate the identical-output thing on our machine.

[yd23]

I followed the guide here, https://haoailab.com/blogs/fastmetal/
On my MBP, the examples could complete after 3 steps of denoising
But it just generates a dark screen of 5kb file mlx_fastwan_prompt_to_video.mp4
export TOKENIZERS_PARALLELISM=false could turn the noise off, resulting in the same blank video file.

the huggingface repo (https://huggingface.co/FastVideo/FastMetal-1.3B-QAD) lacks a transformer/config.json file, complained by the example code, I manually copied the file from here https://huggingface.co/Wan-AI/Wan2.1-T2V-1.3B/blob/main/config.json, and placed it under a manually created transformer folder. Not quite sure if this is the right move, but Gemini says so.

python3.12  examples/inference/basic/mlx_wan_prompt_to_video.py \
>   --model-root ./FastMetal-1.3B-QAD \
>   --mlx-checkpoint ./FastMetal-1.3B-QAD \
>   --prompt "a misty mountain river valley at sunrise"
W0825 15:46:23.547000 97510 site-packages/torch/distributed/elastic/multiprocessing/redirects.py:35] NOTE: Redirects are currently not supported in MacOs.
INFO 08-25 15:46:24.418 [__init__.py:60] MPS (Metal Performance Shaders) is available
INFO 08-25 15:46:25.434 [prompt_cache.py:80] Prompt cache read skipped for /Users/dan/.cache/fastvideo/prompt_embeds/d0426c025a18fad0e9b07a40bb6242c7.npy: [Errno 2] No such file or directory: '/Users/dan/.cache/fastvideo/prompt_embeds/d0426c025a18fad0e9b07a40bb6242c7.npy.json'
Loading weights: 100%|█████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 242/242 [00:14<00:00, 16.18it/s]
denoise step 1/3 complete
denoise step 2/3 complete
denoise step 3/3 complete
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks...
To disable this warning, you can either:
    - Avoid using `tokenizers` before the fork if possible
    - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false)
Prompt encode time: 84.69s
MLX DiT load time: 0.04s
MLX denoise time: 269.66s
Decode/export time: 60.76s
Total prompt-to-video time: 415.61s
MLX load peak memory: 0.00 GiB
MLX denoise peak memory: 3.87 GiB
MLX active memory after denoise: 1.45 GiB
Spatial mode: off
Output written to: video_samples/mlx_fastwan_prompt_to_video.mp4

[aryan5v]

@yd23 Thanks for the detailed follow-up.

transformer/config.json is intentionally missing from FastVideo/FastMetal-1.3B-QAD. FastMetal ships the DiT as mlx_dit.json + mlx_dit.safetensors (the architecture lives in the config field inside mlx_dit.json), not as a Diffusers transformer/ tree. CUDA FastWan-QAD-1.3B does include transformer/config.json, but that is a separate NVIDIA release.

Upstream main had a bug: mlx_wan_prompt_to_video.py tried to read transformer/config.json even when --mlx-checkpoint pointed at a packed FastMetal directory. That is why the example complained. Please do not copy config.json from Wan-AI/Wan2.1-T2V-1.3B — it uses a different schema (WanModel vs WanTransformer3DModel) and is not the right workaround.

Fix is in PR #1758 (uses mlx_dit.json when --mlx-checkpoint is set). Latest commit also clarifies the error message/docs so this is harder to misread.

Please retry on the PR branch (or once #1758 merges):

git fetch origin pull/1758/head:pr-1758 && git checkout pr-1758
hf download FastVideo/FastMetal-1.3B-QAD --local-dir ./FastMetal-1.3B-QAD

python examples/inference/basic/mlx_wan_prompt_to_video.py \
  --model-root ./FastMetal-1.3B-QAD \
  --mlx-checkpoint ./FastMetal-1.3B-QAD \
  --prompt "a misty mountain river valley at sunrise"

No manual transformer/ folder needed. If you still get a blank ~5 KB video after that, it is likely a separate decode/runtime issue — please share the full log (including --decode-backend if you changed it) and we can dig in.