Skip to content

Commit 850db72

Browse files
aryan5varyan5v
committed
[feat]: run FastH3 across two DGX Sparks with Ray sequence parallel
Ray could not start a two-node FastH3 job because the executor was still abstract and NCCL env was not copied to workers. Document the QSFP bring-up, cookbook recipe, and measured 292 s / 587 s pair runs on top of lazy module load.
1 parent 770d2df commit 850db72

16 files changed

Lines changed: 361 additions & 6 deletions

File tree

‎docs/assets/cookbook-recipes.json‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -40,6 +40,14 @@
4040
"source": "examples/inference/basic/basic_wan2_2_ti2v.py",
4141
"command": "python examples/inference/basic/basic_wan2_2_ti2v.py"
4242
},
43+
{
44+
"id": "fasth3-spark-pair",
45+
"task": "Text to video",
46+
"label": "FastH3 on two DGX Sparks (sequence parallel)",
47+
"model": "FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree",
48+
"source": "examples/inference/basic/basic_fasth3_spark_pair.yaml",
49+
"command": "source examples/inference/optimizations/spark_pair_env.sh && FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 FASTVIDEO_VAE_PARALLEL_DECODE=1 fastvideo generate --config examples/inference/basic/basic_fasth3_spark_pair.yaml"
50+
},
4351
{
4452
"id": "matrix-game-2",
4553
"task": "Interactive world",

‎docs/cookbook/index.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -42,3 +42,6 @@ supports:
4242
compilation, and memory tradeoffs.
4343
- [Support matrix](../inference/support_matrix.md) lists supported models and
4444
optimizations.
45+
- Two DGX Sparks: bring up the QSFP Ray cluster first
46+
([pair two Sparks](../getting_started/installation/spark_pair.md)), then pick
47+
the FastH3 two-Spark recipe.

‎docs/getting_started/installation.md‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -4,8 +4,9 @@
44
FastVideo supports the following hardware platforms:
55

66
- [NVIDIA CUDA](installation/gpu.md)
7-
- [NVIDIA DGX Spark / GB10 (ARM64 + CUDA 13)](installation/spark.md)
8-
([performance & tuning](installation/spark_performance.md))
7+
- **NVIDIA DGX Spark / GB10 (ARM64 + CUDA 13)** — [install](installation/spark.md),
8+
[performance](installation/spark_performance.md),
9+
[pair two Sparks](installation/spark_pair.md)
910
- [Apple silicon](installation/mps.md)
1011

1112
## Quick Installation

‎docs/getting_started/installation/spark.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -144,6 +144,9 @@ for which models are practical on the GB10, what makes them faster, and what
144144
won't help on this hardware (and why) — so you don't spend a night tuning knobs
145145
that can't move here.
146146

147+
Two Sparks with QSFP cables: [Pair two NVIDIA DGX Sparks](spark_pair.md) for
148+
one FastH3 clip across both GPUs (`sp_size=2` over Ray).
149+
147150
## Development Environment Setup
148151

149152
If you're planning to contribute to FastVideo please see the

‎docs/getting_started/installation/spark_pair.md‎

Lines changed: 179 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,179 @@
1+
# Pair two NVIDIA DGX Sparks
2+
3+
One GB10 is 128 GB of unified LPDDR5X. FastH3 still fits on a single Spark with
4+
[`--lazy-module-load`](../../inference/offloading.md), but two boxes connected
5+
by the QSFP ConnectX-7 cables can run **one clip faster** and can hold a
6+
**longer clip** (up to the FastH3 15 s cap).
7+
8+
This is FastVideo sequence parallel (`sp_size=2`) over Ray, not a third-party
9+
xDiT vendor. Do not install xDiT for this path.
10+
11+
## What two Sparks buy you
12+
13+
| Goal | How | Use two Sparks? |
14+
|---|---|---|
15+
| Two independent videos at once | One process per box, `num_gpus=1` | Throughput only. Each clip still takes ~6 min. |
16+
| One clip, faster | Ray + `sp_size=2` + parallel VAE | **Yes.** Measured 292 s vs 374 s on the same 124-frame FastH3 recipe. |
17+
| One clip, longer | Same, more frames | **Yes.** 345 frames (~14.4 s at 24 fps) finished in 587 s. |
18+
19+
Sequence parallel **replicates** the DiT (~66 GiB per node). Lazy module load
20+
is still required. FSDP would shard weights; it is untested on this fabric and
21+
is likely slower because every layer gathers over ~21 GB/s RoCE.
22+
23+
## Requirements
24+
25+
- Two DGX Sparks with FastVideo [installed](spark.md) (CUDA 13, `aarch64`).
26+
- The QSFP cables that ship with a dual-Spark kit, **ACTIVE** at 200 Gb/s:
27+
`ibstat` should show the ConnectX-7 ports `LinkUp`.
28+
- The same FastH3 snapshot on **both** NVMes. Copy the Hugging Face cache over
29+
QSFP; do not download 100+ GB twice over Wi-Fi.
30+
- Ray in the FastVideo venv (`uv pip install ray` if it is not already there).
31+
32+
Each Spark has **one** GPU. `num_gpus=2` therefore means two nodes, which is
33+
why the executor must be Ray (`mp` only works inside one process tree).
34+
35+
## 1. Put IPv4 on the QSFP NICs
36+
37+
The RoCE links often come up with no IPv4. Wi-Fi (`192.168.1.x`) is fine for
38+
SSH and must stay the default route. NCCL and Ray must **not** use it.
39+
40+
Pick a /24 that does not collide with your LAN. Example:
41+
42+
| Node | QSFP IPv4 | Interface (typical) |
43+
|---|---|---|
44+
| Spark A | `192.168.23.1/24` | `enp1s0f1np1` |
45+
| Spark B (Ray head) | `192.168.23.2/24` | `enp1s0f1np1` |
46+
47+
Confirm names with `ibdev2netdev` and `ip -br link`. Then, as root, on each
48+
box (NetworkManager likes to steal the NIC; unmanaged is enough for a session):
49+
50+
```bash
51+
sudo nmcli device set enp1s0f1np1 managed no
52+
sudo ip addr replace 192.168.23.1/24 dev enp1s0f1np1 # .2 on the other box
53+
sudo ip link set enp1s0f1np1 mtu 9000 up
54+
```
55+
56+
These addresses do **not** survive reboot. Ping across the cable before
57+
continuing: `ping -c 3 -I enp1s0f1np1 192.168.23.2`.
58+
59+
A healthy fabric on this hardware looks like:
60+
61+
- TCP iperf (jumbo 9000): ~40 Gb/s
62+
- NCCL allreduce 1 GiB × 10: ~21 GB/s busbw (NVIDIA's dual-Spark figure is ~21.7)
63+
64+
## 2. Start a two-node Ray cluster on the cable
65+
66+
On **both** nodes, from the FastVideo repo, with the venv active:
67+
68+
```bash
69+
source examples/inference/optimizations/spark_pair_env.sh
70+
```
71+
72+
That script pins NCCL to the QSFP NIC/HCA, disables NVLink-style P2P (there is
73+
none between boxes), and turns off Ray's memory monitor. The monitor treats
74+
GB10 unified RSS during a 14-shard DiT load as a runaway and SIGTERMs the
75+
worker around shard 11/14.
76+
77+
Cap Ray's object store. The default (~30% of 128 GB) leaves too little room
78+
for the DiT:
79+
80+
```bash
81+
# Spark B — head
82+
export FASTVIDEO_HOST_IP=192.168.23.2
83+
ray start --head --node-ip-address=192.168.23.2 --port=6379 --num-gpus=1 \
84+
--disable-usage-stats --object-store-memory=2147483648 --memory=4294967296
85+
86+
# Spark A — worker
87+
export FASTVIDEO_HOST_IP=192.168.23.1
88+
ray start --address=192.168.23.2:6379 --node-ip-address=192.168.23.1 --num-gpus=1 \
89+
--disable-usage-stats --object-store-memory=2147483648 --memory=4294967296
90+
```
91+
92+
`FASTVIDEO_HOST_IP` **must** match `--node-ip-address`. If you omit it, Ray
93+
advertises the Wi-Fi address, FastVideo builds a placement group for
94+
`node:192.168.1.x`, and the QSFP workers never match.
95+
96+
Check `ray status` on the head: `0.0/2.0 GPU` idle.
97+
98+
## 3. Generate one FastH3 clip on both GPUs
99+
100+
Run the driver on the **head**, same venv, same QSFP IP:
101+
102+
```bash
103+
source examples/inference/optimizations/spark_pair_env.sh
104+
export RAY_ADDRESS=192.168.23.2:6379
105+
export FASTVIDEO_HOST_IP=192.168.23.2
106+
107+
python examples/inference/basic/basic_fasth3.py \
108+
--model-path FastVideo/FastVideo-FastH3-4-step-Preview-v1-VSA-DataFree \
109+
--num-gpus 2 --execution-backend ray \
110+
--lazy-module-load --vsa-kernel triton --no-fa4 \
111+
--repeats 1 --no-warmup --parallel-vae \
112+
--height 768 --width 1344 --num-frames 124 --steps 5 \
113+
--seed 2026 \
114+
--prompt "A wide cinematic shot of an alpine meadow at sunrise, pale pink mountain peaks above a blue valley filled with thin morning mist." \
115+
--output outputs/fasth3_spark_pair
116+
```
117+
118+
`--execution-backend ray` is optional when `RAY_ADDRESS` is already set;
119+
`basic_fasth3.py` selects Ray in that case. GB10 has no FA4 / sm_100a VSA
120+
kernel, so `--vsa-kernel triton --no-fa4` is required.
121+
122+
Config-first equivalent:
123+
124+
```bash
125+
FASTVIDEO_VSA_SM100A=0 FASTVIDEO_FA4=0 \
126+
FASTVIDEO_ATTENTION_BACKEND=VIDEO_SPARSE_ATTN_H3 \
127+
FASTVIDEO_VAE_PARALLEL_DECODE=1 FASTVIDEO_STAGE_LOGGING=1 \
128+
fastvideo generate --config examples/inference/basic/basic_fasth3_spark_pair.yaml
129+
```
130+
131+
Stop the cluster when you are done: `ray stop` on both nodes.
132+
133+
## FastH3 frame counts
134+
135+
H3 is 24 fps. Legal `num_frames` values are `17n+5`. The pipeline rejects
136+
clips longer than **15 s**. The longest legal length is **345 frames**
137+
(14.375 s). 360 frames aligns to 362 and fails the duration check.
138+
139+
## Measured on two GB10s (2026-08-31)
140+
141+
Same alpine prompt, 768×1344, 5 sigma points (4 DiT forwards), Triton VSA,
142+
lazy load, parallel VAE, cold process (no warmup). Denoise times include
143+
deferred DiT load (~35 s).
144+
145+
| Run | GPUs | Frames | E2E | Denoise | VAE decode |
146+
|---|---:|---:|---:|---:|---:|
147+
| One Spark | 1 | 124 | 374–393 s | 180–188 s | 151–156 s |
148+
| Two Sparks, SP=2 | 2 | 124 | **292 s** | **122 s** | **102 s** |
149+
| Two Sparks, SP=2 | 2 | 345 | **587 s** | **351 s** | **173 s** |
150+
151+
The ~330 s one-Spark number from earlier FastH3 bring-up is the same recipe
152+
without this pair path and without TAEH3. 292 s is faster than that 1-GPU
153+
clip. It is **not** a lower bound: the first decode pays `torch.compile` on
154+
the VAE (~1 min of the 102 s); a second `generate()` in the same workers is
155+
cheaper. TAEH3 preview decode ([#1795](https://github.com/hao-ai-lab/FastVideo/pull/1795))
156+
is a separate opt-in and was not used here.
157+
158+
## Troubleshooting
159+
160+
| Symptom | Fix |
161+
|---|---|
162+
| Placement group waits forever / `node:192.168.1.x` | Set `FASTVIDEO_HOST_IP` to the QSFP address on **every** `ray start` **and** on the driver. |
163+
| `RayDistributedExecutor` TypeError / abstract `set_log_queue` | Use a FastVideo build that implements those methods on the Ray executor (this page). |
164+
| Worker SIGTERM during DiT shard 11/14 | `RAY_memory_monitor_refresh_ms=0` **before** `ray start`. Do not leave Ray's default 30% object store. |
165+
| NCCL hangs or uses Wi-Fi | `source spark_pair_env.sh`. Confirm `NCCL_SOCKET_IFNAME` is the QSFP NIC. |
166+
| OOM / `earlyoom` prefers Python | Lazy load must stay on. Peak GPU during 345-frame denoise is ~90 GiB/node. |
167+
| `num_gpus=2` on one Spark | Each Spark has one GPU. Use Ray across two nodes, or `num_gpus=1` on one box. |
168+
169+
## What we are not claiming
170+
171+
- **Throughput of many clips.** Two independent 1-GPU jobs still win if you
172+
want two videos, not one faster video.
173+
- **xDiT PipeFusion / CFG-parallel.** FastH3 is 4-step and has no CFG.
174+
- **FSDP or tensor parallel as a speedup** on this 21 GB/s link.
175+
- **Persistent networking.** The example IPs are session `ip addr replace`.
176+
177+
More GPUs are legal while `num_attention_heads` (56 on FastH3) is divisible by
178+
`sp_size`. Four Sparks would need a four-node fabric that this bring-up did
179+
not exercise.

‎docs/getting_started/installation/spark_performance.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -170,6 +170,11 @@ is power-cycled. To avoid it:
170170
then loads the VAE. Geometry scalars come from checkpoint `config.json`, not
171171
from live weights. A later `generate()` on the same worker reloads from disk.
172172
See [Offloading](../../inference/offloading.md).
173+
- **Two Sparks, one clip.** Sequence parallel (`sp_size=2`) over the QSFP RoCE
174+
link ran the same 768×1344×124 FastH3 recipe in **292 s** vs **374–393 s** on
175+
one GB10, and a 345-frame (~14.4 s) clip in **587 s**. Weights stay replicated,
176+
so lazy load is still required. Bring-up, env vars, and the cookbook recipe:
177+
[Pair two NVIDIA DGX Sparks](spark_pair.md).
173178

174179
## Gotchas specific to the GB10
175180

‎docs/inference/configuration.md‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,22 @@ generator = VideoGenerator.from_pretrained(
1212
)
1313
```
1414

15+
One node uses the multiprocessing executor (`execution_backend: mp`, the
16+
default). Two machines — for example two DGX Sparks, one GPU each — need Ray:
17+
18+
```yaml
19+
generator:
20+
engine:
21+
num_gpus: 2
22+
execution_backend: ray
23+
parallelism:
24+
sp_size: 2
25+
```
26+
27+
Set `RAY_ADDRESS` and `FASTVIDEO_HOST_IP` to the interconnect IPs, not Wi-Fi.
28+
The FastH3 example selects Ray automatically when `RAY_ADDRESS` is set. Full
29+
bring-up: [Pair two NVIDIA DGX Sparks](../getting_started/installation/spark_pair.md).
30+
1531
## Customizing Generation
1632

1733
- `PipelineConfig`: Initialization time parameters

‎docs/inference/offloading.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -33,7 +33,9 @@ safetensors are read onto the accelerator instead of CPU-then-copy. A later
3333
`generate()` on the same worker reloads a released component from disk in
3434
process; it does not need a new generator. The flag is auto-enabled on
3535
unified-memory devices. Pass `--no-lazy-module-load` (or `lazy_module_load=False`)
36-
to keep every component resident.
36+
to keep every component resident. Two-node Spark jobs still need it: sequence
37+
parallel replicates the DiT on each GB10 (~66 GiB of weights plus activations).
38+
See [Pair two NVIDIA DGX Sparks](../getting_started/installation/spark_pair.md).
3739

3840
## Behavior Explanation
3941

‎docs/inference/optimizations.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,8 @@ This page describes the various options for speeding up generation times in Fast
77
Several options on this page behave differently on the GB10's unified-memory
88
hardware — some give little or nothing there. See
99
[DGX Spark: Performance & Tuning](../getting_started/installation/spark_performance.md)
10-
for what actually helps on that platform and why.
10+
for what actually helps on that platform and why. Two Sparks, one clip:
11+
[Pair two NVIDIA DGX Sparks](../getting_started/installation/spark_pair.md).
1112

1213
## Table of Contents
1314

‎docs/inference/support_matrix.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -221,6 +221,8 @@ Per the installation guides:
221221
[GPU install guide](../getting_started/installation/gpu.md).
222222
- **NVIDIA DGX Spark (GB10, aarch64)** — CUDA 13, from-source kernel build; see
223223
the [DGX Spark install guide](../getting_started/installation/spark.md).
224+
Two Sparks over QSFP use Ray sequence parallel; see
225+
[Pair two NVIDIA DGX Sparks](../getting_started/installation/spark_pair.md).
224226
- **Apple silicon** — macOS 14 or newer; FastMetal-QAD via the MLX runtime. See the
225227
[Apple Silicon guide](../getting_started/installation/mps.md). The older
226228
[`basic_mps.py`](https://github.com/hao-ai-lab/FastVideo/blob/main/examples/inference/basic/basic_mps.py)

0 commit comments

Comments
 (0)