[docs] Spark guide: lock numbers to GB10-measured values

Verified on a DGX Spark (GB10, torch 2.12.0+cu130, transformers 5.14.0):
- few-step FastWan gen ~40 s (was ~30 s); ~18x vs full-step
- bf16 VAE decode 1.14x, MS-SSIM 0.9999 (was ~1.2-1.3x); ~5-7% e2e
- Wan2.1-1.3B few-step peaks at ~8.4 GB (measured via pipeline peak_memory_mb)
- spark_benchmark.py reads peak_memory_mb from the result (worker-measured)
  instead of the main-process torch allocator (which reads ~0)

Author: Mister-Raggs

docs/getting_started/installation/spark_performance.md

@@ -7,7 +7,7 @@ tuning knobs that can't move on this hardware.
 
 !!! tip "TL;DR"
     - **Use distilled few-step models** (e.g. `FastVideo/FastWan2.1-T2V-1.3B-Diffusers`).
-      They run in ~30 s/video. Full-step models are 12–47 min on the GB10.
+      They run in ~40 s/video. Full-step models are 12–47 min on the GB10.
     - On few-step models, **VAE decode is the bottleneck**, not attention — it's
       bandwidth-bound on the Spark's unified memory.
     - **bf16 VAE decode** is the real, lossless lever (FastVideo already turns it
@@ -30,11 +30,11 @@ datacenter GPU's HBM**. Two consequences drive every tuning decision:
 ## Use distilled few-step models
 
 The single biggest lever on the GB10 is **model choice**. A 3-step distilled
-model is ~22× faster than the full-step version of the same architecture:
+model is ~18× faster than the full-step version of the same architecture:
 
 | Model | Steps | Time / video | Bottleneck |
 |---|---|---|---|
-| FastWan2.1-T2V-1.3B (distilled) | 3 | **~30 s** | VAE decode |
+| FastWan2.1-T2V-1.3B (distilled) | 3 | **~40 s** | VAE decode |
 | Wan2.1-T2V-1.3B (full-step) | 50 | ~12 min | denoise |
 | Cosmos-Predict2.5-2B (full-step) | 51 | ~47 min | denoise |
 | LTX2.3-distilled (+audio) | 8 | ~6 min | mixed |
@@ -53,8 +53,8 @@ you're paying mostly for VAE decode; above it, mostly for the denoising loop.
 
 Because few-step generation is decode-bound, VAE decode precision is where the
 time is. Decoding in **bf16 instead of fp32 is essentially lossless** (MS-SSIM
-~0.9999 vs fp32 on the identical latent) and ~1.2–1.3× faster — worth roughly
-5–10% end-to-end on a decode-bound few-step model.
+~0.9999 vs fp32 on the identical latent) and ~1.14× faster — worth roughly
+5–7% end-to-end on a decode-bound few-step model.
 
 **FastVideo already defaults Wan's decode to bf16** (`vae_decode_precision="bf16"`,
 with encode kept at fp32), so for the recommended Wan/FastWan models there's
@@ -77,21 +77,23 @@ default — don't lower `vae_precision` blindly for those.)
 The GB10 has **no separate VRAM** — CPU and GPU share one 128 GB LPDDR5X pool
 (~118 GB usable). Two practical consequences:
 
-- **`nvidia-smi` reports memory as `[N/A]`** on the GB10. For a model's real
-  GPU-side footprint use torch's allocator high-water mark
-  (`torch.cuda.max_memory_reserved()`); the system "used" figure conflates
-  CPU + GPU + cache and is only a soft upper bound. `spark_benchmark.py` prints
-  the torch figure for you.
+- **`nvidia-smi` reports memory as `[N/A]`** on the GB10, and the system "used"
+  figure conflates CPU + GPU + cache, so it's only a soft upper bound — treat the
+  whole 128 GB as one shared budget. For a per-run figure, use FastVideo's own
+  `peak_memory_mb` (reported on the generation result and by the performance
+  benchmark), which is measured inside the worker that runs the model.
 - **The 128 GB is a *working-set* ceiling, not storage** — the model cache lives
   on the NVMe (3.7 TB, ample). What has to fit in 128 GB is weights + activations
   + KV, and — critically — the **VAE decode buffers**, which is why tiling matters
   (an untiled high-res decode can spike the pool into swap and lock the box).
 
 The recommended few-step models are comfortable here: their weights are small
-(1.3–2 B) and few-step generation keeps activations modest. The pressure comes
-from **decode resolution/frames**, not the model — a 1080p×121-frame untiled
-decode is what pushes the pool toward its ceiling. Run `spark_benchmark.py` to see
-the peak figure for your exact config.
+(1.3–2 B) and few-step generation keeps activations modest — a Wan2.1-1.3B
+few-step generation peaks at **~8.4 GB** (measured), a small fraction of the pool.
+The pressure comes from **decode resolution/frames**, not the model — a
+1080p×121-frame untiled decode is what pushes the pool toward its ceiling, which
+is why VAE tiling stays
+on by default.
 
 ## What helps vs. what doesn't on the GB10
 
@@ -100,8 +102,8 @@ this hardware, for reasons specific to it:
 
 | Lever | Effect on the GB10 | Use it? |
 |---|---|---|
-| Distilled few-step model | ~22× vs full-step | ✅ **the primary lever** |
-| bf16 VAE decode | ~1.2–1.3×, lossless; ~5–10% e2e on few-step | ✅ default for Wan |
+| Distilled few-step model | ~18× vs full-step | ✅ **the primary lever** |
+| bf16 VAE decode | ~1.14×, lossless; ~5–7% e2e on few-step | ✅ default for Wan |
 | VSA (video sparse attention) | works out of the box (Triton kernel auto-selects on `sm_121`) | ✅ automatic |
 | Building FlashAttention | **no speedup** — Torch SDPA already hits an efficient flash kernel on `sm_121`, and FA2 ties it | ❌ not worth building |
 | `torch.compile` of the VAE decode | recompile storm (per-frame varying shapes) → ~1.1× | ❌ dead end |

examples/inference/optimizations/spark_benchmark.py

@@ -93,30 +93,31 @@ def _gen():
                                           save_video=True,
                                           sampling_param=sampling_param)
         torch.cuda.synchronize()
-        return getattr(video, "generation_time", time.perf_counter() - t0)
+        dt = getattr(video, "generation_time", time.perf_counter() - t0)
+        # Peak memory is measured *inside the worker process* that runs the
+        # pipeline and surfaced on the result; reading torch's allocator in this
+        # (main) process would report ~0 because the allocations aren't here.
+        return dt, getattr(video, "peak_memory_mb", None)
 
     for _ in range(args.warmup):
         _gen()
 
-    # Peak GPU memory: report torch's own allocator high-water mark, not
-    # nvidia-smi. On the GB10's unified pool nvidia-smi reads [N/A] and the
-    # system "used" figure conflates CPU+GPU+cache; torch.cuda.max_memory_reserved
-    # is the model's actual GPU-side footprint.
-    torch.cuda.reset_peak_memory_stats()
-
-    times = []
+    times, peaks = [], []
     for i in range(args.runs):
-        dt = _gen()
+        dt, peak = _gen()
         times.append(dt)
+        if peak:
+            peaks.append(peak)
         _p(f"gen run {i + 1}/{args.runs}: {dt:.2f}s")
 
     med = statistics.median(times)
-    peak_gb = torch.cuda.max_memory_reserved() / 1e9
-    free_b, total_b = torch.cuda.mem_get_info()
     _p(f"median generation time over {args.runs} runs "
        f"({args.warmup} warmup, {args.steps} steps): {med:.2f}s")
-    _p(f"peak GPU memory (torch reserved): {peak_gb:.1f} GB   "
-       f"| unified pool free/total: {free_b / 1e9:.1f}/{total_b / 1e9:.1f} GB")
+    if peaks:
+        _p(f"peak GPU memory (worker, reported by pipeline): {max(peaks):.0f} MB "
+           f"= {max(peaks) / 1024:.1f} GB")
+    else:
+        _p("peak GPU memory: not reported by this pipeline build")
     _p("set FASTVIDEO_STAGE_LOGGING=1 to see the denoise / decode / text split "
        "(few-step generation is VAE-decode-bound on the GB10).")
     generator.shutdown()