[docs] Spark guide: add memory section, gotchas, and peak-memory to the benchmark

- spark_performance.md: "Memory: one unified 128 GB pool" section (nvidia-smi
  N/A, torch reserved as the real footprint, decode buffers as the pressure) and
  a GB10-specific "Gotchas" section.
- spark_benchmark.py: report peak GPU memory (torch reserved) + unified pool
  free/total alongside the timing.

Author: Mister-Raggs

docs/getting_started/installation/spark_performance.md

@@ -72,6 +72,27 @@ Decode is output-only, so lowering its precision is safe. (Encode seeds the
 denoising trajectory for I2V/causal models, so that stays at the pipeline's
 default — don't lower `vae_precision` blindly for those.)
 
+## Memory: one unified 128 GB pool
+
+The GB10 has **no separate VRAM** — CPU and GPU share one 128 GB LPDDR5X pool
+(~118 GB usable). Two practical consequences:
+
+- **`nvidia-smi` reports memory as `[N/A]`** on the GB10. For a model's real
+  GPU-side footprint use torch's allocator high-water mark
+  (`torch.cuda.max_memory_reserved()`); the system "used" figure conflates
+  CPU + GPU + cache and is only a soft upper bound. `spark_benchmark.py` prints
+  the torch figure for you.
+- **The 128 GB is a *working-set* ceiling, not storage** — the model cache lives
+  on the NVMe (3.7 TB, ample). What has to fit in 128 GB is weights + activations
+  + KV, and — critically — the **VAE decode buffers**, which is why tiling matters
+  (an untiled high-res decode can spike the pool into swap and lock the box).
+
+The recommended few-step models are comfortable here: their weights are small
+(1.3–2 B) and few-step generation keeps activations modest. The pressure comes
+from **decode resolution/frames**, not the model — a 1080p×121-frame untiled
+decode is what pushes the pool toward its ceiling. Run `spark_benchmark.py` to see
+the peak figure for your exact config.
+
 ## What helps vs. what doesn't on the GB10
 
 The honest summary — most "obvious" GPU optimizations don't move the needle on
@@ -133,6 +154,22 @@ is power-cycled. To avoid it:
 - Leave `*_cpu_offload` at the example defaults — "CPU" offload is the *same*
   unified RAM on the GB10, so the win is tiling + sane resolution, not offloading.
 
+## Gotchas specific to the GB10
+
+A few things that surprise people on this box (beyond the memory notes above):
+
+- **Don't force `TORCH_SDPA` on a VSA checkpoint** (FastWan, LTX2.3-distilled).
+  The SDPA path builds a model without the gate weights the checkpoint carries and
+  fails to load. Run the model natively — VSA auto-routes to its Triton kernel on
+  `sm_121`.
+- **Few-step timings are noisy run-to-run** (~±30%) — one-time startup dominates a
+  3-step run. Compare in-process / as medians, never two separate single runs (the
+  benchmark script does this).
+- **`nvidia-smi` shows `[N/A]` for memory** — see [Memory](#memory-one-unified-128-gb-pool).
+- **Cosmos-2.5** uses a Qwen2.5-VL text encoder; make sure you're on a FastVideo
+  build recent enough to include its `transformers`-compatibility handling before
+  running it.
+
 ## Reproduce these numbers
 
 Two scripts under `examples/inference/optimizations/` reproduce the claims on

examples/inference/optimizations/spark_benchmark.py

@@ -98,15 +98,25 @@ def _gen():
     for _ in range(args.warmup):
         _gen()
 
+    # Peak GPU memory: report torch's own allocator high-water mark, not
+    # nvidia-smi. On the GB10's unified pool nvidia-smi reads [N/A] and the
+    # system "used" figure conflates CPU+GPU+cache; torch.cuda.max_memory_reserved
+    # is the model's actual GPU-side footprint.
+    torch.cuda.reset_peak_memory_stats()
+
     times = []
     for i in range(args.runs):
         dt = _gen()
         times.append(dt)
         _p(f"gen run {i + 1}/{args.runs}: {dt:.2f}s")
 
     med = statistics.median(times)
+    peak_gb = torch.cuda.max_memory_reserved() / 1e9
+    free_b, total_b = torch.cuda.mem_get_info()
     _p(f"median generation time over {args.runs} runs "
        f"({args.warmup} warmup, {args.steps} steps): {med:.2f}s")
+    _p(f"peak GPU memory (torch reserved): {peak_gb:.1f} GB   "
+       f"| unified pool free/total: {free_b / 1e9:.1f}/{total_b / 1e9:.1f} GB")
     _p("set FASTVIDEO_STAGE_LOGGING=1 to see the denoise / decode / text split "
        "(few-step generation is VAE-decode-bound on the GB10).")
     generator.shutdown()