Skip to content

spark.md

Latest commit

 

History

History
150 lines (114 loc) · 6.21 KB

spark.md

File metadata and controls

150 lines (114 loc) · 6.21 KB
Raw

NVIDIA DGX Spark (GB10)

Instructions to install FastVideo on an NVIDIA DGX Spark — the GB10 Grace-Blackwell platform.

The Spark is ARM64 (aarch64) with CUDA 13, which the standard NVIDIA GPU guide does not cover: there is no prebuilt aarch64 wheel for fastvideo-kernel, so it is compiled from source as part of the install, and the system Python typically lacks the development headers that build needs. The steps below handle both, without requiring sudo.

Requirements

  • OS: Linux (aarch64)
  • GPU: NVIDIA GB10 (compute capability sm_121), at least 1
  • CUDA Toolkit: 13.x at /usr/local/cuda (nvcc on PATH)
  • Python: 3.10–3.12
  • Compilers: gcc/g++ 10+ (the Spark ships gcc 13)

!!! note "Different compute capability?" These instructions target the GB10's sm_121. The kernel build auto-detects it from your GPU, so there is normally nothing to set. To check your device: bash python -c "import torch; print(torch.cuda.get_device_capability(0))" # GB10 -> (12, 1)

Install

Run from the repository root:

# 0. (once) install uv and put it on PATH
curl -LsSf https://astral.sh/uv/install.sh | sh
export PATH="$HOME/.local/bin:$PATH"
git clone https://github.com/hao-ai-lab/FastVideo.git && cd FastVideo

# 1. Create a venv on a uv-managed CPython 3.12 (bundles the dev headers the
#    kernel build needs; the Spark's system python3.12 lacks them).
uv venv .venv --python 3.12 --python-preference only-managed --seed
source .venv/bin/activate

# 2. Initialise the kernel submodules (cutlass + ThunderKittens headers).
git submodule update --init --recursive \
  fastvideo-kernel/include/cutlass fastvideo-kernel/include/tk

# 3. Install FastVideo (editable; compiles the in-tree kernel for the GB10).
UV_TORCH_BACKEND=cu130 uv pip install -e .

Contributors who want the lint/test tooling: UV_TORCH_BACKEND=cu130 uv pip install -e ".[dev]".

Then jump to Verify the install.

Building without a visible GPU (CI / Docker)

With no GPU visible the kernel build can't probe the arch and auto can't detect the driver — name both explicitly:

UV_TORCH_BACKEND=cu130 TORCH_CUDA_ARCH_LIST=12.1a uv pip install -e .

Verify the install

python - <<'PY'
import torch, fastvideo
print("fastvideo:", fastvideo.__version__)
print("torch    :", torch.__version__, "| cuda:", torch.version.cuda, "| avail:", torch.cuda.is_available())
print("device   :", torch.cuda.get_device_name(0))
PY

fastvideo --help

Expected: a +cu130 torch, device: NVIDIA GB10, and the CLI listing generate / serve / router-serve / bench / eval.

To confirm the compiled CUDA kernel actually runs on the GB10 (importing alone doesn't execute the .so):

python - <<'PY'
import torch
from fastvideo_kernel import Int8Linear
lin  = torch.nn.Linear(512, 256, bias=False).cuda().to(torch.bfloat16)
# .cuda() is required: from_linear() leaves the int8 buffers on the CPU
qlin = Int8Linear.from_linear(lin, quantize=True).cuda()  # compiled quant_cuda
x    = torch.randn(128, 512, device="cuda", dtype=torch.bfloat16)
y, ref = qlin(x), lin(x)                                  # compiled gemm_cuda
rel = (y.float() - ref.float()).norm() / ref.float().norm()
print(f"int8 GEMM rel err vs fp32: {rel.item():.4f}  (~0.01 is correct; int8 is lossy)")
PY

Manual build (advanced / fallback)

The one-liner above is the supported path. Build the kernel yourself only if you are iterating on the CUDA source or the auto-build fails:

# install torch + the kernel's build deps into the active venv first
UV_TORCH_BACKEND=cu130 uv pip install torch torchvision torchaudio scikit-build-core cmake ninja setuptools wheel
# build just the kernel against that torch (auto-detects sm_121)
uv pip install ./fastvideo-kernel --no-build-isolation
# then the rest of FastVideo
uv pip install -e .

Or cd fastvideo-kernel && ./build.sh, which auto-detects the arch and does the same compile (it initializes only the kernel's cutlass and tk submodules).

Optional: flash-attn

Not installed by default. FastVideo falls back to other attention backends without it. For the guide's Python 3.12, PyTorch 2.12, and CUDA 13 environment, install the matching prebuilt Linux ARM64 wheel from the FlashAttention prebuilt wheel index:

uv pip install "https://github.com/mjun0812/flash-attention-prebuild-wheels/releases/download/v0.9.22/flash_attn-2.8.3%2Bcu130torch2.12-cp312-cp312-linux_aarch64.whl"

Troubleshooting

Symptom Cause / Fix
Could NOT find Python (missing: ... Development.Module) venv built from system Python without headers. Recreate with --python-preference only-managed (add --clear to reuse the path), or sudo apt install python3.12-dev.
kernel build can't find cutlass headers Submodules not initialised — run the git submodule update step.
fastvideo-kernel: could not determine the target CUDA architecture The build couldn't see a GPU and no arch was given. Build on the Spark itself, or pass TORCH_CUDA_ARCH_LIST=12.1a (see Building without a visible GPU).
sm_121a requires CUDA Toolkit 13.0+ Confirm nvcc --version is 13.x and CUDACXX=/usr/local/cuda/bin/nvcc; architecture-specific sm_120a code cannot run on sm_121.
ninja: command not found (manual build only) uv pip install scikit-build-core cmake ninja setuptools wheel.

If you hit other issues, please open an issue on our GitHub repository. You can also join our Slack community for additional support.

Next: performance & tuning

Installed and verified? See DGX Spark: Performance & Tuning for which models are practical on the GB10, what makes them faster, and what won't help on this hardware (and why) — so you don't spend a night tuning knobs that can't move here.

Development Environment Setup

If you're planning to contribute to FastVideo please see the Contributor Guide.