BENCHMARK_GB200_CUDA_130.md

Benchmark Results

Auto-generated by benchmarks/generate_benchmark_md.py on 2026-05-19.

GPU: NVIDIA GB200 | CUDA: 13.0 | PyTorch: 2.9.1+cu130

FLA baseline: flash-linear-attention v0.5.0

KDA (Kimi Delta Attention)

Fixed-Length (H=64, D=128, bf16)

B	T	FLA Triton (ms)	cuLA (ms)	Speedup
1	512	0.838	0.604	1.39x
1	1024	0.694	0.571	1.22x
1	4096	0.759	0.564	1.35x
1	8192	1.406	1.026	1.37x
1	16384	2.734	1.965	1.39x
2	512	0.665	0.555	1.20x
2	1024	0.695	0.562	1.24x
2	4096	1.408	1.034	1.36x
2	8192	2.733	1.978	1.38x
2	16384	5.354	3.877	1.38x

Summary (10 configs): avg=1.33x, min=1.20x, max=1.39x.

Variable-Length (H=64, D=128, bf16)

Config	FLA Triton (ms)	cuLA (ms)	Speedup
uniform 10seqs T=4096 [409..415] avg=409	0.796	0.600	1.33x
random 10seqs T=4096 [24..1201] avg=409	0.789	0.587	1.34x
skewed 10seqs T=4096 [227..2053] avg=409	0.790	0.590	1.34x
uniform 20seqs T=4096 [204..220] avg=204	0.871	0.649	1.34x
random 20seqs T=4096 [5..787] avg=204	0.843	0.634	1.33x
skewed 20seqs T=4096 [107..2063] avg=204	0.822	0.608	1.35x
uniform 10seqs T=8192 [819..821] avg=819	1.405	1.045	1.34x
random 10seqs T=8192 [48..2401] avg=819	1.433	1.070	1.34x
skewed 10seqs T=8192 [455..4097] avg=819	1.458	1.068	1.37x
uniform 20seqs T=8192 [409..421] avg=409	1.494	1.095	1.36x
random 20seqs T=8192 [9..1574] avg=409	1.494	1.097	1.36x
skewed 20seqs T=8192 [215..4107] avg=409	1.499	1.101	1.36x
uniform 10seqs T=16384 [1638..1642] avg=1638	2.696	1.988	1.36x
random 10seqs T=16384 [95..4802] avg=1638	2.704	1.990	1.36x
skewed 10seqs T=16384 [910..8194] avg=1638	2.715	2.000	1.36x
uniform 20seqs T=16384 [819..823] avg=819	2.718	1.998	1.36x
random 20seqs T=16384 [19..3147] avg=819	2.742	2.023	1.36x
skewed 20seqs T=16384 [431..8195] avg=819	2.723	2.001	1.36x

Summary (18 configs): avg=1.35x, min=1.33x, max=1.37x.

To reproduce:

python benchmarks/bench_kda.py --mode both

Lightning Attention

Prefill (H=64, D=128, bf16)

B	T	FLA Triton (ms)	cuLA (ms)	Speedup
1	1024	0.112	0.073	1.53x
1	4096	0.175	0.157	1.11x
1	8192	0.329	0.292	1.13x
1	16384	0.629	0.563	1.12x
2	1024	0.099	0.068	1.45x
2	4096	0.327	0.176	1.86x
2	8192	0.631	0.327	1.93x
2	16384	1.257	0.632	1.99x

Variable-Length (H=64, D=128, bf16)

Persistent CuTe DSL kernel vs FLA Triton varlen.

N (seqs)	T	cuLA (ms)	FLA Triton (ms)	Speedup
5	1020	0.095	0.199	2.08x
5	2045	0.112	0.219	1.96x
5	4095	0.164	0.262	1.60x
5	8190	0.266	0.410	1.54x
5	16380	0.464	0.698	1.50x
5	32765	0.860	1.289	1.50x
8	1024	0.096	0.165	1.72x
8	2048	0.111	0.197	1.78x
8	4096	0.157	0.248	1.58x
8	8192	0.241	0.389	1.61x
8	16384	0.412	0.680	1.65x
8	32768	0.757	1.250	1.65x
10	1020	0.105	0.159	1.52x
10	2040	0.133	0.199	1.50x
10	4090	0.180	0.261	1.45x
10	8190	0.266	0.403	1.51x
10	16380	0.440	0.688	1.56x
10	32760	0.789	1.264	1.60x
12	1020	0.118	0.164	1.39x
12	2040	0.142	0.190	1.35x
12	4092	0.189	0.260	1.37x
12	8184	0.280	0.401	1.43x
12	16380	0.454	0.697	1.54x
12	32760	0.795	1.250	1.57x
16	1024	0.121	0.162	1.35x
16	2048	0.149	0.186	1.24x
16	4096	0.188	0.254	1.35x
16	8192	0.267	0.398	1.49x
16	16384	0.424	0.688	1.62x
16	32768	0.742	1.242	1.67x
20	1020	0.162	0.173	1.07x
20	2040	0.191	0.203	1.06x
20	4080	0.235	0.283	1.20x
20	8180	0.319	0.415	1.30x
20	16380	0.481	0.691	1.44x
20	32760	0.804	1.262	1.57x
25	1000	0.193	0.184	0.95x
25	2025	0.223	0.225	1.01x
25	4075	0.260	0.288	1.11x
25	8175	0.349	0.450	1.29x
25	16375	0.520	0.718	1.38x
25	32750	0.834	1.275	1.53x

Summary (126 configs across uniform/skewed/random): avg=1.47x, min=0.92x, max=2.16x.

To reproduce:

python benchmarks/bench_lightning_attn_prefill.py --modes no_state varlen

Decode (H=64, D=128, bf16, T=1)

Single-token decode: la_decode (CuTe DSL) vs fla fused_recurrent (Triton).

Kernel-Only

B	FLA Triton (ms)	cuLA (ms)	Speedup
1	0.0728	0.0149	4.88x
4	0.0722	0.0147	4.92x
16	0.0763	0.0209	3.66x
64	0.0997	0.0843	1.18x
256	0.3494	0.3123	1.12x

Wrapper (Full Call Path)

B	FLA Triton (ms)	cuLA (ms)	Speedup
1	0.0953	0.0194	4.91x
4	0.0924	0.0193	4.80x
16	0.0977	0.0233	4.20x
64	0.1029	0.0846	1.22x
256	0.3490	0.3133	1.11x

To reproduce:

python benchmarks/bench_la_decode_vs_fla.py --heads 64 --head-dim 128