Repository navigation
Commit 1d95a2a
authored
[CUDA] Tune hardware-aware split-K decode scheduling (pytorch#22190)
## Stack dependency
Depends on pytorch#22189.
## Summary
Combine the D113 SM-sized split-K work with the final scheduling policy
for the numerically stable online-softmax implementation.
- Derive split counts from the target GPU SM count, `L_kv`, and `B *
H_kv`.
- Keep the launch size static while reading runtime `kv_len` on device,
preserving CUDA graph compatibility.
- Retune the occupancy target from D113's two waves to 16/9 waves.
- Cap splits by both KV-buffer size and a maximum of 128 partitions to
avoid excessive reduction overhead on high-SM GPUs.
- Use contiguous KV partitions instead of D113's persistent strided
traversal; profiling showed that contiguous partitions are faster for
the online-softmax kernel.
- Apply the same hardware-aware policy to both decode and small-query
split-K paths.
- Keep scale arithmetic explicitly in FP32.
The policy contains no model-specific or GPU-specific shape hard-coding.
The SM count is specialized during native export. Under the supported
deployment flow, each target architecture is exported independently and
those variants may then be merged into one PTE; moving a native artifact
to a same-architecture GPU with a different SM count remains correct,
although its scheduling can be suboptimal.
## Operator-level performance
RTX 5090 CUDA Graph replay, BF16 GQA decode (`B=1, Hq=32, Hkv=2,
D=128`), compared with pytorch#22189:
| KV buffer | Runtime KV | pytorch#22189 | This PR | Change |
|---:|---:|---:|---:|---:|
| 4,096 | 512 | 10.86 us | 9.99 us | +8.6% |
| 4,096 | 2,048 | 12.66 us | 10.66 us | +18.8% |
| 4,096 | 4,096 | 13.49 us | 12.51 us | +7.8% |
| 130,048 | 32,768 | 33.21 us | 33.13 us | +0.2% |
| 130,048 | 130,048 | 95.50 us | 96.50 us | -1.0% |
The optimization primarily improves the sliding-window decode family.
The 128-split cap keeps the long global-attention path within
approximately 1% of the preceding PR instead of over-splitting it.
## Test plan
- `lintrunner init && lintrunner -a` (clean).
- Triton split-K and FP8 dispatch suite: 47 passed.
- Standalone D113 decode correctness smoke: 3 passed.
- Covered runtime KV-length tail exclusion, non-power-of-two split
counts, and large-logit stability.
- `git diff --check`.1 parent 645ec92 commit 1d95a2a
2 files changed
Lines changed: 105 additions & 14 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
314 | 314 | | |
315 | 315 | | |
316 | 316 | | |
317 | | - | |
| 317 | + | |
318 | 318 | | |
319 | | - | |
| 319 | + | |
| 320 | + | |
320 | 321 | | |
321 | 322 | | |
322 | 323 | | |
323 | | - | |
324 | | - | |
325 | | - | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
326 | 327 | | |
327 | | - | |
| 328 | + | |
328 | 329 | | |
329 | 330 | | |
330 | 331 | | |
| |||
343 | 344 | | |
344 | 345 | | |
345 | 346 | | |
346 | | - | |
| 347 | + | |
347 | 348 | | |
348 | 349 | | |
349 | 350 | | |
| |||
389 | 390 | | |
390 | 391 | | |
391 | 392 | | |
392 | | - | |
| 393 | + | |
393 | 394 | | |
394 | | - | |
395 | | - | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
396 | 402 | | |
397 | 403 | | |
398 | 404 | | |
| |||
410 | 416 | | |
411 | 417 | | |
412 | 418 | | |
| 419 | + | |
| 420 | + | |
| 421 | + | |
| 422 | + | |
| 423 | + | |
| 424 | + | |
| 425 | + | |
| 426 | + | |
| 427 | + | |
| 428 | + | |
| 429 | + | |
| 430 | + | |
| 431 | + | |
| 432 | + | |
| 433 | + | |
| 434 | + | |
| 435 | + | |
| 436 | + | |
| 437 | + | |
413 | 438 | | |
414 | 439 | | |
415 | 440 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| 34 | + | |
34 | 35 | | |
35 | 36 | | |
36 | 37 | | |
| |||
51 | 52 | | |
52 | 53 | | |
53 | 54 | | |
54 | | - | |
55 | | - | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
56 | 120 | | |
57 | 121 | | |
58 | 122 | | |
| |||
1256 | 1320 | | |
1257 | 1321 | | |
1258 | 1322 | | |
| 1323 | + | |
1259 | 1324 | | |
1260 | 1325 | | |
1261 | 1326 | | |
| |||
1437 | 1502 | | |
1438 | 1503 | | |
1439 | 1504 | | |
1440 | | - | |
| 1505 | + | |
1441 | 1506 | | |
1442 | 1507 | | |
1443 | 1508 | | |
| |||
1717 | 1782 | | |
1718 | 1783 | | |
1719 | 1784 | | |
| 1785 | + | |
1720 | 1786 | | |
1721 | 1787 | | |
1722 | 1788 | | |
| |||
1955 | 2021 | | |
1956 | 2022 | | |
1957 | 2023 | | |
1958 | | - | |
| 2024 | + | |
1959 | 2025 | | |
1960 | 2026 | | |
1961 | 2027 | | |
| |||
0 commit comments