Ternary Bonsai 2 27B and Qwen3.8-27B on one RTX 3090, RTX 4090, RTX 5090 and RTX PRO 6000 each, at
the models' full 262,144-token window where the card holds it, with this line's defaults: the device
route profile of each card (built in), --gdn-state-fp16, text only, one request at a time unless a
section says otherwise. The first section compares this line with the previous master on the same
hosts; the RTX PRO 6000 was measured on this line only.
The public master before this round (8a4c2f18) and this line on the same host with the flags of the reference rows below (Ternary Bonsai 2 at the 262,144-token window, Qwen3.8 at the window its row uses on that card), one request; before → after. With speculation the decode columns also move with the drafts accepted, which differ because the two builds' answers part at near-tied tokens; the rows without speculation show what the kernels changed. Prompts get slower in one place, the RTX 5090: Qwen3.8 takes 8 to 10% longer over 8K to 32K tokens and Bonsai 2 up to 40 ms longer up to 8K; from 64K on both gain:
| model | KV | speculation | short chat, tok/s | TTFT 32K, s | TTFT 131K, s | TTFT 261K, s | decode after 32K | after 131K | after 261K |
|---|---|---|---|---|---|---|---|---|---|
| Bonsai 2 | rk4v4 |
none | 168.3 → 170.3 (+1%) | 5.6 → 5.2 (-8%) | 38.6 → 30.3 (-22%) | 120.0 → 85.1 (-29%) | 155.6 → 157.8 (+1%) | 131.7 → 133.8 (+2%) | 109.6 → 111.6 (+2%) |
| Bonsai 2 | rk8v4 |
DFlash2, 5 drafts | 352.1 → 362.0 (+3%) | 5.6 → 5.2 (-7%) | 37.6 → 29.8 (-21%) | 115.1 → 82.4 (-28%) | 264.5 → 255.8 (-3%) | 193.7 → 205.2 (+6%) | 190.7 → 183.9 (-4%) |
| Bonsai 2 | rk8v4 |
MTP, 3 drafts | 316.1 → 316.6 (+0%) | 5.5 → 5.1 (-7%) | 37.3 → 29.5 (-21%) | 114.6 → 81.8 (-29%) | 288.6 → 272.8 (-5%) | 202.9 → 197.9 (-2%) | 193.7 → 200.6 (+4%) |
| Bonsai 2 | rk8v4 |
none | 168.9 → 170.5 (+1%) | 5.3 → 5.1 (-5%) | 36.9 → 29.4 (-20%) | 114.5 → 81.6 (-29%) | 153.2 → 154.4 (+1%) | 123.0 → 124.0 (+1%) | 98.2 → 99.0 (+1%) |
| Qwen3.8 | rk8v4 |
DFlash2, 5 drafts | 224.9 → 222.6 (-1%) | 7.9 → 8.5 (+8%) | 46.7 → 42.7 (-9%) | 133.3 → 107.5 (-19%) | 214.2 → 210.8 (-2%) | 141.6 → 156.8 (+11%) | 118.2 → 129.7 (+10%) |
| Qwen3.8 | rk8v4 |
none | 89.6 → 91.5 (+2%) | 7.8 → 8.5 (+8%) | 46.2 → 42.2 (-9%) | 132.6 → 106.9 (-19%) | 84.8 → 86.6 (+2%) | 74.8 → 76.2 (+2%) | 64.8 → 66.0 (+2%) |
| model | KV | speculation | short chat, tok/s | TTFT 32K, s | TTFT 131K, s | TTFT 261K, s | decode after 32K | after 131K | after 261K |
|---|---|---|---|---|---|---|---|---|---|
| Bonsai 2 | rk4v4 |
none | 107.8 → 107.9 (+0%) | 6.9 → 6.4 (-8%) | 46.8 → 37.2 (-20%) | 145.2 → 105.2 (-28%) | 97.9 → 99.3 (+1%) | 80.8 → 83.0 (+3%) | 56.5 → 63.8 (+13%) |
| Bonsai 2 | rk8v4 |
DFlash2, 5 drafts | 244.2 → 244.8 (+0%) | 6.9 → 6.4 (-7%) | 45.6 → 36.9 (-19%) | 139.2 → 103.3 (-26%) | 225.5 → 180.0 (-20%) | 140.6 → 125.8 (-11%) | 112.0 → 122.9 (+10%) |
| Bonsai 2 | rk8v4 |
MTP, 3 drafts | 199.5 → 203.4 (+2%) | 6.8 → 6.3 (-7%) | 45.3 → 36.6 (-19%) | 138.7 → 102.6 (-26%) | 183.1 → 164.8 (-10%) | 121.4 → 127.7 (+5%) | 112.4 → 111.0 (-1%) |
| Bonsai 2 | rk8v4 |
none | 108.2 → 108.4 (+0%) | 6.7 → 6.3 (-7%) | 45.0 → 36.2 (-20%) | 138.3 → 102.1 (-26%) | 97.1 → 97.8 (+1%) | 76.0 → 77.3 (+2%) | 55.5 → 56.4 (+2%) |
| Qwen3.8 | rk8v4 |
DFlash2, 5 drafts | 140.4 → 138.1 (-2%) | 13.5 → 13.2 (-2%) | 71.7 → 62.7 (-13%) | - | 111.8 → 140.3 (+25%) | 94.9 → 90.9 (-4%) | - |
| Qwen3.8 | rk8v4 |
none | 54.7 → 54.8 (+0%) | 13.3 → 13.0 (-2%) | 71.1 → 62.2 (-12%) | - | 51.7 → 52.0 (+1%) | 45.1 → 45.6 (+1%) | - |
| model | KV | speculation | short chat, tok/s | TTFT 32K, s | TTFT 131K, s | TTFT 261K, s | decode after 32K | after 131K | after 261K |
|---|---|---|---|---|---|---|---|---|---|
| Bonsai 2 | rk4v4 |
none | 93.5 → 93.8 (+0%) | 15.0 → 13.3 (-11%) | 105.2 → 77.9 (-26%) | 328.9 → 221.1 (-33%) | 81.7 → 83.7 (+3%) | 60.2 → 64.4 (+7%) | 44.5 → 49.2 (+11%) |
| Bonsai 2 | rk8v4 |
DFlash2, 5 drafts | 203.8 → 201.4 (-1%) | 15.1 → 13.4 (-11%) | 103.0 → 77.4 (-25%) | 317.4 → 216.9 (-32%) | 157.0 → 135.4 (-14%) | 90.8 → 92.6 (+2%) | 74.1 → 89.7 (+21%) |
| Bonsai 2 | rk8v4 |
MTP, 3 drafts | 173.6 → 172.2 (-1%) | 14.9 → 13.2 (-11%) | 102.2 → 76.5 (-25%) | 315.9 → 215.4 (-32%) | 134.5 → 150.4 (+12%) | 95.2 → 93.7 (-2%) | 87.5 → 89.3 (+2%) |
| Bonsai 2 | rk8v4 |
none | 93.4 → 94.1 (+1%) | 14.8 → 13.1 (-11%) | 101.4 → 76.2 (-25%) | 315.1 → 214.5 (-32%) | 83.3 → 84.3 (+1%) | 63.4 → 65.3 (+3%) | 47.8 → 50.0 (+5%) |
| Qwen3.8 | rk8v4 |
DFlash2, 5 drafts | 113.4 → 117.1 (+3%) | 23.4 → 23.4 (+0%) | 137.0 → 117.7 (-14%) | - | 113.8 → 110.8 (-3%) | 62.7 → 72.6 (+16%) | - |
| Qwen3.8 | rk8v4 |
none | 48.3 → 48.2 (-0%) | 23.1 → 23.1 (+0%) | 135.7 → 116.4 (-14%) | - | 44.9 → 45.1 (+1%) | 37.7 → 38.4 (+2%) | - |
Window 262,144 tokens, one request.
Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):
| KV | speculation | VRAM | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 12.4 GiB | 171.4 | 170.8 | 165.9 | 159.0 | 150.2 | 134.6 | 122.3 | 112.4 |
rk4v4 |
DFlash2, 5 drafts | 13.8 GiB | 366.2 | 408.8 | 292.7 | 325.6 | 259.2 | 203.3 | 179.4 | 168.4 |
rk4v4 |
MTP, 3 drafts | 13.1 GiB | 321.1 | 268.3 | 279.3 | 291.5 | 228.9 | 217.2 | 203.0 | 217.6 |
rk4v4-e8 |
none | 12.4 GiB | 170.4 | 169.9 | 166.4 | 158.2 | 149.4 | 134.1 | 122.1 | 112.1 |
rk4v4-e8 |
DFlash2, 5 drafts | 13.8 GiB | 387.9 | 315.1 | 332.1 | 301.1 | 241.6 | 219.2 | 219.9 | 197.8 |
rk8v4 |
none | 14.4 GiB | 171.5 | 171.0 | 165.8 | 155.4 | 143.8 | 124.7 | 110.7 | 99.4 |
rk8v4 |
DFlash2, 5 drafts | 15.8 GiB | 368.2 | 336.1 | 305.9 | 343.2 | 248.9 | 215.2 | 238.9 | 189.4 |
rk8v4 |
MTP, 3 drafts | 15.2 GiB | 329.0 | 336.5 | 299.9 | 306.0 | 244.5 | 207.1 | 198.7 | 207.4 |
Time to first token, s (cold prompt of that length):
| KV | speculation | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 0.17 | 1.17 | 5.2 | 12.0 | 30.2 | 54.9 | 85.2 |
rk4v4 |
DFlash2, 5 drafts | 0.18 | 1.20 | 5.3 | 12.1 | 30.6 | 55.4 | 85.9 |
rk4v4 |
MTP, 3 drafts | 0.18 | 1.18 | 5.2 | 12.0 | 30.4 | 55.0 | 85.5 |
rk4v4-e8 |
none | 0.17 | 1.17 | 5.1 | 11.9 | 30.1 | 54.3 | 84.2 |
rk4v4-e8 |
DFlash2, 5 drafts | 0.18 | 1.19 | 5.2 | 12.1 | 30.4 | 54.9 | 85.0 |
rk8v4 |
none | 0.17 | 1.14 | 5.0 | 11.5 | 29.0 | 52.7 | 81.9 |
rk8v4 |
DFlash2, 5 drafts | 0.18 | 1.19 | 5.2 | 11.9 | 29.8 | 53.5 | 82.7 |
rk8v4 |
MTP, 3 drafts | 0.18 | 1.18 | 5.1 | 11.8 | 29.6 | 53.2 | 82.2 |
Draft acceptance (accepted / drafted; tokens per verify round in brackets):
| KV | speculation | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
DFlash2, 5 drafts | 0.39 (2.92) | 0.46 (3.28) | 0.28 (2.40) | 0.38 (2.89) | 0.29 (2.46) | 0.24 (2.20) | 0.24 (2.19) | 0.26 (2.28) |
rk4v4 |
MTP, 3 drafts | 0.50 (2.49) | 0.37 (2.12) | 0.43 (2.29) | 0.50 (2.48) | 0.36 (2.06) | 0.40 (2.19) | 0.43 (2.27) | 0.56 (2.67) |
rk4v4-e8 |
DFlash2, 5 drafts | 0.43 (3.10) | 0.31 (2.53) | 0.35 (2.73) | 0.34 (2.67) | 0.26 (2.29) | 0.28 (2.38) | 0.34 (2.67) | 0.34 (2.67) |
rk8v4 |
DFlash2, 5 drafts | 0.39 (2.93) | 0.34 (2.69) | 0.31 (2.54) | 0.41 (3.02) | 0.27 (2.35) | 0.26 (2.27) | 0.36 (2.81) | 0.29 (2.46) |
rk8v4 |
MTP, 3 drafts | 0.52 (2.55) | 0.55 (2.63) | 0.47 (2.40) | 0.55 (2.65) | 0.43 (2.29) | 0.40 (2.21) | 0.45 (2.36) | 0.57 (2.71) |
Needles (three codes at 33, 66 and 90% of a document):
| KV | speculation | document | found | order | TTFT s |
|---|---|---|---|---|---|
rk4v4 |
none | 261,019 | 3/3 | yes | 85.3 |
rk4v4-e8 |
none | 261,022 | 3/3 | yes | 84.2 |
rk8v4 |
none | 261,021 | 3/3 | yes | 82.0 |
Window 262,144 tokens, one request.
Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):
| KV | speculation | VRAM | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 21.6 GiB | 91.9 | 91.7 | 90.2 | 88.1 | 85.3 | 80.0 | 75.4 | 71.6 |
rk4v4 |
DFlash2, 5 drafts | 23.8 GiB | 214.9 | 193.9 | 197.2 | 204.7 | 151.8 | 129.7 | 136.0 | 116.1 |
rk4v4 |
MTP, 3 drafts | 22.3 GiB | 182.7 | 153.4 | 158.9 | 171.8 | 165.7 | 140.8 | 132.6 | 129.0 |
rk4v4-e8 |
none | 21.6 GiB | 91.5 | 91.3 | 90.2 | 87.8 | 84.9 | 79.7 | 75.2 | 71.3 |
rk4v4-e8 |
DFlash2, 5 drafts | 23.8 GiB | 220.3 | 165.0 | 176.0 | 202.2 | 147.0 | 147.9 | 128.7 | 128.8 |
rk8v4 |
none | 23.6 GiB | 92.0 | 91.7 | 90.2 | 87.1 | 83.3 | 76.5 | 71.0 | 66.2 |
rk8v4 |
DFlash2, 5 drafts | 25.8 GiB | 221.0 | 186.7 | 204.6 | 205.9 | 157.8 | 152.7 | 134.0 | 129.8 |
rk8v4 |
MTP, 3 drafts | 24.4 GiB | 184.7 | 153.7 | 169.8 | 180.1 | 140.5 | 130.3 | 130.5 | 114.0 |
Time to first token, s (cold prompt of that length):
| KV | speculation | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 0.34 | 2.04 | 8.5 | 18.4 | 43.0 | 73.8 | 110.4 |
rk4v4 |
DFlash2, 5 drafts | 0.34 | 2.06 | 8.6 | 18.6 | 43.4 | 74.4 | 111.6 |
rk4v4 |
MTP, 3 drafts | 0.34 | 2.07 | 8.6 | 18.7 | 43.6 | 74.8 | 111.7 |
rk4v4-e8 |
none | 0.34 | 2.05 | 8.5 | 18.5 | 43.0 | 73.7 | 109.8 |
rk4v4-e8 |
DFlash2, 5 drafts | 0.34 | 2.07 | 8.6 | 18.7 | 43.4 | 74.1 | 110.4 |
rk8v4 |
none | 0.33 | 2.03 | 8.4 | 18.2 | 42.1 | 71.9 | 107.0 |
rk8v4 |
DFlash2, 5 drafts | 0.34 | 2.05 | 8.5 | 18.3 | 42.4 | 72.4 | 107.7 |
rk8v4 |
MTP, 3 drafts | 0.33 | 2.03 | 8.4 | 18.2 | 42.2 | 72.0 | 107.1 |
Draft acceptance (accepted / drafted; tokens per verify round in brackets):
| KV | speculation | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
DFlash2, 5 drafts | 0.42 (3.10) | 0.37 (2.81) | 0.38 (2.91) | 0.43 (3.17) | 0.29 (2.44) | 0.25 (2.26) | 0.31 (2.55) | 0.27 (2.34) |
rk4v4 |
MTP, 3 drafts | 0.58 (2.71) | 0.44 (2.31) | 0.48 (2.43) | 0.57 (2.69) | 0.57 (2.68) | 0.48 (2.44) | 0.48 (2.44) | 0.51 (2.51) |
rk4v4-e8 |
DFlash2, 5 drafts | 0.44 (3.19) | 0.28 (2.40) | 0.32 (2.61) | 0.43 (3.15) | 0.28 (2.37) | 0.32 (2.58) | 0.29 (2.42) | 0.32 (2.58) |
rk8v4 |
DFlash2, 5 drafts | 0.44 (3.19) | 0.34 (2.71) | 0.41 (3.05) | 0.43 (3.15) | 0.31 (2.53) | 0.33 (2.63) | 0.30 (2.46) | 0.31 (2.56) |
rk8v4 |
MTP, 3 drafts | 0.59 (2.75) | 0.44 (2.32) | 0.53 (2.57) | 0.61 (2.81) | 0.44 (2.30) | 0.44 (2.30) | 0.49 (2.46) | 0.44 (2.32) |
Needles (three codes at 33, 66 and 90% of a document):
| KV | speculation | document | found | order | TTFT s |
|---|---|---|---|---|---|
rk4v4 |
none | 261,020 | 3/3 | yes | 110.4 |
rk4v4-e8 |
none | 261,022 | 3/3 | yes | 109.7 |
rk8v4 |
none | 261,017 | 3/3 | yes | 107.0 |
Window 262,144 tokens, one request.
Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):
| KV | speculation | VRAM | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 12.3 GiB | 107.9 | 107.6 | 105.1 | 99.3 | 93.3 | 83.0 | 75.1 | 63.8 |
rk4v4 |
DFlash2, 5 drafts | 13.8 GiB | 242.8 | 179.5 | 181.8 | 236.3 | 156.9 | 134.9 | 140.4 | 115.3 |
rk4v4 |
MTP, 3 drafts | 13.0 GiB | 206.9 | 158.6 | 179.9 | 180.7 | 142.6 | 134.7 | 131.6 | 116.3 |
rk4v4-e8 |
none | 12.3 GiB | 107.9 | 107.6 | 105.3 | 99.3 | 93.3 | 83.0 | 75.1 | 63.8 |
rk4v4-e8 |
DFlash2, 5 drafts | 13.8 GiB | 251.5 | 198.5 | 212.9 | 197.2 | 146.2 | 129.0 | 155.6 | 116.4 |
rk8v4 |
none | 14.3 GiB | 108.4 | 107.8 | 105.2 | 97.8 | 89.9 | 77.3 | 67.9 | 56.4 |
rk8v4 |
DFlash2, 5 drafts | 15.8 GiB | 244.8 | 243.6 | 206.7 | 180.0 | 160.9 | 125.8 | 118.7 | 122.9 |
rk8v4 |
MTP, 3 drafts | 15.1 GiB | 203.4 | 191.4 | 184.0 | 164.8 | 153.7 | 127.7 | 121.8 | 111.0 |
Time to first token, s (cold prompt of that length):
| KV | speculation | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 0.21 | 1.44 | 6.4 | 14.7 | 37.2 | 67.6 | 105.2 |
rk4v4 |
DFlash2, 5 drafts | 0.21 | 1.47 | 6.5 | 15.0 | 37.7 | 68.4 | 106.2 |
rk4v4 |
MTP, 3 drafts | 0.21 | 1.45 | 6.4 | 14.8 | 37.4 | 67.9 | 105.5 |
rk4v4-e8 |
none | 0.21 | 1.44 | 6.3 | 14.6 | 36.8 | 66.7 | 103.5 |
rk4v4-e8 |
DFlash2, 5 drafts | 0.22 | 1.47 | 6.5 | 14.8 | 37.3 | 67.6 | 104.6 |
rk8v4 |
none | 0.20 | 1.42 | 6.3 | 14.4 | 36.2 | 65.8 | 102.1 |
rk8v4 |
DFlash2, 5 drafts | 0.22 | 1.47 | 6.4 | 14.7 | 36.9 | 66.8 | 103.3 |
rk8v4 |
MTP, 3 drafts | 0.21 | 1.44 | 6.3 | 14.5 | 36.6 | 66.3 | 102.6 |
Draft acceptance (accepted / drafted; tokens per verify round in brackets):
| KV | speculation | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
DFlash2, 5 drafts | 0.39 (2.91) | 0.24 (2.17) | 0.25 (2.26) | 0.43 (3.11) | 0.25 (2.23) | 0.25 (2.22) | 0.32 (2.60) | 0.30 (2.51) |
rk4v4 |
MTP, 3 drafts | 0.53 (2.57) | 0.34 (2.01) | 0.44 (2.31) | 0.49 (2.49) | 0.36 (2.09) | 0.41 (2.21) | 0.47 (2.40) | 0.53 (2.59) |
rk4v4-e8 |
DFlash2, 5 drafts | 0.41 (3.01) | 0.28 (2.40) | 0.33 (2.63) | 0.32 (2.58) | 0.21 (2.07) | 0.23 (2.12) | 0.37 (2.85) | 0.30 (2.51) |
rk8v4 |
DFlash2, 5 drafts | 0.39 (2.93) | 0.39 (2.94) | 0.31 (2.56) | 0.27 (2.36) | 0.26 (2.27) | 0.21 (2.06) | 0.24 (2.19) | 0.33 (2.63) |
rk8v4 |
MTP, 3 drafts | 0.51 (2.51) | 0.46 (2.38) | 0.45 (2.34) | 0.42 (2.24) | 0.43 (2.28) | 0.39 (2.18) | 0.44 (2.32) | 0.55 (2.64) |
Needles (three codes at 33, 66 and 90% of a document):
| KV | speculation | document | found | order | TTFT s |
|---|---|---|---|---|---|
rk4v4 |
none | 261,015 | 3/3 | yes | 105.2 |
rk4v4-e8 |
none | 261,018 | 3/3 | yes | 103.5 |
rk8v4 |
none | 261,022 | 3/3 | yes | 102.2 |
One request. Each KV coding runs at the largest window DFlash2 fits beside it on this card, at most 262,144, for every drafter; rk4v4-e8 without speculation runs the full window.
Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):
| KV | speculation | window | VRAM | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 245,760 | 21.3 GiB | 54.7 | 54.6 | 53.9 | 52.4 | 50.7 | 47.5 | 44.8 | - |
rk4v4 |
DFlash2, 5 drafts | 245,760 | 23.5 GiB | 137.8 | 108.4 | 115.3 | 142.7 | 93.8 | 86.4 | 80.4 | - |
rk4v4 |
MTP, 3 drafts | 245,760 | 21.9 GiB | 108.9 | 93.5 | 102.0 | 105.3 | 92.8 | 85.0 | 83.3 | - |
rk4v4-e8 |
none | 262,144 | 21.5 GiB | 54.7 | 54.6 | 54.0 | 52.4 | 50.7 | 47.5 | 44.8 | 40.6 |
rk8v4 |
none | 167,936 | 21.2 GiB | 54.8 | 54.7 | 54.0 | 52.0 | 49.7 | 45.6 | - | - |
rk8v4 |
DFlash2, 5 drafts | 167,936 | 23.5 GiB | 138.1 | 127.1 | 129.8 | 140.3 | 102.2 | 90.9 | - | - |
rk8v4 |
MTP, 3 drafts | 167,936 | 21.9 GiB | 109.0 | 102.6 | 101.7 | 108.5 | 94.9 | 77.3 | - | - |
Time to first token, s (cold prompt of that length):
| KV | speculation | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 0.56 | 3.24 | 13.1 | 27.8 | 63.0 | 106.0 | - |
rk4v4 |
DFlash2, 5 drafts | 0.57 | 3.27 | 13.2 | 28.1 | 63.5 | 106.7 | - |
rk4v4 |
MTP, 3 drafts | 0.57 | 3.25 | 13.1 | 27.9 | 63.2 | 106.2 | - |
rk4v4-e8 |
none | 0.56 | 3.23 | 13.1 | 27.7 | 62.6 | 105.0 | 154.1 |
rk8v4 |
none | 0.56 | 3.23 | 13.0 | 27.6 | 62.2 | - | - |
rk8v4 |
DFlash2, 5 drafts | 0.57 | 3.26 | 13.2 | 27.8 | 62.7 | - | - |
rk8v4 |
MTP, 3 drafts | 0.57 | 3.24 | 13.1 | 27.6 | 62.3 | - | - |
Draft acceptance (accepted / drafted; tokens per verify round in brackets):
| KV | speculation | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
DFlash2, 5 drafts | 0.44 (3.15) | 0.30 (2.51) | 0.34 (2.71) | 0.49 (3.47) | 0.27 (2.36) | 0.28 (2.37) | 0.28 (2.37) | - |
rk4v4 |
MTP, 3 drafts | 0.57 (2.69) | 0.45 (2.36) | 0.53 (2.62) | 0.59 (2.79) | 0.50 (2.48) | 0.48 (2.45) | 0.52 (2.56) | - |
rk8v4 |
DFlash2, 5 drafts | 0.44 (3.16) | 0.39 (2.95) | 0.41 (3.05) | 0.48 (3.40) | 0.32 (2.56) | 0.30 (2.48) | - | - |
rk8v4 |
MTP, 3 drafts | 0.57 (2.68) | 0.52 (2.57) | 0.52 (2.55) | 0.61 (2.80) | 0.53 (2.56) | 0.42 (2.29) | - | - |
Needles (three codes at 33, 66 and 90% of a document):
| KV | speculation | document | found | order | TTFT s |
|---|---|---|---|---|---|
rk4v4 |
none | 244,635 | 3/3 | yes | 142.3 |
rk4v4-e8 |
none | 261,017 | 3/3 | yes | 154.1 |
rk8v4 |
none | 166,815 | 3/3 | yes | 84.1 |
Window 262,144 tokens, one request.
Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):
| KV | speculation | VRAM | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 12.2 GiB | 93.8 | 93.2 | 90.3 | 83.7 | 76.1 | 64.4 | 55.5 | 49.2 |
rk4v4 |
DFlash2, 5 drafts | 13.7 GiB | 200.8 | 172.3 | 180.3 | 174.7 | 116.5 | 97.5 | 73.5 | 78.6 |
rk4v4 |
MTP, 3 drafts | 12.9 GiB | 173.4 | 145.1 | 138.3 | 149.7 | 116.9 | 96.5 | 89.4 | 83.7 |
rk4v4-e8 |
none | 12.2 GiB | 93.6 | 93.2 | 90.4 | 84.0 | 76.5 | 64.9 | 56.0 | 49.8 |
rk4v4-e8 |
DFlash2, 5 drafts | 13.7 GiB | 202.8 | 157.2 | 173.3 | 140.0 | 119.4 | 109.2 | 113.9 | 76.3 |
rk8v4 |
none | 14.2 GiB | 94.1 | 93.5 | 90.8 | 84.3 | 76.8 | 65.3 | 56.3 | 50.0 |
rk8v4 |
DFlash2, 5 drafts | 15.7 GiB | 201.4 | 198.4 | 161.1 | 135.4 | 124.9 | 92.6 | 85.3 | 89.7 |
rk8v4 |
MTP, 3 drafts | 15.0 GiB | 172.2 | 143.6 | 184.8 | 150.4 | 109.4 | 93.7 | 94.7 | 89.3 |
Time to first token, s (cold prompt of that length):
| KV | speculation | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 0.42 | 2.99 | 13.3 | 30.7 | 77.9 | 142.0 | 221.1 |
rk4v4 |
DFlash2, 5 drafts | 0.43 | 3.06 | 13.5 | 31.2 | 79.2 | 143.8 | 223.7 |
rk4v4 |
MTP, 3 drafts | 0.42 | 3.00 | 13.3 | 30.8 | 78.3 | 142.7 | 222.0 |
rk4v4-e8 |
none | 0.42 | 2.98 | 13.2 | 30.4 | 77.1 | 140.2 | 218.2 |
rk4v4-e8 |
DFlash2, 5 drafts | 0.43 | 3.06 | 13.5 | 31.0 | 78.3 | 141.8 | 220.5 |
rk8v4 |
none | 0.42 | 2.97 | 13.1 | 30.2 | 76.2 | 138.3 | 214.5 |
rk8v4 |
DFlash2, 5 drafts | 0.43 | 3.05 | 13.4 | 30.8 | 77.4 | 139.8 | 216.9 |
rk8v4 |
MTP, 3 drafts | 0.42 | 2.99 | 13.2 | 30.3 | 76.5 | 138.8 | 215.4 |
Draft acceptance (accepted / drafted; tokens per verify round in brackets):
| KV | speculation | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
DFlash2, 5 drafts | 0.39 (2.91) | 0.31 (2.52) | 0.36 (2.78) | 0.40 (3.03) | 0.25 (2.26) | 0.27 (2.34) | 0.23 (2.13) | 0.33 (2.63) |
rk4v4 |
MTP, 3 drafts | 0.52 (2.54) | 0.39 (2.17) | 0.38 (2.15) | 0.51 (2.53) | 0.41 (2.22) | 0.41 (2.23) | 0.50 (2.47) | 0.56 (2.65) |
rk4v4-e8 |
DFlash2, 5 drafts | 0.39 (2.94) | 0.26 (2.30) | 0.34 (2.67) | 0.27 (2.34) | 0.25 (2.22) | 0.29 (2.44) | 0.41 (3.07) | 0.27 (2.33) |
rk8v4 |
DFlash2, 5 drafts | 0.39 (2.92) | 0.39 (2.91) | 0.30 (2.46) | 0.26 (2.27) | 0.27 (2.35) | 0.22 (2.12) | 0.27 (2.33) | 0.36 (2.80) |
rk8v4 |
MTP, 3 drafts | 0.51 (2.51) | 0.37 (2.12) | 0.62 (2.90) | 0.50 (2.49) | 0.34 (2.04) | 0.37 (2.10) | 0.50 (2.51) | 0.55 (2.65) |
Needles (three codes at 33, 66 and 90% of a document):
| KV | speculation | document | found | order | TTFT s |
|---|---|---|---|---|---|
rk4v4 |
none | 261,020 | 3/3 | yes | 221.4 |
rk4v4-e8 |
none | 261,020 | 3/3 | yes | 218.1 |
rk8v4 |
none | 261,019 | 3/3 | yes | 214.8 |
One request. Each KV coding runs at the largest window DFlash2 fits beside it on this card, at most 262,144, for every drafter; rk4v4-e8 without speculation runs the full window.
Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):
| KV | speculation | window | VRAM | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 258,048 | 21.3 GiB | 48.0 | 47.9 | 47.1 | 45.0 | 42.3 | 38.0 | 34.4 | - |
rk4v4 |
DFlash2, 5 drafts | 258,048 | 23.6 GiB | 115.9 | 88.5 | 86.1 | 93.2 | 70.6 | 60.5 | 54.1 | - |
rk4v4 |
MTP, 3 drafts | 258,048 | 22.0 GiB | 95.9 | 79.0 | 88.1 | 89.4 | 77.5 | 66.3 | 61.5 | - |
rk4v4-e8 |
none | 262,144 | 21.4 GiB | 48.0 | 47.9 | 47.1 | 45.2 | 42.5 | 38.3 | 34.7 | 32.0 |
rk8v4 |
none | 176,128 | 21.3 GiB | 48.2 | 48.0 | 47.2 | 45.1 | 42.6 | 38.4 | - | - |
rk8v4 |
DFlash2, 5 drafts | 176,128 | 23.5 GiB | 117.1 | 101.4 | 101.3 | 110.8 | 78.7 | 72.6 | - | - |
rk8v4 |
MTP, 3 drafts | 176,128 | 22.0 GiB | 97.2 | 90.6 | 90.5 | 83.6 | 83.9 | 60.9 | - | - |
Time to first token, s (cold prompt of that length):
| KV | speculation | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 1.15 | 5.63 | 23.2 | 50.2 | 118.3 | 203.4 | - |
rk4v4 |
DFlash2, 5 drafts | 1.17 | 5.73 | 23.6 | 51.0 | 119.4 | 205.2 | - |
rk4v4 |
MTP, 3 drafts | 1.17 | 5.68 | 23.3 | 50.3 | 118.6 | 203.9 | - |
rk4v4-e8 |
none | 1.15 | 5.62 | 23.1 | 50.0 | 117.3 | 201.4 | 300.9 |
rk8v4 |
none | 1.14 | 5.61 | 23.1 | 49.9 | 116.4 | - | - |
rk8v4 |
DFlash2, 5 drafts | 1.17 | 5.73 | 23.4 | 50.6 | 117.7 | - | - |
rk8v4 |
MTP, 3 drafts | 1.16 | 5.65 | 23.1 | 49.9 | 117.0 | - | - |
Draft acceptance (accepted / drafted; tokens per verify round in brackets):
| KV | speculation | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
DFlash2, 5 drafts | 0.45 (3.24) | 0.30 (2.51) | 0.30 (2.51) | 0.38 (2.86) | 0.27 (2.35) | 0.26 (2.31) | 0.27 (2.35) | - |
rk4v4 |
MTP, 3 drafts | 0.58 (2.73) | 0.42 (2.26) | 0.53 (2.60) | 0.59 (2.75) | 0.53 (2.57) | 0.50 (2.50) | 0.54 (2.61) | - |
rk8v4 |
DFlash2, 5 drafts | 0.46 (3.27) | 0.38 (2.86) | 0.39 (2.92) | 0.48 (3.39) | 0.32 (2.57) | 0.34 (2.68) | - | - |
rk8v4 |
MTP, 3 drafts | 0.59 (2.76) | 0.53 (2.59) | 0.55 (2.65) | 0.52 (2.55) | 0.58 (2.74) | 0.42 (2.25) | - | - |
Needles (three codes at 33, 66 and 90% of a document):
| KV | speculation | document | found | order | TTFT s |
|---|---|---|---|---|---|
rk4v4 |
none | 256,924 | 3/3 | yes | 297.5 |
rk4v4-e8 |
none | 261,016 | 3/3 | yes | 300.9 |
rk8v4 |
none | 175,006 | 3/3 | yes | 170.6 |
Window 262,144 tokens, one request.
Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):
| KV | speculation | VRAM | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 12.6 GiB | 167.2 | 166.6 | 161.3 | 155.1 | 146.4 | 131.3 | 119.5 | 109.8 |
rk4v4 |
DFlash2, 5 drafts | 14.0 GiB | 360.9 | 401.3 | 317.4 | 284.2 | 242.7 | 197.9 | 179.4 | 193.0 |
rk4v4 |
MTP, 3 drafts | 13.3 GiB | 330.4 | 266.0 | 306.9 | 229.8 | 226.2 | 208.6 | 198.3 | 217.9 |
rk4v4-e8 |
none | 12.6 GiB | 167.3 | 166.9 | 161.8 | 155.2 | 146.6 | 131.3 | 119.7 | 110.0 |
rk4v4-e8 |
DFlash2, 5 drafts | 14.0 GiB | 367.4 | 285.1 | 307.0 | 319.9 | 222.8 | 203.7 | 180.8 | 189.7 |
rk8v4 |
none | 14.6 GiB | 167.5 | 166.8 | 162.2 | 151.5 | 139.8 | 121.2 | 107.3 | 96.4 |
rk8v4 |
DFlash2, 5 drafts | 16.0 GiB | 361.7 | 304.6 | 318.1 | 309.6 | 250.7 | 204.6 | 233.5 | 200.9 |
rk8v4 |
MTP, 3 drafts | 15.4 GiB | 314.7 | 313.0 | 293.3 | 258.1 | 229.7 | 191.4 | 192.0 | 210.5 |
Time to first token, s (cold prompt of that length):
| KV | speculation | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 0.16 | 1.11 | 4.9 | 11.4 | 28.9 | 52.7 | 82.0 |
rk4v4 |
DFlash2, 5 drafts | 0.17 | 1.12 | 4.9 | 11.5 | 29.2 | 53.1 | 82.5 |
rk4v4 |
MTP, 3 drafts | 0.17 | 1.11 | 4.9 | 11.4 | 29.0 | 52.9 | 82.1 |
rk4v4-e8 |
none | 0.16 | 1.10 | 4.9 | 11.3 | 28.7 | 52.1 | 81.1 |
rk4v4-e8 |
DFlash2, 5 drafts | 0.17 | 1.12 | 4.9 | 11.4 | 28.9 | 52.5 | 81.6 |
rk8v4 |
none | 0.16 | 1.07 | 4.7 | 10.8 | 27.4 | 50.1 | 78.3 |
rk8v4 |
DFlash2, 5 drafts | 0.17 | 1.12 | 4.9 | 11.2 | 28.3 | 51.2 | 79.0 |
rk8v4 |
MTP, 3 drafts | 0.16 | 1.11 | 4.8 | 11.1 | 28.0 | 50.8 | 78.5 |
Draft acceptance (accepted / drafted; tokens per verify round in brackets):
| KV | speculation | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
DFlash2, 5 drafts | 0.39 (2.91) | 0.46 (3.26) | 0.33 (2.62) | 0.31 (2.54) | 0.26 (2.31) | 0.23 (2.14) | 0.24 (2.17) | 0.32 (2.57) |
rk4v4 |
MTP, 3 drafts | 0.54 (2.61) | 0.38 (2.13) | 0.52 (2.60) | 0.33 (1.99) | 0.36 (2.06) | 0.38 (2.14) | 0.41 (2.25) | 0.56 (2.67) |
rk4v4-e8 |
DFlash2, 5 drafts | 0.40 (2.96) | 0.26 (2.31) | 0.31 (2.53) | 0.37 (2.84) | 0.23 (2.12) | 0.24 (2.19) | 0.24 (2.18) | 0.30 (2.51) |
rk8v4 |
DFlash2, 5 drafts | 0.39 (2.91) | 0.30 (2.47) | 0.33 (2.65) | 0.35 (2.75) | 0.28 (2.38) | 0.24 (2.19) | 0.36 (2.81) | 0.33 (2.63) |
rk8v4 |
MTP, 3 drafts | 0.50 (2.48) | 0.50 (2.48) | 0.48 (2.46) | 0.42 (2.24) | 0.39 (2.15) | 0.35 (2.05) | 0.44 (2.30) | 0.60 (2.78) |
Needles (three codes at 33, 66 and 90% of a document):
| KV | speculation | document | found | order | TTFT s |
|---|---|---|---|---|---|
rk4v4 |
none | 261,020 | 3/3 | yes | 82.0 |
rk4v4-e8 |
none | 261,016 | 3/3 | yes | 81.0 |
rk8v4 |
none | 261,019 | 3/3 | yes | 78.4 |
Window 262,144 tokens, one request.
Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):
| KV | speculation | VRAM | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 21.8 GiB | 89.6 | 89.5 | 88.0 | 85.9 | 83.2 | 78.0 | 73.7 | 69.9 |
rk4v4 |
DFlash2, 5 drafts | 24.0 GiB | 219.9 | 189.4 | 179.6 | 183.7 | 154.0 | 142.4 | 166.4 | 116.1 |
rk4v4 |
MTP, 3 drafts | 22.5 GiB | 186.4 | 164.5 | 169.6 | 149.0 | 161.0 | 143.4 | 142.9 | 128.7 |
rk4v4-e8 |
none | 21.8 GiB | 89.7 | 89.5 | 88.0 | 85.9 | 83.2 | 78.1 | 73.8 | 70.0 |
rk4v4-e8 |
DFlash2, 5 drafts | 24.0 GiB | 222.7 | 184.6 | 171.7 | 214.9 | 148.5 | 142.2 | 150.3 | 121.6 |
rk8v4 |
none | 23.8 GiB | 89.6 | 89.4 | 88.0 | 84.9 | 81.1 | 74.5 | 69.1 | 64.4 |
rk8v4 |
DFlash2, 5 drafts | 26.0 GiB | 219.1 | 195.9 | 186.0 | 217.2 | 156.9 | 160.2 | 164.5 | 120.9 |
rk8v4 |
MTP, 3 drafts | 24.6 GiB | 187.5 | 154.3 | 172.4 | 176.4 | 164.2 | 135.8 | 129.6 | 120.4 |
Time to first token, s (cold prompt of that length):
| KV | speculation | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|
rk4v4 |
none | 0.30 | 1.87 | 7.9 | 17.2 | 40.5 | 70.0 | 104.9 |
rk4v4 |
DFlash2, 5 drafts | 0.31 | 1.88 | 7.9 | 17.3 | 40.9 | 70.5 | 105.6 |
rk4v4 |
MTP, 3 drafts | 0.31 | 1.87 | 7.8 | 17.2 | 40.6 | 70.1 | 105.0 |
rk4v4-e8 |
none | 0.30 | 1.87 | 7.8 | 17.1 | 40.3 | 69.4 | 104.0 |
rk4v4-e8 |
DFlash2, 5 drafts | 0.31 | 1.88 | 7.9 | 17.2 | 40.6 | 69.9 | 104.5 |
rk8v4 |
none | 0.30 | 1.86 | 7.7 | 16.9 | 39.4 | 67.7 | 100.8 |
rk8v4 |
DFlash2, 5 drafts | 0.31 | 1.87 | 7.8 | 17.0 | 39.7 | 68.3 | 101.7 |
rk8v4 |
MTP, 3 drafts | 0.30 | 1.86 | 7.8 | 16.9 | 39.6 | 68.1 | 101.5 |
Draft acceptance (accepted / drafted; tokens per verify round in brackets):
| KV | speculation | short chat | 1,024 | 8,192 | 32,768 | 65,536 | 131,072 | 196,608 | 261,120 |
|---|---|---|---|---|---|---|---|---|---|
rk4v4 |
DFlash2, 5 drafts | 0.44 (3.18) | 0.36 (2.75) | 0.33 (2.66) | 0.37 (2.85) | 0.30 (2.47) | 0.30 (2.48) | 0.42 (3.11) | 0.26 (2.31) |
rk4v4 |
MTP, 3 drafts | 0.60 (2.78) | 0.49 (2.46) | 0.54 (2.62) | 0.45 (2.34) | 0.54 (2.62) | 0.50 (2.49) | 0.54 (2.61) | 0.50 (2.48) |
rk4v4-e8 |
DFlash2, 5 drafts | 0.45 (3.21) | 0.34 (2.70) | 0.31 (2.54) | 0.47 (3.33) | 0.28 (2.38) | 0.29 (2.47) | 0.36 (2.78) | 0.28 (2.40) |
rk8v4 |
DFlash2, 5 drafts | 0.44 (3.16) | 0.37 (2.84) | 0.35 (2.77) | 0.47 (3.37) | 0.30 (2.51) | 0.35 (2.75) | 0.41 (3.04) | 0.28 (2.38) |
rk8v4 |
MTP, 3 drafts | 0.60 (2.80) | 0.44 (2.31) | 0.54 (2.62) | 0.60 (2.85) | 0.57 (2.74) | 0.48 (2.45) | 0.49 (2.46) | 0.48 (2.44) |
Needles (three codes at 33, 66 and 90% of a document):
| KV | speculation | document | found | order | TTFT s |
|---|---|---|---|---|---|
rk4v4 |
none | 261,021 | 3/3 | yes | 104.8 |
rk4v4-e8 |
none | 261,018 | 3/3 | yes | 104.0 |
rk8v4 |
none | 261,021 | 3/3 | yes | 100.9 |
The largest --max-context (= --kv-capacity, one request, --rope-yarn past 262,144) the server starts with, found by bisection in 4,096-token steps. Each cell gives it without speculation / with MTP (3 drafts) / with DFlash2 (5 drafts); 1,048,576 is the engine's ceiling. The RTX PRO 6000 starts every configuration at the ceiling with at least 50 GiB free.
| model | KV | RTX PRO 6000 | RTX 5090 | RTX 4090 | RTX 3090 |
|---|---|---|---|---|---|
| Ternary Bonsai 2 27B | rk8v4 |
1,048,576 / 1,048,576 / 1,048,576 | 978,944 / 901,120 / 901,120 | 659,456 / 602,112 / 581,632 | 663,552 / 606,208 / 585,728 |
| Ternary Bonsai 2 27B | rk4v4 |
1,048,576 / 1,048,576 / 1,048,576 | 1,048,576 / 1,048,576 / 1,048,576 | 958,464 / 876,544 / 831,488 | 970,752 / 884,736 / 839,680 |
| Ternary Bonsai 2 27B | rk4v4-e8 |
1,048,576 / - / - | 1,048,576 / - / - | 958,464 / - / - | 970,752 / - / - |
| Qwen3.8-27B | rk8v4 |
1,048,576 / 1,048,576 / 1,048,576 | 598,016 / 544,768 / 491,520 | 278,528 / 241,664 / 167,936 | 286,720 / 249,856 / 176,128 |
| Qwen3.8-27B | rk4v4 |
1,048,576 / 1,048,576 / 1,048,576 | 872,448 / 794,624 / 716,800 | 405,504 / 356,352 / 245,760 | 417,792 / 364,544 / 258,048 |
| Qwen3.8-27B | rk4v4-e8 |
1,048,576 / - / - | 872,448 / - / - | 405,504 / - / - | 417,792 / - / - |
Filled to that length (one request: a document with three codes at 33, 66 and 90%, asked back in order), without speculation:
| card | model | KV | tokens | needles | TTFT | decode after it |
|---|---|---|---|---|---|---|
| RTX PRO 6000 | Ternary Bonsai 2 27B | rk4v4 |
1,048,576 | 2/3 | 910 s | 55.3 tok/s |
| RTX PRO 6000 | Qwen3.8-27B | rk4v4 |
1,048,576 | 2/3 | 999 s | 42.9 tok/s |
| RTX 5090 | Ternary Bonsai 2 27B | rk4v4 |
1,048,576 | 2/3 | 937 s | 56.2 tok/s |
| RTX 5090 | Ternary Bonsai 2 27B | rk8v4 |
978,944 | 3/3 | 772 s | 46.9 tok/s |
| RTX 5090 | Qwen3.8-27B | rk4v4 |
872,448 | 3/3 | 755 s | 47.6 tok/s |
| RTX 5090 | Qwen3.8-27B | rk8v4 |
598,016 | 3/3 | 378 s | 49.1 tok/s |
| RTX 4090 | Ternary Bonsai 2 27B | rk4v4 |
958,464 | 3/3 | 991 s | 30.9 tok/s |
| RTX 4090 | Ternary Bonsai 2 27B | rk8v4 |
659,456 | 3/3 | 484 s | 32.9 tok/s |
| RTX 4090 | Qwen3.8-27B | rk4v4 |
405,504 | 3/3 | 293 s | 35.4 tok/s |
| RTX 4090 | Qwen3.8-27B | rk8v4 |
278,528 | 3/3 | 166 s | 36.8 tok/s |
| RTX 3090 | Ternary Bonsai 2 27B | rk4v4 |
970,752 | 3/3 | 2,359 s | 17.7 tok/s |
| RTX 3090 | Ternary Bonsai 2 27B | rk8v4 |
663,552 | 3/3 | 1,137 s | 24.4 tok/s |
| RTX 3090 | Qwen3.8-27B | rk4v4 |
417,792 | 3/3 | 644 s | 23.8 tok/s |
| RTX 3090 | Qwen3.8-27B | rk8v4 |
286,720 | 3/3 | 353 s | 31.0 tok/s |
At 1,048,576 tokens the RTX 5090 finds 2 of the three codes with --rope-yarn and 2 with plain RoPE (TTFT 937 and 939 s, decode 56.2 tok/s), missing the same code, the one at 90%, both times. The largest window that returned all three on that card is 978,944 tokens with rk8v4, whose last code sits near 881K. On the RTX PRO 6000 both models miss that code at 1,048,576 tokens too.
Decode tokens/s with the draft acceptance in brackets, rk8v4, one request: short chat and a ~400-word answer after 32,768 and 131,072 tokens of document.
262,144-token window.
| drafter | drafts | short chat | 32,768 | 131,072 |
|---|---|---|---|---|
| none | - | 171.6 | 155.2 | 124.5 |
| MTP | 1 | 265.2 (0.74) | 245.6 (0.78) | 190.6 (0.71) |
| MTP | 2 | 312.1 (0.63) | 283.4 (0.63) | 198.7 (0.48) |
| MTP | 3 | 320.0 (0.50) | 295.6 (0.52) | 207.3 (0.40) |
| MTP | 4 | 312.4 (0.41) | 228.8 (0.28) | 178.8 (0.26) |
| MTP | 5 | 321.5 (0.38) | 288.3 (0.38) | 188.0 (0.27) |
| MTP | 6 | 302.1 (0.33) | 233.1 (0.26) | 172.3 (0.22) |
| MTP | 7 | 282.6 (0.28) | 220.5 (0.22) | 169.8 (0.20) |
| MTP | 8 | 262.7 (0.25) | 229.4 (0.25) | 129.8 (0.16) |
| MTP | 10 | 232.2 (0.20) | 172.1 (0.15) | 111.4 (0.12) |
| MTP | 12 | 218.7 (0.18) | 188.8 (0.17) | 121.0 (0.13) |
| MTP | 15 | 181.1 (0.13) | 138.7 (0.10) | 104.0 (0.10) |
| DFlash2 | 1 | 230.9 (0.74) | 199.3 (0.63) | 171.8 (0.69) |
| DFlash2 | 2 | 281.8 (0.59) | 230.0 (0.46) | 194.8 (0.48) |
| DFlash2 | 3 | 323.4 (0.50) | 305.8 (0.53) | 192.9 (0.32) |
| DFlash2 | 4 | 345.6 (0.43) | 338.9 (0.48) | 216.0 (0.31) |
| DFlash2 | 5 | 358.7 (0.38) | 276.5 (0.29) | 184.3 (0.19) |
| DFlash2 | 6 | 385.0 (0.36) | 303.5 (0.29) | 216.8 (0.23) |
| DFlash2 | 7 | 396.7 (0.33) | 253.2 (0.19) | 200.6 (0.17) |
| DFlash2 | 8 | 372.4 (0.29) | 234.3 (0.17) | 156.1 (0.15) |
| DFlash2 | 10 | 349.7 (0.23) | 210.9 (0.13) | 152.0 (0.12) |
| DFlash2 | 12 | 341.7 (0.20) | 241.8 (0.14) | 156.2 (0.11) |
| DFlash2 | 15 | 310.0 (0.15) | 205.3 (0.10) | 144.1 (0.09) |
| drafter | drafts | short chat | 32,768 | 131,072 |
|---|---|---|---|---|
| none | - | 108.3 | 97.9 | 77.3 |
| MTP | 1 | 167.7 (0.74) | 139.7 (0.61) | 116.3 (0.68) |
| MTP | 2 | 191.1 (0.59) | 175.0 (0.60) | 128.3 (0.51) |
| MTP | 3 | 201.2 (0.50) | 175.2 (0.47) | 130.9 (0.41) |
| MTP | 4 | 198.9 (0.42) | 142.8 (0.28) | 119.0 (0.30) |
| MTP | 5 | 196.9 (0.37) | 153.4 (0.29) | 113.7 (0.26) |
| MTP | 6 | 187.6 (0.32) | 145.6 (0.25) | 109.3 (0.22) |
| MTP | 7 | 176.9 (0.28) | 135.1 (0.21) | 99.4 (0.18) |
| MTP | 8 | 171.3 (0.26) | 128.9 (0.20) | 86.6 (0.18) |
| MTP | 10 | 152.3 (0.20) | 127.0 (0.18) | 80.5 (0.15) |
| MTP | 12 | 137.7 (0.17) | 100.1 (0.12) | 73.4 (0.12) |
| MTP | 15 | 120.5 (0.13) | 90.5 (0.10) | 56.1 (0.07) |
| DFlash2 | 1 | 151.0 (0.72) | 140.3 (0.75) | 105.9 (0.61) |
| DFlash2 | 2 | 191.1 (0.60) | 162.4 (0.53) | 121.7 (0.43) |
| DFlash2 | 3 | 215.4 (0.50) | 205.6 (0.54) | 144.3 (0.41) |
| DFlash2 | 4 | 228.3 (0.42) | 220.4 (0.47) | 129.9 (0.26) |
| DFlash2 | 5 | 242.9 (0.38) | 171.8 (0.25) | 130.5 (0.23) |
| DFlash2 | 6 | 250.7 (0.34) | 183.1 (0.24) | 144.9 (0.24) |
| DFlash2 | 7 | 256.4 (0.31) | 178.7 (0.20) | 131.9 (0.18) |
| DFlash2 | 8 | 248.9 (0.28) | 160.0 (0.17) | 110.9 (0.17) |
| DFlash2 | 10 | 254.4 (0.24) | 178.9 (0.17) | 101.2 (0.12) |
| DFlash2 | 12 | 244.3 (0.19) | 158.5 (0.12) | 106.9 (0.11) |
| DFlash2 | 15 | 238.5 (0.15) | 163.4 (0.11) | 98.0 (0.08) |
| drafter | drafts | short chat | 32,768 | 131,072 |
|---|---|---|---|---|
| none | - | 92.5 | 82.5 | 63.4 |
| MTP | 1 | 139.5 (0.74) | 121.9 (0.69) | 96.9 (0.73) |
| MTP | 2 | 164.4 (0.63) | 123.1 (0.47) | 89.4 (0.46) |
| MTP | 3 | 171.4 (0.52) | 128.6 (0.40) | 93.6 (0.39) |
| MTP | 4 | 162.5 (0.42) | 128.2 (0.35) | 81.9 (0.26) |
| MTP | 5 | 167.1 (0.39) | 121.2 (0.30) | 72.4 (0.21) |
| MTP | 6 | 156.4 (0.33) | 115.1 (0.25) | 77.3 (0.22) |
| MTP | 7 | 145.9 (0.28) | 93.8 (0.17) | 65.9 (0.16) |
| MTP | 8 | 133.7 (0.25) | 98.0 (0.21) | 56.9 (0.15) |
| MTP | 10 | 118.6 (0.20) | 90.7 (0.18) | 59.1 (0.17) |
| MTP | 12 | 109.0 (0.18) | 85.1 (0.16) | 50.8 (0.13) |
| MTP | 15 | 95.2 (0.14) | 62.9 (0.10) | 37.8 (0.07) |
| DFlash2 | 1 | 126.3 (0.72) | 116.7 (0.75) | 88.1 (0.65) |
| DFlash2 | 2 | 160.7 (0.62) | 132.6 (0.55) | 90.0 (0.44) |
| DFlash2 | 3 | 184.6 (0.54) | 163.0 (0.54) | 97.2 (0.36) |
| DFlash2 | 4 | 189.5 (0.44) | 155.6 (0.39) | 96.6 (0.28) |
| DFlash2 | 5 | 191.3 (0.37) | 176.2 (0.41) | 93.6 (0.24) |
| DFlash2 | 6 | 198.7 (0.34) | 131.8 (0.22) | 89.7 (0.19) |
| DFlash2 | 7 | 202.0 (0.31) | 161.5 (0.28) | 85.5 (0.16) |
| DFlash2 | 8 | 188.6 (0.29) | 152.1 (0.28) | 65.0 (0.13) |
| DFlash2 | 10 | 173.6 (0.23) | 114.2 (0.17) | 60.1 (0.11) |
| DFlash2 | 12 | 166.7 (0.20) | 124.3 (0.17) | 61.5 (0.10) |
| DFlash2 | 15 | 149.1 (0.15) | 118.5 (0.15) | 55.8 (0.08) |
262,144-token window on the RTX 5090, 139,264 on the 24 GB cards (DFlash2 and MTP 15 fit there).
| drafter | drafts | short chat | 32,768 | 131,072 |
|---|---|---|---|---|
| none | - | 91.6 | 86.7 | 76.3 |
| MTP | 1 | 144.0 (0.81) | 141.7 (0.87) | 114.4 (0.71) |
| MTP | 2 | 173.6 (0.70) | 155.9 (0.63) | 127.9 (0.55) |
| MTP | 3 | 184.2 (0.59) | 181.1 (0.63) | 130.8 (0.44) |
| MTP | 4 | 183.2 (0.50) | 176.2 (0.52) | 134.2 (0.40) |
| MTP | 5 | 181.2 (0.44) | 183.3 (0.50) | 123.7 (0.32) |
| MTP | 6 | 179.3 (0.40) | 159.1 (0.37) | 118.6 (0.28) |
| MTP | 7 | 168.0 (0.34) | 158.8 (0.34) | 109.4 (0.23) |
| MTP | 8 | 148.5 (0.30) | 139.5 (0.30) | 92.3 (0.21) |
| MTP | 10 | 144.0 (0.27) | 112.8 (0.21) | 84.7 (0.17) |
| MTP | 12 | 134.2 (0.23) | 95.1 (0.16) | 83.1 (0.16) |
| MTP | 15 | 125.8 (0.20) | 106.4 (0.17) | 71.1 (0.12) |
| DFlash2 | 1 | 127.7 (0.77) | 126.6 (0.84) | 108.1 (0.75) |
| DFlash2 | 2 | 164.3 (0.65) | 169.0 (0.75) | 127.6 (0.54) |
| DFlash2 | 3 | 192.4 (0.57) | 202.5 (0.67) | 135.7 (0.41) |
| DFlash2 | 4 | 213.7 (0.52) | 198.1 (0.50) | 164.0 (0.44) |
| DFlash2 | 5 | 218.1 (0.44) | 217.9 (0.48) | 147.9 (0.31) |
| DFlash2 | 6 | 222.2 (0.38) | 226.0 (0.43) | 143.3 (0.25) |
| DFlash2 | 7 | 236.2 (0.36) | 217.6 (0.35) | 145.0 (0.23) |
| DFlash2 | 8 | 215.0 (0.32) | 202.2 (0.33) | 121.7 (0.20) |
| DFlash2 | 10 | 224.3 (0.28) | 195.0 (0.26) | 129.8 (0.19) |
| DFlash2 | 12 | 226.1 (0.24) | 152.5 (0.16) | 122.2 (0.15) |
| DFlash2 | 15 | 220.6 (0.19) | 181.6 (0.17) | 125.4 (0.13) |
| drafter | drafts | short chat | 32,768 | 131,072 |
|---|---|---|---|---|
| none | - | 54.8 | 52.0 | 45.6 |
| MTP | 1 | 87.8 (0.79) | 84.1 (0.81) | 67.9 (0.66) |
| MTP | 2 | 104.3 (0.68) | 94.0 (0.62) | 75.4 (0.51) |
| MTP | 3 | 110.6 (0.58) | 114.7 (0.66) | 80.3 (0.45) |
| MTP | 4 | 113.5 (0.52) | 114.4 (0.56) | 78.9 (0.38) |
| MTP | 5 | 111.0 (0.44) | 109.4 (0.47) | 69.7 (0.28) |
| MTP | 6 | 109.6 (0.41) | 107.1 (0.43) | 72.8 (0.29) |
| MTP | 7 | 106.8 (0.36) | 102.8 (0.37) | 64.3 (0.22) |
| MTP | 8 | 90.4 (0.31) | 93.4 (0.35) | 56.1 (0.22) |
| MTP | 10 | 87.0 (0.27) | 82.5 (0.27) | 47.5 (0.15) |
| MTP | 12 | 77.3 (0.22) | 77.0 (0.24) | 50.4 (0.16) |
| MTP | 15 | 71.5 (0.19) | 65.8 (0.18) | 41.3 (0.11) |
| DFlash2 | 1 | 79.4 (0.77) | 78.6 (0.84) | 66.3 (0.73) |
| DFlash2 | 2 | 101.6 (0.65) | 105.7 (0.76) | 79.2 (0.55) |
| DFlash2 | 3 | 120.6 (0.58) | 120.6 (0.63) | 88.7 (0.45) |
| DFlash2 | 4 | 128.1 (0.49) | 127.7 (0.52) | 90.8 (0.36) |
| DFlash2 | 5 | 137.1 (0.43) | 138.7 (0.47) | 90.8 (0.30) |
| DFlash2 | 6 | 142.7 (0.39) | 139.1 (0.41) | 98.6 (0.29) |
| DFlash2 | 7 | 148.8 (0.35) | 151.9 (0.39) | 102.3 (0.27) |
| DFlash2 | 8 | 137.7 (0.34) | 121.5 (0.31) | 80.7 (0.23) |
| DFlash2 | 10 | 135.4 (0.27) | 116.8 (0.24) | 83.6 (0.20) |
| DFlash2 | 12 | 138.1 (0.23) | 115.5 (0.20) | 78.0 (0.15) |
| DFlash2 | 15 | 137.4 (0.19) | 114.5 (0.16) | 75.0 (0.12) |
| drafter | drafts | short chat | 32,768 | 131,072 |
|---|---|---|---|---|
| none | - | 47.9 | 44.8 | 38.1 |
| MTP | 1 | 74.2 (0.79) | 72.6 (0.87) | 57.2 (0.72) |
| MTP | 2 | 91.7 (0.70) | 85.4 (0.71) | 63.8 (0.59) |
| MTP | 3 | 95.9 (0.58) | 93.2 (0.63) | 68.5 (0.51) |
| MTP | 4 | 93.0 (0.49) | 88.1 (0.50) | 65.4 (0.42) |
| MTP | 5 | 93.7 (0.44) | 87.7 (0.47) | 57.0 (0.32) |
| MTP | 6 | 91.7 (0.40) | 84.4 (0.41) | 55.7 (0.29) |
| MTP | 7 | 85.1 (0.35) | 77.0 (0.34) | 51.3 (0.24) |
| MTP | 8 | 76.6 (0.31) | 70.5 (0.33) | 42.7 (0.21) |
| MTP | 10 | 71.6 (0.27) | 46.5 (0.16) | 38.3 (0.17) |
| MTP | 12 | 67.5 (0.23) | 58.5 (0.22) | 35.4 (0.14) |
| MTP | 15 | 56.3 (0.17) | 52.1 (0.18) | 28.2 (0.10) |
| DFlash2 | 1 | 68.1 (0.79) | 66.2 (0.84) | 54.9 (0.75) |
| DFlash2 | 2 | 89.4 (0.67) | 83.0 (0.67) | 64.3 (0.57) |
| DFlash2 | 3 | 103.4 (0.58) | 102.3 (0.65) | 74.9 (0.52) |
| DFlash2 | 4 | 108.0 (0.50) | 103.9 (0.53) | 79.0 (0.45) |
| DFlash2 | 5 | 115.0 (0.45) | 107.6 (0.47) | 74.1 (0.36) |
| DFlash2 | 6 | 115.0 (0.39) | 109.3 (0.41) | 63.5 (0.24) |
| DFlash2 | 7 | 117.5 (0.37) | 107.6 (0.37) | 61.6 (0.21) |
| DFlash2 | 8 | 108.8 (0.33) | 101.1 (0.35) | 52.3 (0.19) |
| DFlash2 | 10 | 104.5 (0.26) | 88.1 (0.25) | 51.0 (0.16) |
| DFlash2 | 12 | 105.5 (0.23) | 87.3 (0.21) | 47.1 (0.12) |
| DFlash2 | 15 | 99.9 (0.19) | 79.0 (0.16) | 42.5 (0.10) |
Total generated tokens per second of wall time when N requests arrive together: the short-chat prompts in turn (the essay alone at 1, the essay and the Python module at 2, and so on), 512 tokens each, rk8v4, --max-concurrency 8 --max-context 16384 --kv-capacity 131072. Speculation pays less as the batch grows, since the verify widths add up to more of the card's compute. Batched answers keep their quality but can part from a lone request's at near-tied tokens (batch composition). A dash at 8: eight lanes of 16,384 tokens do not fit beside that model and drafter on the card, and the row ran with four:
| card | model | speculation | 1 at once | 2 at once | 4 at once | 8 at once |
|---|---|---|---|---|---|---|
| RTX PRO 6000 | Ternary Bonsai 2 27B | none | 165.5 | 321.4 | 599.1 | 1,043.5 |
| RTX PRO 6000 | Ternary Bonsai 2 27B | MTP, 3 drafts | 274.4 | 516.0 | 805.0 | 1,155.3 |
| RTX PRO 6000 | Ternary Bonsai 2 27B | DFlash2, 5 drafts | 270.6 | 523.9 | 735.5 | 890.4 |
| RTX PRO 6000 | Qwen3.8-27B | none | 88.8 | 172.6 | 327.9 | 569.8 |
| RTX PRO 6000 | Qwen3.8-27B | MTP, 3 drafts | 154.4 | 300.9 | 515.8 | 739.3 |
| RTX PRO 6000 | Qwen3.8-27B | DFlash2, 5 drafts | 176.1 | 319.4 | 473.7 | 534.6 |
| RTX 5090 | Ternary Bonsai 2 27B | none | 169.8 | 327.4 | 608.6 | 1,053.3 |
| RTX 5090 | Ternary Bonsai 2 27B | MTP, 3 drafts | 263.3 | 511.3 | 771.1 | 1,063.0 |
| RTX 5090 | Ternary Bonsai 2 27B | DFlash2, 5 drafts | 278.1 | 511.7 | 631.9 | 799.1 |
| RTX 5090 | Qwen3.8-27B | none | 91.0 | 172.0 | 324.7 | 569.1 |
| RTX 5090 | Qwen3.8-27B | MTP, 3 drafts | 160.6 | 299.0 | 490.0 | 690.0 |
| RTX 5090 | Qwen3.8-27B | DFlash2, 5 drafts | 171.6 | 311.1 | 430.9 | 469.4 |
| RTX 4090 | Ternary Bonsai 2 27B | none | 107.4 | 206.9 | 383.5 | 670.6 |
| RTX 4090 | Ternary Bonsai 2 27B | MTP, 3 drafts | 182.1 | 318.3 | 533.8 | 824.1 |
| RTX 4090 | Ternary Bonsai 2 27B | DFlash2, 5 drafts | 197.1 | 354.2 | 504.9 | 673.5 |
| RTX 4090 | Qwen3.8-27B | none | 54.3 | 105.6 | 198.4 | 356.7 |
| RTX 4090 | Qwen3.8-27B | MTP, 3 drafts | 92.8 | 174.9 | 300.3 | 442.2 |
| RTX 4090 | Qwen3.8-27B | DFlash2, 5 drafts | 107.7 | 185.1 | 295.2 | - |
| RTX 3090 | Ternary Bonsai 2 27B | none | 93.1 | 176.9 | 324.0 | 546.4 |
| RTX 3090 | Ternary Bonsai 2 27B | MTP, 3 drafts | 141.7 | 267.3 | 413.7 | 550.8 |
| RTX 3090 | Ternary Bonsai 2 27B | DFlash2, 5 drafts | 154.7 | 256.0 | 334.1 | 386.2 |
| RTX 3090 | Qwen3.8-27B | none | 48.1 | 91.9 | 172.2 | 290.2 |
| RTX 3090 | Qwen3.8-27B | MTP, 3 drafts | 88.2 | 154.8 | 238.4 | 328.9 |
| RTX 3090 | Qwen3.8-27B | DFlash2, 5 drafts | 89.1 | 155.4 | 206.0 | - |
Routes each card's calibration moved off the compiled tables, with the largest speedup of the operation (its time on the compiled route over its time on the chosen one, not end to end):
| route | RTX 3090 | RTX 4090 | RTX 5090 | RTX PRO 6000 | PRO 6000 Max-Q | PRO 6000 Server |
|---|---|---|---|---|---|---|
rk8v4 decode/verify attention |
query widths 2-5: up to 1.16× | query widths 2-8: up to 1.16× | query widths 3-8: up to 1.08× | query widths 3-4, 6-8: up to 1.08× | - | query widths 6-8: up to 1.06× |
rk4v4 decode/verify attention |
query widths 1-5: up to 3.20× | query widths 1, 3-7: up to 1.46× | query widths 1-8: up to 1.13× | query widths 1-8: up to 1.19× | query widths 1, 3-8: up to 1.11× | query widths 1, 3-8: up to 1.16× |
| fast prompt kernel | 1.41× | 1.43× | 1.24× | 1.20× | 1.19× | 1.23× |
| FP16 PV accumulation | 1.17× | 1.20× | 1.19× | - | - | - |
| ternary small-T projections | 12 widths, up to 1.09× | 15 widths, up to 1.27× | 34 widths, up to 1.38× | 31 widths, up to 1.28× | 33 widths, up to 1.33× | 33 widths, up to 1.16× |
| groupwise Q4/Q5 projections | - | 13 widths, up to 1.26× | 9 widths, up to 1.14× | 11 widths, up to 1.23× | 12 widths, up to 1.24× | 7 widths, up to 1.25× |
Hardware. Rented single-GPU hosts, each GPU at its board's default power limit. RTX 3090: 390 W
for the Ternary Bonsai 2 rows, their comparison with master and several requests at once; two
other boards at 370 W for the Qwen3.8 rows and their comparison with master, the draft sweeps,
the largest context and the fills. RTX 4090: 450 W (the largest-context bisection ran on a second
board capped at 400 W; it does not depend on power). RTX 5090: 575 W, on two hosts. RTX PRO 6000:
the Workstation Edition at 600 W; the Max-Q (300 W) and Server (600 W) editions were calibrated for
their built-in profiles and not measured here. The server runs pinned to the CPUs of the GPU's NUMA
node.
Software. This repository at the commit that ships the built-in profiles, built for the card
(sm_86, sm_89, sm_120a on the mma.sync path), CUDA 13.1. Artifacts:
Ternary-Bonsai-2-27B-NInfer-v3 and
Qwen3.8-27B-NInfer (groupwise-int).
Server. ninfer-serve <artifact> --max-context W --kv-capacity W --max-concurrency 1 --kv-dtype KV --gdn-state-fp16, plus --spec dflash2 --draft-tokens 5 or --spec mtp --draft-tokens 3, plus
--rope-yarn past 262,144 tokens. Everything else is at its default, so prefix reuse is on but
never hit: every request starts with a fresh nonce.
Requests (tools/bench/refbench.py), greedy, thinking off,
non-streaming, against the OpenAI Chat Completions endpoint:
- Short chat: five prompts (an essay, a Python module with tests, a word problem, a Russian explanation, a list of forty items), 512 generated tokens each; the table gives the mean decode rate.
- Depth: a cold document of the given length (PG-19, WikiText and this repository's text, cut by tokens), then the request to retell its final part in about 400 words, at most 512 tokens. TTFT is the server's prompt time; decode is the rest of the answer over its decode time.
- Needles: three codes planted at 33, 66 and 90% of a document 1,024 tokens short of the window, asked back in order.
VRAM is the device memory the server holds when idle: NInfer reserves the whole KV cache, workspaces and graphs at startup, so it does not grow under load.
Largest context. --max-context = --kv-capacity bisected in 4,096-token steps on whether the
server starts; the engine plans every byte at startup, and the filled runs confirm that the found
window also serves one request that fills it.
Draft length. The same server with --draft-tokens from 1 to 15; short chat and the depth
requests at 32,768 and 131,072 tokens.
Several requests at once. --max-concurrency 8 --max-context 16384 --kv-capacity 131072; N
short-chat requests sent together, 512 tokens each, and their total generated tokens over the
wall time from the first request to the last answer.
Against the previous master. The public master at 8a4c2f18, built the same way on the same
host and served with the same flags; it has no device profiles, no fast prompt route and no
wave-aligned prefill chunks, and MTP with ten or more drafts fails there at startup.