reference-2026-09.md

Reference measurements, September 2026

Ternary Bonsai 2 27B and Qwen3.8-27B on one RTX 3090, RTX 4090, RTX 5090 and RTX PRO 6000 each, at the models' full 262,144-token window where the card holds it, with this line's defaults: the device route profile of each card (built in), --gdn-state-fp16, text only, one request at a time unless a section says otherwise.

RTX 5090

Ternary Bonsai 2 27B

Window 262,144 tokens, one request.

Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):

KV	speculation	VRAM	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	12.4 GiB	171.4	170.8	165.9	159.0	150.2	134.6	122.3	112.4
rk4v4	DFlash2, 5 drafts	13.8 GiB	366.2	408.8	292.7	325.6	259.2	203.3	179.4	168.4
rk4v4	MTP, 3 drafts	13.1 GiB	321.1	268.3	279.3	291.5	228.9	217.2	203.0	217.6
rk4v4-e8	none	12.4 GiB	170.4	169.9	166.4	158.2	149.4	134.1	122.1	112.1
rk4v4-e8	DFlash2, 5 drafts	13.8 GiB	387.9	315.1	332.1	301.1	241.6	219.2	219.9	197.8
rk8v4	none	14.4 GiB	171.5	171.0	165.8	155.4	143.8	124.7	110.7	99.4
rk8v4	DFlash2, 5 drafts	15.8 GiB	368.2	336.1	305.9	343.2	248.9	215.2	238.9	189.4
rk8v4	MTP, 3 drafts	15.2 GiB	329.0	336.5	299.9	306.0	244.5	207.1	198.7	207.4

Time to first token, s (cold prompt of that length):

KV	speculation	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	0.17	1.17	5.2	12.0	30.2	54.9	85.2
rk4v4	DFlash2, 5 drafts	0.18	1.20	5.3	12.1	30.6	55.4	85.9
rk4v4	MTP, 3 drafts	0.18	1.18	5.2	12.0	30.4	55.0	85.5
rk4v4-e8	none	0.17	1.17	5.1	11.9	30.1	54.3	84.2
rk4v4-e8	DFlash2, 5 drafts	0.18	1.19	5.2	12.1	30.4	54.9	85.0
rk8v4	none	0.17	1.14	5.0	11.5	29.0	52.7	81.9
rk8v4	DFlash2, 5 drafts	0.18	1.19	5.2	11.9	29.8	53.5	82.7
rk8v4	MTP, 3 drafts	0.18	1.18	5.1	11.8	29.6	53.2	82.2

Draft acceptance (accepted / drafted; tokens per verify round in brackets):

KV	speculation	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	DFlash2, 5 drafts	0.39 (2.92)	0.46 (3.28)	0.28 (2.40)	0.38 (2.89)	0.29 (2.46)	0.24 (2.20)	0.24 (2.19)	0.26 (2.28)
rk4v4	MTP, 3 drafts	0.50 (2.49)	0.37 (2.12)	0.43 (2.29)	0.50 (2.48)	0.36 (2.06)	0.40 (2.19)	0.43 (2.27)	0.56 (2.67)
rk4v4-e8	DFlash2, 5 drafts	0.43 (3.10)	0.31 (2.53)	0.35 (2.73)	0.34 (2.67)	0.26 (2.29)	0.28 (2.38)	0.34 (2.67)	0.34 (2.67)
rk8v4	DFlash2, 5 drafts	0.39 (2.93)	0.34 (2.69)	0.31 (2.54)	0.41 (3.02)	0.27 (2.35)	0.26 (2.27)	0.36 (2.81)	0.29 (2.46)
rk8v4	MTP, 3 drafts	0.52 (2.55)	0.55 (2.63)	0.47 (2.40)	0.55 (2.65)	0.43 (2.29)	0.40 (2.21)	0.45 (2.36)	0.57 (2.71)

Needles (three codes at 33, 66 and 90% of a document):

KV	speculation	document	found	order	TTFT s
rk4v4	none	261,019	3/3	yes	85.3
rk4v4-e8	none	261,022	3/3	yes	84.2
rk8v4	none	261,021	3/3	yes	82.0

Qwen3.8-27B

Window 262,144 tokens, one request.

Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):

KV	speculation	VRAM	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	21.6 GiB	91.9	91.7	90.2	88.1	85.3	80.0	75.4	71.6
rk4v4	DFlash2, 5 drafts	23.8 GiB	214.9	193.9	197.2	204.7	151.8	129.7	136.0	116.1
rk4v4	MTP, 3 drafts	22.3 GiB	182.7	153.4	158.9	171.8	165.7	140.8	132.6	129.0
rk4v4-e8	none	21.6 GiB	91.5	91.3	90.2	87.8	84.9	79.7	75.2	71.3
rk4v4-e8	DFlash2, 5 drafts	23.8 GiB	220.3	165.0	176.0	202.2	147.0	147.9	128.7	128.8
rk8v4	none	23.6 GiB	92.0	91.7	90.2	87.1	83.3	76.5	71.0	66.2
rk8v4	DFlash2, 5 drafts	25.8 GiB	221.0	186.7	204.6	205.9	157.8	152.7	134.0	129.8
rk8v4	MTP, 3 drafts	24.4 GiB	184.7	153.7	169.8	180.1	140.5	130.3	130.5	114.0

Time to first token, s (cold prompt of that length):

KV	speculation	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	0.34	2.04	8.5	18.4	43.0	73.8	110.4
rk4v4	DFlash2, 5 drafts	0.34	2.06	8.6	18.6	43.4	74.4	111.6
rk4v4	MTP, 3 drafts	0.34	2.07	8.6	18.7	43.6	74.8	111.7
rk4v4-e8	none	0.34	2.05	8.5	18.5	43.0	73.7	109.8
rk4v4-e8	DFlash2, 5 drafts	0.34	2.07	8.6	18.7	43.4	74.1	110.4
rk8v4	none	0.33	2.03	8.4	18.2	42.1	71.9	107.0
rk8v4	DFlash2, 5 drafts	0.34	2.05	8.5	18.3	42.4	72.4	107.7
rk8v4	MTP, 3 drafts	0.33	2.03	8.4	18.2	42.2	72.0	107.1

Draft acceptance (accepted / drafted; tokens per verify round in brackets):

KV	speculation	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	DFlash2, 5 drafts	0.42 (3.10)	0.37 (2.81)	0.38 (2.91)	0.43 (3.17)	0.29 (2.44)	0.25 (2.26)	0.31 (2.55)	0.27 (2.34)
rk4v4	MTP, 3 drafts	0.58 (2.71)	0.44 (2.31)	0.48 (2.43)	0.57 (2.69)	0.57 (2.68)	0.48 (2.44)	0.48 (2.44)	0.51 (2.51)
rk4v4-e8	DFlash2, 5 drafts	0.44 (3.19)	0.28 (2.40)	0.32 (2.61)	0.43 (3.15)	0.28 (2.37)	0.32 (2.58)	0.29 (2.42)	0.32 (2.58)
rk8v4	DFlash2, 5 drafts	0.44 (3.19)	0.34 (2.71)	0.41 (3.05)	0.43 (3.15)	0.31 (2.53)	0.33 (2.63)	0.30 (2.46)	0.31 (2.56)
rk8v4	MTP, 3 drafts	0.59 (2.75)	0.44 (2.32)	0.53 (2.57)	0.61 (2.81)	0.44 (2.30)	0.44 (2.30)	0.49 (2.46)	0.44 (2.32)

Needles (three codes at 33, 66 and 90% of a document):

KV	speculation	document	found	order	TTFT s
rk4v4	none	261,020	3/3	yes	110.4
rk4v4-e8	none	261,022	3/3	yes	109.7
rk8v4	none	261,017	3/3	yes	107.0

RTX 4090

Ternary Bonsai 2 27B

Window 262,144 tokens, one request.

Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):

KV	speculation	VRAM	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	12.3 GiB	107.9	107.6	105.1	99.3	93.3	83.0	75.1	63.8
rk4v4	DFlash2, 5 drafts	13.8 GiB	242.8	179.5	181.8	236.3	156.9	134.9	140.4	115.3
rk4v4	MTP, 3 drafts	13.0 GiB	206.9	158.6	179.9	180.7	142.6	134.7	131.6	116.3
rk4v4-e8	none	12.3 GiB	107.9	107.6	105.3	99.3	93.3	83.0	75.1	63.8
rk4v4-e8	DFlash2, 5 drafts	13.8 GiB	251.5	198.5	212.9	197.2	146.2	129.0	155.6	116.4
rk8v4	none	14.3 GiB	108.4	107.8	105.2	97.8	89.9	77.3	67.9	56.4
rk8v4	DFlash2, 5 drafts	15.8 GiB	244.8	243.6	206.7	180.0	160.9	125.8	118.7	122.9
rk8v4	MTP, 3 drafts	15.1 GiB	203.4	191.4	184.0	164.8	153.7	127.7	121.8	111.0

Time to first token, s (cold prompt of that length):

KV	speculation	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	0.21	1.44	6.4	14.7	37.2	67.6	105.2
rk4v4	DFlash2, 5 drafts	0.21	1.47	6.5	15.0	37.7	68.4	106.2
rk4v4	MTP, 3 drafts	0.21	1.45	6.4	14.8	37.4	67.9	105.5
rk4v4-e8	none	0.21	1.44	6.3	14.6	36.8	66.7	103.5
rk4v4-e8	DFlash2, 5 drafts	0.22	1.47	6.5	14.8	37.3	67.6	104.6
rk8v4	none	0.20	1.42	6.3	14.4	36.2	65.8	102.1
rk8v4	DFlash2, 5 drafts	0.22	1.47	6.4	14.7	36.9	66.8	103.3
rk8v4	MTP, 3 drafts	0.21	1.44	6.3	14.5	36.6	66.3	102.6

Draft acceptance (accepted / drafted; tokens per verify round in brackets):

KV	speculation	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	DFlash2, 5 drafts	0.39 (2.91)	0.24 (2.17)	0.25 (2.26)	0.43 (3.11)	0.25 (2.23)	0.25 (2.22)	0.32 (2.60)	0.30 (2.51)
rk4v4	MTP, 3 drafts	0.53 (2.57)	0.34 (2.01)	0.44 (2.31)	0.49 (2.49)	0.36 (2.09)	0.41 (2.21)	0.47 (2.40)	0.53 (2.59)
rk4v4-e8	DFlash2, 5 drafts	0.41 (3.01)	0.28 (2.40)	0.33 (2.63)	0.32 (2.58)	0.21 (2.07)	0.23 (2.12)	0.37 (2.85)	0.30 (2.51)
rk8v4	DFlash2, 5 drafts	0.39 (2.93)	0.39 (2.94)	0.31 (2.56)	0.27 (2.36)	0.26 (2.27)	0.21 (2.06)	0.24 (2.19)	0.33 (2.63)
rk8v4	MTP, 3 drafts	0.51 (2.51)	0.46 (2.38)	0.45 (2.34)	0.42 (2.24)	0.43 (2.28)	0.39 (2.18)	0.44 (2.32)	0.55 (2.64)

Needles (three codes at 33, 66 and 90% of a document):

KV	speculation	document	found	order	TTFT s
rk4v4	none	261,015	3/3	yes	105.2
rk4v4-e8	none	261,018	3/3	yes	103.5
rk8v4	none	261,022	3/3	yes	102.2

Qwen3.8-27B

One request. Each KV coding runs at the largest window DFlash2 fits beside it on this card, at most 262,144, for every drafter; rk4v4-e8 without speculation runs the full window.

Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):

KV	speculation	window	VRAM	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	245,760	21.3 GiB	54.7	54.6	53.9	52.4	50.7	47.5	44.8	-
rk4v4	DFlash2, 5 drafts	245,760	23.5 GiB	137.8	108.4	115.3	142.7	93.8	86.4	80.4	-
rk4v4	MTP, 3 drafts	245,760	21.9 GiB	108.9	93.5	102.0	105.3	92.8	85.0	83.3	-
rk4v4-e8	none	262,144	21.5 GiB	54.7	54.6	54.0	52.4	50.7	47.5	44.8	40.6
rk8v4	none	167,936	21.2 GiB	54.8	54.7	54.0	52.0	49.7	45.6	-	-
rk8v4	DFlash2, 5 drafts	167,936	23.5 GiB	138.1	127.1	129.8	140.3	102.2	90.9	-	-
rk8v4	MTP, 3 drafts	167,936	21.9 GiB	109.0	102.6	101.7	108.5	94.9	77.3	-	-

Time to first token, s (cold prompt of that length):

KV	speculation	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	0.56	3.24	13.1	27.8	63.0	106.0	-
rk4v4	DFlash2, 5 drafts	0.57	3.27	13.2	28.1	63.5	106.7	-
rk4v4	MTP, 3 drafts	0.57	3.25	13.1	27.9	63.2	106.2	-
rk4v4-e8	none	0.56	3.23	13.1	27.7	62.6	105.0	154.1
rk8v4	none	0.56	3.23	13.0	27.6	62.2	-	-
rk8v4	DFlash2, 5 drafts	0.57	3.26	13.2	27.8	62.7	-	-
rk8v4	MTP, 3 drafts	0.57	3.24	13.1	27.6	62.3	-	-

Draft acceptance (accepted / drafted; tokens per verify round in brackets):

KV	speculation	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	DFlash2, 5 drafts	0.44 (3.15)	0.30 (2.51)	0.34 (2.71)	0.49 (3.47)	0.27 (2.36)	0.28 (2.37)	0.28 (2.37)	-
rk4v4	MTP, 3 drafts	0.57 (2.69)	0.45 (2.36)	0.53 (2.62)	0.59 (2.79)	0.50 (2.48)	0.48 (2.45)	0.52 (2.56)	-
rk8v4	DFlash2, 5 drafts	0.44 (3.16)	0.39 (2.95)	0.41 (3.05)	0.48 (3.40)	0.32 (2.56)	0.30 (2.48)	-	-
rk8v4	MTP, 3 drafts	0.57 (2.68)	0.52 (2.57)	0.52 (2.55)	0.61 (2.80)	0.53 (2.56)	0.42 (2.29)	-	-

Needles (three codes at 33, 66 and 90% of a document):

KV	speculation	document	found	order	TTFT s
rk4v4	none	244,635	3/3	yes	142.3
rk4v4-e8	none	261,017	3/3	yes	154.1
rk8v4	none	166,815	3/3	yes	84.1

RTX 3090

Ternary Bonsai 2 27B

Window 262,144 tokens, one request.

Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):

KV	speculation	VRAM	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	12.2 GiB	93.8	93.2	90.3	83.7	76.1	64.4	55.5	49.2
rk4v4	DFlash2, 5 drafts	13.7 GiB	200.8	172.3	180.3	174.7	116.5	97.5	73.5	78.6
rk4v4	MTP, 3 drafts	12.9 GiB	173.4	145.1	138.3	149.7	116.9	96.5	89.4	83.7
rk4v4-e8	none	12.2 GiB	93.6	93.2	90.4	84.0	76.5	64.9	56.0	49.8
rk4v4-e8	DFlash2, 5 drafts	13.7 GiB	202.8	157.2	173.3	140.0	119.4	109.2	113.9	76.3
rk8v4	none	14.2 GiB	94.1	93.5	90.8	84.3	76.8	65.3	56.3	50.0
rk8v4	DFlash2, 5 drafts	15.7 GiB	201.4	198.4	161.1	135.4	124.9	92.6	85.3	89.7
rk8v4	MTP, 3 drafts	15.0 GiB	172.2	143.6	184.8	150.4	109.4	93.7	94.7	89.3

Time to first token, s (cold prompt of that length):

KV	speculation	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	0.42	2.99	13.3	30.7	77.9	142.0	221.1
rk4v4	DFlash2, 5 drafts	0.43	3.06	13.5	31.2	79.2	143.8	223.7
rk4v4	MTP, 3 drafts	0.42	3.00	13.3	30.8	78.3	142.7	222.0
rk4v4-e8	none	0.42	2.98	13.2	30.4	77.1	140.2	218.2
rk4v4-e8	DFlash2, 5 drafts	0.43	3.06	13.5	31.0	78.3	141.8	220.5
rk8v4	none	0.42	2.97	13.1	30.2	76.2	138.3	214.5
rk8v4	DFlash2, 5 drafts	0.43	3.05	13.4	30.8	77.4	139.8	216.9
rk8v4	MTP, 3 drafts	0.42	2.99	13.2	30.3	76.5	138.8	215.4

Draft acceptance (accepted / drafted; tokens per verify round in brackets):

KV	speculation	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	DFlash2, 5 drafts	0.39 (2.91)	0.31 (2.52)	0.36 (2.78)	0.40 (3.03)	0.25 (2.26)	0.27 (2.34)	0.23 (2.13)	0.33 (2.63)
rk4v4	MTP, 3 drafts	0.52 (2.54)	0.39 (2.17)	0.38 (2.15)	0.51 (2.53)	0.41 (2.22)	0.41 (2.23)	0.50 (2.47)	0.56 (2.65)
rk4v4-e8	DFlash2, 5 drafts	0.39 (2.94)	0.26 (2.30)	0.34 (2.67)	0.27 (2.34)	0.25 (2.22)	0.29 (2.44)	0.41 (3.07)	0.27 (2.33)
rk8v4	DFlash2, 5 drafts	0.39 (2.92)	0.39 (2.91)	0.30 (2.46)	0.26 (2.27)	0.27 (2.35)	0.22 (2.12)	0.27 (2.33)	0.36 (2.80)
rk8v4	MTP, 3 drafts	0.51 (2.51)	0.37 (2.12)	0.62 (2.90)	0.50 (2.49)	0.34 (2.04)	0.37 (2.10)	0.50 (2.51)	0.55 (2.65)

Needles (three codes at 33, 66 and 90% of a document):

KV	speculation	document	found	order	TTFT s
rk4v4	none	261,020	3/3	yes	221.4
rk4v4-e8	none	261,020	3/3	yes	218.1
rk8v4	none	261,019	3/3	yes	214.8

Qwen3.8-27B

One request. Each KV coding runs at the largest window DFlash2 fits beside it on this card, at most 262,144, for every drafter; rk4v4-e8 without speculation runs the full window.

Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):

KV	speculation	window	VRAM	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	258,048	21.3 GiB	48.0	47.9	47.1	45.0	42.3	38.0	34.4	-
rk4v4	DFlash2, 5 drafts	258,048	23.6 GiB	115.9	88.5	86.1	93.2	70.6	60.5	54.1	-
rk4v4	MTP, 3 drafts	258,048	22.0 GiB	95.9	79.0	88.1	89.4	77.5	66.3	61.5	-
rk4v4-e8	none	262,144	21.4 GiB	48.0	47.9	47.1	45.2	42.5	38.3	34.7	32.0
rk8v4	none	176,128	21.3 GiB	48.2	48.0	47.2	45.1	42.6	38.4	-	-
rk8v4	DFlash2, 5 drafts	176,128	23.5 GiB	117.1	101.4	101.3	110.8	78.7	72.6	-	-
rk8v4	MTP, 3 drafts	176,128	22.0 GiB	97.2	90.6	90.5	83.6	83.9	60.9	-	-

Time to first token, s (cold prompt of that length):

KV	speculation	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	1.15	5.63	23.2	50.2	118.3	203.4	-
rk4v4	DFlash2, 5 drafts	1.17	5.73	23.6	51.0	119.4	205.2	-
rk4v4	MTP, 3 drafts	1.17	5.68	23.3	50.3	118.6	203.9	-
rk4v4-e8	none	1.15	5.62	23.1	50.0	117.3	201.4	300.9
rk8v4	none	1.14	5.61	23.1	49.9	116.4	-	-
rk8v4	DFlash2, 5 drafts	1.17	5.73	23.4	50.6	117.7	-	-
rk8v4	MTP, 3 drafts	1.16	5.65	23.1	49.9	117.0	-	-

Draft acceptance (accepted / drafted; tokens per verify round in brackets):

KV	speculation	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	DFlash2, 5 drafts	0.45 (3.24)	0.30 (2.51)	0.30 (2.51)	0.38 (2.86)	0.27 (2.35)	0.26 (2.31)	0.27 (2.35)	-
rk4v4	MTP, 3 drafts	0.58 (2.73)	0.42 (2.26)	0.53 (2.60)	0.59 (2.75)	0.53 (2.57)	0.50 (2.50)	0.54 (2.61)	-
rk8v4	DFlash2, 5 drafts	0.46 (3.27)	0.38 (2.86)	0.39 (2.92)	0.48 (3.39)	0.32 (2.57)	0.34 (2.68)	-	-
rk8v4	MTP, 3 drafts	0.59 (2.76)	0.53 (2.59)	0.55 (2.65)	0.52 (2.55)	0.58 (2.74)	0.42 (2.25)	-	-

Needles (three codes at 33, 66 and 90% of a document):

KV	speculation	document	found	order	TTFT s
rk4v4	none	256,924	3/3	yes	297.5
rk4v4-e8	none	261,016	3/3	yes	300.9
rk8v4	none	175,006	3/3	yes	170.6

RTX PRO 6000

Ternary Bonsai 2 27B

Window 262,144 tokens, one request.

Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):

KV	speculation	VRAM	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	12.6 GiB	167.2	166.6	161.3	155.1	146.4	131.3	119.5	109.8
rk4v4	DFlash2, 5 drafts	14.0 GiB	360.9	401.3	317.4	284.2	242.7	197.9	179.4	193.0
rk4v4	MTP, 3 drafts	13.3 GiB	330.4	266.0	306.9	229.8	226.2	208.6	198.3	217.9
rk4v4-e8	none	12.6 GiB	167.3	166.9	161.8	155.2	146.6	131.3	119.7	110.0
rk4v4-e8	DFlash2, 5 drafts	14.0 GiB	367.4	285.1	307.0	319.9	222.8	203.7	180.8	189.7
rk8v4	none	14.6 GiB	167.5	166.8	162.2	151.5	139.8	121.2	107.3	96.4
rk8v4	DFlash2, 5 drafts	16.0 GiB	361.7	304.6	318.1	309.6	250.7	204.6	233.5	200.9
rk8v4	MTP, 3 drafts	15.4 GiB	314.7	313.0	293.3	258.1	229.7	191.4	192.0	210.5

Time to first token, s (cold prompt of that length):

KV	speculation	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	0.16	1.11	4.9	11.4	28.9	52.7	82.0
rk4v4	DFlash2, 5 drafts	0.17	1.12	4.9	11.5	29.2	53.1	82.5
rk4v4	MTP, 3 drafts	0.17	1.11	4.9	11.4	29.0	52.9	82.1
rk4v4-e8	none	0.16	1.10	4.9	11.3	28.7	52.1	81.1
rk4v4-e8	DFlash2, 5 drafts	0.17	1.12	4.9	11.4	28.9	52.5	81.6
rk8v4	none	0.16	1.07	4.7	10.8	27.4	50.1	78.3
rk8v4	DFlash2, 5 drafts	0.17	1.12	4.9	11.2	28.3	51.2	79.0
rk8v4	MTP, 3 drafts	0.16	1.11	4.8	11.1	28.0	50.8	78.5

Draft acceptance (accepted / drafted; tokens per verify round in brackets):

KV	speculation	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	DFlash2, 5 drafts	0.39 (2.91)	0.46 (3.26)	0.33 (2.62)	0.31 (2.54)	0.26 (2.31)	0.23 (2.14)	0.24 (2.17)	0.32 (2.57)
rk4v4	MTP, 3 drafts	0.54 (2.61)	0.38 (2.13)	0.52 (2.60)	0.33 (1.99)	0.36 (2.06)	0.38 (2.14)	0.41 (2.25)	0.56 (2.67)
rk4v4-e8	DFlash2, 5 drafts	0.40 (2.96)	0.26 (2.31)	0.31 (2.53)	0.37 (2.84)	0.23 (2.12)	0.24 (2.19)	0.24 (2.18)	0.30 (2.51)
rk8v4	DFlash2, 5 drafts	0.39 (2.91)	0.30 (2.47)	0.33 (2.65)	0.35 (2.75)	0.28 (2.38)	0.24 (2.19)	0.36 (2.81)	0.33 (2.63)
rk8v4	MTP, 3 drafts	0.50 (2.48)	0.50 (2.48)	0.48 (2.46)	0.42 (2.24)	0.39 (2.15)	0.35 (2.05)	0.44 (2.30)	0.60 (2.78)

Needles (three codes at 33, 66 and 90% of a document):

KV	speculation	document	found	order	TTFT s
rk4v4	none	261,020	3/3	yes	82.0
rk4v4-e8	none	261,016	3/3	yes	81.0
rk8v4	none	261,019	3/3	yes	78.4

Qwen3.8-27B

Window 262,144 tokens, one request.

Decode, tokens/s (short chat = five 512-token answers; at depth, a ~400-word answer after a document of that length):

KV	speculation	VRAM	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	21.8 GiB	89.6	89.5	88.0	85.9	83.2	78.0	73.7	69.9
rk4v4	DFlash2, 5 drafts	24.0 GiB	219.9	189.4	179.6	183.7	154.0	142.4	166.4	116.1
rk4v4	MTP, 3 drafts	22.5 GiB	186.4	164.5	169.6	149.0	161.0	143.4	142.9	128.7
rk4v4-e8	none	21.8 GiB	89.7	89.5	88.0	85.9	83.2	78.1	73.8	70.0
rk4v4-e8	DFlash2, 5 drafts	24.0 GiB	222.7	184.6	171.7	214.9	148.5	142.2	150.3	121.6
rk8v4	none	23.8 GiB	89.6	89.4	88.0	84.9	81.1	74.5	69.1	64.4
rk8v4	DFlash2, 5 drafts	26.0 GiB	219.1	195.9	186.0	217.2	156.9	160.2	164.5	120.9
rk8v4	MTP, 3 drafts	24.6 GiB	187.5	154.3	172.4	176.4	164.2	135.8	129.6	120.4

Time to first token, s (cold prompt of that length):

KV	speculation	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	none	0.30	1.87	7.9	17.2	40.5	70.0	104.9
rk4v4	DFlash2, 5 drafts	0.31	1.88	7.9	17.3	40.9	70.5	105.6
rk4v4	MTP, 3 drafts	0.31	1.87	7.8	17.2	40.6	70.1	105.0
rk4v4-e8	none	0.30	1.87	7.8	17.1	40.3	69.4	104.0
rk4v4-e8	DFlash2, 5 drafts	0.31	1.88	7.9	17.2	40.6	69.9	104.5
rk8v4	none	0.30	1.86	7.7	16.9	39.4	67.7	100.8
rk8v4	DFlash2, 5 drafts	0.31	1.87	7.8	17.0	39.7	68.3	101.7
rk8v4	MTP, 3 drafts	0.30	1.86	7.8	16.9	39.6	68.1	101.5

Draft acceptance (accepted / drafted; tokens per verify round in brackets):

KV	speculation	short chat	1,024	8,192	32,768	65,536	131,072	196,608	261,120
rk4v4	DFlash2, 5 drafts	0.44 (3.18)	0.36 (2.75)	0.33 (2.66)	0.37 (2.85)	0.30 (2.47)	0.30 (2.48)	0.42 (3.11)	0.26 (2.31)
rk4v4	MTP, 3 drafts	0.60 (2.78)	0.49 (2.46)	0.54 (2.62)	0.45 (2.34)	0.54 (2.62)	0.50 (2.49)	0.54 (2.61)	0.50 (2.48)
rk4v4-e8	DFlash2, 5 drafts	0.45 (3.21)	0.34 (2.70)	0.31 (2.54)	0.47 (3.33)	0.28 (2.38)	0.29 (2.47)	0.36 (2.78)	0.28 (2.40)
rk8v4	DFlash2, 5 drafts	0.44 (3.16)	0.37 (2.84)	0.35 (2.77)	0.47 (3.37)	0.30 (2.51)	0.35 (2.75)	0.41 (3.04)	0.28 (2.38)
rk8v4	MTP, 3 drafts	0.60 (2.80)	0.44 (2.31)	0.54 (2.62)	0.60 (2.85)	0.57 (2.74)	0.48 (2.45)	0.49 (2.46)	0.48 (2.44)

Needles (three codes at 33, 66 and 90% of a document):

KV	speculation	document	found	order	TTFT s
rk4v4	none	261,021	3/3	yes	104.8
rk4v4-e8	none	261,018	3/3	yes	104.0
rk8v4	none	261,021	3/3	yes	100.9

Re-check, October 2026

The PRO 6000's rows were measured again on 2026-10-06, next to an RTX 5090.

• Cards. Another RTX PRO 6000 Workstation Edition at 600 W (RunPod), and one card of an RTX 5090 pair whose boards default to 600 W (Vast.ai).
• Build. 796f98500, sm_120a, CUDA 13.1.
• Method. The same server flags and refbench.py suites as above, the server pinned to the GPU's NUMA node.

Two other hosts were rejected before measuring:

• a PRO 6000 host capped at 500 W;
• a 5090 host whose clocks stopped at 1.98 GHz under load. Bonsai 2 short chat decoded at 132 tok/s there, 23% below this card.

Decode in tok/s, prompt time in seconds, acceptance in brackets:

model, KV, speculation, request	RTX PRO 6000, September	RTX PRO 6000, October	RTX 5090, October
Bonsai 2, rk8v4, none, short chat	167.5	167.7	170.4
Bonsai 2, rk8v4, DFlash2 7 drafts, short chat	381	378.8	372.1
Bonsai 2, rk8v4, MTP 3 drafts, short chat	314.7	313.9	321.2
Bonsai 2, rk8v4, none, prompt of 261,120 tokens	78.3 s	75.0 s	76.7 s
Bonsai 2, rk8v4, none, decode after it	96.4	96.7	99.0
Bonsai 2, rk4v4, MTP 3 drafts, decode after it	217.9 (0.56)	220.9 (0.56)	194.8 (0.46)
Bonsai 2, rk8v4, MTP 3 drafts, decode after it	210.5 (0.60)	186.4 (0.49)	207.6 (0.57)
Bonsai 2, rk8v4, MTP 3 drafts, eight requests at once	1,155.3	1,159.6	1,081.5
Qwen3.8, rk8v4, none, short chat	89.6	90.0	92.1
Qwen3.8, rk8v4, DFlash2 7 drafts, short chat	237	238.5	233.5
Qwen3.8, rk8v4, MTP 3 drafts, eight requests at once	739.3	749.5	709.9

The PRO 6000 reproduces September within 2% wherever the drafts accepted match. The 261K prompt is 4% faster on the newer build. The two MTP rows after 261K follow their acceptance, which moves with the answer.

The PRO 6000 decodes one request no faster than the RTX 5090 because that decode is bound by memory bandwidth, and both cards have 1.79 TB/s of GDDR7: 167.7 against 170.4 tok/s for Bonsai 2 without speculation. Its 188 SMs against the RTX 5090's 170 show where compute decides. It prefills the 261K prompt 2% faster, and runs eight requests at once 7% faster with Bonsai 2 and 6% faster with Qwen3.8.

Largest context per card

The largest --max-context (= --kv-capacity, one request, --rope-yarn past 262,144) the server starts with, found by bisection in 4,096-token steps. Each cell gives it without speculation / with MTP (3 drafts) / with DFlash2 (5 drafts); 1,048,576 is the engine's ceiling. The RTX PRO 6000 starts every configuration at the ceiling with at least 50 GiB free.

model	KV	RTX PRO 6000	RTX 5090	RTX 4090	RTX 3090
Ternary Bonsai 2 27B	rk8v4	1,048,576 / 1,048,576 / 1,048,576	978,944 / 901,120 / 901,120	659,456 / 602,112 / 581,632	663,552 / 606,208 / 585,728
Ternary Bonsai 2 27B	rk4v4	1,048,576 / 1,048,576 / 1,048,576	1,048,576 / 1,048,576 / 1,048,576	958,464 / 876,544 / 831,488	970,752 / 884,736 / 839,680
Ternary Bonsai 2 27B	rk4v4-e8	1,048,576 / - / -	1,048,576 / - / -	958,464 / - / -	970,752 / - / -
Qwen3.8-27B	rk8v4	1,048,576 / 1,048,576 / 1,048,576	598,016 / 544,768 / 491,520	278,528 / 241,664 / 167,936	286,720 / 249,856 / 176,128
Qwen3.8-27B	rk4v4	1,048,576 / 1,048,576 / 1,048,576	872,448 / 794,624 / 716,800	405,504 / 356,352 / 245,760	417,792 / 364,544 / 258,048
Qwen3.8-27B	rk4v4-e8	1,048,576 / - / -	872,448 / - / -	405,504 / - / -	417,792 / - / -

Filled to that length (one request: a document with three codes at 33, 66 and 90%, asked back in order), without speculation:

card	model	KV	tokens	needles	TTFT	decode after it
RTX PRO 6000	Ternary Bonsai 2 27B	rk4v4	1,048,576	2/3	910 s	55.3 tok/s
RTX PRO 6000	Qwen3.8-27B	rk4v4	1,048,576	2/3	999 s	42.9 tok/s
RTX 5090	Ternary Bonsai 2 27B	rk4v4	1,048,576	2/3	937 s	56.2 tok/s
RTX 5090	Ternary Bonsai 2 27B	rk8v4	978,944	3/3	772 s	46.9 tok/s
RTX 5090	Qwen3.8-27B	rk4v4	872,448	3/3	755 s	47.6 tok/s
RTX 5090	Qwen3.8-27B	rk8v4	598,016	3/3	378 s	49.1 tok/s
RTX 4090	Ternary Bonsai 2 27B	rk4v4	958,464	3/3	991 s	30.9 tok/s
RTX 4090	Ternary Bonsai 2 27B	rk8v4	659,456	3/3	484 s	32.9 tok/s
RTX 4090	Qwen3.8-27B	rk4v4	405,504	3/3	293 s	35.4 tok/s
RTX 4090	Qwen3.8-27B	rk8v4	278,528	3/3	166 s	36.8 tok/s
RTX 3090	Ternary Bonsai 2 27B	rk4v4	970,752	3/3	2,359 s	17.7 tok/s
RTX 3090	Ternary Bonsai 2 27B	rk8v4	663,552	3/3	1,137 s	24.4 tok/s
RTX 3090	Qwen3.8-27B	rk4v4	417,792	3/3	644 s	23.8 tok/s
RTX 3090	Qwen3.8-27B	rk8v4	286,720	3/3	353 s	31.0 tok/s

At 1,048,576 tokens the RTX 5090 finds 2 of the three codes with --rope-yarn and 2 with plain RoPE (TTFT 937 and 939 s, decode 56.2 tok/s), missing the same code, the one at 90%, both times. The largest window that returned all three on that card is 978,944 tokens with rk8v4, whose last code sits near 881K. On the RTX PRO 6000 both models miss that code at 1,048,576 tokens too.

Draft length

Decode tokens/s with the draft acceptance in brackets, rk8v4, one request: short chat and a ~400-word answer after 32,768 and 131,072 tokens of document.

Ternary Bonsai 2 27B

262,144-token window.

RTX 5090

drafter	drafts	short chat	32,768	131,072
none	-	171.6	155.2	124.5
MTP	1	265.2 (0.74)	245.6 (0.78)	190.6 (0.71)
MTP	2	312.1 (0.63)	283.4 (0.63)	198.7 (0.48)
MTP	3	320.0 (0.50)	295.6 (0.52)	207.3 (0.40)
MTP	4	312.4 (0.41)	228.8 (0.28)	178.8 (0.26)
MTP	5	321.5 (0.38)	288.3 (0.38)	188.0 (0.27)
MTP	6	302.1 (0.33)	233.1 (0.26)	172.3 (0.22)
MTP	7	282.6 (0.28)	220.5 (0.22)	169.8 (0.20)
MTP	8	262.7 (0.25)	229.4 (0.25)	129.8 (0.16)
MTP	10	232.2 (0.20)	172.1 (0.15)	111.4 (0.12)
MTP	12	218.7 (0.18)	188.8 (0.17)	121.0 (0.13)
MTP	15	181.1 (0.13)	138.7 (0.10)	104.0 (0.10)
DFlash2	1	230.9 (0.74)	199.3 (0.63)	171.8 (0.69)
DFlash2	2	281.8 (0.59)	230.0 (0.46)	194.8 (0.48)
DFlash2	3	323.4 (0.50)	305.8 (0.53)	192.9 (0.32)
DFlash2	4	345.6 (0.43)	338.9 (0.48)	216.0 (0.31)
DFlash2	5	358.7 (0.38)	276.5 (0.29)	184.3 (0.19)
DFlash2	6	385.0 (0.36)	303.5 (0.29)	216.8 (0.23)
DFlash2	7	396.7 (0.33)	253.2 (0.19)	200.6 (0.17)
DFlash2	8	372.4 (0.29)	234.3 (0.17)	156.1 (0.15)
DFlash2	10	349.7 (0.23)	210.9 (0.13)	152.0 (0.12)
DFlash2	12	341.7 (0.20)	241.8 (0.14)	156.2 (0.11)
DFlash2	15	310.0 (0.15)	205.3 (0.10)	144.1 (0.09)

RTX 4090

drafter	drafts	short chat	32,768	131,072
none	-	108.3	97.9	77.3
MTP	1	167.7 (0.74)	139.7 (0.61)	116.3 (0.68)
MTP	2	191.1 (0.59)	175.0 (0.60)	128.3 (0.51)
MTP	3	201.2 (0.50)	175.2 (0.47)	130.9 (0.41)
MTP	4	198.9 (0.42)	142.8 (0.28)	119.0 (0.30)
MTP	5	196.9 (0.37)	153.4 (0.29)	113.7 (0.26)
MTP	6	187.6 (0.32)	145.6 (0.25)	109.3 (0.22)
MTP	7	176.9 (0.28)	135.1 (0.21)	99.4 (0.18)
MTP	8	171.3 (0.26)	128.9 (0.20)	86.6 (0.18)
MTP	10	152.3 (0.20)	127.0 (0.18)	80.5 (0.15)
MTP	12	137.7 (0.17)	100.1 (0.12)	73.4 (0.12)
MTP	15	120.5 (0.13)	90.5 (0.10)	56.1 (0.07)
DFlash2	1	151.0 (0.72)	140.3 (0.75)	105.9 (0.61)
DFlash2	2	191.1 (0.60)	162.4 (0.53)	121.7 (0.43)
DFlash2	3	215.4 (0.50)	205.6 (0.54)	144.3 (0.41)
DFlash2	4	228.3 (0.42)	220.4 (0.47)	129.9 (0.26)
DFlash2	5	242.9 (0.38)	171.8 (0.25)	130.5 (0.23)
DFlash2	6	250.7 (0.34)	183.1 (0.24)	144.9 (0.24)
DFlash2	7	256.4 (0.31)	178.7 (0.20)	131.9 (0.18)
DFlash2	8	248.9 (0.28)	160.0 (0.17)	110.9 (0.17)
DFlash2	10	254.4 (0.24)	178.9 (0.17)	101.2 (0.12)
DFlash2	12	244.3 (0.19)	158.5 (0.12)	106.9 (0.11)
DFlash2	15	238.5 (0.15)	163.4 (0.11)	98.0 (0.08)

RTX 3090

drafter	drafts	short chat	32,768	131,072
none	-	92.5	82.5	63.4
MTP	1	139.5 (0.74)	121.9 (0.69)	96.9 (0.73)
MTP	2	164.4 (0.63)	123.1 (0.47)	89.4 (0.46)
MTP	3	171.4 (0.52)	128.6 (0.40)	93.6 (0.39)
MTP	4	162.5 (0.42)	128.2 (0.35)	81.9 (0.26)
MTP	5	167.1 (0.39)	121.2 (0.30)	72.4 (0.21)
MTP	6	156.4 (0.33)	115.1 (0.25)	77.3 (0.22)
MTP	7	145.9 (0.28)	93.8 (0.17)	65.9 (0.16)
MTP	8	133.7 (0.25)	98.0 (0.21)	56.9 (0.15)
MTP	10	118.6 (0.20)	90.7 (0.18)	59.1 (0.17)
MTP	12	109.0 (0.18)	85.1 (0.16)	50.8 (0.13)
MTP	15	95.2 (0.14)	62.9 (0.10)	37.8 (0.07)
DFlash2	1	126.3 (0.72)	116.7 (0.75)	88.1 (0.65)
DFlash2	2	160.7 (0.62)	132.6 (0.55)	90.0 (0.44)
DFlash2	3	184.6 (0.54)	163.0 (0.54)	97.2 (0.36)
DFlash2	4	189.5 (0.44)	155.6 (0.39)	96.6 (0.28)
DFlash2	5	191.3 (0.37)	176.2 (0.41)	93.6 (0.24)
DFlash2	6	198.7 (0.34)	131.8 (0.22)	89.7 (0.19)
DFlash2	7	202.0 (0.31)	161.5 (0.28)	85.5 (0.16)
DFlash2	8	188.6 (0.29)	152.1 (0.28)	65.0 (0.13)
DFlash2	10	173.6 (0.23)	114.2 (0.17)	60.1 (0.11)
DFlash2	12	166.7 (0.20)	124.3 (0.17)	61.5 (0.10)
DFlash2	15	149.1 (0.15)	118.5 (0.15)	55.8 (0.08)

Qwen3.8-27B

262,144-token window on the RTX 5090, 139,264 on the 24 GB cards (DFlash2 and MTP 15 fit there).

RTX 5090

drafter	drafts	short chat	32,768	131,072
none	-	91.6	86.7	76.3
MTP	1	144.0 (0.81)	141.7 (0.87)	114.4 (0.71)
MTP	2	173.6 (0.70)	155.9 (0.63)	127.9 (0.55)
MTP	3	184.2 (0.59)	181.1 (0.63)	130.8 (0.44)
MTP	4	183.2 (0.50)	176.2 (0.52)	134.2 (0.40)
MTP	5	181.2 (0.44)	183.3 (0.50)	123.7 (0.32)
MTP	6	179.3 (0.40)	159.1 (0.37)	118.6 (0.28)
MTP	7	168.0 (0.34)	158.8 (0.34)	109.4 (0.23)
MTP	8	148.5 (0.30)	139.5 (0.30)	92.3 (0.21)
MTP	10	144.0 (0.27)	112.8 (0.21)	84.7 (0.17)
MTP	12	134.2 (0.23)	95.1 (0.16)	83.1 (0.16)
MTP	15	125.8 (0.20)	106.4 (0.17)	71.1 (0.12)
DFlash2	1	127.7 (0.77)	126.6 (0.84)	108.1 (0.75)
DFlash2	2	164.3 (0.65)	169.0 (0.75)	127.6 (0.54)
DFlash2	3	192.4 (0.57)	202.5 (0.67)	135.7 (0.41)
DFlash2	4	213.7 (0.52)	198.1 (0.50)	164.0 (0.44)
DFlash2	5	218.1 (0.44)	217.9 (0.48)	147.9 (0.31)
DFlash2	6	222.2 (0.38)	226.0 (0.43)	143.3 (0.25)
DFlash2	7	236.2 (0.36)	217.6 (0.35)	145.0 (0.23)
DFlash2	8	215.0 (0.32)	202.2 (0.33)	121.7 (0.20)
DFlash2	10	224.3 (0.28)	195.0 (0.26)	129.8 (0.19)
DFlash2	12	226.1 (0.24)	152.5 (0.16)	122.2 (0.15)
DFlash2	15	220.6 (0.19)	181.6 (0.17)	125.4 (0.13)

RTX 4090

drafter	drafts	short chat	32,768	131,072
none	-	54.8	52.0	45.6
MTP	1	87.8 (0.79)	84.1 (0.81)	67.9 (0.66)
MTP	2	104.3 (0.68)	94.0 (0.62)	75.4 (0.51)
MTP	3	110.6 (0.58)	114.7 (0.66)	80.3 (0.45)
MTP	4	113.5 (0.52)	114.4 (0.56)	78.9 (0.38)
MTP	5	111.0 (0.44)	109.4 (0.47)	69.7 (0.28)
MTP	6	109.6 (0.41)	107.1 (0.43)	72.8 (0.29)
MTP	7	106.8 (0.36)	102.8 (0.37)	64.3 (0.22)
MTP	8	90.4 (0.31)	93.4 (0.35)	56.1 (0.22)
MTP	10	87.0 (0.27)	82.5 (0.27)	47.5 (0.15)
MTP	12	77.3 (0.22)	77.0 (0.24)	50.4 (0.16)
MTP	15	71.5 (0.19)	65.8 (0.18)	41.3 (0.11)
DFlash2	1	79.4 (0.77)	78.6 (0.84)	66.3 (0.73)
DFlash2	2	101.6 (0.65)	105.7 (0.76)	79.2 (0.55)
DFlash2	3	120.6 (0.58)	120.6 (0.63)	88.7 (0.45)
DFlash2	4	128.1 (0.49)	127.7 (0.52)	90.8 (0.36)
DFlash2	5	137.1 (0.43)	138.7 (0.47)	90.8 (0.30)
DFlash2	6	142.7 (0.39)	139.1 (0.41)	98.6 (0.29)
DFlash2	7	148.8 (0.35)	151.9 (0.39)	102.3 (0.27)
DFlash2	8	137.7 (0.34)	121.5 (0.31)	80.7 (0.23)
DFlash2	10	135.4 (0.27)	116.8 (0.24)	83.6 (0.20)
DFlash2	12	138.1 (0.23)	115.5 (0.20)	78.0 (0.15)
DFlash2	15	137.4 (0.19)	114.5 (0.16)	75.0 (0.12)

RTX 3090

drafter	drafts	short chat	32,768	131,072
none	-	47.9	44.8	38.1
MTP	1	74.2 (0.79)	72.6 (0.87)	57.2 (0.72)
MTP	2	91.7 (0.70)	85.4 (0.71)	63.8 (0.59)
MTP	3	95.9 (0.58)	93.2 (0.63)	68.5 (0.51)
MTP	4	93.0 (0.49)	88.1 (0.50)	65.4 (0.42)
MTP	5	93.7 (0.44)	87.7 (0.47)	57.0 (0.32)
MTP	6	91.7 (0.40)	84.4 (0.41)	55.7 (0.29)
MTP	7	85.1 (0.35)	77.0 (0.34)	51.3 (0.24)
MTP	8	76.6 (0.31)	70.5 (0.33)	42.7 (0.21)
MTP	10	71.6 (0.27)	46.5 (0.16)	38.3 (0.17)
MTP	12	67.5 (0.23)	58.5 (0.22)	35.4 (0.14)
MTP	15	56.3 (0.17)	52.1 (0.18)	28.2 (0.10)
DFlash2	1	68.1 (0.79)	66.2 (0.84)	54.9 (0.75)
DFlash2	2	89.4 (0.67)	83.0 (0.67)	64.3 (0.57)
DFlash2	3	103.4 (0.58)	102.3 (0.65)	74.9 (0.52)
DFlash2	4	108.0 (0.50)	103.9 (0.53)	79.0 (0.45)
DFlash2	5	115.0 (0.45)	107.6 (0.47)	74.1 (0.36)
DFlash2	6	115.0 (0.39)	109.3 (0.41)	63.5 (0.24)
DFlash2	7	117.5 (0.37)	107.6 (0.37)	61.6 (0.21)
DFlash2	8	108.8 (0.33)	101.1 (0.35)	52.3 (0.19)
DFlash2	10	104.5 (0.26)	88.1 (0.25)	51.0 (0.16)
DFlash2	12	105.5 (0.23)	87.3 (0.21)	47.1 (0.12)
DFlash2	15	99.9 (0.19)	79.0 (0.16)	42.5 (0.10)

Several requests at once

Total generated tokens per second of wall time when N requests arrive together: the short-chat prompts in turn (the essay alone at 1, the essay and the Python module at 2, and so on), 512 tokens each, rk8v4, --max-concurrency 8 --max-context 16384 --kv-capacity 131072. Speculation pays less as the batch grows, since the verify widths add up to more of the card's compute. Batched answers keep their quality but can part from a lone request's at near-tied tokens (batch composition). A dash at 8: eight lanes of 16,384 tokens do not fit beside that model and drafter on the card, and the row ran with four:

card	model	speculation	1 at once	2 at once	4 at once	8 at once
RTX PRO 6000	Ternary Bonsai 2 27B	none	165.5	321.4	599.1	1,043.5
RTX PRO 6000	Ternary Bonsai 2 27B	MTP, 3 drafts	274.4	516.0	805.0	1,155.3
RTX PRO 6000	Ternary Bonsai 2 27B	DFlash2, 5 drafts	270.6	523.9	735.5	890.4
RTX PRO 6000	Qwen3.8-27B	none	88.8	172.6	327.9	569.8
RTX PRO 6000	Qwen3.8-27B	MTP, 3 drafts	154.4	300.9	515.8	739.3
RTX PRO 6000	Qwen3.8-27B	DFlash2, 5 drafts	176.1	319.4	473.7	534.6
RTX 5090	Ternary Bonsai 2 27B	none	169.8	327.4	608.6	1,053.3
RTX 5090	Ternary Bonsai 2 27B	MTP, 3 drafts	263.3	511.3	771.1	1,063.0
RTX 5090	Ternary Bonsai 2 27B	DFlash2, 5 drafts	278.1	511.7	631.9	799.1
RTX 5090	Qwen3.8-27B	none	91.0	172.0	324.7	569.1
RTX 5090	Qwen3.8-27B	MTP, 3 drafts	160.6	299.0	490.0	690.0
RTX 5090	Qwen3.8-27B	DFlash2, 5 drafts	171.6	311.1	430.9	469.4
RTX 4090	Ternary Bonsai 2 27B	none	107.4	206.9	383.5	670.6
RTX 4090	Ternary Bonsai 2 27B	MTP, 3 drafts	182.1	318.3	533.8	824.1
RTX 4090	Ternary Bonsai 2 27B	DFlash2, 5 drafts	197.1	354.2	504.9	673.5
RTX 4090	Qwen3.8-27B	none	54.3	105.6	198.4	356.7
RTX 4090	Qwen3.8-27B	MTP, 3 drafts	92.8	174.9	300.3	442.2
RTX 4090	Qwen3.8-27B	DFlash2, 5 drafts	107.7	185.1	295.2	-
RTX 3090	Ternary Bonsai 2 27B	none	93.1	176.9	324.0	546.4
RTX 3090	Ternary Bonsai 2 27B	MTP, 3 drafts	141.7	267.3	413.7	550.8
RTX 3090	Ternary Bonsai 2 27B	DFlash2, 5 drafts	154.7	256.0	334.1	386.2
RTX 3090	Qwen3.8-27B	none	48.1	91.9	172.2	290.2
RTX 3090	Qwen3.8-27B	MTP, 3 drafts	88.2	154.8	238.4	328.9
RTX 3090	Qwen3.8-27B	DFlash2, 5 drafts	89.1	155.4	206.0	-

What the built-in profiles change

Routes each card's calibration moved off the compiled tables, with the largest speedup of the operation (its time on the compiled route over its time on the chosen one, not end to end):

route	RTX 3090	RTX 4090	RTX 5090	RTX PRO 6000	PRO 6000 Max-Q	PRO 6000 Server
rk8v4 decode/verify attention	query widths 2-5: up to 1.16×	query widths 2-8: up to 1.16×	query widths 3-8: up to 1.08×	query widths 3-4, 6-8: up to 1.08×	-	query widths 6-8: up to 1.06×
rk4v4 decode/verify attention	query widths 1-5: up to 3.20×	query widths 1, 3-7: up to 1.46×	query widths 1-8: up to 1.13×	query widths 1-8: up to 1.19×	query widths 1, 3-8: up to 1.11×	query widths 1, 3-8: up to 1.16×
fast prompt kernel	1.41×	1.43×	1.24×	1.20×	1.19×	1.23×
FP16 PV accumulation	1.17×	1.20×	1.19×	-	-	-
ternary small-T projections	12 widths, up to 1.09×	15 widths, up to 1.27×	34 widths, up to 1.38×	31 widths, up to 1.28×	33 widths, up to 1.33×	33 widths, up to 1.16×
groupwise Q4/Q5 projections	-	13 widths, up to 1.26×	9 widths, up to 1.14×	11 widths, up to 1.23×	12 widths, up to 1.24×	7 widths, up to 1.25×

How these were measured

Hardware. Rented single-GPU hosts, each GPU at its board's default power limit. RTX 3090: 390 W for the Ternary Bonsai 2 rows and several requests at once; two other boards at 370 W for the Qwen3.8 rows, the draft sweeps, the largest context and the fills. RTX 4090: 450 W (the largest-context bisection ran on a second board capped at 400 W; it does not depend on power). RTX 5090: 575 W, on two hosts. RTX PRO 6000: the Workstation Edition at 600 W; the Max-Q (300 W) and Server (600 W) editions were calibrated for their built-in profiles and not measured here. The server runs pinned to the CPUs of the GPU's NUMA node.

Software. This repository at the commit that ships the built-in profiles, built for the card (sm_86, sm_89, sm_120a on the mma.sync path), CUDA 13.1. Artifacts: Ternary-Bonsai-2-27B-NInfer-v3 and Qwen3.8-27B-NInfer (groupwise-int).

Server. ninfer-serve <artifact> --max-context W --kv-capacity W --max-concurrency 1 --kv-dtype KV --gdn-state-fp16, plus --spec dflash2 --draft-tokens 5 or --spec mtp --draft-tokens 3, plus --rope-yarn past 262,144 tokens. Everything else is at its default, so prefix reuse is on but never hit: every request starts with a fresh nonce.

Requests (tools/bench/refbench.py), greedy, thinking off, non-streaming, against the OpenAI Chat Completions endpoint:

• Short chat: five prompts (an essay, a Python module with tests, a word problem, a Russian explanation, a list of forty items), 512 generated tokens each; the table gives the mean decode rate.
• Depth: a cold document of the given length (PG-19, WikiText and this repository's text, cut by tokens), then the request to retell its final part in about 400 words, at most 512 tokens. TTFT is the server's prompt time; decode is the rest of the answer over its decode time.
• Needles: three codes planted at 33, 66 and 90% of a document 1,024 tokens short of the window, asked back in order.

VRAM is the device memory the server holds when idle: NInfer reserves the whole KV cache, workspaces and graphs at startup, so it does not grow under load.

Largest context. --max-context = --kv-capacity bisected in 4,096-token steps on whether the server starts; the engine plans every byte at startup, and the filled runs confirm that the found window also serves one request that fills it.

Draft length. The same server with --draft-tokens from 1 to 15; short chat and the depth requests at 32,768 and 131,072 tokens.

Several requests at once. --max-concurrency 8 --max-context 16384 --kv-capacity 131072; N short-chat requests sent together, 512 tokens each, and their total generated tokens over the wall time from the first request to the last answer.