Skip to content

Commit 7bbfcaa

Browse files
committed
docs: document the perplexity benchmark in full
Adds the per-domain perplexity for rk4v4 and rk2v4-e8 measured on the RTX 3080, mean NLL, scoring speed and run time, the corpus protocol, what the fixed-window metric does and does not measure, and the command that reproduces it.
1 parent 789b591 commit 7bbfcaa

1 file changed

Lines changed: 26 additions & 8 deletions

File tree

‎README.md‎

Lines changed: 26 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -64,17 +64,35 @@ Tokens per second. "Nearly full" is one request whose prompt fills about 92% of
6464
- **The desktop moves the ceiling.** The monitor's share of the card varied from 0.4 to 1.3 GB during
6565
these runs. At the high end the 64K MTP window does not fit, and `ninfer-switch` starts it at 56K.
6666

67-
### KV cache quality
67+
### Perplexity (KV cache quality)
6868

69-
Perplexity on the repository's fixed corpus (`--quick`, 4,096-token context, 2,048 stride, 261,223
70-
scored tokens). Lower is better.
69+
The two KV caches the profiles use, scored by `ninfer-perplexity` on the repository's fixed corpus
70+
`ninfer-ppl-1m-v1` in `--quick` mode: one stream from each of four domains, a 4,096-token context and a
71+
2,048-token stride, 261,223 scored tokens per run. Lower is better.
7172

72-
| KV cache | Perplexity | Tokens per GiB of KV |
73-
|---|---:|---:|
74-
| `rk4v4` | 5.887 | 59,900 |
75-
| `rk2v4-e8` | 6.088 (+3.4%) | 77,700 |
73+
| Domain | Scored tokens | `rk4v4` | `rk2v4-e8` | Change |
74+
|---|---:|---:|---:|---:|
75+
| English reference | 65,328 | 8.034 | 8.308 | +3.4% |
76+
| English long-form | 65,484 | 9.378 | 9.608 | +2.5% |
77+
| Chinese reference | 65,513 | 8.266 | 8.604 | +4.1% |
78+
| NInfer C++/CUDA code | 64,898 | 1.910 | 1.982 | +3.7% |
79+
| **Overall** | **261,223** | **5.887** | **6.088** | **+3.4%** |
7680

77-
`rk2v4-e8` buys 30% more context for a small quality cost. Use `mtp48` when that trade is not worth it.
81+
| KV cache | Mean NLL | Tokens per GiB of KV | Scoring speed | Run time |
82+
|---|---:|---:|---:|---:|
83+
| `rk4v4` | 1.7728 | 59,900 | 1,108 tok/s | 4.2 min |
84+
| `rk2v4-e8` | 1.8063 | 77,700 | 1,088 tok/s | 4.4 min |
85+
86+
`rk2v4-e8` buys 30% more context for a 3.4% rise in perplexity, spread evenly across domains. Use
87+
`mtp48` (`rk4v4`) when that trade is not worth it. Every window starts from empty state, so this
88+
measures quality within a 4K context; it does not test recall across a long one.
89+
90+
To reproduce, with `--kv-dtype` set to the cache under test:
91+
92+
```powershell
93+
.\build-ninja\apps\ninfer-perplexity.exe models\Ternary-Bonsai-2-27B-Heretic-ninfer.ninfer `
94+
--corpus eval\corpora\perplexity-1m\manifest.json --quick --kv-dtype rk2v4-e8 --gdn-state-fp16
95+
```
7896

7997
### Other engines on the same card
8098

0 commit comments

Comments
 (0)