Skip to content

Commit 2d6f56b

Browse files
authored
Arm backend: Add KV cache SmolLM2 VGF demo flow (pytorch#21937)
Unify the SmolLM2 VGF generation and perplexity helpers so the same entry points support static non-KV full-logits and KV-cache exports. Recommend linear16a8w with an FP32 KV cache for the default quantized workflow. Document calibrated static INT8 KV storage as experimental while retaining FP32 and static non-KV alternatives. AI-assisted-by: Codex cc @digantdesai @freddan80 @per @zingo @oscarandersson8218 @mansnils @Sebastian-Larsson @robell @rascani Signed-off-by: Xingguo Li <xingguo.li@arm.com>
1 parent a48ff9f commit 2d6f56b

3 files changed

Lines changed: 403 additions & 104 deletions

File tree

Lines changed: 134 additions & 81 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
# SmolLM2 → VGF Quickstart
22

3-
> **Heads-up:** The current VGF PTQ flow is still experimental. Use FP32 as the baseline, expect `linear8a8w` to be accuracy-sensitive, and treat `linear16a8w` as the preferred quantized path to try first.
3+
> **Heads-up:** Use FP32 as the accuracy baseline and `linear16a8w` with an
4+
> FP32 KV cache as the recommended quantized path. Treat static INT8 KV-cache
5+
> storage and `linear8a8w` as experimental.
46
57
This is a host-only VGF workflow built around `executor_runner`. Run the
68
commands from the root of an ExecuTorch source checkout.
@@ -34,7 +36,7 @@ If you want the broader Arm backend setup flow, see
3436
## 1. Tokenizer (one-time)
3537
```bash
3638
mkdir -p data/tokenizers/smollm2
37-
huggingface-cli download HuggingFaceTB/SmolLM2-135M-Instruct tokenizer.json \
39+
huggingface-cli download HuggingFaceTB/SmolLM2-135M tokenizer.json \
3840
--local-dir data/tokenizers/smollm2
3941
```
4042
The download lives at `data/tokenizers/smollm2/tokenizer.json`. Use this path in the export and sampling commands below.
@@ -48,90 +50,120 @@ export CXX=/usr/bin/g++-12
4850
export CUDAHOSTCXX=$CXX
4951
```
5052

51-
## 2. Recommended: FP32 export
52-
Produces a stable `.pte` for experimentation and sampling.
53+
## 2. Baseline: full-FP32 model and KV cache
54+
Produces a stable `.pte` for experimentation and sampling. KV cache is the
55+
preferred runtime flow because each decode step reuses cached attention state
56+
instead of recomputing the full prompt window.
57+
5358
```bash
5459
python -m extension.llm.export.export_llm \
5560
base.model_class=smollm2 \
5661
base.params=examples/models/smollm2/135M_config.json \
5762
base.tokenizer_path=data/tokenizers/smollm2/tokenizer.json \
58-
export.output_dir=outputs/$(date +%F)/$(date +%H-%M-%S)_fp32 \
59-
export.output_name=smollm2_vgf_fp32_full_logits.pte \
60-
export.max_seq_length=64 \
61-
export.max_context_length=64 \
63+
export.output_dir=outputs/$(date +%F)/$(date +%H-%M-%S)_kv_fp32 \
64+
export.output_name=smollm2_vgf_kv_fp32.pte \
65+
export.max_seq_length=128 \
66+
export.max_context_length=128 \
6267
backend.vgf.enabled=True \
63-
backend.vgf.compile_spec=TOSA-1.0+FP \
64-
model.use_kv_cache=False \
68+
backend.vgf.compile_spec=TOSA-1.0+FP+INT \
69+
model.use_kv_cache=True \
6570
model.enable_dynamic_shape=False \
66-
debug.verbose=True \
67-
debug.generate_full_logits=True
71+
debug.verbose=True
6872
```
6973

74+
## 3. Recommended quantized path: `linear16a8w` with FP32 KV storage
75+
This quantizes `torch.nn.Linear` modules with the Arm VGF PT2E quantizer and
76+
keeps the persistent KV cache in FP32. This is the recommended starting point
77+
because static INT8 cache storage has not yet demonstrated a performance
78+
benefit on VGF hardware.
7079

71-
## 3. Experimental: 8-bit PTQ (Linear-only)
72-
This quantizes only `torch.nn.Linear` modules using the Arm VGF PT2E quantizer.
73-
74-
Supported calibration inputs:
75-
- `quantization.calibration_data=@...` for a text corpus
76-
- `quantization.calibration_tasks=[wikitext]` for LM-Eval tasks
77-
78-
For this static non-KV-cache flow, keep `debug.generate_full_logits=True` for
79-
calibrated exports. Calibration uses padded fixed-shape prefixes, and full
80-
logits let the calibration/eval helpers select the last real-token logits row
81-
instead of accidentally using the padded tail.
80+
Example (16-bit activations, 8-bit weights, Linear-only):
8281

83-
Example (LM-Eval wikitext calibration):
8482
```bash
8583
python -m extension.llm.export.export_llm \
8684
base.model_class=smollm2 \
8785
base.params=examples/models/smollm2/135M_config.json \
8886
base.tokenizer_path=data/tokenizers/smollm2/tokenizer.json \
89-
export.output_dir=outputs/$(date +%F)/$(date +%H-%M-%S)_linear8a8w \
90-
export.output_name=smollm2_vgf_linear8a8w_wikitext_full_logits.pte \
91-
export.max_seq_length=64 \
92-
export.max_context_length=64 \
93-
quantization.pt2e_quantize=vgf_8a8w \
87+
export.output_dir=outputs/$(date +%F)/$(date +%H-%M-%S)_kv_linear16a8w \
88+
export.output_name=smollm2_vgf_kv_linear16a8w.pte \
89+
export.max_seq_length=128 \
90+
export.max_context_length=128 \
91+
quantization.pt2e_quantize=vgf_16a8w \
9492
quantization.calibration_tasks=\[wikitext\] \
9593
quantization.calibration_limit=64 \
96-
quantization.calibration_seq_length=64 \
94+
quantization.calibration_seq_length=128 \
9795
backend.vgf.enabled=True \
98-
backend.vgf.compile_spec=TOSA-1.0+FP+INT \
96+
backend.vgf.compile_spec=TOSA-1.0+FP+INT+int16 \
9997
backend.vgf.quantize_scope=linear \
100-
model.use_kv_cache=False \
98+
model.use_kv_cache=True \
10199
model.enable_dynamic_shape=False \
102-
debug.verbose=True \
103-
debug.generate_full_logits=True
100+
debug.verbose=True
104101
```
105102

106-
Example (16-bit activations, 8-bit weights, Linear-only):
103+
### 3.1 Experimental: `linear16a8w` with static INT8 KV storage
104+
105+
This path stores the persistent KV cache as calibrated INT8 data. It is
106+
different from `model.quantize_kv_cache=True`, which performs dynamic per-token
107+
KV quantization and currently leaves cache QDQ/update maintenance outside the
108+
VGF delegate.
109+
110+
When `quantization.calibration_tasks` is set, export first runs the task
111+
through a float KV cache and derives separate symmetric per-head-dimension K/V
112+
scales for every layer. The same task is then reused for normal PT2E Linear
113+
calibration. Without a calibration task, static KV storage uses
114+
`model.static_quantize_kv_cache_scale` as a fixed fallback;
115+
`calibration_data` alone only calibrates the PT2E operators.
116+
117+
Use the recommended command above with these changes:
118+
119+
```bash
120+
export.output_dir=outputs/$(date +%F)/$(date +%H-%M-%S)_static_kvq_linear16a8w \
121+
export.output_name=smollm2_vgf_static_kvq_linear16a8w.pte \
122+
model.static_quantize_kv_cache=True
123+
```
124+
125+
The static KV-cache path uses symmetric per-head-dimension scales with
126+
zero-point `0` and dequantizes the cache back to float for the existing
127+
attention path. Treat its accuracy and performance as experimental until they
128+
have been validated for the target hardware and context length.
129+
130+
Alternative `linear8a8w` mode:
107131

108132
```bash
109133
python -m extension.llm.export.export_llm \
110134
base.model_class=smollm2 \
111135
base.params=examples/models/smollm2/135M_config.json \
112136
base.tokenizer_path=data/tokenizers/smollm2/tokenizer.json \
113-
export.output_dir=outputs/$(date +%F)/$(date +%H-%M-%S)_linear16a8w \
114-
export.output_name=smollm2_vgf_linear16a8w_wikitext_full_logits.pte \
115-
export.max_seq_length=64 \
116-
export.max_context_length=64 \
117-
quantization.pt2e_quantize=vgf_16a8w \
137+
export.output_dir=outputs/$(date +%F)/$(date +%H-%M-%S)_kv_linear8a8w \
138+
export.output_name=smollm2_vgf_kv_linear8a8w.pte \
139+
export.max_seq_length=128 \
140+
export.max_context_length=128 \
141+
quantization.pt2e_quantize=vgf_8a8w \
118142
quantization.calibration_tasks=\[wikitext\] \
119143
quantization.calibration_limit=64 \
120-
quantization.calibration_seq_length=64 \
144+
quantization.calibration_seq_length=128 \
121145
backend.vgf.enabled=True \
122-
backend.vgf.compile_spec=TOSA-1.0+FP+INT+int16 \
146+
backend.vgf.compile_spec=TOSA-1.0+FP+INT \
123147
backend.vgf.quantize_scope=linear \
124-
model.use_kv_cache=False \
148+
model.use_kv_cache=True \
125149
model.enable_dynamic_shape=False \
126-
debug.verbose=True \
127-
debug.generate_full_logits=True
150+
debug.verbose=True
128151
```
129152

130153
`quantization.pt2e_quantize` selects the numeric mode.
131154
`backend.vgf.quantize_scope=linear` keeps quantization limited to
132155
`torch.nn.Linear` modules. The compile spec still includes FP because the rest
133156
of the graph remains floating point.
134157

158+
### 3.2 Alternative: static non-KV full-logits exports
159+
Use the static non-KV path when you specifically need full-window logits, for
160+
example faster perplexity evaluation or debugging padded fixed-shape behavior.
161+
For calibrated non-KV exports, keep `debug.generate_full_logits=True`; this lets
162+
the helpers select the last real-token logits row instead of the padded tail.
163+
Set `model.use_kv_cache=False`, keep `export.max_seq_length` and
164+
`quantization.calibration_seq_length` aligned, and add
165+
`debug.generate_full_logits=True` to the export commands above.
166+
135167
## 4. Sampling with `executor_runner`
136168

137169
### 4.0 Build `executor_runner` (VKML)
@@ -149,8 +181,9 @@ CPU kernel coverage.
149181

150182
### 4.1 Greedy and `T=0.8` sampling
151183
`examples/arm/smollm2_example_vgf/generate_sampled.py` wraps
152-
`cmake-out-vkml/executor_runner`, keeps a sliding fixed-length token window,
153-
and can print the top-5 logits each step.
184+
`cmake-out-vkml/executor_runner` and supports both KV-cache and static non-KV
185+
exports. The recommended KV-cache path uses `--use-kv-cache`; the static
186+
non-KV path uses a fixed token window plus `--full-logits`.
154187

155188
Greedy generation (`--temperature 0`) always chooses the highest-logit next
156189
token, which is useful for deterministic comparisons. Stochastic generation
@@ -160,46 +193,47 @@ with a fixed `--seed`.
160193

161194
Notes:
162195
- `--max-seq-length` must match the export `export.max_seq_length` (otherwise you will hit input size mismatch).
163-
- The exported SmolLM2 VGF input is `int32[1, max_seq_length]`; the helper writes
164-
token windows as `int32` binary inputs for `executor_runner`.
165-
- Use `--persistent-runner` for faster multi-token generation (loads the model once).
196+
- KV-cache exports take `token,input_pos` inputs and should pass `--use-kv-cache`.
197+
- Static non-KV exports take an `int32[1, max_seq_length]` token window and should pass `--full-logits`.
198+
- The KV-cache path keeps `executor_runner` alive in server mode automatically.
166199
- The documented examples use `--temperature 0` (greedy) and `--temperature 0.8`.
167200
- For deterministic comparisons against saved `temp0` outputs, use `--seed 0`, `--repetition-penalty 1.1`, and `--no-topk-print`. At `--temperature 0`, token selection is greedy, so `--top-p` does not affect the chosen token.
168201

169202
Greedy example (`T=0`):
170203
```bash
171204
python examples/arm/smollm2_example_vgf/generate_sampled.py \
172-
--persistent-runner \
173205
--runner cmake-out-vkml/executor_runner \
174-
--pte smollm2_vgf_fp32_full_logits.pte \
206+
--pte smollm2_vgf_kv_linear16a8w.pte \
175207
--tokenizer data/tokenizers/smollm2/tokenizer.json \
176208
--prompt "Once upon a time in a small village," \
177-
--max-seq-length 64 \
209+
--max-seq-length 128 \
210+
--max-context-length 128 \
178211
--max-new-tokens 10 \
179212
--seed 0 \
180213
--temperature 0 \
181214
--repetition-penalty 1.1 \
182-
--full-logits
215+
--use-kv-cache
183216
```
184217

185218
Stochastic example (`T=0.8`):
186219
```bash
187220
python examples/arm/smollm2_example_vgf/generate_sampled.py \
188-
--persistent-runner \
189221
--runner cmake-out-vkml/executor_runner \
190-
--pte smollm2_vgf_fp32_full_logits.pte \
222+
--pte smollm2_vgf_kv_linear16a8w.pte \
191223
--tokenizer data/tokenizers/smollm2/tokenizer.json \
192224
--prompt "Once upon a time in a small village," \
193-
--max-seq-length 64 \
225+
--max-seq-length 128 \
226+
--max-context-length 128 \
194227
--max-new-tokens 10 \
195228
--seed 0 \
196229
--temperature 0.8 \
197230
--top-p 0.9 \
198231
--repetition-penalty 1.1 \
199-
--full-logits
232+
--use-kv-cache
200233
```
201-
> Swap `--pte` to the quantized build to compare behaviour. `linear8a8w` still
202-
> tends to drift more than `linear16a8w`.
234+
> Swap `--pte` to `smollm2_vgf_static_kvq_linear16a8w.pte` to compare
235+
> experimental static INT8 KV storage, or to `smollm2_vgf_kv_fp32.pte` for the
236+
> full-FP32 baseline.
203237
204238

205239

@@ -209,56 +243,73 @@ To generate for *all* prompts in `default_prompts.txt` and save to a file:
209243

210244
```bash
211245
python examples/arm/smollm2_example_vgf/generate_sampled.py \
212-
--persistent-runner \
213246
--runner cmake-out-vkml/executor_runner \
214-
--pte smollm2_vgf_fp32_full_logits.pte \
247+
--pte smollm2_vgf_kv_linear16a8w.pte \
215248
--tokenizer data/tokenizers/smollm2/tokenizer.json \
216249
--prompt-file examples/arm/smollm2_example_vgf/default_prompts.txt \
217250
--prompt-all \
218-
--max-seq-length 64 \
251+
--max-seq-length 128 \
252+
--max-context-length 128 \
219253
--max-new-tokens 64 \
220254
--temperature 0.8 \
221255
--top-p 0.9 \
222256
--repetition-penalty 1.1 \
223-
--full-logits \
257+
--use-kv-cache \
224258
--save-generations outputs/$(date +%F)/$(date +%H-%M-%S)_smollm2_gen.txt
225259
```
226260

227261
## 5. Wikitext prompts and perplexity
228262

263+
For a smoke test before a dedicated comparison, run the `default_prompts.txt`
264+
command above with `smollm2_vgf_kv_linear16a8w.pte` and inspect the
265+
generations. To evaluate the experimental static INT8 cache artifact alone,
266+
pass `smollm2_vgf_static_kvq_linear16a8w.pte` to any one of the PTE options
267+
below.
268+
229269
Build a reusable 1000-prompt file from `wikitext-2-raw-v1` and evaluate
230-
perplexity on the first 100 prompts for FP32, `linear8a8w`, and `linear16a8w`:
270+
perplexity on the first 100 prompts for KV-cache FP32, `linear8a8w`, and
271+
`linear16a8w`:
231272

232273
```bash
233274
OUT_DIR=outputs/$(date +%F)/$(date +%H-%M-%S)_smollm2_vgf_eval
234275

235276
python examples/arm/smollm2_example_vgf/eval_wikitext_perplexity.py \
236277
--runner cmake-out-vkml/executor_runner \
237-
--pte-fp32 "${OUT_DIR}/smollm2_vgf_fp32_full_logits.pte" \
238-
--pte-linear8a8w "${OUT_DIR}/smollm2_vgf_linear8a8w_wikitext_full_logits.pte" \
239-
--pte-linear16a8w "${OUT_DIR}/smollm2_vgf_linear16a8w_wikitext_full_logits.pte" \
278+
--pte-fp32 "${OUT_DIR}/smollm2_vgf_kv_fp32.pte" \
279+
--pte-linear8a8w "${OUT_DIR}/smollm2_vgf_kv_linear8a8w.pte" \
280+
--pte-linear16a8w "${OUT_DIR}/smollm2_vgf_kv_linear16a8w.pte" \
240281
--tokenizer data/tokenizers/smollm2/tokenizer.json \
241282
--prompts-file "${OUT_DIR}/wikitext_prompts_1000.txt" \
242283
--num-prompts 1000 \
243284
--ppl-prompts 100 \
244-
--max-seq-length 64 \
245-
--max-prompt-tokens 64 \
285+
--max-seq-length 128 \
286+
--max-prompt-tokens 128 \
287+
--max-tokens-per-prompt 64 \
288+
--use-kv-cache \
246289
--refresh-prompts
247290
```
248291

249292
Notes:
250293
- This script downloads `wikitext-2-raw-v1` via Hugging Face `datasets`.
251294
- The prompts file is reusable; omit `--refresh-prompts` on later runs.
252295
- Perplexity is computed on the first 100 prompts from that file.
253-
- Each prompt is capped to 64 tokens and scored from one full-logits
254-
`executor_runner` invocation per prompt, rather than one invocation per token.
296+
- KV-cache prompts are scored step-by-step with `token` and `input_pos`
297+
inputs, so this path is slower but matches the recommended decode contract.
298+
- `--max-tokens-per-prompt` controls how many tokens are scored from each
299+
prompt. Use `64` for comparison with the static-window baseline, or a larger
300+
value to stress a longer KV context.
301+
- For static non-KV PTEs, omit `--use-kv-cache` and pass full-logits PTEs; each
302+
prompt is then scored from one full-logits invocation rather than one
303+
invocation per token.
255304

256305
## 6. Notes
257-
- This flow keeps KV cache disabled and uses a fixed token window. KV-cache
258-
support is the expected next step for faster generation, but it is outside
259-
this static VGF quickstart.
260-
- Without KV cache, the model recomputes the entire token window for each
261-
generated token.
306+
- The quickstart exports use KV cache. Static non-KV full-logits exports are
307+
still useful for faster perplexity experiments and fixed-window debugging.
308+
- The recommended quantized export keeps KV storage in FP32. Static INT8 KV
309+
storage remains available for experimental memory and performance studies.
310+
- With KV cache, generation replays one token plus `input_pos` per runner
311+
invocation; without KV cache, the model recomputes the full token window for
312+
each generated token.
262313
- `linear8a8w` still shows noticeably more quality loss than `linear16a8w`.
263314
- When you change `max_seq_length`, regenerate any cached prompt inputs to match the new window size.
264315
- On hosts with multiple Vulkan devices, use `vulkaninfo --summary` to check
@@ -267,7 +318,9 @@ Notes:
267318
### Implementation details
268319
- The VKML runner is `examples/portable/executor_runner/executor_runner.cpp`,
269320
built here as `cmake-out-vkml/executor_runner`.
270-
- `generate_sampled.py` tokenizes prompts, prepares the fixed token window,
271-
invokes `executor_runner`, reads logits, and decodes sampled tokens.
272-
- The sampling and perplexity commands pass `--full-logits` to match the
273-
exported full-logits PTEs.
321+
- `generate_sampled.py` tokenizes prompts, prepares either the fixed non-KV
322+
token window or KV-cache `token,input_pos` inputs, invokes `executor_runner`,
323+
reads logits, and decodes sampled tokens.
324+
- The non-KV sampling and perplexity commands pass `--full-logits` to match the
325+
exported full-logits PTEs. KV-cache PTEs return next-token logits directly
326+
and should use `--use-kv-cache` instead.

0 commit comments

Comments
 (0)