11# SmolLM2 → VGF Quickstart
22
3- > ** Heads-up:** The current VGF PTQ flow is still experimental. Use FP32 as the baseline, expect ` linear8a8w ` to be accuracy-sensitive, and treat ` linear16a8w ` as the preferred quantized path to try first.
3+ > ** Heads-up:** Use FP32 as the accuracy baseline and ` linear16a8w ` with an
4+ > FP32 KV cache as the recommended quantized path. Treat static INT8 KV-cache
5+ > storage and ` linear8a8w ` as experimental.
46
57This is a host-only VGF workflow built around ` executor_runner ` . Run the
68commands from the root of an ExecuTorch source checkout.
@@ -34,7 +36,7 @@ If you want the broader Arm backend setup flow, see
3436## 1. Tokenizer (one-time)
3537``` bash
3638mkdir -p data/tokenizers/smollm2
37- huggingface-cli download HuggingFaceTB/SmolLM2-135M-Instruct tokenizer.json \
39+ huggingface-cli download HuggingFaceTB/SmolLM2-135M tokenizer.json \
3840 --local-dir data/tokenizers/smollm2
3941```
4042The download lives at ` data/tokenizers/smollm2/tokenizer.json ` . Use this path in the export and sampling commands below.
@@ -48,90 +50,120 @@ export CXX=/usr/bin/g++-12
4850export CUDAHOSTCXX=$CXX
4951```
5052
51- ## 2. Recommended: FP32 export
52- Produces a stable ` .pte ` for experimentation and sampling.
53+ ## 2. Baseline: full-FP32 model and KV cache
54+ Produces a stable ` .pte ` for experimentation and sampling. KV cache is the
55+ preferred runtime flow because each decode step reuses cached attention state
56+ instead of recomputing the full prompt window.
57+
5358``` bash
5459python -m extension.llm.export.export_llm \
5560 base.model_class=smollm2 \
5661 base.params=examples/models/smollm2/135M_config.json \
5762 base.tokenizer_path=data/tokenizers/smollm2/tokenizer.json \
58- export.output_dir=outputs/$( date +%F) /$( date +%H-%M-%S) _fp32 \
59- export.output_name=smollm2_vgf_fp32_full_logits .pte \
60- export.max_seq_length=64 \
61- export.max_context_length=64 \
63+ export.output_dir=outputs/$( date +%F) /$( date +%H-%M-%S) _kv_fp32 \
64+ export.output_name=smollm2_vgf_kv_fp32 .pte \
65+ export.max_seq_length=128 \
66+ export.max_context_length=128 \
6267 backend.vgf.enabled=True \
63- backend.vgf.compile_spec=TOSA-1.0+FP \
64- model.use_kv_cache=False \
68+ backend.vgf.compile_spec=TOSA-1.0+FP+INT \
69+ model.use_kv_cache=True \
6570 model.enable_dynamic_shape=False \
66- debug.verbose=True \
67- debug.generate_full_logits=True
71+ debug.verbose=True
6872```
6973
74+ ## 3. Recommended quantized path: ` linear16a8w ` with FP32 KV storage
75+ This quantizes ` torch.nn.Linear ` modules with the Arm VGF PT2E quantizer and
76+ keeps the persistent KV cache in FP32. This is the recommended starting point
77+ because static INT8 cache storage has not yet demonstrated a performance
78+ benefit on VGF hardware.
7079
71- ## 3. Experimental: 8-bit PTQ (Linear-only)
72- This quantizes only ` torch.nn.Linear ` modules using the Arm VGF PT2E quantizer.
73-
74- Supported calibration inputs:
75- - ` quantization.calibration_data=@... ` for a text corpus
76- - ` quantization.calibration_tasks=[wikitext] ` for LM-Eval tasks
77-
78- For this static non-KV-cache flow, keep ` debug.generate_full_logits=True ` for
79- calibrated exports. Calibration uses padded fixed-shape prefixes, and full
80- logits let the calibration/eval helpers select the last real-token logits row
81- instead of accidentally using the padded tail.
80+ Example (16-bit activations, 8-bit weights, Linear-only):
8281
83- Example (LM-Eval wikitext calibration):
8482``` bash
8583python -m extension.llm.export.export_llm \
8684 base.model_class=smollm2 \
8785 base.params=examples/models/smollm2/135M_config.json \
8886 base.tokenizer_path=data/tokenizers/smollm2/tokenizer.json \
89- export.output_dir=outputs/$( date +%F) /$( date +%H-%M-%S) _linear8a8w \
90- export.output_name=smollm2_vgf_linear8a8w_wikitext_full_logits .pte \
91- export.max_seq_length=64 \
92- export.max_context_length=64 \
93- quantization.pt2e_quantize=vgf_8a8w \
87+ export.output_dir=outputs/$( date +%F) /$( date +%H-%M-%S) _kv_linear16a8w \
88+ export.output_name=smollm2_vgf_kv_linear16a8w .pte \
89+ export.max_seq_length=128 \
90+ export.max_context_length=128 \
91+ quantization.pt2e_quantize=vgf_16a8w \
9492 quantization.calibration_tasks=\[ wikitext\] \
9593 quantization.calibration_limit=64 \
96- quantization.calibration_seq_length=64 \
94+ quantization.calibration_seq_length=128 \
9795 backend.vgf.enabled=True \
98- backend.vgf.compile_spec=TOSA-1.0+FP+INT \
96+ backend.vgf.compile_spec=TOSA-1.0+FP+INT+int16 \
9997 backend.vgf.quantize_scope=linear \
100- model.use_kv_cache=False \
98+ model.use_kv_cache=True \
10199 model.enable_dynamic_shape=False \
102- debug.verbose=True \
103- debug.generate_full_logits=True
100+ debug.verbose=True
104101```
105102
106- Example (16-bit activations, 8-bit weights, Linear-only):
103+ ### 3.1 Experimental: ` linear16a8w ` with static INT8 KV storage
104+
105+ This path stores the persistent KV cache as calibrated INT8 data. It is
106+ different from ` model.quantize_kv_cache=True ` , which performs dynamic per-token
107+ KV quantization and currently leaves cache QDQ/update maintenance outside the
108+ VGF delegate.
109+
110+ When ` quantization.calibration_tasks ` is set, export first runs the task
111+ through a float KV cache and derives separate symmetric per-head-dimension K/V
112+ scales for every layer. The same task is then reused for normal PT2E Linear
113+ calibration. Without a calibration task, static KV storage uses
114+ ` model.static_quantize_kv_cache_scale ` as a fixed fallback;
115+ ` calibration_data ` alone only calibrates the PT2E operators.
116+
117+ Use the recommended command above with these changes:
118+
119+ ``` bash
120+ export.output_dir=outputs/$( date +%F) /$( date +%H-%M-%S) _static_kvq_linear16a8w \
121+ export.output_name=smollm2_vgf_static_kvq_linear16a8w.pte \
122+ model.static_quantize_kv_cache=True
123+ ```
124+
125+ The static KV-cache path uses symmetric per-head-dimension scales with
126+ zero-point ` 0 ` and dequantizes the cache back to float for the existing
127+ attention path. Treat its accuracy and performance as experimental until they
128+ have been validated for the target hardware and context length.
129+
130+ Alternative ` linear8a8w ` mode:
107131
108132``` bash
109133python -m extension.llm.export.export_llm \
110134 base.model_class=smollm2 \
111135 base.params=examples/models/smollm2/135M_config.json \
112136 base.tokenizer_path=data/tokenizers/smollm2/tokenizer.json \
113- export.output_dir=outputs/$( date +%F) /$( date +%H-%M-%S) _linear16a8w \
114- export.output_name=smollm2_vgf_linear16a8w_wikitext_full_logits .pte \
115- export.max_seq_length=64 \
116- export.max_context_length=64 \
117- quantization.pt2e_quantize=vgf_16a8w \
137+ export.output_dir=outputs/$( date +%F) /$( date +%H-%M-%S) _kv_linear8a8w \
138+ export.output_name=smollm2_vgf_kv_linear8a8w .pte \
139+ export.max_seq_length=128 \
140+ export.max_context_length=128 \
141+ quantization.pt2e_quantize=vgf_8a8w \
118142 quantization.calibration_tasks=\[ wikitext\] \
119143 quantization.calibration_limit=64 \
120- quantization.calibration_seq_length=64 \
144+ quantization.calibration_seq_length=128 \
121145 backend.vgf.enabled=True \
122- backend.vgf.compile_spec=TOSA-1.0+FP+INT+int16 \
146+ backend.vgf.compile_spec=TOSA-1.0+FP+INT \
123147 backend.vgf.quantize_scope=linear \
124- model.use_kv_cache=False \
148+ model.use_kv_cache=True \
125149 model.enable_dynamic_shape=False \
126- debug.verbose=True \
127- debug.generate_full_logits=True
150+ debug.verbose=True
128151```
129152
130153` quantization.pt2e_quantize ` selects the numeric mode.
131154` backend.vgf.quantize_scope=linear ` keeps quantization limited to
132155` torch.nn.Linear ` modules. The compile spec still includes FP because the rest
133156of the graph remains floating point.
134157
158+ ### 3.2 Alternative: static non-KV full-logits exports
159+ Use the static non-KV path when you specifically need full-window logits, for
160+ example faster perplexity evaluation or debugging padded fixed-shape behavior.
161+ For calibrated non-KV exports, keep ` debug.generate_full_logits=True ` ; this lets
162+ the helpers select the last real-token logits row instead of the padded tail.
163+ Set ` model.use_kv_cache=False ` , keep ` export.max_seq_length ` and
164+ ` quantization.calibration_seq_length ` aligned, and add
165+ ` debug.generate_full_logits=True ` to the export commands above.
166+
135167## 4. Sampling with ` executor_runner `
136168
137169### 4.0 Build ` executor_runner ` (VKML)
@@ -149,8 +181,9 @@ CPU kernel coverage.
149181
150182### 4.1 Greedy and ` T=0.8 ` sampling
151183` examples/arm/smollm2_example_vgf/generate_sampled.py ` wraps
152- ` cmake-out-vkml/executor_runner ` , keeps a sliding fixed-length token window,
153- and can print the top-5 logits each step.
184+ ` cmake-out-vkml/executor_runner ` and supports both KV-cache and static non-KV
185+ exports. The recommended KV-cache path uses ` --use-kv-cache ` ; the static
186+ non-KV path uses a fixed token window plus ` --full-logits ` .
154187
155188Greedy generation (` --temperature 0 ` ) always chooses the highest-logit next
156189token, which is useful for deterministic comparisons. Stochastic generation
@@ -160,46 +193,47 @@ with a fixed `--seed`.
160193
161194Notes:
162195- ` --max-seq-length ` must match the export ` export.max_seq_length ` (otherwise you will hit input size mismatch).
163- - The exported SmolLM2 VGF input is ` int32[1, max_seq_length] ` ; the helper writes
164- token windows as ` int32 ` binary inputs for ` executor_runner ` .
165- - Use ` --persistent-runner ` for faster multi-token generation (loads the model once) .
196+ - KV-cache exports take ` token,input_pos ` inputs and should pass ` --use-kv-cache ` .
197+ - Static non-KV exports take an ` int32[1, max_seq_length] ` token window and should pass ` --full-logits ` .
198+ - The KV-cache path keeps ` executor_runner ` alive in server mode automatically .
166199- The documented examples use ` --temperature 0 ` (greedy) and ` --temperature 0.8 ` .
167200- For deterministic comparisons against saved ` temp0 ` outputs, use ` --seed 0 ` , ` --repetition-penalty 1.1 ` , and ` --no-topk-print ` . At ` --temperature 0 ` , token selection is greedy, so ` --top-p ` does not affect the chosen token.
168201
169202Greedy example (` T=0 ` ):
170203``` bash
171204python examples/arm/smollm2_example_vgf/generate_sampled.py \
172- --persistent-runner \
173205 --runner cmake-out-vkml/executor_runner \
174- --pte smollm2_vgf_fp32_full_logits .pte \
206+ --pte smollm2_vgf_kv_linear16a8w .pte \
175207 --tokenizer data/tokenizers/smollm2/tokenizer.json \
176208 --prompt " Once upon a time in a small village," \
177- --max-seq-length 64 \
209+ --max-seq-length 128 \
210+ --max-context-length 128 \
178211 --max-new-tokens 10 \
179212 --seed 0 \
180213 --temperature 0 \
181214 --repetition-penalty 1.1 \
182- --full-logits
215+ --use-kv-cache
183216```
184217
185218Stochastic example (` T=0.8 ` ):
186219``` bash
187220python examples/arm/smollm2_example_vgf/generate_sampled.py \
188- --persistent-runner \
189221 --runner cmake-out-vkml/executor_runner \
190- --pte smollm2_vgf_fp32_full_logits .pte \
222+ --pte smollm2_vgf_kv_linear16a8w .pte \
191223 --tokenizer data/tokenizers/smollm2/tokenizer.json \
192224 --prompt " Once upon a time in a small village," \
193- --max-seq-length 64 \
225+ --max-seq-length 128 \
226+ --max-context-length 128 \
194227 --max-new-tokens 10 \
195228 --seed 0 \
196229 --temperature 0.8 \
197230 --top-p 0.9 \
198231 --repetition-penalty 1.1 \
199- --full-logits
232+ --use-kv-cache
200233```
201- > Swap ` --pte ` to the quantized build to compare behaviour. ` linear8a8w ` still
202- > tends to drift more than ` linear16a8w ` .
234+ > Swap ` --pte ` to ` smollm2_vgf_static_kvq_linear16a8w.pte ` to compare
235+ > experimental static INT8 KV storage, or to ` smollm2_vgf_kv_fp32.pte ` for the
236+ > full-FP32 baseline.
203237
204238
205239
@@ -209,56 +243,73 @@ To generate for *all* prompts in `default_prompts.txt` and save to a file:
209243
210244``` bash
211245python examples/arm/smollm2_example_vgf/generate_sampled.py \
212- --persistent-runner \
213246 --runner cmake-out-vkml/executor_runner \
214- --pte smollm2_vgf_fp32_full_logits .pte \
247+ --pte smollm2_vgf_kv_linear16a8w .pte \
215248 --tokenizer data/tokenizers/smollm2/tokenizer.json \
216249 --prompt-file examples/arm/smollm2_example_vgf/default_prompts.txt \
217250 --prompt-all \
218- --max-seq-length 64 \
251+ --max-seq-length 128 \
252+ --max-context-length 128 \
219253 --max-new-tokens 64 \
220254 --temperature 0.8 \
221255 --top-p 0.9 \
222256 --repetition-penalty 1.1 \
223- --full-logits \
257+ --use-kv-cache \
224258 --save-generations outputs/$( date +%F) /$( date +%H-%M-%S) _smollm2_gen.txt
225259```
226260
227261## 5. Wikitext prompts and perplexity
228262
263+ For a smoke test before a dedicated comparison, run the ` default_prompts.txt `
264+ command above with ` smollm2_vgf_kv_linear16a8w.pte ` and inspect the
265+ generations. To evaluate the experimental static INT8 cache artifact alone,
266+ pass ` smollm2_vgf_static_kvq_linear16a8w.pte ` to any one of the PTE options
267+ below.
268+
229269Build a reusable 1000-prompt file from ` wikitext-2-raw-v1 ` and evaluate
230- perplexity on the first 100 prompts for FP32, ` linear8a8w ` , and ` linear16a8w ` :
270+ perplexity on the first 100 prompts for KV-cache FP32, ` linear8a8w ` , and
271+ ` linear16a8w ` :
231272
232273``` bash
233274OUT_DIR=outputs/$( date +%F) /$( date +%H-%M-%S) _smollm2_vgf_eval
234275
235276python examples/arm/smollm2_example_vgf/eval_wikitext_perplexity.py \
236277 --runner cmake-out-vkml/executor_runner \
237- --pte-fp32 " ${OUT_DIR} /smollm2_vgf_fp32_full_logits .pte" \
238- --pte-linear8a8w " ${OUT_DIR} /smollm2_vgf_linear8a8w_wikitext_full_logits .pte" \
239- --pte-linear16a8w " ${OUT_DIR} /smollm2_vgf_linear16a8w_wikitext_full_logits .pte" \
278+ --pte-fp32 " ${OUT_DIR} /smollm2_vgf_kv_fp32 .pte" \
279+ --pte-linear8a8w " ${OUT_DIR} /smollm2_vgf_kv_linear8a8w .pte" \
280+ --pte-linear16a8w " ${OUT_DIR} /smollm2_vgf_kv_linear16a8w .pte" \
240281 --tokenizer data/tokenizers/smollm2/tokenizer.json \
241282 --prompts-file " ${OUT_DIR} /wikitext_prompts_1000.txt" \
242283 --num-prompts 1000 \
243284 --ppl-prompts 100 \
244- --max-seq-length 64 \
245- --max-prompt-tokens 64 \
285+ --max-seq-length 128 \
286+ --max-prompt-tokens 128 \
287+ --max-tokens-per-prompt 64 \
288+ --use-kv-cache \
246289 --refresh-prompts
247290```
248291
249292Notes:
250293- This script downloads ` wikitext-2-raw-v1 ` via Hugging Face ` datasets ` .
251294- The prompts file is reusable; omit ` --refresh-prompts ` on later runs.
252295- Perplexity is computed on the first 100 prompts from that file.
253- - Each prompt is capped to 64 tokens and scored from one full-logits
254- ` executor_runner ` invocation per prompt, rather than one invocation per token.
296+ - KV-cache prompts are scored step-by-step with ` token ` and ` input_pos `
297+ inputs, so this path is slower but matches the recommended decode contract.
298+ - ` --max-tokens-per-prompt ` controls how many tokens are scored from each
299+ prompt. Use ` 64 ` for comparison with the static-window baseline, or a larger
300+ value to stress a longer KV context.
301+ - For static non-KV PTEs, omit ` --use-kv-cache ` and pass full-logits PTEs; each
302+ prompt is then scored from one full-logits invocation rather than one
303+ invocation per token.
255304
256305## 6. Notes
257- - This flow keeps KV cache disabled and uses a fixed token window. KV-cache
258- support is the expected next step for faster generation, but it is outside
259- this static VGF quickstart.
260- - Without KV cache, the model recomputes the entire token window for each
261- generated token.
306+ - The quickstart exports use KV cache. Static non-KV full-logits exports are
307+ still useful for faster perplexity experiments and fixed-window debugging.
308+ - The recommended quantized export keeps KV storage in FP32. Static INT8 KV
309+ storage remains available for experimental memory and performance studies.
310+ - With KV cache, generation replays one token plus ` input_pos ` per runner
311+ invocation; without KV cache, the model recomputes the full token window for
312+ each generated token.
262313- ` linear8a8w ` still shows noticeably more quality loss than ` linear16a8w ` .
263314- When you change ` max_seq_length ` , regenerate any cached prompt inputs to match the new window size.
264315- On hosts with multiple Vulkan devices, use ` vulkaninfo --summary ` to check
@@ -267,7 +318,9 @@ Notes:
267318### Implementation details
268319- The VKML runner is ` examples/portable/executor_runner/executor_runner.cpp ` ,
269320 built here as ` cmake-out-vkml/executor_runner ` .
270- - ` generate_sampled.py ` tokenizes prompts, prepares the fixed token window,
271- invokes ` executor_runner ` , reads logits, and decodes sampled tokens.
272- - The sampling and perplexity commands pass ` --full-logits ` to match the
273- exported full-logits PTEs.
321+ - ` generate_sampled.py ` tokenizes prompts, prepares either the fixed non-KV
322+ token window or KV-cache ` token,input_pos ` inputs, invokes ` executor_runner ` ,
323+ reads logits, and decodes sampled tokens.
324+ - The non-KV sampling and perplexity commands pass ` --full-logits ` to match the
325+ exported full-logits PTEs. KV-cache PTEs return next-token logits directly
326+ and should use ` --use-kv-cache ` instead.
0 commit comments