Skip to content

Commit 02bb177

Browse files
authored
[docs] Clarify LLM generation configuration behavior (pytorch#22939)
### Summary The public `GenerationConfig` comments imply grammar constraints are implemented, and the runner README suggests lowering `seq_len` can reduce allocated memory. Clarify that grammar fields are currently ignored and generation limits do not resize the KV cache. The C++ guide also explains how to use settings chosen offline, reset the runner between independent evaluations, and avoid overriding `config.temperature` through the deprecated factory argument. Related to pytorch#22761. Authored with Codex (OpenAI). ### Test plan The following checks passed: ```bash lintrunner docs/source/llm/run-with-c-plus-plus.md extension/llm/runner/README.md extension/llm/runner/irunner.h git diff --check ``` The changes affect documentation and C++ comments only; no runtime tests were added or run. cc @nil-is-all
1 parent 1fbe60b commit 02bb177

3 files changed

Lines changed: 22 additions & 6 deletions

File tree

‎docs/source/llm/run-with-c-plus-plus.md‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -105,6 +105,13 @@ int main() {
105105
}
106106
```
107107
108+
Applications can choose generation settings offline, for example by comparing
109+
quality and latency on representative prompts, then populate the same
110+
`GenerationConfig` shown above. The selection tool does not need to be a runtime
111+
dependency. Validate the selected values with the exported `.pte`, matching
112+
tokenizer, and intended backend and device. Call `runner->reset()` between
113+
independent evaluations so earlier prompts do not carry over in the KV cache.
114+
108115
## The Runner API Architecture
109116
110117
The ExecuTorch LLM runner library is designed with a modular architecture that separates concerns between different components of the text generation pipeline.
@@ -221,6 +228,14 @@ The `resolve_max_new_tokens` method handles the logic of determining how many to
221228
- The number of token positions already occupied in the context window
222229
- The user-specified maximum sequence length and maximum new tokens
223230
231+
`max_new_tokens` and `seq_len` limit generation within the exported model's
232+
context capacity. They do not change the model's exported shapes or resize its
233+
allocated KV cache. To change that capacity, configure it during model export.
234+
235+
When setting `config.temperature`, leave the deprecated `temperature` argument
236+
to `create_text_llm_runner()` at its default of `-1.0f`. An explicit value for
237+
that argument overrides `config.temperature` during decoding.
238+
224239
### Implementation Components
225240
226241
The runner library consists of several specialized components that work together:

‎extension/llm/runner/README.md‎

Lines changed: 2 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -799,7 +799,8 @@ either field does not constrain generated output.
799799
**Memory Optimization**:
800800
- Use KV cache for efficient autoregressive generation
801801
- Enable parallel prefill for faster prompt processing
802-
- Set appropriate `seq_len` based on available memory
802+
- Choose the exported context capacity based on available memory; lowering
803+
`GenerationConfig::seq_len` does not resize the allocated KV cache
803804

804805
**Sampling Strategies**:
805806
- Low temperature (0.1-0.3) for factual, deterministic output

‎extension/llm/runner/irunner.h‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -33,12 +33,12 @@ struct GenerationConfig {
3333
// Whether to echo the input prompt in the output
3434
bool echo = true;
3535

36-
// Grammar definition for constrained decoding (e.g. a JSON schema, regex,
37-
// Lark CFG, or GBNF grammar). Empty string means no constraint.
36+
// Reserved grammar definition for constrained decoding.
37+
// Currently ignored by in-tree runners.
3838
std::string grammar;
3939

40-
// Grammar format: "json_schema", "regex", "lark", or "gbnf".
41-
// Only used when grammar is non-empty.
40+
// Reserved grammar format (e.g. "json_schema", "regex", "lark", or "gbnf").
41+
// Currently ignored by in-tree runners.
4242
std::string grammar_type;
4343

4444
// Whether to ignore EOS token and continue generating until max_new_tokens
@@ -55,7 +55,7 @@ struct GenerationConfig {
5555
// Whether this is a warmup run (affects perf benchmarking)
5656
bool warming = false;
5757

58-
// Maximum number of total tokens
58+
// Maximum number of total tokens. Does not resize the allocated KV cache.
5959
// If the .pte file contains the max_context_len metadata, it will override
6060
// this value if it's smaller. If this field is -1, we will use the
6161
// max_context_len metadata directly. Check resolve_max_new_tokens for

0 commit comments

Comments
 (0)