Repository navigation
Commit 02bb177
authored
[docs] Clarify LLM generation configuration behavior (pytorch#22939)
### Summary
The public `GenerationConfig` comments imply grammar constraints are
implemented, and the runner README suggests lowering `seq_len` can
reduce allocated memory. Clarify that grammar fields are currently
ignored and generation limits do not resize the KV cache. The C++ guide
also explains how to use settings chosen offline, reset the runner
between independent evaluations, and avoid overriding
`config.temperature` through the deprecated factory argument.
Related to pytorch#22761.
Authored with Codex (OpenAI).
### Test plan
The following checks passed:
```bash
lintrunner docs/source/llm/run-with-c-plus-plus.md extension/llm/runner/README.md extension/llm/runner/irunner.h
git diff --check
```
The changes affect documentation and C++ comments only; no runtime tests
were added or run.
cc @nil-is-all1 parent 1fbe60b commit 02bb177
3 files changed
Lines changed: 22 additions & 6 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
105 | 105 | | |
106 | 106 | | |
107 | 107 | | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
108 | 115 | | |
109 | 116 | | |
110 | 117 | | |
| |||
221 | 228 | | |
222 | 229 | | |
223 | 230 | | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
224 | 239 | | |
225 | 240 | | |
226 | 241 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
799 | 799 | | |
800 | 800 | | |
801 | 801 | | |
802 | | - | |
| 802 | + | |
| 803 | + | |
803 | 804 | | |
804 | 805 | | |
805 | 806 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
36 | | - | |
37 | | - | |
| 36 | + | |
| 37 | + | |
38 | 38 | | |
39 | 39 | | |
40 | | - | |
41 | | - | |
| 40 | + | |
| 41 | + | |
42 | 42 | | |
43 | 43 | | |
44 | 44 | | |
| |||
55 | 55 | | |
56 | 56 | | |
57 | 57 | | |
58 | | - | |
| 58 | + | |
59 | 59 | | |
60 | 60 | | |
61 | 61 | | |
| |||
0 commit comments