Repository navigation
Commit ee45eb8
authored
Add off-graph KV-cache ring policy (pytorch#21447)
Summary
Adds the sliding-window (ring) layout policy. RingPolicy keeps a layer's
last window tokens in a fixed ring of window + max_write − 1 slots —
oversized so a multi-token prefill step fits without clobbering context
earlier queries in the step still need — and evicts the oldest.
SequenceCache dispatches windowed layers to it, so a mixed model
(gemma4's alternating full/sliding-window layers) shares one logical
length. Backend- and tensor-free: it emits integer write/read runs; the
per-query window mask stays in the backend.
Files
- extension/llm/cache/cache.h — LayerPolicy::Kind::Ring; max_write on
CacheConfig (max tokens/step; default 1 = decode-only ring).
- extension/llm/cache/sequence_cache.h — split_ring_runs (wrap into ≤2
runs), RingPolicy over the oversized ring (retained_from still
window-based), the
make_policy ring branch, and a ring-specific T ≤ max_write guard.
- extension/llm/cache/test/cache_test.cpp — ring wrap/evict, mixed
flat/ring shared length, rewind bounded by the window, and
step-over-max_write rejected.
Testing
C++ unit tests for the ring path (wrap, eviction, mixed flat/ring,
window-bounded rewind, max_write guard). Built and ran via the standard
flow, all pass:
cmake -B cmake-out -DEXECUTORCH_BUILD_EXTENSION_LLM=ON
-DEXECUTORCH_BUILD_TESTS=ON
cmake --build cmake-out --target extension_llm_cache_test -j
ctest --test-dir cmake-out -R extension_llm_cache --output-on-failure1 parent 42a178f commit ee45eb8
3 files changed
Lines changed: 159 additions & 24 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
16 | | - | |
17 | | - | |
18 | | - | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
19 | 20 | | |
20 | 21 | | |
21 | 22 | | |
| |||
58 | 59 | | |
59 | 60 | | |
60 | 61 | | |
61 | | - | |
62 | | - | |
63 | | - | |
64 | | - | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
65 | 66 | | |
66 | 67 | | |
67 | 68 | | |
| |||
84 | 85 | | |
85 | 86 | | |
86 | 87 | | |
87 | | - | |
88 | | - | |
| 88 | + | |
| 89 | + | |
89 | 90 | | |
90 | 91 | | |
91 | 92 | | |
92 | 93 | | |
93 | 94 | | |
94 | 95 | | |
95 | | - | |
96 | | - | |
97 | | - | |
| 96 | + | |
| 97 | + | |
98 | 98 | | |
99 | 99 | | |
100 | 100 | | |
101 | 101 | | |
102 | 102 | | |
103 | | - | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
104 | 107 | | |
105 | | - | |
106 | | - | |
| 108 | + | |
107 | 109 | | |
108 | 110 | | |
109 | 111 | | |
| |||
114 | 116 | | |
115 | 117 | | |
116 | 118 | | |
117 | | - | |
118 | | - | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
119 | 124 | | |
120 | 125 | | |
121 | 126 | | |
122 | 127 | | |
123 | 128 | | |
| 129 | + | |
124 | 130 | | |
125 | 131 | | |
126 | 132 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
15 | 15 | | |
16 | 16 | | |
17 | 17 | | |
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
46 | 90 | | |
47 | 91 | | |
48 | | - | |
| 92 | + | |
| 93 | + | |
49 | 94 | | |
50 | 95 | | |
51 | 96 | | |
52 | 97 | | |
53 | | - | |
| 98 | + | |
| 99 | + | |
54 | 100 | | |
55 | 101 | | |
56 | 102 | | |
| |||
105 | 151 | | |
106 | 152 | | |
107 | 153 | | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
108 | 161 | | |
109 | 162 | | |
110 | 163 | | |
| |||
129 | 182 | | |
130 | 183 | | |
131 | 184 | | |
132 | | - | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
133 | 191 | | |
134 | 192 | | |
135 | 193 | | |
136 | 194 | | |
| 195 | + | |
137 | 196 | | |
138 | 197 | | |
139 | 198 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
33 | 33 | | |
34 | 34 | | |
35 | 35 | | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
36 | 39 | | |
37 | 40 | | |
38 | 41 | | |
| |||
66 | 69 | | |
67 | 70 | | |
68 | 71 | | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
69 | 124 | | |
70 | 125 | | |
71 | 126 | | |
| |||
89 | 144 | | |
90 | 145 | | |
91 | 146 | | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
92 | 162 | | |
93 | 163 | | |
94 | 164 | | |
| |||
0 commit comments