Repository navigation
Commit e60faa2
authored
Fix off-graph KV-cache causal alignment: add explicit masks (pytorch#21749)
**Summary**
Adds an EXPLICIT mask kind to the eager KV-cache reference: AttendSpec
can now carry a bool mask (true = attend) for the patterns the fused
kinds can't express, which is what the cell and tree caches need. Its
first user is the contiguous cache's continuation step, and that fixes a
silent bug — the reference declared CAUSAL for every multi-token step
and attend() applied torch's is_causal, which is upper-left aligned when
the query and key lengths differ, while the design's causal is
lower-right (new tokens at the tail of the read window, as MLX's
"causal"). So a chunked or multi-turn prefill hid the prior context: the
second chunk's first query attended key 0 alone instead of the whole
history. The runtime was already correct; MLX aligns "causal"
lower-right and MLXSequenceCache returns it; so this is the oracle only,
and it would have surfaced as MLX disagreeing with the reference on any
chat turn after the first.
**Files**
- extension/llm/cache/reference_cache.py — MaskKind gains EXPLICIT and
AttendSpec an optional mask, bool with true = attend, broadcastable to
[B, H_q, q_len, total]. The kind choice moves out of update_and_fetch
into _spec: NONE for a decode, CAUSAL for a fresh square prefill,
EXPLICIT with a lower-right band for a continuation, where torch offers
no fused lower-right causal. attend() applies one mechanism per kind and
raises on a non-square CAUSAL rather than mis-aligning it.
- extension/llm/cache/test_update_and_attend.py — six cases.
**Testing**
Six cases added to the existing unittest suite (11 total):
- a two-chunk prefill matches the cacheless baseline end-to-end, the
case the merged tests could not reach since
prefill-from-empty is square and decode is maskless
- the kind declared for each step shape: NONE for a decode, CAUSAL for a
fresh prefill, EXPLICIT for a continuation
- the continuation band's exact contents, against tril(total - q_len)
- attend() rejects a non-square CAUSAL instead of silently applying the
wrong alignment
- an explicit mask attends exactly its true cells, checked against
attending that subset unmasked, under GQA
pytest extension/llm/cache/test_update_and_attend.py -q1 parent 161d9f9 commit e60faa2
2 files changed
Lines changed: 128 additions & 7 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
27 | 27 | | |
28 | 28 | | |
29 | 29 | | |
30 | | - | |
| 30 | + | |
31 | 31 | | |
32 | 32 | | |
33 | 33 | | |
| |||
43 | 43 | | |
44 | 44 | | |
45 | 45 | | |
| 46 | + | |
46 | 47 | | |
47 | 48 | | |
48 | 49 | | |
49 | 50 | | |
50 | 51 | | |
| 52 | + | |
51 | 53 | | |
52 | 54 | | |
53 | 55 | | |
| |||
150 | 152 | | |
151 | 153 | | |
152 | 154 | | |
153 | | - | |
154 | | - | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
155 | 172 | | |
156 | 173 | | |
157 | 174 | | |
| |||
165 | 182 | | |
166 | 183 | | |
167 | 184 | | |
168 | | - | |
169 | | - | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
170 | 190 | | |
171 | 191 | | |
172 | 192 | | |
173 | 193 | | |
174 | 194 | | |
175 | | - | |
| 195 | + | |
| 196 | + | |
176 | 197 | | |
177 | 198 | | |
178 | 199 | | |
| |||
186 | 207 | | |
187 | 208 | | |
188 | 209 | | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
189 | 217 | | |
190 | 218 | | |
191 | 219 | | |
192 | 220 | | |
193 | | - | |
| 221 | + | |
| 222 | + | |
194 | 223 | | |
195 | 224 | | |
196 | 225 | | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
9 | 9 | | |
10 | 10 | | |
11 | 11 | | |
| 12 | + | |
| 13 | + | |
12 | 14 | | |
13 | 15 | | |
14 | 16 | | |
| 17 | + | |
15 | 18 | | |
16 | 19 | | |
17 | 20 | | |
| |||
205 | 208 | | |
206 | 209 | | |
207 | 210 | | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
208 | 232 | | |
209 | 233 | | |
210 | 234 | | |
| |||
234 | 258 | | |
235 | 259 | | |
236 | 260 | | |
| 261 | + | |
| 262 | + | |
| 263 | + | |
| 264 | + | |
| 265 | + | |
| 266 | + | |
| 267 | + | |
| 268 | + | |
| 269 | + | |
| 270 | + | |
| 271 | + | |
| 272 | + | |
| 273 | + | |
| 274 | + | |
| 275 | + | |
| 276 | + | |
| 277 | + | |
| 278 | + | |
| 279 | + | |
| 280 | + | |
| 281 | + | |
| 282 | + | |
| 283 | + | |
| 284 | + | |
| 285 | + | |
| 286 | + | |
| 287 | + | |
| 288 | + | |
| 289 | + | |
| 290 | + | |
| 291 | + | |
| 292 | + | |
| 293 | + | |
| 294 | + | |
| 295 | + | |
| 296 | + | |
| 297 | + | |
| 298 | + | |
| 299 | + | |
| 300 | + | |
| 301 | + | |
| 302 | + | |
| 303 | + | |
| 304 | + | |
| 305 | + | |
| 306 | + | |
| 307 | + | |
| 308 | + | |
| 309 | + | |
| 310 | + | |
| 311 | + | |
| 312 | + | |
| 313 | + | |
| 314 | + | |
| 315 | + | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
237 | 329 | | |
238 | 330 | | |
0 commit comments