Skip to content

Commit e60faa2

Browse files
Fix off-graph KV-cache causal alignment: add explicit masks (pytorch#21749)
**Summary** Adds an EXPLICIT mask kind to the eager KV-cache reference: AttendSpec can now carry a bool mask (true = attend) for the patterns the fused kinds can't express, which is what the cell and tree caches need. Its first user is the contiguous cache's continuation step, and that fixes a silent bug — the reference declared CAUSAL for every multi-token step and attend() applied torch's is_causal, which is upper-left aligned when the query and key lengths differ, while the design's causal is lower-right (new tokens at the tail of the read window, as MLX's "causal"). So a chunked or multi-turn prefill hid the prior context: the second chunk's first query attended key 0 alone instead of the whole history. The runtime was already correct; MLX aligns "causal" lower-right and MLXSequenceCache returns it; so this is the oracle only, and it would have surfaced as MLX disagreeing with the reference on any chat turn after the first. **Files** - extension/llm/cache/reference_cache.py — MaskKind gains EXPLICIT and AttendSpec an optional mask, bool with true = attend, broadcastable to [B, H_q, q_len, total]. The kind choice moves out of update_and_fetch into _spec: NONE for a decode, CAUSAL for a fresh square prefill, EXPLICIT with a lower-right band for a continuation, where torch offers no fused lower-right causal. attend() applies one mechanism per kind and raises on a non-square CAUSAL rather than mis-aligning it. - extension/llm/cache/test_update_and_attend.py — six cases. **Testing** Six cases added to the existing unittest suite (11 total): - a two-chunk prefill matches the cacheless baseline end-to-end, the case the merged tests could not reach since prefill-from-empty is square and decode is maskless - the kind declared for each step shape: NONE for a decode, CAUSAL for a fresh prefill, EXPLICIT for a continuation - the continuation band's exact contents, against tril(total - q_len) - attend() rejects a non-square CAUSAL instead of silently applying the wrong alignment - an explicit mask attends exactly its true cells, checked against attending that subset unmasked, under GQA pytest extension/llm/cache/test_update_and_attend.py -q
1 parent 161d9f9 commit e60faa2

2 files changed

Lines changed: 128 additions & 7 deletions

File tree

‎extension/llm/cache/reference_cache.py‎

Lines changed: 36 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,7 @@
2727

2828
from dataclasses import dataclass
2929
from enum import Enum
30-
from typing import List, Tuple
30+
from typing import List, Optional, Tuple
3131

3232
import torch
3333
import torch.nn.functional as F
@@ -43,11 +43,13 @@ class CacheSizing(Enum):
4343
class MaskKind(Enum):
4444
NONE = "none" # decode: q_len == 1, the single query sees all of history
4545
CAUSAL = "causal" # prefill/continuation: query i sees keys up to its position
46+
EXPLICIT = "explicit" # anything the other two cannot express: a mask tensor
4647

4748

4849
@dataclass
4950
class AttendSpec:
5051
kind: MaskKind
52+
mask: Optional[torch.Tensor] = None # EXPLICIT only: bool, true = attend
5153

5254

5355
@experimental(
@@ -150,8 +152,23 @@ def update_and_fetch(
150152
v_hist = self._v[layer_id]
151153
self._used[layer_id] = new_used
152154

153-
kind = MaskKind.NONE if q_len == 1 else MaskKind.CAUSAL
154-
return k_hist, v_hist, AttendSpec(kind=kind)
155+
return k_hist, v_hist, self._spec(q_len, new_used, k.device)
156+
157+
@staticmethod
158+
def _spec(q_len: int, total: int, device: torch.device) -> AttendSpec:
159+
"""The mask semantic for q_len new cells at the tail of a total window."""
160+
if q_len == 1:
161+
return AttendSpec(kind=MaskKind.NONE)
162+
if q_len == total:
163+
return AttendSpec(kind=MaskKind.CAUSAL)
164+
# Continuation (chunked / multi-turn): the new cells are at the tail, so
165+
# query i attends keys up to i + total - q_len. A backend whose causal is
166+
# lower-right aligned fuses that as CAUSAL; torch's is_causal
167+
# is upper-left, so the reference hands back the band itself.
168+
offsets = torch.arange(total, device=device) - torch.arange(
169+
q_len, device=device
170+
).unsqueeze(-1)
171+
return AttendSpec(kind=MaskKind.EXPLICIT, mask=offsets <= total - q_len)
155172

156173

157174
def attend(
@@ -165,14 +182,18 @@ def attend(
165182
"""Eager attend mechanism: SDPA over fetched K/V per the mask semantic.
166183
167184
Repeats K/V heads for GQA/MQA (``H_q`` a multiple of ``H_kv``), casts to fp32,
168-
and calls ``F.scaled_dot_product_attention`` -- causal for CAUSAL (the cache is
169-
contiguous, so causal alignment matches the prior length), unmasked for NONE.
185+
and calls ``F.scaled_dot_product_attention`` -- one mechanism per kind:
186+
unmasked for NONE, ``is_causal`` for CAUSAL, the cache's own bool mask for
187+
EXPLICIT. CAUSAL is square-only here. The design's causal is lower-right
188+
aligned, which torch's upper-left ``is_causal`` matches only on a fresh full prefill,
189+
so a cache must declare EXPLICIT for a chunked or multi-turn step.
170190
171191
Args (BHSD):
172192
q: ``[B, H_q, q_len, head_dim]`` -- queries (already RoPE-rotated).
173193
k: ``[B, H_kv, total, head_dim]`` -- key history.
174194
v: ``[B, H_kv, total, v_head_dim]`` -- value history.
175-
spec: mask semantic (NONE = attend all; CAUSAL = causal).
195+
spec: mask semantic (NONE = attend all; CAUSAL = causal; EXPLICIT = the
196+
spec's bool mask).
176197
scale: attention softmax scale.
177198
out_dtype: output dtype.
178199
@@ -186,11 +207,19 @@ def attend(
186207
k = k.repeat_interleave(rep, dim=1)
187208
v = v.repeat_interleave(rep, dim=1)
188209

210+
if spec.kind == MaskKind.CAUSAL and q.shape[-2] != k.shape[-2]:
211+
raise ValueError(
212+
"CAUSAL over a non-square window: torch's is_causal is upper-left "
213+
"aligned and would hide the prior cells. The cache must declare "
214+
"EXPLICIT with a lower-right band for a continuation."
215+
)
216+
189217
out = F.scaled_dot_product_attention(
190218
q.to(torch.float32),
191219
k.to(torch.float32),
192220
v.to(torch.float32),
193-
is_causal=spec.kind != MaskKind.NONE,
221+
attn_mask=spec.mask if spec.kind == MaskKind.EXPLICIT else None,
222+
is_causal=spec.kind == MaskKind.CAUSAL,
194223
scale=scale,
195224
)
196225
return out.to(out_dtype)

‎extension/llm/cache/test_update_and_attend.py‎

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -9,9 +9,12 @@
99
import torch
1010
import torch.nn.functional as F
1111
from executorch.extension.llm.cache.reference_cache import (
12+
attend,
13+
AttendSpec,
1214
CacheConfig,
1315
CacheSizing,
1416
ContiguousReferenceCache,
17+
MaskKind,
1518
)
1619
from executorch.extension.llm.cache.update_and_attend import REGISTRY, update_and_attend
1720

@@ -205,6 +208,27 @@ def test_incremental_decode_matches_baseline(self):
205208
out[:, 0, :], ref[:, step, :], atol=1e-4, rtol=1e-4
206209
)
207210

211+
def test_chunked_prefill_matches_baseline(self):
212+
# Prefill in chunks: each chunk's queries must attend every earlier
213+
# chunk's keys, which holds only if CAUSAL is lower-right aligned.
214+
chunk, total = 3, 6
215+
x = torch.randn(1, total, self.hidden)
216+
ref = self.model.reference_forward(x, torch.arange(total))
217+
218+
ep = self._export(chunk)
219+
cache = ContiguousReferenceCache(self._config(CacheSizing.DYNAMIC, total))
220+
REGISTRY.install(self.cache_key, cache)
221+
with REGISTRY.active(self.cache_key):
222+
for start in range(0, total, chunk):
223+
out = ep.module()(
224+
x[:, start : start + chunk, :],
225+
_positions(start, chunk),
226+
torch.arange(chunk),
227+
)
228+
torch.testing.assert_close(
229+
out, ref[:, total - chunk :, :], atol=1e-4, rtol=1e-4
230+
)
231+
208232
def test_static_overflow_raises(self):
209233
ep = self._export(seq_len=5)
210234
cache = ContiguousReferenceCache(self._config(CacheSizing.STATIC, capacity=3))
@@ -234,5 +258,73 @@ def forward(self, q, k, v, position):
234258
self.assertEqual(tuple(node.meta["val"].shape), (1, 4, 3, 5))
235259

236260

261+
class ContiguousSpecTest(unittest.TestCase):
262+
# Which mask semantic the cache declares for each shape of step.
263+
264+
def setUp(self):
265+
torch.manual_seed(0)
266+
self.cache = ContiguousReferenceCache(
267+
CacheConfig(n_layers=1, n_kv_heads=2, head_dim=4, capacity=8)
268+
)
269+
270+
def _step(self, start, q_len):
271+
kv = torch.randn(1, 2, q_len, 4)
272+
return self.cache.update_and_fetch(0, kv, kv, _positions(start, q_len))[2]
273+
274+
def test_decode_is_unmasked(self):
275+
self.assertEqual(self._step(0, 1).kind, MaskKind.NONE)
276+
277+
def test_fresh_prefill_is_causal(self):
278+
self.assertEqual(self._step(0, 4).kind, MaskKind.CAUSAL)
279+
280+
def test_continuation_is_an_explicit_lower_right_band(self):
281+
self._step(0, 4)
282+
spec = self._step(4, 3) # 3 new cells at the tail of a 7-cell window
283+
self.assertEqual(spec.kind, MaskKind.EXPLICIT)
284+
self.assertEqual(spec.mask.dtype, torch.bool)
285+
torch.testing.assert_close(
286+
spec.mask, torch.ones(3, 7, dtype=torch.bool).tril(7 - 3)
287+
)
288+
289+
290+
class AttendExplicitTest(unittest.TestCase):
291+
# No cache emits MaskKind.EXPLICIT yet; these pin the spec's contract (bool,
292+
# true = attend, broadcast over batch/heads) that cell/tree caches rely on.
293+
294+
def setUp(self):
295+
torch.manual_seed(0)
296+
self.q_len, self.total, self.head_dim = 3, 5, 8
297+
self.q = torch.randn(1, 4, self.q_len, self.head_dim) # GQA: 4 q heads
298+
self.k = torch.randn(1, 2, self.total, self.head_dim) # over 2 kv heads
299+
self.v = torch.randn(1, 2, self.total, self.head_dim)
300+
self.scale = self.head_dim**-0.5
301+
302+
def _attend(self, spec, k=None, v=None):
303+
k = self.k if k is None else k
304+
v = self.v if v is None else v
305+
return attend(self.q, k, v, spec, self.scale, torch.float32)
306+
307+
def test_causal_rejects_a_non_square_window(self):
308+
# torch's is_causal is upper-left, so it cannot serve a continuation;
309+
# a cache must declare EXPLICIT there rather than CAUSAL.
310+
with self.assertRaises(ValueError):
311+
self._attend(AttendSpec(kind=MaskKind.CAUSAL))
312+
313+
def test_explicit_attends_the_true_cells(self):
314+
# Polarity: masking to cells {0, 2} must equal attending over just those
315+
# keys. The inverted convention would instead select {1, 3, 4}.
316+
keep = torch.tensor([0, 2])
317+
mask = torch.zeros(self.q_len, self.total, dtype=torch.bool)
318+
mask[:, keep] = True
319+
torch.testing.assert_close(
320+
self._attend(AttendSpec(kind=MaskKind.EXPLICIT, mask=mask)),
321+
self._attend(
322+
AttendSpec(kind=MaskKind.NONE),
323+
self.k.index_select(2, keep),
324+
self.v.index_select(2, keep),
325+
),
326+
)
327+
328+
237329
if __name__ == "__main__":
238330
unittest.main()

0 commit comments

Comments
 (0)