Repository navigation
Commit 32f56be
authored
Add off-graph KV-cache SequenceCache and flat policy (pytorch#21412)
**Summary**
Adds the neutral single-sequence cache: a SequenceCache controller that
owns one logical length for the whole model and produces per-layer,
integer-only layout plans, plus its full-history FlatPolicy. Backend and
tensor-free the cache logic stays ET-independent, with a thin adapter
for ExecuTorch callers. The SeqStepPlan and config shapes are already
general for sliding-window policies, but only full-history flat is
implemented here.
**Files**
- extension/llm/cache/cache.h — adds the two neutral faces recovered
from CacheBase without RTTI (SequenceControl runner face;
SequencePlanner backend face), the SeqStepPlan planner→backend handoff,
the LayoutPolicy interface, and per-layer config (LayerPolicy,
LayerConfig, CacheConfig.layers).
- extension/llm/cache/sequence_cache.h — the SequenceCache controller
(single logical length, admission, rewind,
per-layer plan dispatch with policy dedup) and FlatPolicy (full-history
layout). Header-only, ET-independent.
- extension/llm/cache/cache_et.h — thin ExecuTorch adapter mapping the
core's bool/std::optional results to
Error/Result (et::plan → OutOfResources, et::rewind → InvalidArgument).
- extension/llm/cache/cache_registry.h / cache_registry.cpp —
CacheSession gains control() (the typed runner face);
builder registry now keyed by std::pair<backend_id, kind> instead of a
concatenated string.
- extension/llm/cache/test/cache_test.cpp — tests for flat plan
(append/read-all-history), admission/rewind, face
recovery, the registry with a real SequenceCache, and the ET adapter.
**Testing**
Added C++ unit tests covering the flat bookkeeping
(plan/can_extend/rewind/clear), face recovery, the registry with
a real cache, and the ET adapter's result/code mapping. Built and ran
through the standard cmake/ctest flow, all
pass:
cmake -B cmake-out -DEXECUTORCH_BUILD_EXTENSION_LLM=ON
-DEXECUTORCH_BUILD_TESTS=ON
cmake --build cmake-out --target extension_llm_cache_test -j
ctest --test-dir cmake-out -R extension_llm_cache --output-on-failure1 parent 2a64e2d commit 32f56be
6 files changed
Lines changed: 420 additions & 33 deletions
File tree
- extension/llm/cache
- test
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
8 | 8 | | |
9 | 9 | | |
10 | 10 | | |
11 | | - | |
12 | | - | |
13 | | - | |
14 | | - | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
15 | 23 | | |
16 | 24 | | |
17 | 25 | | |
18 | 26 | | |
19 | 27 | | |
20 | 28 | | |
21 | | - | |
22 | | - | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
23 | 35 | | |
24 | 36 | | |
25 | 37 | | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
26 | 114 | | |
27 | 115 | | |
28 | | - | |
29 | | - | |
30 | | - | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
31 | 119 | | |
32 | 120 | | |
33 | 121 | | |
| 122 | + | |
34 | 123 | | |
35 | 124 | | |
36 | 125 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
| 1 | + | |
| 2 | + | |
| 3 | + | |
| 4 | + | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
| 18 | + | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
50 | 50 | | |
51 | 51 | | |
52 | 52 | | |
53 | | - | |
| 53 | + | |
54 | 54 | | |
55 | 55 | | |
56 | 56 | | |
| |||
60 | 60 | | |
61 | 61 | | |
62 | 62 | | |
63 | | - | |
| 63 | + | |
64 | 64 | | |
65 | 65 | | |
66 | 66 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
16 | | - | |
17 | | - | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
18 | 18 | | |
19 | 19 | | |
| 20 | + | |
20 | 21 | | |
21 | 22 | | |
22 | 23 | | |
23 | 24 | | |
| 25 | + | |
24 | 26 | | |
25 | 27 | | |
26 | 28 | | |
| |||
76 | 78 | | |
77 | 79 | | |
78 | 80 | | |
79 | | - | |
80 | | - | |
| 81 | + | |
| 82 | + | |
81 | 83 | | |
82 | 84 | | |
83 | 85 | | |
| |||
86 | 88 | | |
87 | 89 | | |
88 | 90 | | |
89 | | - | |
| 91 | + | |
90 | 92 | | |
91 | 93 | | |
92 | 94 | | |
| |||
100 | 102 | | |
101 | 103 | | |
102 | 104 | | |
103 | | - | |
104 | | - | |
| 105 | + | |
| 106 | + | |
105 | 107 | | |
106 | 108 | | |
107 | 109 | | |
| |||
0 commit comments