Skip to content

Commit 32f56be

Browse files
Add off-graph KV-cache SequenceCache and flat policy (pytorch#21412)
**Summary** Adds the neutral single-sequence cache: a SequenceCache controller that owns one logical length for the whole model and produces per-layer, integer-only layout plans, plus its full-history FlatPolicy. Backend and tensor-free the cache logic stays ET-independent, with a thin adapter for ExecuTorch callers. The SeqStepPlan and config shapes are already general for sliding-window policies, but only full-history flat is implemented here. **Files** - extension/llm/cache/cache.h — adds the two neutral faces recovered from CacheBase without RTTI (SequenceControl runner face; SequencePlanner backend face), the SeqStepPlan planner→backend handoff, the LayoutPolicy interface, and per-layer config (LayerPolicy, LayerConfig, CacheConfig.layers). - extension/llm/cache/sequence_cache.h — the SequenceCache controller (single logical length, admission, rewind, per-layer plan dispatch with policy dedup) and FlatPolicy (full-history layout). Header-only, ET-independent. - extension/llm/cache/cache_et.h — thin ExecuTorch adapter mapping the core's bool/std::optional results to Error/Result (et::plan → OutOfResources, et::rewind → InvalidArgument). - extension/llm/cache/cache_registry.h / cache_registry.cpp — CacheSession gains control() (the typed runner face); builder registry now keyed by std::pair<backend_id, kind> instead of a concatenated string. - extension/llm/cache/test/cache_test.cpp — tests for flat plan (append/read-all-history), admission/rewind, face recovery, the registry with a real SequenceCache, and the ET adapter. **Testing** Added C++ unit tests covering the flat bookkeeping (plan/can_extend/rewind/clear), face recovery, the registry with a real cache, and the ET adapter's result/code mapping. Built and ran through the standard cmake/ctest flow, all pass: cmake -B cmake-out -DEXECUTORCH_BUILD_EXTENSION_LLM=ON -DEXECUTORCH_BUILD_TESTS=ON cmake --build cmake-out --target extension_llm_cache_test -j ctest --test-dir cmake-out -R extension_llm_cache --output-on-failure
1 parent 2a64e2d commit 32f56be

6 files changed

Lines changed: 420 additions & 33 deletions

File tree

‎extension/llm/cache/cache.h‎

Lines changed: 98 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -8,29 +8,118 @@
88

99
#pragma once
1010

11-
// Neutral, ET-independent ownership handle for an off-graph KV cache. The
12-
// registry owns a cache as a CacheBase* -- an opaque, cache-agnostic anchor --
13-
// and hands it back by key. The typed faces a runner and backend use to drive
14-
// the cache are added by the concrete cache implementation.
11+
// Neutral, tensor-free, ET-independent KV-cache core shared across backends. A
12+
// cache exposes two faces recovered from the owning CacheBase* via
13+
// as_control()/as_planner() (static upcasts -- no dynamic_cast/RTTI, no
14+
// diamond): a runner-facing control face (SequenceControl) and a backend-facing
15+
// planner face (SequencePlanner). One controller (SequenceCache) drives all
16+
// layers and dispatches per-layer layout to a LayoutPolicy (full-history flat
17+
// today), so a multi-layer model shares one logical length. Errors are plain
18+
// C++ (bool / std::optional); cache_et.h adapts them to Error/Result for ET
19+
// consumers.
20+
21+
#include <optional>
22+
#include <vector>
1523

1624
namespace executorch {
1725
namespace extension {
1826
namespace llm {
1927
namespace cache {
2028

21-
// Ownership / erasure anchor: the registry owns and deletes a cache through
22-
// this base, staying agnostic to the concrete cache type.
29+
class SequenceControl;
30+
class SequencePlanner;
31+
32+
// Registry ownership / erasure anchor. The registry owns a cache as a
33+
// CacheBase*; the runner recovers the control face and the backend recovers the
34+
// planner face through these accessors (each concrete cache returns `this`).
2335
class CacheBase {
2436
public:
2537
virtual ~CacheBase() = default;
38+
virtual SequenceControl* as_control() = 0;
39+
virtual SequencePlanner* as_planner() = 0;
40+
};
41+
42+
// Application (runner) face: lifecycle + admission, tensor-free.
43+
class SequenceControl {
44+
public:
45+
virtual ~SequenceControl() = default;
46+
virtual bool can_extend(int n = 1) const = 0; // admission / hard-stop
47+
virtual int capacity() const = 0; // logical cap
48+
// Truncate to new_len (agent backtracking); false = cannot grow, or the
49+
// target is older than an evicting layer still retains.
50+
virtual bool rewind(int new_len) = 0;
51+
virtual void clear() = 0; // reset for reuse
52+
};
53+
54+
// A contiguous span of physical rows in a layer's pool.
55+
struct Run {
56+
int start;
57+
int len;
58+
};
59+
60+
// Integer-only handoff from the planner to the backend byte layer. Runs are in
61+
// logical order (oldest -> newest); a flat layer uses one run, a windowing
62+
// layer up to two (a write/read that wraps its buffer splits in two).
63+
// read_base_pos is the logical position of read[0].start (0 for flat), so the
64+
// backend can align RoPE / the attention mask.
65+
struct SeqStepPlan {
66+
Run write[2];
67+
int n_write;
68+
Run read[2];
69+
int n_read;
70+
int read_base_pos;
71+
};
72+
73+
// Backend face. plan() is pure -- it computes a layer's layout for a step
74+
// without changing state; commit() advances the shared logical length once the
75+
// step is accepted. `layer` selects the layer's policy. nullopt = the step
76+
// would exceed capacity or `layer` is out of range.
77+
class SequencePlanner {
78+
public:
79+
virtual ~SequencePlanner() = default;
80+
virtual std::optional<SeqStepPlan> plan(int layer, int position, int T)
81+
const = 0;
82+
// Advance the logical length past this step. Idempotent (commits the max), so
83+
// calling it once per step -- not per layer -- suffices.
84+
virtual void commit(const SeqStepPlan& plan) = 0;
85+
};
86+
87+
// Per-layer layout behavior (e.g. full-history flat). Pure: plan() has no side
88+
// effects, so the controller (SequenceCache) owns length.
89+
class LayoutPolicy {
90+
public:
91+
virtual ~LayoutPolicy() = default;
92+
// Write/read runs for T cells at logical `position`. Precondition: T fits the
93+
// policy's window (the runner chunks prefill so a step fits).
94+
virtual SeqStepPlan plan(int position, int T) const = 0;
95+
// Oldest logical position still retained given the current length (0 for a
96+
// full-history policy; a windowing policy retains only its last window). Used
97+
// to bound rewind.
98+
virtual int retained_from(int length) const = 0;
99+
};
100+
101+
// Per-layer cache kind and its parameters.
102+
struct LayerPolicy {
103+
enum class Kind : int { Flat = 0 }; // serialized values: append-only
104+
Kind kind = Kind::Flat;
105+
int window =
106+
0; // sliding-window size for windowing policies; 0 = full history
107+
};
108+
109+
// Per-layer architecture facts + cache policy.
110+
struct LayerConfig {
111+
LayerPolicy policy; // default Flat
112+
int n_kv_heads;
113+
int head_dim;
26114
};
27115

28-
// Model facts a cache factory builds from. capacity is the logical cap;
29-
// n_layers is the number of attention layers; initial_capacity is the byte
30-
// layer's starting pool size before it grows (by doubling) toward capacity.
116+
// Model facts + runtime policy the byte layer sizes its pools from. capacity is
117+
// the logical cap; initial_capacity tunes the byte layer's lazy-doubling pool.
118+
// `layers` is per-layer: size 1 == uniform across all layers, else == n_layers.
31119
struct CacheConfig {
32120
int capacity;
33121
int n_layers;
122+
std::vector<LayerConfig> layers;
34123
int initial_capacity = 512;
35124
};
36125

‎extension/llm/cache/cache_et.h‎

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
/*
2+
* Copyright (c) Meta Platforms, Inc. and affiliates.
3+
* All rights reserved.
4+
*
5+
* This source code is licensed under the BSD-style license found in the
6+
* LICENSE file in the root directory of this source tree.
7+
*/
8+
9+
#pragma once
10+
11+
// ExecuTorch adapter for the neutral cache core. The core (cache.h /
12+
// sequence_cache.h) is ET-independent and reports failures as
13+
// bool/std::optional so it is usable outside an ET runner. These thin inline
14+
// adapters map those results to ExecuTorch Error/Result (logging on failure)
15+
// for ET consumers -- the runner and the delegate byte layer. (The registry is
16+
// delegate-specific and already returns Result directly, so it needs no
17+
// adapter.)
18+
19+
#include <optional>
20+
21+
#include <executorch/extension/llm/cache/cache.h>
22+
#include <executorch/runtime/core/error.h>
23+
#include <executorch/runtime/core/result.h>
24+
25+
namespace executorch {
26+
namespace extension {
27+
namespace llm {
28+
namespace cache {
29+
namespace et {
30+
31+
using ::executorch::runtime::Error;
32+
using ::executorch::runtime::Result;
33+
34+
// Plan a layer's step, or OutOfResources if it would exceed capacity (or the
35+
// layer is out of range).
36+
inline Result<SeqStepPlan>
37+
plan(const SequencePlanner& planner, int layer, int position, int T) {
38+
std::optional<SeqStepPlan> p = planner.plan(layer, position, T);
39+
ET_CHECK_OR_RETURN_ERROR(
40+
p.has_value(),
41+
OutOfResources,
42+
"cache: plan(layer=%d, position=%d, T=%d) exceeds capacity or bad layer",
43+
layer,
44+
position,
45+
T);
46+
return *p;
47+
}
48+
49+
// Truncate the history, or InvalidArgument if new_len would grow it (or is
50+
// older than an evicting layer retains).
51+
inline Error rewind(SequenceControl& control, int new_len) {
52+
ET_CHECK_OR_RETURN_ERROR(
53+
control.rewind(new_len),
54+
InvalidArgument,
55+
"rewind: cannot grow to %d",
56+
new_len);
57+
return Error::Ok;
58+
}
59+
60+
} // namespace et
61+
} // namespace cache
62+
} // namespace llm
63+
} // namespace extension
64+
} // namespace executorch

‎extension/llm/cache/cache_registry.cpp‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,7 @@ void CacheBuilderRegistry::register_builder(
5050
const std::string& kind,
5151
CacheBuilder builder) {
5252
std::lock_guard<std::mutex> lock(mu_);
53-
builders_[backend_id + ":" + kind] = std::move(builder);
53+
builders_[{backend_id, kind}] = std::move(builder);
5454
}
5555

5656
Result<std::shared_ptr<CacheBase>> CacheBuilderRegistry::build(
@@ -60,7 +60,7 @@ Result<std::shared_ptr<CacheBase>> CacheBuilderRegistry::build(
6060
CacheBuilder builder;
6161
{
6262
std::lock_guard<std::mutex> lock(mu_);
63-
const auto it = builders_.find(backend_id + ":" + kind);
63+
const auto it = builders_.find({backend_id, kind});
6464
ET_CHECK_OR_RETURN_ERROR(
6565
it != builders_.end(),
6666
NotFound,

‎extension/llm/cache/cache_registry.h‎

Lines changed: 10 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -12,15 +12,17 @@
1212
// is opaque to the host, so the runner (which knows the cache kind) creates the
1313
// cache and binds it to the delegate through a process-global registry; the two
1414
// sides rendezvous on a cache_key passed as a runtime backend-load option.
15-
// Caches are owned as CacheBase*, keeping the registry agnostic to the concrete
16-
// cache type. This layer is delegate-specific and may use ExecuTorch
17-
// Error/Result directly; the cache handle (cache.h) stays ET-free.
15+
// Caches are owned as CacheBase* and the faces are recovered via
16+
// as_control()/as_planner() (no RTTI). This layer is delegate-specific and may
17+
// use ExecuTorch Error/Result directly; the cache core (cache.h) stays ET-free.
1818

1919
#include <functional>
20+
#include <map>
2021
#include <memory>
2122
#include <mutex>
2223
#include <string>
2324
#include <unordered_map>
25+
#include <utility>
2426

2527
#include <executorch/extension/llm/cache/cache.h>
2628
#include <executorch/runtime/core/error.h>
@@ -76,8 +78,8 @@ class CacheBuilderRegistry {
7678
CacheBuilderRegistry() = default;
7779

7880
mutable std::mutex mu_;
79-
std::unordered_map<std::string, CacheBuilder>
80-
builders_; // backend_id + ":" + kind
81+
std::map<std::pair<std::string, std::string>, CacheBuilder>
82+
builders_; // keyed by (backend_id, kind)
8183
};
8284

8385
// Process-global atomic counter -> "cache-N"; centralizes key generation so
@@ -86,7 +88,7 @@ std::string make_unique_key();
8688

8789
// RAII: installs the cache into the global registry under a unique key on
8890
// construction and erases it on destruction (no leak on any exit path). Holds
89-
// the runner's shared_ptr for the lifetime of the generation loop.
91+
// the runner's shared_ptr and exposes the control face for the generation loop.
9092
class CacheSession {
9193
public:
9294
CacheSession(std::string key, std::shared_ptr<CacheBase> cache)
@@ -100,8 +102,8 @@ class CacheSession {
100102
CacheSession(const CacheSession&) = delete;
101103
CacheSession& operator=(const CacheSession&) = delete;
102104

103-
const std::shared_ptr<CacheBase>& cache() const {
104-
return cache_;
105+
SequenceControl* control() const {
106+
return cache_->as_control();
105107
}
106108
const std::string& key() const {
107109
return key_;

0 commit comments

Comments
 (0)