Skip to content

README.md

Latest commit

 

History

History
526 lines (415 loc) · 23.6 KB

README.md

File metadata and controls

526 lines (415 loc) · 23.6 KB
Raw

SmartEvict

Python License Release DOI

Research artifact accompanying the SmartEvict preprint: a reproducible software and benchmark suite for learned and heuristic cost-aware eviction policies in semantic LLM caches.

📄 Paper

SmartEvict: An Empirical Study of Learned and Heuristic Eviction Policies for Semantic LLM Caches

The accompanying preprint presents the methodology, experiments, results, and analysis behind SmartEvict.

Semantic LLM caches require effective eviction policies to manage limited capacity while minimizing regeneration cost. Most deployed systems rely on simple recency-based heuristics such as LRU or FIFO. Recent work suggests that learned, cost-aware eviction policies could outperform these baselines, but it remains unclear whether such policies outperform strong cost-aware heuristics under realistic conversational workloads.
We present SmartEvict, a learned cost-aware semantic cache eviction policy, and perform an extensive empirical evaluation against LRU, FIFO, GDSF, and CostWeightedRecency across synthetic workloads and three real conversational traces (LMSYS-Chat-1M, WildChat-1M, and a customer-support corpus).
Across the evaluated workloads, cost-aware policies—learned and heuristic alike—consistently outperform recency-only baselines. However, strong cost-aware heuristics, particularly GDSF, remain highly competitive with and frequently exceed the learned policy. Across the three evaluated real-world conversational traces, we observe a relationship, consistent in direction and magnitude but based on only three independent data points, between a workload’s reuse density (measured as the fraction of eviction candidates never previously reused) and the relative advantage of the heuristic policy. We additionally identify cache pressure as a second, partially understood axis that can shift this relationship independently of reuse density.
These results suggest that selecting a semantic cache eviction policy should depend on measurable workload characteristics rather than assuming that learned policies universally dominate heuristic approaches. We find that strong cost-aware heuristics remain difficult to surpass, and identify measurable workload characteristics associated with their advantage—providing empirical guidance for policy selection in semantic caching systems.

Overview

SmartEvict evaluates how semantic LLM caches can retain the most valuable entries under limited capacity by comparing learned and heuristic eviction policies. The repository provides a reproducible reference implementation and benchmark suite for the paper's main finding: the paper analyzes when each approach performs best and when cost-aware eviction outperforms recency-only policies.

Why SmartEvict?

Traditional semantic caches use heuristic eviction policies such as LRU and FIFO. These policies only consider when an entry was last accessed, not how expensive it would be to regenerate.

SmartEvict predicts which cache entries are worth keeping by considering:

  • Regeneration cost
  • Access history
  • Semantic reuse patterns
  • Response size

Design Goals

  • Increase regeneration-token savings
  • Incorporate regeneration cost into eviction decisions
  • Keep learned-policy inference lightweight
  • Provide deterministic LRU fallback

Architecture

Architecture of SmartEvict

The figure above summarizes the end-to-end flow from a user query to cache hit/miss handling and eviction decisions.

                  User Query
                      │
                      ▼
               Embedding Model
                      │
                      ▼
             Semantic Similarity Search
                      │
             ┌────────┴────────┐
             │                 │
          Cache Hit        Cache Miss
             │                 │
             ▼                 ▼
      Return Response      Call LLM
                                │
                                ▼
                     Store Prompt–Response Pair
                                │
                                ▼
                     SmartEvict Policy Engine
                                │
                                ▼
                   Select Entry to Evict

Key Features

  • Lightweight learned eviction model (~9.5K parameters)
  • Cost-aware eviction decisions
  • Offline training using replayed request traces
  • Exact LRU fallback for production safety
  • GPTCache integration
  • Backend-agnostic wrapper
  • FAISS or NumPy vector search backend
  • Fully reproducible benchmarking pipeline

Why Not LRU?

Traditional cache eviction policies assume that recently used entries are the most valuable to keep. While this works well for conventional caches, semantic LLM caches have an additional consideration: regeneration cost.

Cached Entry Regeneration Cost LRU Decision

FAQ answer Low Keep if recent Multi-page analysis High Evict if old

Although both entries may be equally old, regenerating the multi-page analysis is significantly more expensive. SmartEvict learns to prioritize cache entries based on their expected future value rather than recency alone.

Project Goals

  • Improve regeneration-token savings over classical eviction policies
  • Maintain production safety through deterministic LRU fallback
  • Keep inference lightweight enough for CPU-only deployment
  • Provide a reproducible benchmark for learned semantic cache eviction

Workflow

Incoming Prompt
      │
      ▼
Generate Embedding
      │
      ▼
Semantic Cache Lookup
      │
 ┌────┴─────┐
 │          │
Hit        Miss
 │          │
 ▼          ▼
Return     Call LLM
              │
              ▼
      Store Response
              │
              ▼
    SmartEvict decides
    whether another cache
    entry should be evicted

Semantic LLM caches (GPTCache, LangChain's cache, Redis semantic caching) evict by recency alone: LRU treats a cheap cached FAQ answer and an expensive multi-page cached analysis as equally disposable the moment neither has been touched in a while. But they aren't equally disposable — one costs orders of magnitude more to regenerate than the other. This project replaces recency-only eviction with a learned, cost-aware policy: a ~9.5K-param model that predicts which entries are worth keeping warm based on the regeneration cost they'd save if reused, not just how recently they were touched.

It follows a Cold-RL-style pattern (arXiv:2508.12485: K-tail candidate sampling + tiny dueling network + hard fallback to a classical policy), trains offline on replayed traces, is safe by default (falls back to LRU exactly if the model is absent or errors), and is benchmarked honestly against LRU, FIFO, GDSF, CostWeightedRecency, and a clairvoyant oracle rather than against itself.

Scope note: this project decides which cached entries to keep warm. It deliberately does not touch semantic-match correctness (whether a cached answer is actually valid for a new prompt) — that is a separate problem.

Results at a glance

Every cost-aware policy tested — learned or not — beats recency-only LRU/FIFO by a wide margin, on both synthetic and real data: the token-cost signal is what matters. A simple non-learned heuristic (GDSF: evict by hit-count × cost) matches or beats the learned RL policy in most regimes tested, including on the real trace below; the learned net's edge over GDSF only shows up in the synthetic high-duplicate-density regime. See results/RESULTS.md for the full breakdown, baselines, and a mechanistic explanation (with two follow-up experiments) of why GDSF wins where it does.

On a 20K-request synthetic conversational workload (held-out split, cache size 400): the learned policy saves +3.6% to +6.1% more regeneration tokens than LRU depending on duplicate density (GDSF: +5.4% to +8.0%), with zero fallbacks fired.

On a real 50K-request trace from LMSYS-Chat-1M (held-out 20K-request tail), averaged across 5 training seeds (fifo/lru/gdsf/oracle are deterministic, so only the learned net varies): learned reaches +16.7% ± 1.1% more regeneration tokens than LRU with the local HashingEmbedder and +17.0% ± 1.4% with real MiniLM sentence embeddings (--embedder minilm) — but GDSF reaches +27.7% and +19.5% respectively on the same trace. Full tables, baselines, and all caveats: results/RESULTS.md. For feature/architecture/ hyperparameter ablations digging into why, see results/ABLATIONS.md.

Two more real traces (WildChat-1M, Bitext customer support) plus a cache-size sweep extend this into a general finding: GDSF's advantage over the learned policy tracks a measurable workload property — the fraction of eviction candidates never previously reused — consistently across all three real traces, with cache pressure as a second, independent factor. ./reproduce.sh regenerates every table behind this finding from scratch. A full write-up covering the methodology, experiments, threats to validity, and discussion is available in the SmartEvict preprint.

Installation

Not published on PyPI (still under active benchmarking/validation) — install straight from GitHub or a local clone:

# directly from GitHub, no clone needed
pip install "git+https://github.com/Shikha-code36/SmartEvict-Semantic-Cache-Eviction.git"
pip install "smartevict[all] @ git+https://github.com/Shikha-code36/SmartEvict-Semantic-Cache-Eviction.git"

# or, if you already have a local clone
git clone https://github.com/Shikha-code36/SmartEvict-Semantic-Cache-Eviction.git
cd SmartEvict-Semantic-Cache-Eviction
pip install -e .              # core package (numpy only)
pip install -e ".[all]"       # + faiss, LMSYS download, MiniLM, GPTCache adapter

Extras are also installable individually: .[faiss], .[lmsys], .[minilm], .[gptcache].

Quick Start

from smartevict.features.embeddings import HashingEmbedder   # or your own embed fn
from smartevict.policies.wrapper import LearnedSemanticCache

cache = LearnedSemanticCache(
    embedding_fn=HashingEmbedder(dim=64).embed,   # list[str] -> normalized np.ndarray
    max_size=1000,
    eviction_policy="learned",                     # or "lru" / "fifo"
    model_path="results/learned_policy.npz",
    sim_threshold=0.8,
)

resp = cache.get(prompt)          # None on miss
if resp is None:
    resp = call_llm(prompt)       # your model call
    cache.set(prompt, resp)

print(cache.stats())              # hits, evictions, fallbacks, tokens_saved, backend

Backend: FAISS (IndexFlatIP over normalized vectors) if installed, transparent brute-force NumPy store otherwise. The wrapper is backend-agnostic by design (Plan §9): it only owns the eviction decision.

For real semantic quality, swap the embedder:

from smartevict.features.embeddings import sentence_transformers_embedder
cache = LearnedSemanticCache(embedding_fn=sentence_transformers_embedder(), ...)

Advanced Usage

Use it inside GPTCache

Already using GPTCache? You don't need to switch caching libraries to try this — GPTCache's own eviction backend (gptcache.manager.eviction.memory_cache.MemoryCacheEviction) is hardcoded to a fixed set of cachetools policies (LRU/LFU/FIFO/RR), but the interface underneath it is a plain 3-method ABC, and get_data_manager(..., eviction_base=...) accepts an already-built instance of it. smartevict/policies/gptcache_adapter.py implements that interface using the same net + hard-fallback logic benchmarked in this repo:

pip install -e ".[gptcache]"

from gptcache import Cache
from gptcache.manager import get_data_manager, CacheBase, VectorBase
from smartevict.policies.gptcache_adapter import LearnedEviction

eviction = LearnedEviction(model_path="results/learned_policy.npz", maxsize=1000)
data_manager = get_data_manager(CacheBase("sqlite"),
                                VectorBase("faiss", dimension=384),
                                eviction_base=eviction, max_size=1000)

cache = Cache()
cache.init(data_manager=data_manager, ...)  # your usual embedding_func / similarity_evaluation

Caveat: GPTCache's put/get eviction hooks only pass opaque row ids, not the prompt/response text or token counts — this adapter tracks its own per-id age/hit-count/idle-time bookkeeping, but has no visibility into regeneration cost unless you tell it. Call eviction.note_cost(id, response_tokens) right after each insert to get the full cost-aware behavior benchmarked in results/RESULTS.md; without it, cost defaults to a flat value and the policy degrades to a recency/frequency-only signal (still safe — falls back to plain LRU exactly if the model is missing or errors, same as the standalone wrapper).

Benchmarks

pip install -e ".[all]"
python tests/test_all.py                                # ~30s sanity suite
smartevict-benchmark                                     # full 3-regime sweep, ~5 min CPU
smartevict-benchmark --quick                             # fast sanity version

(smartevict-benchmark is a console script installed by pip install -e .; equivalently python -m smartevict.benchmark.run_benchmark.)

Outputs results/benchmark.json + results/learned_policy.npz.

To reproduce every table across all four workloads and all ablations in one step, run ./reproduce.sh from the repo root — it's idempotent, so it's safe to rerun after an interruption (e.g. while waiting on LMSYS-Chat-1M's Hugging Face access approval, see below). See datasets.md for what each of the four evaluated workloads is and how to get it individually.

Datasets

The benchmark covers four workloads: a synthetic trace, LMSYS-Chat-1M, WildChat-1M, and a Bitext customer-support trace. See datasets.md for dataset descriptions, preprocessing notes, and download commands.

Run on real data (LMSYS-Chat-1M)

lmsys/lmsys-chat-1m is a gated dataset — installing the lmsys extra is not enough on its own, you need an approved Hugging Face access request and a token:

  1. Create a free account at https://huggingface.co if you don't have one.
  2. Visit https://huggingface.co/datasets/lmsys/lmsys-chat-1m while logged in and submit the access request (fills in affiliation/use-case). This is reviewed manually by the dataset owner and is not instant — it can take anywhere from minutes to a day or more to be approved. Recheck the page until it shows you have access.
  3. Generate a read-scoped token at https://huggingface.co/settings/tokens.
  4. Put the token in a .env file in the repo root (gitignored, never commit it): 
    HF_TOKEN=hf_your_token_here
  5. Install deps and download: 
    pip install -e ".[lmsys]"
smartevict-download-lmsys --n 50000 --out data/lmsys_trace.json
smartevict-benchmark --trace data/lmsys_trace.json

If step 5 fails with DatasetNotFoundError: ... is a gated dataset, the token is either missing/invalid or your access request from step 2 hasn't been approved yet — it is not a code/setup bug.

To benchmark with real sentence embeddings instead of the local hashing proxy, add pip install -e ".[minilm]" (first run downloads all-MiniLM-L6-v2, ~90MB) and pass --embedder minilm:

pip install -e ".[minilm]"
smartevict-benchmark --trace data/lmsys_trace.json --embedder minilm --out results/benchmark_minilm.json

This writes to results/benchmark_minilm.json and results/minilm_learned_policy.npz rather than the default filenames, so it won't overwrite the hashing-embedder results/model.

To check the result isn't a lucky single seed, add --seeds: fifo/lru/oracle are deterministic so they're computed once, and only the learned net is retrained per seed, reporting mean ± std vs LRU:

smartevict-benchmark --trace data/lmsys_trace.json --seeds 0 1 2 3 4 --out results/benchmark_multiseed_hashing.json
smartevict-benchmark --trace data/lmsys_trace.json --embedder minilm --seeds 0 1 2 3 4 --out results/benchmark_multiseed_minilm.json

Run on other real workloads (WildChat-1M, Bitext customer support)

Two more real traces are used in the paper (cache-pressure sweep and the reuse-density cross-dataset comparison) — neither is gated, so no HF access request is needed:

smartevict-download-wildchat --n 50000 --out data/wildchat_trace.json
smartevict-download-bitext --out data/bitext_trace.json

See datasets.md for what each dataset is, how it's preprocessed, and which paper section it feeds into.

How it works (Cold-RL → semantic cache mapping)

Cold-RL (NGINX) Here
HTTP object Cached prompt–response pair
age, size, hits, inter-arrival, TTL, RTT age, response tokens (cost proxy), hits, idle time, mean inter-access gap, staleness ratio
K-tail from LRU list K=8 coldest entries
Dueling DQN (~10K params) Dueling net, 9,474 params, pure NumPy, CPU-trains in seconds
ONNX sidecar, 500µs SLO In-process inference (~0.2ms/decision; semantic-cache evictions are rare)
Hard timeout → LRU fallback try/except → LRU fallback; fallback path unit-tested to match LRU exactly
Reward: +1 if reused before TTL Reward: discounted future regeneration tokens saved

Training is offline fitted-Q on decision points collected from an LRU replay, with targets from an infinite-cache "demand" pre-pass (every request's future matches, computed policy-free) — this sidesteps the off-policy counterfactual problem of never observing reuse of entries the behavior policy evicted. Because eviction terminates an entry's episode, the 1-step target reduces to discounted future demand; that simplification (vs. full multi-step Q-learning) is deliberate for this POC and stated here so nobody mistakes it for the full algorithm.

Repo layout

pyproject.toml       package metadata; `pip install -e .` / `.[all]`
reproduce.sh          single entry point: regenerates every table + figure below
datasets.md           what each of the 4 evaluated workloads is + how to get it
smartevict/
  data/         synthetic workload generator + LMSYS/WildChat/Bitext download scripts
  simulator/    bounded semantic cache simulator, replay harness
  features/     embeddings (hashing / sentence-transformers) + 6-feature extractor
  model/        NumPy dueling net + linear net (architecture ablation) +
                offline training pipeline
  policies/     LRU, FIFO, GDSF, Cost-weighted-recency, Learned (w/ fallback),
                Oracle, LearnedSemanticCache wrapper, GPTCache EvictionBase adapter
  benchmark/    comparison script + feature/architecture/hparam ablation scripts
tests/          sanity suite (simulator, training, fallback, wrapper, FAISS, GPTCache)
results/        benchmark output + honest write-ups (RESULTS.md, ABLATIONS.md)

The full paper (methods, threats to validity, discussion, figures) is available as a Zenodo preprint: https://doi.org/10.5281/zenodo.21643364. This README + results/RESULTS.md + results/ABLATIONS.md are the code-adjacent source of truth for the same findings.

Known limitations

  • A simple non-learned heuristic (GDSF) beats the learned policy in most regimes tested, including the real LMSYS trace — see results/RESULTS.md for the full comparison and a mechanistic explanation. Framing this project as "learning beats heuristics" is not supported by the current evidence; "cost-aware beats recency-only" is.
  • The synthetic-data tables still only use HashingEmbedder. The LMSYS real-trace benchmark has now been run with both HashingEmbedder and real MiniLM sentence embeddings (--embedder minilm), each averaged across 5 training seeds — see results/RESULTS.md for the full comparison.
  • LMSYS results are still from a single trace/split (only the training seed is varied, not the data); a different 50K-request sample or a different train/test split isn't yet covered.
  • The model generalizes across duplicate-density regimes here, but was not tested across time-scale shifts (e.g., traces with very different arrival rates); features use log-scaled absolute times, so a per-deployment fine-tune (seconds of CPU) is recommended.
  • The GPTCache adapter (smartevict/policies/gptcache_adapter.py) is tested against a real in-process GPTCache Cache (sqlite + FAISS), but not against every backend combination GPTCache supports (Redis, Milvus, etc.), and cost weighting there requires the caller to call note_cost() explicitly — GPTCache's eviction hooks don't expose response length on their own.
  • Single-threaded wrapper; no persistence of cache contents across restarts.

Future Work

The current implementation focuses on demonstrating the effectiveness of a lightweight, learned eviction policy for semantic LLM caches. Future improvements include:

  • Support additional semantic cache backends (Redis, Milvus, ChromaDB, Qdrant, etc.)
  • Benchmark on additional real-world conversational and enterprise workloads
  • Evaluate alternative reinforcement learning algorithms (e.g., PPO, SAC, Offline RL variants)
  • Online learning and adaptive fine-tuning from production traffic
  • Multi-threaded and distributed cache wrapper
  • Persistent cache storage and recovery across restarts
  • Additional cost models beyond response-token count (latency, API cost, energy)
  • Integration with production LLM serving frameworks such as vLLM and SGLang

References

This project builds upon ideas and tools from the following works:

Citation

If you use SmartEvict in your research or build upon this project, please cite the accompanying preprint:

Pandey, Shikha. (2026). SmartEvict: An Empirical Study of Learned and Heuristic Eviction Policies for Semantic LLM Caches (Version v2.0.0). Zenodo. https://doi.org/10.5281/zenodo.21643364

@misc{pandey2026smartevict,
  author    = {Shikha Pandey},
  title     = {SmartEvict: An Empirical Study of Learned and Heuristic Eviction Policies for Semantic LLM Caches},
  year      = {2026},
  publisher = {Zenodo},
  version   = {v2.0.0},
  doi       = {10.5281/zenodo.21643364}
}