Repository navigation
Persistent Agent Memory & Cross-Session Decision Reuse in optillm #299
Description
I’m exploring a research direction that extends optillm from per-request optimization to cross-session decision reuse via persistent agent memory (i.e., turning past trajectories into reusable policies, rather than a policy-search setup like OpenEvolve or LLM4AD).
I wish to check whether the current architecture supports the following:
1. Persistent State
I may need a mechanism to accumulate knowledge across sessions (e.g., successful traces, failure→fix mappings, routing policies).
- Is there any built-in shared/global state?
- If not, is an external memory layer (KV / vector DB) the intended pattern?
- Where should it integrate: strategy layer, middleware, or outside the proxy?
2. Feedback Loops
My setup depends on post-hoc signals (e.g., execution results, tests, errors) to inform future decisions.
Is there any support (or recommended pattern) for closing the loop:
response → execution → feedback → memory update → future decisions?
Are there hooks for post-response feedback ingestion?
3. Hybrid Decision Layer
I’m considering a tiered setup:
- fast path: local logic / small model for known patterns
- slow path: LLM via optillm
Would this be better implemented inside optillm as a custom strategy,
or as an external pre-routing layer before requests enter the proxy?
Any expected bottlenecks with a high-frequency local decision layer?
4. Scope Alignment
Does adding persistent agent memory and decision reuse across sessions align with optillm’s design direction, or is it outside the intended scope?
If aligned, I'd be interested in contributing a reusable “agent memory” extension. Thanks!
Activity
codelion commentedon Mar 21, 2026
Thanks for the thoughtful writeup! These are great questions. Let me address each briefly:
There's no built-in shared/global state right now. optillm is stateless by design (each request is independent). For what you're describing, an external memory layer (vector DB or even a simple SQLite/KV store) would be the right pattern. I'd recommend integrating it at the strategy layer, since that's where you have the most context about what approach was chosen and why. You could implement a custom strategy that reads from / writes to this store.
There's no built-in post-response feedback mechanism currently. A reasonable approach would be to build a lightweight wrapper service that sits around optillm: it captures the response, passes it to your execution environment, collects the signal, and writes back to the memory store. Alternatively, you could implement a custom strategy that exposes a
/feedbackendpoint to accept post-hoc signals and update its internal state. Theexecuteplugin approach we have could be a starting point here since it already runs code and captures results.I'd suggest an external pre-routing layer. Keeping the fast path outside optillm avoids adding latency to the proxy for known-pattern lookups. Your pre-router can check the memory store, and if it finds a high-confidence cached policy, bypass optillm entirely. For the slow path, route through optillm as normal. This keeps concerns cleanly separated and avoids bottlenecks.
This is an interesting research direction and broadly aligned with where we see optillm going. We'd be open to a contribution that adds an optional "agent memory" extension, especially if it's modular (doesn't force persistence on users who don't want it). A plugin-style architecture where this is opt-in would be ideal.
yuxuan-z19 commentedon Mar 21, 2026
Thanks @codelion, your timely clarification is very helpful!