Skip to content

feat(memory): rebuild agent memory on one Postgres fact store - #284

Open
eastriverlee wants to merge 11 commits into
mainfrom
feat/memory-store
Open

feat(memory): rebuild agent memory on one Postgres fact store#284
eastriverlee wants to merge 11 commits into
mainfrom
feat/memory-store

Conversation

@eastriverlee

@eastriverlee eastriverlee commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

What to look at

Seven commits, in order:

  1. c55737db the store: migration 030 (memory_episode, memory_fact, memory_profile, memory_job, pg_trgm, and a memory_fact_embedding side table created only where pgvector exists), the repositories, the hybrid search (pgvector cosine + pg_trgm word similarity, reciprocal rank fusion), and the job outbox with FOR UPDATE SKIP LOCKED claims.
  2. 0daf5dd4 ingest: one pipeline for what the agent writes (memory_remember) and what it did (a finished task run). The runtime embeds the episode, offers the nearest live facts the requester may read as candidates, and a low-tier model returns facts related as new / supersedes / reinforces. The runtime rejects any relation to a fact it did not offer. Three tools for the model: memory_search, memory_remember {content}, memory_forget {factIDs} (IDs only from a search in the same task).
  3. b6bc9cfe removal: Graphiti sidecar, Kuzu, MEMORY.md, the in-process queue, admin graph endpoints, health probe, config; migration 031 drops the graphiti tables; the namespace list becomes one memory.SecurityLabel per launch.
  4. 7c408aca admin: GET /admin/api/memory/facts (a person's profile and the live facts they may read) and POST /admin/api/memory/facts/forget, for the device web page.
  5. 922d1abb fix: lexical search when the vector table is absent (CI runs plain Postgres).
  6. 27326c18 the store moves to its own module, bluememo, vendored at .dependency/bluememo like the harness; internal/memory keeps only the host adapters. Circles nest: a circle fact names one or more circles, memberCircles on a circle policy says which circles belong to it, and a member of a containing circle reads the contained circle's facts, transitively. Writing to a circle still needs direct membership. migrations/030_memory_store.sql is a verbatim copy of bluememo's schema with a drift test.
  7. d45e13bc no company-wide scope: a fact with no circles is read by its owner alone; a fact that names circles is read by their members (and containing circles) too; sharing with everyone is sharing with the member circle. memory_search returns each fact's circles instead of a scope.

Design: docs/internal/memory-redesign.md in the internkim repository (arrives with the submodule bump, internkim PR feat/memory-store).

Evidence

  • go vet ./... and go test ./... green; catalogs regenerated. bluememo's own suite (unit + Postgres on pgvector/pgvector:pg16 and plain postgres:16) green, covering multi-circle facts, transitive containment with a cycle, and the reader predicate in SQL and Go saying the same thing.
  • Postgres integration (BLUECLAW_TEST_POSTGRES_URL, run on pgvector/pgvector:pg16 and on plain postgres:16): reader filter across scope, rank and classes; supersede hides the old row and keeps it; expiry; scoped forget; vector ranking; job dedupe/claim/lease/retry; a finished task run observed through the real transition observer ends as rows in all three tables and Recall returns them.
  • Live model check (BLUECLAW_LIVE_LLM_TEST=1, real low-tier model + real embedding model): a Korean transcript yields a preference and a temporary fact with its expiry; a correcting transcript supersedes the team fact and reinforces the preference; recall ranks the corrected fact first; a profile builds. It passed once on 2026-09-02 after fixing a ranking fault it found (a multiplier for reinforced preferences flipped adjacent reciprocal ranks; reinforcement is now a tie-breaker). Not run in CI.
  • Embedding benchmark (BLUECLAW_EMBEDDING_BENCHMARK=1, 30 Korean facts, 20 hand-marked queries, 1,024 dims): qwen3-embedding-8b, qwen3-embedding-4b and pplx-embed-v1-4b all at R@1 1.000; pplx-embed-v1-0.6b 0.950; bge-m3 R@1 0.950 and R@3 1.000. bluememo's library default is baai/bge-m3; the host names the model it actually uses (a device: bge-m3 from its own llama.cpp; a direct-execution host: pplx-embed-v1-4b on OpenRouter). Vectors from two models do not compare, so search ranks vectors only from the store's own model and a reembed job, enqueued at every worker start, moves live facts onto it; the memory_guided_followup fleet scenario now calls memory_remember and requires the remembered sentence in the next task's model context. The live model check passed again against bge-m3 on 2026-09-03.

What the model now sees

memory_search 217 bytes, memory_remember 428 bytes, memory_forget 179 bytes of tool description (was 599 bytes for two tools). The ingest and profile instructions are sent only on the background low-tier calls, never in the agent turn.

Not done here

  • pgvector on the bookworm guest rootfs and the capabilityd remote embedding default are in the internkim PR.
  • Import of existing MEMORY.md files: none exist on a device this ships to first; the design keeps the import job kind for when one does.

🤖 Generated with Claude Code

eastriverlee and others added 11 commits September 3, 2026 13:31
…b queue

The memory subsystem is being rebuilt from scratch; this is the store,
nothing reads it yet. Facts are atomic sentences extracted from an
immutable episode, scoped private/circle/workspace, labelled with the
security rank and classes of their origin conversation, and retired by a
supersede pointer, a forget timestamp, or a valid_until, never by
deletion. The reader's clearance is one SQL filter shared by search,
lookup and forget, so there is no second gate in Go.

Search fuses a pgvector cosine list and a pg_trgm word_similarity list
with reciprocal rank fusion, then adjusts deterministically: episodes
decay with age, reinforced preferences rank higher. The substring
re-ranking that undid the old sidecar's ordering has no successor.

Vectors sit in a side table the migration creates only when the vector
extension is available, because two databases this runtime boots on have
none: the bookworm guest rootfs (Debian ships no pgvector before trixie)
and the embedded-postgres binaries behind the standalone command. Both
apply the migration and answer lexically; the search result names the
mode so a lexical answer is never mistaken for a hybrid one.

memory_job is an outbox with FOR UPDATE SKIP LOCKED claims, a lease,
exponential backoff and a partial unique index that collapses repeated
enqueues of one subject into one pending row. The worker takes handlers
per kind and claims only kinds it can run; extraction and profile
handlers come with the next change.

The capability embedding client gains inputType, batch input and
outputDimensions, the fields the sidecar already sent and Qwen3-style
models need.

Not done here: tool wiring, launch-time recall, extraction, and the
removal of the Graphiti and markdown stores.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Writing memory and extracting it from finished work were going to be two
mechanisms with two schemas. They are now one: an episode (a task
transcript, or the sentence a person asked the assistant to keep) is
embedded, the nearest live facts the requester may read become
candidates, and a low-tier model returns the facts the memory should
hold afterwards, each related to a candidate as new, supersedes, or
reinforces. The runtime rejects a relation to any fact it did not offer,
narrows a circle fact to private when no active circle is at hand, and
strips the security label from private facts because their scope is
already one person. No similarity threshold ever merges two facts by
itself; that judgment stays with the model, and the runtime only feeds
it what exists.

The agent sees three verbs: memory_search returns facts with IDs,
memory_remember takes one sentence and reports what was created,
superseded, or reinforced, and memory_forget takes IDs that
memory_search returned in the same task, failing closed on any other.
The write path carries no kind, scope, or ID, so a model does not have
to look a fact up before correcting it.

Every finished, failed, or cancelled task run is enqueued for
extraction by a transition observer; the worker reads the label,
requester name, active circle, and platform from a
memory.extraction_context event the launcher wrote, so it never guesses
who could read what. Launch loads the requester's profile and a hybrid
recall of the prompt under character budgets and records what it spent
in memory.recall_injected.

The live check against the real low-tier model and embedding model
caught a ranking fault: a mild multiplier for reinforced preferences
flipped adjacent reciprocal ranks, whose gap is under two percent, and
put a repeated preference ahead of the fact that answered the question.
Reinforcement is now a tie-breaker.

The old Graphiti and markdown path stays wired only where no database
is configured and for the simulate harness's virtual sessions; its
removal, migration 031, and the virtual-session store on the in-memory
repository are the next change.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The Graphiti sidecar, the Kuzu graph, the per-person MEMORY.md and the
in-process update queue are gone, with the admin graph endpoints, the
health probe, the healthd flag, the backup root and the configuration
that named them. Migration 031 drops graphiti_namespace and
graphiti_episode; the current-schema probe now looks for memory_fact.
Person canonicalisation and the direct-conversation reset follow the
memory tables, and a reset forgets the facts that came out of the
conversation with a stated reason instead of deleting rows.

The namespace vocabulary went with the sidecar. A launch carries one
memory.SecurityLabel, resolved by the connector from the channel policy
(or the requester's own clearance in a direct message), and the tool
catalog reads that label; the five-scope namespace list every caller
had to assemble is not needed by a store whose reader filter is one SQL
predicate.

The simulate harness's virtual sessions run on the in-memory
repository with a passthrough ingest model, so scenarios that remember
and search still read back what they wrote. The default embedding model
is qwen/qwen3-embedding-4b: a first benchmark on Korean facts separated
the 8b, the 4b and pplx-embed-v1-4b on nothing but latency, and the
Qwen3 models are the ones a self-hosting company can serve from
llama.cpp with vendor-published GGUF files. The benchmark stays in the
tree as an on-demand test.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The web memory page read the Graphiti graph endpoints, which the
previous change removed. It now has GET /admin/api/memory/facts, which
returns a person's profile and the live facts that person may read,
newest first, under the same reader filter the agent uses, and
POST /admin/api/memory/facts/forget, which soft-forgets facts with a
reason and refuses when nothing readable was forgotten. Both take the
person as readerPersonID because the device web app resolves the
signed-in member before it proxies.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The hybrid query joins memory_fact_embedding, which the migration
creates only where the vector extension exists. A caller that already
had a query embedding (ingest, on every host) hit a missing-table error
on a plain Postgres, the extraction job retried forever, and no fact
was written. CI runs plain Postgres, which is how it surfaced. The
repository now checks for the table before choosing the hybrid query.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
The memory store becomes its own module, github.com/yeomyeonggeori/bluememo,
vendored at .dependency/bluememo the way the harness is. It depends on
nothing but pgx; this host adapts its task, model and identity types in
internal/memory (a reader from PersonAccess, a label from a channel
policy, a transcript from a task run, the harness's structured-output
provider behind bluememo's one-method interface). Migration 030 is a
verbatim copy of bluememo's schema and a test fails when it drifts.

Circle facts name one or more circles, and circles nest. A circle policy
gains memberCircles, the projection turns that into a containment map,
and a reader's readable set is their own circles plus everything those
contain, transitively: a fact shared with platform is readable from
engineering when engineering contains platform. Writing to a circle
still needs direct membership, so the model may name several circles
for one fact but never one the requester is not in.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bluememo dropped the scope field: every fact has an owner who always
reads it, and zero or more circles whose members read it too. There is
no company-wide scope; sharing with everyone is sharing with the member
circle everyone is in. memory_search returns each fact's circles instead
of a scope, the admin listing carries the owner and circles, and the
loop's memory context groups a fact under "user" when it names no
circle and "circle" otherwise.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bluememo's ingest instruction now says memory is for what a different
conversation will need once the transcript is gone, and that an empty
list is the normal answer. The live check gains a mundane task run, a
file rename with its shell step and reply, and asserts it yields no
fact, alongside the transcripts that must. It passed once on
2026-09-03 against the real low-tier model and embedding model.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bluememo's default moves to qwen3-embedding-8b because OpenRouter serves
the 4b through one provider only; the example configuration and the tests
that spell the name follow it.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bluememo's default moves to bge-m3, the model the device already serves
locally, so local and remote embeddings share one vector space. The
example configuration and the tests that spell the name follow it, and
the on-demand benchmark measures bge-m3 beside the models it replaced.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
bluememo now ranks vectors only from the store's own model and carries a
reembed job for the rest. The application registers that handler and
enqueues one reembed each time the worker starts, so a host that changed
its embedding model brings its memory back into vector search on its own;
the job finishes at once when nothing needs moving.

The memory_guided_followup scenario used to finish both turns from a
script and proved nothing about memory. It now calls memory_remember in
the first turn and, in the second, requires the memory.recall_injected
event and the remembered sentence in the model's context.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant