Skip to content

README.md

Latest commit

 

History

History
631 lines (453 loc) · 34.7 KB

README.md

File metadata and controls

631 lines (453 loc) · 34.7 KB
Raw

CueMap

CueMap Rust Engine

Fast, accurate, and explainable temporal-associative memory for agents.

CI Coverage License

High-performance temporal-associative memory store designed for dynamic contextual retrieval.

Overview

CueMap uses temporal-associative retrieval: lexical and structural candidate generation, with optional semantic reranking. Its main components are:

  1. Intersection (Context Filter): Triangulates relevant memories by overlapping cues
  2. Structural Extraction: Emits deterministic cues for observable evidence such as dates, numbers, lists, source metadata, and surface entities.
  3. Recency & Salience (Signal Dynamics): Balances fresh data with salient, high-signal events prioritized by an adaptive impact scoring module.
  4. Reinforcement (Access-based Learning): Frequently accessed memories gain signal strength, remaining highly accessible even as they age.
  5. Sparse Recall: Uses normalized lexical cues, structural facets, recency, salience, and bounded deterministic reranking.

As of v0.7.2+, CueMap's default core path is deterministic and ontology-free. GloVe/Ollama cue generation, WordNet/POS expansion, semantic bridges, pattern completion, external lexicon graphs, context expansion/speculation endpoints, and autonomous consolidation have been removed from the default engine path. v0.7.2+ bundles a qint8 paraphrase-MiniLM-L3-v2 vector layer for semantic reranking, intent classification, and query embeddings; the edge profile selects a q4 build of the same model. The encoder can still be disabled for constrained builds or deployments.

v0.7.2+ also uses numeric per-project memory IDs everywhere. If callers need deterministic upsert/dedupe identity, pass source_key; memory IDs remain compact runtime addresses.

v0.7.3 adds Tree-sitter-backed ingestion for Swift, Dart, Objective-C, Kotlin, C, C++, C#, and Bash source files, plus structured TOML files.

Built with Rust for maximum performance and reliability.

Quick Start

Build & Run

# Production (optimized)
cargo build --release
./target/release/cuemap start --port 8735

# Development
cargo run -- start

CueMap treats the nlprule tokenizer as a runtime asset, not a build artifact. Set TOKENIZER_PATH to a compiled tokenizer .bin file, or place en_tokenizer.bin under ~/.cuemap/data. The production Docker image downloads the same checksum-pinned tokenizer used by the install script and packages it at /app/assets/en_tokenizer.bin.

Docker

docker build -t cuemap/engine:0.7.5 .
docker run -p 127.0.0.1:8735:8735 -v "$(pwd)/local_snapshot_dir:/app/data" cuemap/engine:0.7.5

The container runs as the unprivileged cuemap user. Ensure a bind-mounted data directory is writable by UID/GID 10001, or use a Docker-managed volume. Runtime defaults can be overridden with CUEMAP_HOST, CUEMAP_PORT, CUEMAP_DATA_DIR, CUEMAP_SNAPSHOT_INTERVAL_SECONDS, CUEMAP_PROJECT_INACTIVITY_TIMEOUT_SECONDS, CUEMAP_PROJECT_UNLOAD_CHECK_INTERVAL_SECONDS, TOKENIZER_PATH, and RUST_LOG.

Network access and read-only operation

The native server binds to 127.0.0.1:8735 by default. Set server.host or CUEMAP_HOST to a numeric IPv4 or IPv6 address to change the bind address. The Docker image binds to 0.0.0.0 inside the container; the examples publish its port only on the host loopback interface.

Browser origins are denied by default, including simple cross-origin requests. To allow a browser application, list its exact origin in configuration:

[security]
allowed_origins = ["http://localhost:3000"]

API-key authentication still applies to allowed browser clients.

server.read_only = true and static loading disable write routes, recall reinforcement, automatic ingestion, and snapshot writes. Existing snapshots can still be loaded for queries. Ordinary requests and multipart uploads have a 64 MiB body limit; project-package uploads to /projects/load have a 1 GiB limit. These limits apply to the request body, not the expanded contents of archives.

Set CUEMAP_HOME to isolate the engine's configuration and PID files. It defaults to ~/.cuemap; CUEMAP_DATA_DIR separately controls the data directory.

Native npm packages

Build the Darwin ARM64, Darwin x64, Linux x64, and Linux ARM64 native packages locally without publishing them:

./scripts/build-npm-native-packages.sh
./scripts/verify-npm-native-packages.sh

The local packager builds Linux on Debian Bookworm, bundles the checksum-pinned tokenizer, and writes package tarballs plus SHA256SUMS under dist/npm-native/tarballs. The Windows x64 package is built and published by the GitHub Actions release workflow.

Release validation

Run the local consumer preflight on macOS and Windows before publishing, then run the read-only public-registry smoke test from GitHub Actions after publishing. See RELEASE.md for the exact commands and release order.

CLI Commands

CueMap provides a unified CLI for server management, ingestion, and interaction.

Install CLI:

# Install CLI
cargo install --path . --locked
cuemap <COMMAND> [OPTIONS]

Core Commands

  • start: Start the CueMap server.
  • stop: Stop the background server instance.
  • status: Check server health, metrics, and background jobs.
  • logs: View or tail server logs.

Interaction

  • add: Add a memory via natural language.
  • recall: Search memories (supports Grounded Recall and Web Recall).
  • ingest: Ingest data from files or URLs.
  • project: Manage projects, portable packages, and sync (projects remains an alias).
  • set-project: Set the default project for the current session.
  • set-watch-dir: Set a watch directory for a project (enables agent).

Deterministic Semantics

  • lexicon: Inspect lexicon entries and wire/unwire cues.
  • alias: Manage explicit deterministic aliases.

Hint: Use cuemap --help to see available commands and options.

Configuration

For agent-facing memory inspection, GET /memories/{id}?decoded=true with X-Project-ID returns readable content, IDs, source key, cues, metadata, and timestamps. It resolves compressed, encrypted, and disk-backed content through the engine's content reader and omits storage internals such as vectors. Omitting decoded=true preserves the existing raw storage response. This option affects individual memory reads, not recall or its scoring path.

CueMap uses a layered configuration system that prioritizes settings in the following order: CLI Args > Env Vars > server_config.toml > Defaults.

This system allows you to:

  1. Centralize Settings: Manage server options, security keys, and engine tuning in ~/.cuemap/server_config.toml.
  2. Fine-Tune Performance: Adjust critical engine parameters like scoring weights, scan limits, and bounded fallback thresholds via the [tuning] section without recompiling.
  3. Manage Project Context: Use set-watch-dir to persist project-specific settings (like agent watch paths) in .meta.json files alongside your data.

Self-Learning Agent (Zero-Friction Ingestion)

CueMap includes a Self-Learning Agent that automatically watches local directories, extracts structured "facts", and ingests them into your memory store.

Automated Bootstrapping

On startup, if --agent-dir is provided, CueMap initializes the Self-Learning Agent.

Example

# Point CueMap at your project
./target/release/cuemap start --agent-dir ~/projects/my-app

# The agent will automatically:
# 1. Supercharged Structural Ingestion (Rust, Python, Go, JS/TS, PHP, Java, Swift, Dart, Objective-C, Kotlin, C/C++, C#, Bash, and TOML).
#    - Native tree-sitter queries capture definitions, calls, and imports as grounded cues.
# 2. Document & Data Parsing (PDF, Word, Excel, JSON, CSV, YAML, XML).
#    - Extracts headers, keys, and metadata as structural metadata.
# 3. Source-aware chunking: related chunks share stable parent/session/order metadata for optional bounded reconstruction during recall.

AI Agent Integration (MCP Server)

CueMap provides a native Model Context Protocol (MCP) server, allowing AI coding assistants (like Claude Desktop, Cursor, and Windsurf) to instantly recall codebase context using the engine.

Setup

We provide a zero-config NPM package that automatically downloads and manages the CueMap background engine.

Add the MCP server to your AI agent's configuration (e.g., Claude Desktop, Cursor, or Windsurf):

{
  "mcpServers": {
    "cuemap": {
      "command": "npx",
      "args": [
        "-y",
        "cuemap-mcp"
      ],
      "env": {
        "CUEMAP_PORT": "8735"
      }
    }
  }
}

Once configured, the AI Agent can use the cuemap_init and cuemap_recall tools to query your codebase memories natively. There is no need to manually start the engine.

Project Management & Persistence

CueMap provides complete project isolation with automatic persistence:

Features

  • Project Isolation: Each project has its own memory space, identified by X-Project-ID header.
  • Auto-Save on Shutdown: All projects are saved on graceful shutdown when persistence is enabled.
  • Auto-Load on Startup: Snapshots are restored from the configured data directory when persistence is enabled.
  • Memory-Aware Residency: Loaded project contexts are automatically unloaded after a configurable inactivity period while their snapshots remain available on disk. A request for an unloaded project transparently loads it again.
  • Zero Configuration: Works out of the box

Usage

CueMap runs in multi-tenant mode by default. Select a project for CLI commands with cuemap set-project or pass --project to an individual command.

# Start the server
./target/release/cuemap start --port 8735

# Choose a project and use the local CLI
cuemap set-project my-project
cuemap add "Important data"
cuemap recall "What is important?"

# Stop server (Ctrl+C) - saves all projects when persistence is enabled
# Restart server - loads persisted snapshots
# Data persists across restarts unless snapshots are disabled.

Project memory residency

By default, the engine checks loaded projects every 60 seconds and unloads projects that have had no activity for one day. Configure this in server_config.toml:

[project_lifecycle]
inactivity_timeout_seconds = 86400
unload_check_interval_seconds = 60

Set inactivity_timeout_seconds = 0 to disable automatic unloading. Project snapshots are written before an unload, and ordinary recall, ingestion, and other project requests demand-load the project when needed. The first request after a reload can therefore have additional snapshot/index reconstruction latency. Use POST /projects/{project_id}/load to warm a project explicitly or POST /projects/{project_id}/unload to persist and release it immediately. GET /projects includes loaded: true|false for each project. Explicit unload returns a conflict while active work still holds the project context.

Memory optimization

CueMap has two complementary ways to reduce memory usage. Choose between them based on whether the memory pressure comes from large content payloads or from having many inactive projects loaded at once.

Content-level optimization: --disk-content

Start the engine with --disk-content to keep memory content on disk instead of retaining the raw content bytes in RAM:

./target/release/cuemap start --disk-content

The project’s cues, metadata, indexes, and semantic vectors remain loaded, so recall stays warm. CueMap reads the content from <data-dir>/contents/<project-id>/ when it needs to return a result. This is useful for a frequently accessed project with many large memories, but content results incur disk I/O. It is not a project unload mechanism.

Project-level optimization: load/unload

Project unloading persists the project snapshot and releases the complete in-memory project context, including its indexes and metadata. It is useful when an instance contains many repositories but only a few are active. A request for an unloaded project loads it automatically; the first request can therefore have higher latency. See Project memory residency for the inactivity policy and explicit endpoints.

The two options can be enabled together: --disk-content reduces the RAM used by each loaded project, while project unloading reduces the number of loaded projects. Disk-backed content lives outside the project snapshots, so backups must include both the snapshots directory and <data-dir>/contents/.

Portable project packages

A .cuemap file carries a ready-to-query project—snapshots, disk-backed content, and CueBridge artifacts—so another server can load it without re-ingestion.

cuemap project pack my-project --output my-project.cuemap
cuemap project load my-project.cuemap
cuemap project push my-project s3://my-bucket/cuemap/
cuemap project pull s3://my-bucket/cuemap/my-project.cuemap
cuemap project sync my-project s3://my-bucket/team

HTTP clients use the matching POST /projects/{id}/pack, /projects/load, /projects/{id}/push, and /projects/pull endpoints.

pack/push flush the running server first; use --offline only for a current stopped instance. Imports verify SHA-256 checksums and snapshot compatibility and refuse overwrite unless --force is used while the server is stopped. Packages exclude machine-specific watch settings and are point-in-time, sensitive copies. Encrypted projects require the same master key on the target. S3 commands use the configured AWS CLI and incur normal AWS charges.

sync adds immutable commits and a conditionally updated S3 head. It pushes or pulls only fast-forwards and refuses divergent or concurrently changed state. HTTP clients use POST /projects/{id}/sync with {"remote":"s3://..."}.

Snapshot Management

Snapshots are automatically managed:

  • Created: Periodically and on graceful shutdown (SIGINT/Ctrl+C) when persistence is enabled.
  • Loaded: On server startup
  • Disabled: --disable-snapshots turns off periodic and shutdown snapshot saves.
  • Location: ~/.cuemap/data/snapshots/ by default, or <--data-dir>/snapshots when --data-dir is set. Older installs may also be discovered under the legacy sibling snapshots/ directory.
  • Format: zstd-compressed JSON inside .bin files. This preserves arbitrary metadata reliably while keeping snapshots compact; older uncompressed bincode snapshots remain readable when their metadata can be decoded.
  • Migration note: Some pre-v0.7.2 bincode snapshots that contain dynamic JSON metadata cannot be decoded by bincode's deserialize_any limitation. Those projects are reported at startup and must be reingested or exported from a compatible older binary before upgrading.
  • Files: {project-id}.bin, {project-id}_lexicon.bin, {project-id}_aliases.bin

Cloud Backup

CueMap supports secure offsite backups to AWS S3, Google Cloud Storage, and Azure Blob Storage.

Configuration: Enable cloud backup via CLI flags or ~/.cuemap/server_config.toml.

# S3 Example
./target/release/cuemap start \
  --cloud-backup s3 \
  --cloud-bucket my-backup-bucket \
  --cloud-region us-east-1

Supported Providers:

  • s3: AWS S3 or compatible (MinIO, DigitalOcean Spaces)
  • gcs: Google Cloud Storage
  • azure: Azure Blob Storage
  • local: Local path (for testing/replication)

Management: Manual backup operations are documented in the HTTP API reference.

Authentication

Secure your CueMap instance with API key authentication.

Enable Authentication

Set an API key via environment variable:

# Single API key
CUEMAP_API_KEY=your-secret-key ./target/release/cuemap start --port 8735

# Multiple API keys (comma-separated)
CUEMAP_API_KEYS=key1,key2,key3 ./target/release/cuemap start --port 8735

Or configure keys in ~/.cuemap/server_config.toml:

[security]
api_keys = ["your-secret-key"]

Clients send the configured key in the X-API-Key header. See the HTTP API reference for request headers and SDK examples.

Docker with Authentication

docker run -p 127.0.0.1:8735:8735 -v "$(pwd)/local_snapshot_dir:/app/data" \
  -e CUEMAP_API_KEY=your-secret-key \
  cuemap/engine

Security Notes

  • Authentication is disabled by default (no keys = no auth required, unless security.require_auth=true is set in config)
  • Keys can be loaded from security.api_keys in ~/.cuemap/server_config.toml or from CUEMAP_API_KEY / CUEMAP_API_KEYS.
  • Use strong, randomly generated keys in production
  • Rotate keys regularly
  • Use HTTPS in production to protect keys in transit

Encryption

CueMap supports encryption-at-rest for all memory content using modern authenticated encryption.

  • Algorithm: ChaCha20-Poly1305 (IETF) (via chacha20poly1305 crate).
  • Master Key: Uses a 256-bit (32-byte) key for encryption.
  • Key Derivation (PBKDF2): Supports deriving the master key from a human-readable passphrase using PBKDF2-HMAC-SHA256 with 100,000 iterations and a persistent installation-unique salt.
  • Nonce: A random 12-byte nonce is generated for every memory encryption operation and stored alongside the ciphertext.
  • Key Handling: The engine does not write the master key into CueMap data files. Encryption-at-rest is opt-in and the key must be provided at startup via environment variables (CUEMAP_MASTER_KEY or CUEMAP_MASTER_PASSWORD) or security.master_key in the configuration file. If you store the key in server_config.toml, protect that file like any other secret.

Compression

To optimize storage efficiency, especially for large textual memories, CueMap employs transparent compression.

  • Algorithm: Zstandard (Zstd), configured for a balanced compression level (3).
  • Strategy: Content is compressed before encryption. This ensures maximum entropy reduction before the data is scrambled, often resulting in 40-60% storage savings for English text.
  • Performance: Zstd provides extremely fast decompression speeds, ensuring that the "hot path" for reading memories remains sub-millisecond even with compression enabled.

Performance

Benchmark Results (v0.7.5)

Tests performed on Real-World Data (Wikipedia Articles), processing full natural language sentences with the complete NLP pipeline.

Hardware: MacBook Pro M-series, 64GB RAM, single node. The v0.7.5 release runs cover lexical and hybrid writes and reads at 10K, 100K, and 1M memories. Lexical runs isolate the sparse core with the semantic encoder disabled; hybrid runs use the v0.7.5 defaults, including the bundled local encoder. P95 is the release headline percentile, while P99 remains available in the JSON diagnostics.

Benchmark Methodology

The NL benchmark script lives at benchmarks/benchmark_nl.py.

Benchmark setup:

  • Uses the public Wikipedia Plaintext (2023-07-01) Kaggle dataset as the release corpus. The benchmark script downloads it automatically when --wikipedia-path is omitted, caches it under ~/.cache/cuemap/benchmarks/wikipedia-20230701, and samples parquet files with a text column. Install the downloader first with python -m pip install kagglehub; configure Kaggle access if Kaggle prompts for authentication. To avoid the download or use another corpus, pass --wikipedia-path /path/to/parquet-or-directory.
  • Deduplicates sampled snippets and consumes them without replacement, so 100K and 1M write runs do not reuse the same text.
  • Writes use HTTP POST /memories with minimal_response=true and no explicit cues, forcing CueMap to run deterministic cue/facet extraction and indexing.
  • Reads generate keyword-style natural-language queries from retained ingested snippets.
  • Recall numbers use the script's lean recall mode: semantic_mode=lexical, auto_reinforce=false, salience disabled, alias expansion disabled, CueBridge artifacts disabled, depth=1, expansion_depth=1, and parent/order/evidence reconstruction disabled. This isolates the core sparse recall path from the bundled semantic encoder and reranker.
  • Each requested size runs in its own run-scoped project, so a 1M pass is not layered on top of a previous 100K pass or stale state from an earlier invocation.
  • --trace-timing records engine timing breakdowns but is not required for throughput measurements.

Example run with the checked-in release fixture:

CUEMAP_SEMANTIC_ENCODER_ENABLED=false cuemap start --disable-snapshots --disable-bg-jobs

python benchmarks/benchmark_nl.py \
  --sizes 10000,100000,1000000 \
  --project-id nl_test \
  --semantic-mode lexical \
  --wiki-reservoir-size 20000 \
  --query-sample-size 5000 \
  --payload-buffer-size 500

Restart the engine without CUEMAP_SEMANTIC_ENCODER_ENABLED=false, then run the same command with --semantic-mode hybrid --expansion-depth default to measure the v0.7.5 defaults (the bundled encoder on the CPU, semantic reranking of up to 200 candidates, and the default conversation_round expansion, which leaves Wikipedia snippets unchanged because they carry no conversation metadata). The dataset is downloaded only once and reused from the local cache on subsequent runs.

v0.7.5 latency

Mode Scale Write avg Write P50 Write P95 Write throughput Read avg Read P50 Read P95 Read throughput
Lexical 10,000 2.44 ms 2.04 ms 4.25 ms 409 ops/s 1.05 ms 0.99 ms 1.64 ms 947 ops/s
Lexical 100,000 2.65 ms 2.20 ms 4.89 ms 378 ops/s 1.94 ms 1.77 ms 3.07 ms 511 ops/s
Lexical 1,000,000 2.83 ms 2.33 ms 5.30 ms 353 ops/s 2.43 ms 1.91 ms 3.64 ms 409 ops/s
Hybrid 10,000 3.74 ms 3.30 ms 6.34 ms 267 ops/s 1.73 ms 1.68 ms 2.48 ms 574 ops/s
Hybrid 100,000 3.79 ms 3.35 ms 6.49 ms 264 ops/s 2.32 ms 2.22 ms 3.35 ms 430 ops/s
Hybrid 1,000,000 4.03 ms 3.49 ms 7.28 ms 248 ops/s 3.64 ms 3.09 ms 5.68 ms 273 ops/s

Writes measure HTTP ingestion, deterministic cue/facet extraction, memory allocation, and indexing (plus one local embedding per memory in hybrid mode). Reads measure query parsing, deterministic cue resolution, sparse candidate scoring and, in hybrid mode, query encoding (about 0.5 ms per query) and semantic reranking of up to 200 candidates. Write latency stays mostly flat with project size; read latency grows slowly with it.

Against v0.7.3 at 1M memories:

Mode v0.7.3 write avg v0.7.5 write avg v0.7.3 read avg / P95 v0.7.5 read avg / P95
Lexical 3.33 ms 2.83 ms 2.63 / 3.72 ms 2.43 / 3.64 ms
Hybrid 11.28 ms 4.03 ms 8.36 / 10.67 ms 3.64 / 5.68 ms

The hybrid speed-up comes from running the bundled encoder on the CPU instead of CoreML by default (CUEMAP_SEMANTIC_COREML_ENABLED=true re-enables CoreML).

Key Metrics:

  • Low-latency recall: 1M memories in 2.43 ms average (3.64 ms p95) lexical and 3.64 ms average (5.68 ms p95) with the default hybrid reranking.
  • Numeric ID memory reduction: 1M in-memory footprint dropped from about 5.25GB to about 1.93GB after the v0.7 numeric memory-ID refactor.
  • Controlled hot path: no LLMs, network services, or disk scans on the recall path; candidate generation is deterministic, and the bundled local encoder only reranks a bounded candidate set.

Retrieval evaluations (v0.7.5)

Raw retrieval metrics (no LLM reader or judge): Any@k is the share of questions with at least one evidence item in the first k results, All@k the share with every evidence item, and Frac@k the mean fraction of evidence items. Tokens are approximate tokens in the first 20 results. Settings:

  • Matched only (expansion_depth: 1): the matched memories alone.
  • Default (conversation_round, the v0.7.5 default): each result also gets the message it replies to, and the top result gets its answer.
  • Replies on 5 (CUEMAP_ROUND_REPLY_TOP=5): the first five results get their answers.
  • Window ±9 (expansion_depth: 10, LoCoMo only): nine source-ordered neighbours on each side, the setting behind earlier LoCoMo numbers.
Benchmark Setting Any@1 Any@20 All@20 Frac@20 Tokens in top 20 Recall (mean)
LoCoMo (1,986 questions) Matched only 58.3 92.7 84.4 88.7 650
Default 59.7 93.0 84.7 89.0 1,211
Replies on 5 59.8 93.1 84.9 89.1 1,303
Window ±9 65.9 96.2 88.9 92.8 10,043
LongMemEval (470 questions) Matched only 57.0 96.2 87.9 92.8 3,902
Default 61.1 97.9 90.4 94.6 4,135
Replies on 5 61.1 97.9 90.4 94.6 4,616
BEAM 128K (355 questions) Matched only 49.3 87.3 58.0 73.3 6,874 4.4 ms
Default 58.6 90.7 63.4 78.2 8,076 3.2 ms
Replies on 5 58.6 91.5 65.9 79.4 10,328 4.9 ms
BEAM 1M (625 questions) Matched only 37.3 81.9 34.6 55.4 3,298 5.3 ms
Default 50.6 86.7 37.3 59.3 4,228 4.4 ms
Replies on 5 50.6 88.2 39.0 62.7 7,437 5.9 ms
BEAM 10M (176 questions) Matched only 33.5 67.6 25.0 39.2 1,592 13.5 ms
Default 38.6 68.2 26.1 40.2 2,420 12.0 ms
Replies on 5 38.6 71.0 27.3 42.2 6,127 14.2 ms

All runs ingest one memory per message with conversation metadata, use hybrid recall, and score the same projects under each setting. Recall latency is the engine's mean per query on the MacBook Pro above (BEAM 10M excludes the first query after each engine start, which loads the 10M-token project's caches). Harnesses and run scripts live in evals/; evals/run_release_evals.sh reproduces this table.

Architecture

Core Components

  • Axum: Minimal overhead async web framework
  • DashMap + aHash: Lock-free concurrent hash map with high-speed hashing
  • IndexSet: O(1) move-to-front operations
  • Bincode: Fast binary serialization for persistence
  • Zstd: High-ratio, real-time compression for storage
  • ChaCha20-Poly1305: Authenticated encryption at rest

Optimizations

  • Zero-copy: Efficient memory management with Arc
  • Pre-allocated collections: Capacity hints eliminate reallocation
  • Unstable sorting: 2-3x faster than stable sort
  • Iterative deepening: Early termination on hot paths

HTTP API and SDK documentation

The complete HTTP/OpenAPI contract, request and response schemas, authentication headers, ingestion and recall routes, and Python/TypeScript SDK examples live in the CueMap documentation.

The website docs are the source of truth for endpoint behavior and are kept aligned with the checked-in Rust router. This README stays focused on building, operating, and understanding the engine; use the CLI and MCP sections above for the fastest local workflows.

System Architecture

The system diagrams are maintained separately to keep this README focused:

Advanced Capabilities

1. Self-Learning Ingestion Agent

The agent transforms your local filesystem into a deterministic structural knowledge base with zero manual effort.

  • Universal Format Support: Deeply integrates with dozens of formats:
    • Languages: Rust, Python, TypeScript, JavaScript, Go, Java, PHP, HTML, CSS, Swift, Dart, Objective-C, Kotlin, C, C++, C#, and Bash (via Tree-sitter).
    • Documents: PDF (text extraction), Word (DOCX), Excel (XLSX).
    • Data: CSV (row-aware), JSON (key-aware), YAML, XML, TOML.
  • Tree-sitter Powered Chunking: Smartly splits code into functions, classes, and modules while preserving context.
  • Deterministic Knowledge Extraction: Uses tree-sitter structure, document parsers, metadata facets, and token normalization; no runtime model call is required.
  • Idempotent Updates: Uses content-aware hashing (file:<path>:<hash>) to prevent memory duplication and ensure stale memories are pruned.
  • Background Verification Loop: Continuously verifies that memories in the engine still exist on disk, pruning stale references automatically.

2. Deterministic Natural Language Engine

CueMap bridges unstructured text to sparse deterministic recall without vector search, runtime models, or background semantic expansion by default. Optional vector retrieval can add externally computed semantic candidates without changing the structural extraction path.

How It Works

At add-time, CueMap extracts cues synchronously from real structure:

  • normalized lexical cues
  • surface entity, quote, model-like, and structural evidence cues
  • evidence facets such as numbers, money, dates, durations, and lists
  • source facets from metadata such as role, channel, session, and order

At query-time, CueMap uses the same deterministic normalization path, then applies only bounded in-memory expansions:

  • explicit aliases when enabled
  • installed CueBridge AliasPack entries during query cue resolution
  • installed CueBridge GapPack entries only when exact recall is weak
  • optional ordered/evidence reconstruction passes when explicitly requested

Semantic Boundary

CueMap Core does not try to infer broad semantic relationships from local co-occurrence or ontology rules. That keeps the default recall fast, deterministic, and inspectable. Semantic gap closure can come from externally precomputed vectors or explicit artifacts:

  • Manual Lexicon Wiring: explicit token-to-canonical cue connections for project owners.
  • CueBridge Artifacts: offline-compiled GapPack/AliasPack files generated by CueBridge Local or Cloud and loaded into CueMap.

This split is intentional: CueMap Core stays lean and latency-stable, while CueBridge can use heavier local or cloud models offline to generate static lexical-gap artifacts.

3. Advanced Contextual Recall

CueMap keeps advanced recall behavior deterministic and inspectable:

Source-Order Context and Episodes

Long-form ingests, chat logs, files, and agent chunks preserve parent/session/order metadata. Recall can use expansion_depth to include nearby source-order chunks, and add-time temporal episode cues can be disabled on memory writes with disable_temporal_chunking: true.

Conversation Rounds

/ingest/content with "segmenter": "conversation" stores a chat history as one memory per message (long messages become ordered parts), with source_session_id, source_turn_index, source_role and part metadata. It reads Claude and ChatGPT exports (keeping the displayed branch, without system messages or tool traffic), JSON message lists, and transcripts whose messages start with User:/Assistant: or speaker names.

Recall uses expansion_depth: "conversation_round" by default (pass a number to get the previous behaviour; 1 returns matched memories only). It prepends to each result the message it replies to: for an assistant reply, the user message that opened its round; in human-to-human chats, the nearest earlier message from someone else. The attachment is capped at 512 approx-tokens and leaves the order of results unchanged. A message already ranked above its reply is not repeated; one ranked below is moved up into the reply, so a round appears at the rank of its best result and fewer results than limit can come back. Results gain expansion and expanded_memory_ids metadata. The first result also gets the message that replies to it, appended in full: the assistant answer to a user prompt, or in human-to-human chats the next message from someone else. A reply ranked lower is moved up the same way, and those results carry reply_memory_ids; CUEMAP_ROUND_REPLY_TOP=n changes how many of the top results get replies (0 turns it off; 5 gives more context per recall, at a token cost). Messages are matched by session, turn and role metadata, never by ingestion order, so memories without that metadata are returned as they are.

Adaptive Salience Bias

Not all memories are created equal. The engine calculates a Salience Multiplier based on cue density, reinforcement frequency, and rare cue combinations. High-signal memories rank above routine events when other structural evidence is comparable. Can be disabled per-recall via disable_salience_bias: true.

Match Integrity

Every recall result now includes a Match Integrity score. This internal diagnostic combines intersection strength, reinforcement history, and context agreement to tell you how structurally reliable a specific recall result is.

Bounded Reconstruction

For long-form chat logs, tickets, transcripts, and benchmark records, recall can optionally run bounded reconstruction passes:

  • parent_fusion: stitch related chunks that share source parent metadata.
  • ordered_reconstruction: retrieve ordered evidence from a small number of matching sessions.
  • evidence_coverage: diversify results across multiple evidence cues for summary-style queries.

These modes are off by default and are designed for diagnostics or workloads that explicitly trade a bounded second pass for higher evidence coverage.

License

CueMap Rust Engine and its native engine packages are licensed under Apache-2.0 from v0.7.3 onward. Earlier releases remain under BSL-1.1. See LICENSE and NOTICE for details.

Recall previews

The engine's POST /recall accepts response_mode: "preview" and optional preview_chars (100–2000 UTF-16 code units, default 200). Full content remains the default. Previews replace each hit's content with a leading preview, content_truncated, and content_length, preserving metadata and ranking. Use previews for broad discovery, then fetch a selected memory with GET /memories/{id}?decoded=true or read its source. Metadata and diagnostics are not capped. TypeScript request objects and Python sync/async recall accept these same options; Python returns RecallPreviewResult for ungrouped preview results. The updated engine is required.