Native Lightweight RAG & Vector Memory: Bridging the Gap from Creative Engine to Enterprise/Reference Workflows

[mamq144]

Hi @LostRuins and KoboldCpp Contributors,

First of all, huge respect for building and maintaining what is undeniably the most resource-efficient, hardware-optimized, and granular local inference engine available today. The integration of Quantized KV cache, Context Shifting, Flash Attention, and deep sampler controls (Min-P, DRY) makes KoboldCpp unmatched in raw inference performance and memory management.

Currently, KoboldCpp is widely recognized and celebrated for creative writing, roleplay, and open chat. However, its exceptional backend capabilities position it as an ideal candidate for professional reference and enterprise-grade use cases—if paired with native document retrieval.

The Problem

Tools like AnythingLLM provide automated RAG and document orchestration, but rely on heavy runtimes (Electron/Node/abstracted backends) and completely lack KoboldCpp’s microscopic parameter control, memory locking, and efficient GPU offloading. Meanwhile, KoboldCpp users who want to query local technical manuals, business documents, or long-form reference texts must either rely on complex third-party stacks or hit context window constraints.

The Proposal: Native "Micro-RAG" Module

Rather than introducing heavy external dependencies (which would contradict KoboldCpp’s single-binary philosophy), what if KoboldCpp introduced an embedded, lightweight vector/retrieval pipeline?

• Zero-Dependency Vector Storage: Integrating a lightweight, self-contained embedding/retrieval engine (e.g., embedded C/C++ vector search or an extension like sqlite-vec).
• Leveraging Embedded Inference: Reusing the built-in GGUF engine to run small, dedicated embedding models (like bge-small or nomic-embed-text) locally on the same backend.
• Lite UI Document Ingestion: Extending KoboldAI Lite to allow simple drag-and-drop of text/PDF/markdown files that get automatically chunked, embedded, and injected as dynamic grounding context.
• Reference/Business Mode: Complementing Story and Adventure modes with a dedicated "Reference / RAG" workspace mode.

Collaboration & Benchmarking

I run local testing and benchmark environments on high-end hardware (i9, RTX 4090 24GB VRAM, multi-display setups) and have authored detailed comparative studies analyzing KoboldCpp's architecture against RAG-focused platforms. I have also configured dedicated local server pipelines integrating KoboldCpp with external orchestrators.

I would love to collaborate, contribute test cases, share architectural benchmarks, or assist in testing experimental builds if this aligns with the project’s roadmap.

What are your thoughts on introducing an ultra-lightweight, native retrieval layer to elevate KoboldCpp into the reference/enterprise domain?

Best regards,
Mohammed

[LostRuins]

We actually have some basic document storage/retrieval via a built-in feature called "TextDB".

Have you tried that?

[mamq144]

Thanks for pointing that out, @LostRuins!

I've tested TextDB, and while it's a great lightweight solution for injecting plain-text excerpts into the prompt context, it has natural boundaries when dealing with production-grade reference workflows:

1. Format Handling: It requires manually pasting and formatting raw text, whereas enterprise/reference workflows depend on parsing rich, structured files (.pdf, .docx, .xlsx, .csv).
2. Context & Ingestion Overhead: Injecting retrieved snippets directly into context can trigger continuous reprocessing overhead on every turn.
3. Autonomous Retrieval via MCP: In contrast, adopting an MCP-based architecture allows the LLM to dynamically trigger semantic search tools (query_knowledge_base / index_documents) only when factual context is genuinely needed, backed by persistent vector storage (like ChromaDB).

To bridge this gap without bloating KoboldCpp's core binary, I actually built and open-sourced an external, zero-leak MCP RAG server tailored specifically for KoboldCpp's tool-calling interface:
https://github.com/mamq144/KoboldCpp-RAG-MCP

It handles multi-format parsing and persistent vector embeddings locally while letting KoboldCpp do what it does best: fast, granular inference.

Looking forward to seeing how MCP tool calling evolves in KoboldCpp—it provides the ideal decoupled foundation for these advanced use cases!