feat: DedupVectorStore + LlmExtractor + FactExtractor (#40, #41)

[0xvasanth]

Summary

Implements issues #40 and #41 as a single integrated feature pair — the two primitives that every persistent-memory system needs at write time.

• DedupVectorStore<S, F> (cognis-rag) — content-fingerprinting decorator over any VectorStore. Silently drops documents whose normalised text has already been seen. Generic over both the inner store and the fingerprint function so callers can supply a custom key at zero runtime cost.

• LlmExtractor<O> (cognis) — generic Runnable<String, O> for any O: DeserializeOwned + JsonSchema. Derives format instructions from the JSON Schema automatically. The reusable building block underneath FactExtractor.

• FactExtractor (cognis) — ready-to-use specialisation of LlmExtractor that maps FactExtractionInput → Vec<Fact>. Parse failures are swallowed (Ok(vec![]) + WARN log) so the memory write path never stalls on a badly-formatted model response.

What's new

File	What
crates/cognis-rag/src/vectorstore/dedup.rs	DedupVectorStore<S,F>, normalized_fingerprint — 15 tests
crates/cognis/src/agent/fact_extractor.rs	LlmExtractor<O>, FactExtractor, Fact, FactKind, FactExtractionInput — 10 tests
examples/memory/fact_extraction.rs	End-to-end demo (Ollama-verified)

All types re-exported from cognis::agent and cognis top-level.

Design decisions

Generic fingerprint function — DedupVectorStore<S, F> uses a type parameter for the fingerprint function rather than Arc<dyn Fn> so there is no heap allocation or vtable call on the hot add path. DedupVectorStore::new default-types F to fn(&str) -> String with normalized_fingerprint as the value, keeping the simple case ergonomic.

Restart persistence — the seen-set is in-memory only. DedupVectorStore::with_seen and the public normalized_fingerprint function let callers pre-seed hashes from storage on startup; seen_fingerprints() exposes the set for serialisation on shutdown.

Error-swallowing on FactExtractor — intentional. The write path to a memory store must not block because of a chatty model response. LlmExtractor propagates errors normally; only FactExtractor swallows them.

LlmExtractor as public primitive — any extraction task (sentiment, entity, classification, custom schema) uses it directly without going through FactExtractor. This was the generic-at-library-level requirement from the issue.

Test plan

• cargo test -p cognis-rag --lib vectorstore::dedup — 15/15
• cargo test -p cognis --lib agent::fact_extractor — 10/10
• cargo build --workspace — clean
• COGNIS_PROVIDER=ollama COGNIS_OLLAMA_MODEL=qwen2.5:3b cargo run -p cognis-examples --example memory_fact_extraction — verified end-to-end

🤖 Generated with Claude Code

---

Summary by cubic

Adds write-time memory deduplication and a generic LLM-based extractor to prevent duplicate index entries and enable schema-driven extraction. Implements #40 and #41.

• New Features
  • cognis-rag: DedupVectorStore<S, F> — drops duplicate documents by a content fingerprint; default normalized_fingerprint (lowercase + whitespace collapse + FNV-1a); supports custom keys; with_seen/seen_fingerprints() for restart persistence; preserves output length by returning dedup:skipped:{fingerprint} for skipped items.
  • cognis: LlmExtractor<O> — Runnable<String, O> that derives JSON Schema format instructions and parses prose-wrapped JSON; builder for prompt and parser config.
  • cognis: FactExtractor — Runnable<FactExtractionInput, Vec<Fact>> producing Fact { content, kind, importance }; swallows parse errors (returns []) to keep memory writes non-blocking.
  • Re-exports from cognis::agent and crate root; adds example memory_fact_extraction showing FactExtractor + DedupVectorStore.

^{Written for commit 9ade2ab. Summary will update on new commits. Review in cubic}

[cubic-dev-ai]

0 issues found across 5 files (changes from recent commits).

_{Re-trigger cubic}