Kalkan

Kalkan (Turkish for shield) is an integrity layer between an AI agent and your document: the agent can only propose structured patches — a deterministic engine validates them, applies them atomically, snapshots every version, and rolls back byte-exactly without a single model call.

Türkçe sürüm: README.tr.md

The problem

In April 2026, Microsoft Research published "LLMs Corrupt Your Documents When You Delegate" (Philippe Laban, Tobias Schnabel, Jennifer Neville), introducing the DELEGATE-52 benchmark: 19 LLMs editing documents across 52 professional domains over sessions of 10 edit/revert round-trips. Even frontier models corrupted about a quarter of document content by the end of long delegated sessions — and the damage is often silent: the agent reports success while the file quietly loses rows, wording, and structure. Kalkan attacks the mechanism behind that failure mode: it removes the model from the write path entirely.

Results

Measured with our DELEGATE-style harness — 6 documents × 10 edit+revert pairs (20 interactions each) under three conditions, Claude Sonnet via claude -p. Full tables: eval/RAPOR-sonnet-k2.md.

Condition	Final similarity (avg, 1 = intact)	Untouched-region damage % (avg)	Allowed-region residual drift % (avg)	Catastrophic events
A1 — full rewrite (paste doc, ask for edited copy)	0.7381	30.38	25.61	4
A2 — plain file tools (read_lines + str_replace, no validation)	0.9946	0.00	5.98	0
B — Kalkan (validated patches + versioned rollback)	1.0000	0.00	0.00	0

Honesty note: plain file tools (A2) performed well most of the time, but with no guarantee — its worst run ended at 0.9823 final similarity, and the allowed-region residual drift metric exists precisely to catch what its failed reverts leave behind inside the edited regions.

How it works

Four pieces, all deterministic:

1. MCP tool set — six tools over stdio (open_document, read_section, propose_patch, list_versions, revert, get_report). Reading is windowed (max 200 lines per call, deliberately no full-document dump); propose_patch is the only write path.
2. Patch engine — pure TypeScript, no model, no filesystem. Every operation (replace / insert_after / delete_section) must anchor on text that occurs exactly once; ambiguous or missing anchors are rejected with the closest matching passage and its line number so the agent can self-correct. Overlapping operations are rejected by name; a single patch may not touch more than 40% of the document (blast-radius cap). A patch list is all-or-nothing.
3. Version store — a hidden .kalkan/ folder next to the document keeps a full copy of every version plus a manifest (id, timestamp, operation summary, agent rationale). No git dependency. revert restores a version byte-exactly as a file operation — the model never regenerates "restored" content.
4. Diff & session report — every rejection carries an actionable reason; get_report summarizes patches, rejections, changed line ranges, and version count, so you can audit what the agent actually did.

flowchart LR
  A[AI agent] -->|MCP / stdio| T[Kalkan tool set]
  T --> P[Patch engine<br/>exactly-once anchors, overlap check,<br/>blast-radius cap, atomic apply]
  P -->|validated write| D[(document.md)]
  P -->|snapshot each version| V[(.kalkan/ version store)]
  V -->|revert: byte-exact restore,<br/>no model call| D
  T --> R[Diff / session report]

Loading

Quickstart

npm install
npm run build   # compiles to dist/
npm test        # vitest suite

Claude Desktop

Add to claude_desktop_config.json (adjust the path):

{
  "mcpServers": {
    "kalkan": {
      "command": "node",
      "args": ["C:/path/to/kalkan/dist/index.js"]
    }
  }
}

Claude Code

claude mcp add kalkan -- node /path/to/kalkan/dist/index.js

Or project-scoped in .mcp.json:

{
  "mcpServers": {
    "kalkan": {
      "command": "node",
      "args": ["./dist/index.js"]
    }
  }
}

3-minute walkthrough

A sample document ships in ornek/ornek-belge.md. Try this conversation with your agent:

1. Open — "Open ornek/ornek-belge.md with kalkan." → metadata arrives; .kalkan/ now holds v0.
2. Patch — "In the İletişim section, change the contact sentence to mention e-mail, with a rationale." → applied deterministically, v1 saved.
3. Fail on purpose — "Replace the sentence 'this text does not exist'." → rejected atomically; the error suggests the closest real passage; the file is byte-identical to before.
4. Roll back — "Roll back to v0." → the document is byte-identical to v0; the restore itself is recorded as v2.
5. Audit — "Get the session report." → 1 patch, 1 rejection with its reason, 3 versions.

Measurement

Full methodology: eval/METHODOLOGY.md. In short: every condition runs the same documents, same instructions, same order — 10 edit+revert pairs per document, each interaction a fresh conversation. A1 pastes the document and asks for a full edited copy back (the unprotected baseline). A2 gives the model unvalidated read_lines/str_replace file tools — today's typical agent setup. B gives it Kalkan's tools, where reverts are version restores instead of model output. Metrics are line-LCS based and fully deterministic: final similarity to the original, damage outside the regions the scenario was allowed to touch, residual drift inside allowed regions (failed reverts), and a catastrophe counter (>10% of the document changed in a single interaction).

Reproduce it yourself:

npm run eval:pilot    # 1 document × 3 rounds × 3 conditions, asks confirmation first
npm run eval:full     # full run: 6 documents × 10 rounds × 3 conditions
npm run eval:report   # regenerates eval/RAPOR-<model>[-<label>].md from result JSONs

The harness runs over your installed Claude Code subscription (claude -p, no API key) or the Anthropic API (eval/config.json → saglayici). Results land in eval/sonuclar/ as raw JSON — including every model response verbatim — so every number in the report can be audited. If you run it with a different model or provider, we'd love to see your RAPOR-*.md (see CONTRIBUTING.md).

Known limits & roadmap

• Whole-document patches are capped, not scoped — a single operation touching >40% of the document is rejected, but finer blast-radius enforcement (per-section scopes) is v2.
• No semantic invariants yet — Kalkan protects structure and untouched regions; rules like "totals must stay consistent" are v2.
• Markdown / plain text only — no DOCX, no spreadsheets, no code-aware patching.
• Single-model measurement — published numbers are one model on six Turkish documents; the harness supports re-running with other models, and cross-model results are exactly the contribution we're hoping for.
• v2 ordering will follow community feedback — open an issue if one of these blocks you.

Contributing & license

See CONTRIBUTING.md — bug reports, patch-engine edge cases, and especially your own measurement runs are welcome. Licensed under MIT.