aakit: assumption-aware measurement kit

Measures whether an AI agent's silent assumption-making actually costs anything on your real work, and whether asking or retracting beats assuming.

Three experiments, one number each:

#	Question	The number	Kills the thesis if
1	How often are silent assumptions wrong and consequential?	load-bearing-wrong rate	95% CI upper bound < 5%
2	Does divergence-gated asking beat never-ask and always-ask?	success × questions/task	gating doesn't beat both
3	Is targeted retraction cheaper than starting over?	tokens at equal pass rate	targeted isn't cheaper-and-as-good

As far as I can find, #1 has not been published for organic workloads. The numbers I could locate come from adversarially injected ambiguity, which bounds the problem from above by an unknown margin. That gap is the reason this exists. If you know of a measurement on unprompted, real-workload data, open an issue.

None of the three has a publishable number yet. One run exists, it is small, and most of what it produced is a finding about the instrument. See Results so far below before you read any further into the design.

This kit is designed to be able to come back negative. If it does, that is the useful outcome. See PROTOCOL.md.

---

The idea it implements

Assumption-Aware Intelligence: a system that knows when to assume, when to abstain, and when to imagine, and says which one it just did.

An assumption-aware agent emits a record alongside its work:

statement:    the input CSV has a header row
provenance:   invented | from_context | from_evidence | from_convention
support:      the quote, file, or command it rests on. Empty if guessed.
justifies:    parser.py:L14-L38, tests/test_load.py
defeater:     head -1 data.csv
entrenchment: 0.7   (how much gets redone if this is wrong)

Then one loop. When evidence arrives that contradicts a record, fire, compute the blast radius, regenerate only that. Evidence means a failing test, a file whose contents disagree, or a correction you type three hours later.

Note the shape. It is event-driven, so it does not tax every interaction to catch the rare one.

Prior art, stated up front

None of the theory here is new. Value of information: Howard, 1966. The optimal error-reject tradeoff: Chow, 1970. Deciding how much to think: Russell & Wefald, 1991. Belief dependency tracking: Doyle's TMS, 1979, and de Kleer's ATMS, 1986. The formal licence to assume: Reiter, 1980.

What appears to be new is that no shipped AI product implements any of it. I searched coding agents, agent frameworks, observability vendors and spec-driven tools. I found nothing that maintains a machine-checkable, automatically-revisable record of what was assumed. GitHub's Spec Kit comes closest, with a mandatory Assumptions section and a capped, ranked set of clarifying questions. But its assumptions are prose bullets that nothing ever re-checks.

If you know of a counterexample, open an issue. I would rather be wrong than first.

---

Results so far

One complete run. Experiment 2 has a full pre-registered result. Experiment 1 has no number. Experiment 3 has never been run.

Experiment 2: ask policy, 120 trials

10 tasks (5 information-limited, 2 default-right, 3 controls) x 4 policies x --repeats 3, at --max-questions 3, on claude-sonnet-4-5 via the cli backend.

Policy	Success	95% CI	Questions/task	Tokens/trial
gated_multi	19/30 (63.3%)	[45.5, 78.1]	1.90	4,181
always	18/30 (60.0%)	[42.3, 75.4]	2.47	3,186
divergence_gated	13/30 (43.3%)	[27.4, 60.8]	0.67	3,971
never	10/30 (33.3%)	[19.2, 51.2]	0.00	1,606

Verdict: NO WIN. Against the pre-registered kill criteria in PROTOCOL.md, that is a fail.

Nothing separates from never-ask. Every interval overlaps never's [19.2, 51.2]. The closest is gated_multi, still overlapping by 5.7 points. divergence_gated, the one-question ClarifyDelphi gate, overlaps by 23.8. The point estimates all favour asking, and every one of them is inside the noise at n=30 per policy.

The finding that does not depend on the success column: the gate costs more than the questions it saves.

always asks 2.47 questions per task and spends 3,186 tokens. divergence_gated asks 0.67, which is 3.7x fewer questions, and spends 3,971 tokens, 25% more. Running the divergence check, generating the top-two readings of the request and comparing them, costs more than simply asking. Against never-ask the gate runs at 2.47x baseline, and gated_multi at 2.60x, on every request including the ones the gate correctly waves through.

A gate is normally pitched as a cheap pre-filter that pays for itself by suppressing unnecessary questions. On this task set it is the most expensive policy that beats nothing.

What this run cannot tell you. Whether asking genuinely helps. Every point estimate says yes and no interval will support it. Separating gated_multi from never at these rates needs roughly n=40 per policy. That is a separate, newly pre-registered run, not an extension of this one.

A note on interim peeking. At 81% complete this run projected gated_multi clear of never by 4.4 points. The final 22 trials reversed it to a 5.7 point overlap. The interim projection would have supported the opposite conclusion. Do not read these tables before they finish.

Experiment 1: no base rate yet

An earlier pass produced 0 assumptions across 2 tasks. That is not a base rate. n=2, and both tasks were heavily clarified up front. Worth one line only because of the direction: on real traces the LLM extractor found nothing where the heuristic backend found 44. The anti-inflation measures do something, and the heuristic floor is as poor as advertised.

aakit metrics reports insufficient_n. A real Experiment 1 number needs at least 25 adjudicated tasks. It does not exist yet.

Experiment 3: not run

The defeater loop has never been run on real data. The code is here and the smoke test covers it, but there is no result, no defeat-precision number, and no targeted-versus-full comparison. Treat the Experiment 3 section below as a design and a set of commands, not as a validated claim.

Three instrument bugs, all found by running it

1. Head-only truncation of tool results. The 1,800-char cap cut the tail off a file read, which was exactly where the lines supporting the claim under test lived. The extractor correctly labelled the assumption invented with no support, because the evidence had been deleted before it ever saw it. The instrument was inflating the number it exists to measure. Fixed to head+tail at 4,000 chars. The assumption disappeared entirely after the fix. On that sample the bug inflated the count by 100% (1 to 0).
2. The kit ingested its own LLM calls as tasks. 2 real tasks became 5, with Audit the following trace… and RUBRIC: 1) Dedupes on event_id… scored as user work.
3. Session-id exclusion does not fix #2. Under AAKIT_BACKEND=cli, claude -p can inherit the parent session id and append to the very transcript being measured. Filtering by session id would have discarded the real data along with the noise. It uses a sentinel in the prompt now, with prefix matching as a fallback for transcripts recorded before the sentinel existed.

Bugs 1 and 3 both bias the headline number upward. Both were caught by the "read three traces by hand" step in PROTOCOL.md days 1 and 2. Do not skip that step, and re-read traces after any change to truncation or caps.

---

Install

Stdlib only. Python 3.10+.

git clone https://github.com/abhixhek/aakit && cd aakit
python3 -m venv .venv && source .venv/bin/activate
pip install -e .
aakit init                  # creates ~/.aakit
aakit status

Optional extras:

pip install -e ".[api]"     # anthropic SDK, for AAKIT_BACKEND=api
pip install -e ".[duck]"    # duckdb, for `aakit export` to parquet

Backends

AAKIT_BACKEND	Needs	Notes
cli (default)	claude on PATH	Uses your existing Claude Code auth. No API key. Slowest.
api	ANTHROPIC_API_KEY	Fastest for batch runs.
heuristic	nothing	No model at all. Runs the full pipeline offline so you can debug plumbing without spending tokens. Its output is deliberately poor. Treat it as a floor, not a method.

---

Experiment 1: the base rate

aakit ingest --since-days 14     # parse Claude Code transcripts into tasks
aakit extract                    # LLM extracts assumptions per task
aakit calibrate                  # READ THIS BEFORE GOING FURTHER
aakit review                     # you adjudicate: wrong? material?
aakit metrics
aakit report --out run1.html

aakit calibrate exists because extractor bias is the biggest threat to this measurement. If the extractor is primed to find assumptions it will find them in tasks that had none, the base rate inflates, and the experiment confirms what it was built to confirm. Calibrate prints the provenance split and warns you when the distribution looks like padding: fewer than 15% zero-assumption tasks, more than 60% labelled invented, more than 6 per task on average.

The provenance split is the counter-argument made measurable. Only invented is a pure guess. from_evidence means the agent read something and extrapolated. If load-bearing-wrong concentrates there, the problem is misreading rather than assuming, and an assumption ledger is the wrong fix.

Why human adjudication

If a model both extracts and grades, the headline number is a model artifact and worth nothing. Only adjudicated_by='human' verdicts feed the headline. The kit also reports pre-pass agreement. Once that clears roughly 85% on verdict+material over 25 or more items, automating later batches with spot-checks is defensible. Until then it is not.

aakit auto-review --yes-i-know exists for smoke-testing only. Its output is marked provisional and excluded from headline metrics.

Running it continuously

Install the hook so tasks accumulate without you thinking about it:

chmod +x hooks/aakit_hook.py
# merge hooks/settings.snippet.json into ~/.claude/settings.json
# (replace /ABSOLUTE/PATH/TO)
export AAKIT_HOOK_EXTRACT=0     # ingest only; batch extraction manually

The hook never blocks or fails a session. A measurement tool that interferes with the work it measures contaminates the thing being measured.

---

Experiment 2: the ask-budget curve

aakit ab --tasks tasks/dataeng_ambiguity.jsonl --repeats 3 --max-questions 3
aakit metrics

Four policies over the same underspecified tasks:

• never: answer immediately, assume the rest
• always: ask up to --max-questions first
• divergence_gated: ask exactly one question, and only when the top-two readings of the request produce materially different artifacts
• gated_multi: the same gate, but the same budget as always

divergence_gated is the ClarifyDelphi criterion. A good question is one whose answer changes what you do.

gated_multi exists because divergence_gated confounds two things. It asks one question where always asks three, so when it loses you cannot tell whether the gate picked the wrong moment or simply ran out of budget. Running both separates discrimination from budget, and on a spec hiding several independent facts that distinction is the whole result. Report all four or the comparison is not interpretable.

Fairness details that matter, because this comparison is trivial to rig:

• The oracle answers only what the hidden spec covers. Anything else gets "No preference, your call", exactly like a real user who has not thought about it. So an always-ask policy burns budget on questions with no information. That is the real-world cost of over-asking, and the whole reason never is a respectable baseline.
• The grader never sees which policy produced the artifact.
• The bundled task set is 7 ambiguous + 5 adequately specified controls. Without controls, always-ask looks free and gating has nothing to be right about.
• --max-questions is the single most load-bearing knob. Report it with any result.
• Use --repeats 3 minimum. Single-sample LLM policy comparisons are noise.

Writing your own task set

JSONL, one object per line:

{"id": "...",
 "prompt": "the underspecified request the user actually sends",
 "hidden_spec": "what the user knows but did not say",
 "rubric": "checkable criteria a correct artifact must satisfy"}

Keep roughly a third as fully-specified controls.

Say how many facts each ambiguous task hides. The one thing the run above showed clearly is that a one-question budget fails on multi-fact gaps, so a task set where every ambiguous task hides exactly one fact will flatter the gate.

---

Experiment 3: the defeater loop

The piece nobody has shipped. A monitor that watches incoming evidence, decides which recorded assumptions it kills, computes the blast radius, and regenerates only that.

# 1. record evidence (a test run, or a correction you type)
aakit observe --task task_abc123 --repo ~/code/myproj --cmd "pytest -q"
aakit observe --task task_abc123 --text "the events table is partitioned by event_date, not ingest_date"

# 2. see which assumptions it defeats, and the blast radius
aakit defeats --evidence ev_xyz789
aakit confirm                       # you say whether each defeat was real

# 3. targeted repair vs full rerun, both in disposable repo copies
aakit repair --task task_abc123 --repo ~/code/myproj \
  --verify "pytest -q" \
  --correction "the events table is partitioned by event_date"

Structure is a stripped-down ATMS (de Kleer 1986):

assumption --justifies--> artifact
evidence   --defeats---->  assumption   (undermining | undercutting | rebutting)
blast radius = artifacts of defeated assumptions
             + artifacts of assumptions sharing those files (one hop)

--scope all tests every assumption in the task rather than only those whose artifacts the evidence mentions. It is slower and more expensive, but it tells you how many defeats the cheap file-based filter misses. That is the honest way to report monitor recall.

Everything runs against a copy of the repo in a temp dir. The kit never writes to the directory it is measuring.

Watch defeat_precision. Below roughly 70%, the monitor generates more pointless rework than it saves, and the loop is net negative regardless of token savings.

---

Layout

aakit/
  schema.py       records; vocabulary borrowed from Reiter/Doyle/de Kleer/AGM/ASPIC+
  config.py       paths, backend selection, redaction patterns
  llm.py          cli / api / heuristic backends, JSON coaxing with repair retry
  store.py        SQLite (stdlib), optional parquet export
  transcripts.py  Claude Code JSONL -> Tasks, with middle-out trace truncation
  extract.py      the extractor prompt (and its three anti-inflation measures)
  review.py       human adjudication loop with LLM pre-pass
  defeater.py     ATMS-lite: index, defeat detection, repair comparison
  askpolicy.py    experiment 2 runner + oracle + blind grader
  metrics.py      Wilson intervals, kill criteria
  report.py       HTML report
  cli.py          aakit entry point
hooks/            Claude Code SessionEnd hook
tasks/            example task set for experiment 2
PROTOCOL.md       the two-week run protocol and kill criteria. Read this first.

Privacy

Everything stays local. SQLite in ~/.aakit, traces on disk next to it. The only thing leaving your machine is what you send to whichever model backend you chose. config.redact_patterns strips API keys, tokens, private keys and emails from traces before extraction. Review that list against your own repos before the first run. It is a starting point, not a guarantee.