For someone deciding whether to trust an autonomous coding agent with a real codebase. Every claim below has a command next to it. Run them; do not take our word for it.
We have deliberately not written a feature grid scoring ourselves against ten competitors. Those grids are written by the vendor being scored, the criteria are chosen by the vendor, and no reader can check a single cell. This page only makes claims you can falsify in a terminal.
npx loki-mode tour # no install, no API key, no spend, no networkOutput includes:
Headline: VERIFIED WITH GAPS
| Files changed | 8 |
| Diff sha256 | c2be6fff3e774c387f276277b25fc424f07b667… |
| Tests | verified (node-test) |
| Build | not_run |
| Security | findings |
| Cost | $10.3218 |
What to notice: the headline is not "SUCCESS". Build was not run. Security has findings. The receipt says so on its own front page.
Why that is the product. Every coding agent reports its own completion, and self-reporting is the thing they are structurally worst at. The receipt separates deterministic FACTS (diff hash, test result, cost) from AI ASSESSMENTS, because only four of our eight quality gates are agent-independent and a receipt implying otherwise would be marketing.
Check it yourself: recompute the diff sha256 over the same range and confirm it matches. If it does not, the receipt is worthless and you should not use us.
Deterministic verification makes zero network calls, so it runs inside a perimeter on code that may never leave the building.
bash tests/test-airgap-verify.shThat test blackholes every proxy variable, strips the environment, and asserts a real verdict still comes back. Measured: 8.43 ms.
Scope, stated honestly: verification is air-gapped. Generation is not. Every provider we ship calls a hosted API, and local-weight generation needs models you would supply. We are not claiming the generation half, and any vendor who claims a fully air-gapped LLM agent without shipping weights is worth a second question.
Brownfield is the harder problem, and the reason to distrust an agent near it is obvious. So the entry point writes nothing:
loki modernize heal ./your-repo --assess
git status # clean. no scratch files, no .loki/, no commits.Enforced, not promised:
bash tests/test-brownfield-assess-readonly.shThat test hashes every file before and after, compares HEAD, and requires a clean working tree. It is content-addressed, so it does not care how a write might happen.
cat benchmarks/results/cross-model-eval.json- Claimed: the same gates run, the same acceptance is checked, and the same receipt semantics apply regardless of which model is behind it.
- Explicitly NOT claimed: identical quality or identical speed across models. That is not deliverable and we do not assert it.
Measured runs are in that file with wall-clock and iteration counts. Two runs is two runs; it is not a benchmark suite, and the file says so.
python3 autonomy/lib/fast_verify.py --path . --diff-base HEAD~1Measured on this repository, 1,932 tracked source files: 11,040 ms before, 19 ms diff-scoped (298 ms cold, 87 ms warm). That is the difference between a check you run at the end and a check something else can call as a dependency.
The one competitive comparison we will publish, because you can rerun it:
bash tests/test-competitor-verify-surface.shIt runs --help on whatever agent CLIs are installed and checks whether any of
them exposes a command that verifies the agent's OWN output.
Measured 2026-07-30 against every one of the six named competitors that ships a local CLI:
| CLI | Verifies its own output |
|---|---|
| opencode 1.18.9 | no |
| aider 0.86.2 | no |
| codex-cli 0.146.0 | no |
| Claude Code 2.1.220 | no |
| cursor-agent 2026.05.24 | no |
| loki-mode 8.3.3 | yes (loki proof verify) |
Devin and Replit Agent ship no local CLI, so they are not covered and no claim is made about them.
The detail that makes this honest. aider matches eight times on
verify/proof/attest terms. All eight are --verify-ssl (TLS certificate
validation) and --git-commit-verify (git pre-commit hooks). Neither verifies
the agent's output. A count-based comparison would have scored aider 8 against
our 2 and concluded the opposite of the truth, so the test reads every hit
instead of counting them.
Scope. This inspects the CLI surface of locally installed tools. Devin and
Replit Agent ship no local CLI and are not covered. Absence from a --help
listing is also not proof of absence from a product: a web UI or an API could
expose something the CLI does not. If a competitor does ship output
verification, this test is how we would find out, and the claim above would be
corrected rather than defended.
Same task, same machine, artifact verified by content, five trials each:
| CLI | Success | Median | Range |
|---|---|---|---|
| opencode 1.18.9 | 5/5 | 11s | 4s to 265s |
| codex-cli 0.146.0 | 4/5 | 69s | 48s to 80s (one 300s timeout) |
Read the range, not just the median. opencode's first trial took 265 seconds and the next four took 4 to 12. A new user experiences the 265, not the 11. Reporting only the median would hide the thing they will actually feel.
The timeout is kept in. Dropping codex's one failure would turn 4-of-5 into an implied 5-of-5 and overstate reliability.
Five trials, not one, and that is the point. An earlier single-shot run of this same codex command timed out, and publishing it would have recorded "codex: timeout" as a fact about a competitor when the identical command completed in 51 seconds minutes later. The spread is the finding.
What this does not measure. One trivial file-creation task is not a proxy for build quality, multi-iteration work, or brownfield capability. It measures headless invocation latency and nothing else. aider, Claude Code and cursor-agent have not been run on this task; Devin and Replit Agent ship no local CLI. Cost is not compared: codex ran on a free tier and opencode's per-call cost was not recorded.
Stating this plainly, because you will find it out anyway and it is cheaper for both of us if you find it here.
- No published enterprise case studies. We have adoption signal (fork ratio well above the norm for a tool this size) but no named enterprise references.
- No independent third-party benchmark placement. The SWE-bench Verified leaderboard is months stale and every entry on it is self-reported, ours would be too.
- Only the CLI surface is audited, not whole products. Section 6 measures
every named competitor that ships a local CLI (opencode, aider, codex, Claude
Code, cursor-agent) and none exposes output verification. That is a real
measurement you can rerun, but it is a measurement of
--help, not of a product: a web UI or an API could expose something the CLI does not. Devin and Replit Agent ship no local CLI and are not covered at all. Treat this as "unclaimed on every surface we can reach", which is stronger than the earlier "unclaimed as far as we can verify" and still short of a proven first. - Generation is not air-gapped. See section 2.
When your agent says it finished, what artifact can I check that does not come from the agent's own narrative?
Ours is the Evidence Receipt, and section 1 is a two-minute test of whether the answer holds up. Ask the same question everywhere else.