Durable vulnerability-response decisioning for open-source dependencies, operating under incomplete, evolving, and externally-modelled evidence.
Dependency scanning is a solved and commoditised problem. Deciding what to do about the output is not. An enterprise with a mature toolchain has more findings than it can action, and the work of choosing between them is done by senior engineers under three conditions that make it genuinely hard:
- Evidence is incomplete when the decision is made. CVEs are routinely published before enrichment, before machine-readable version identity, sometimes before a fix exists.
- Evidence keeps changing afterwards. NVD records are still being modified at a median of (MEASURED) 490.7 days after publication.
- Part of the evidence is a third-party model that gets replaced without consumer consent. EPSS is retrained and re-released on a schedule its consumers do not control.
Sentinel takes findings as input and produces defensible, auditable response decisions. It does not scan, and it does not research offence. See ADR-0001.
Measurement precedes architecture, and a measurement is allowed to kill the design. No
component enters the system without a stated measurement that forces it. Every number
carries a provenance label, and tools/check_claims.py enforces that mechanically in
make verify — an unlabelled figure in a findings document fails the build.
The reasoning is in ADR-0002. The
decision records are in docs/adr/. Findings are in
docs/findings/, one per measurement, written only after the results
JSON exists.
| Phase | State |
|---|---|
| 0 — Data thesis | Complete, tagged phase-0 |
| 1 — Decision core | Complete, tagged phase-1 |
| 2 — Executable plane | Complete, tagged phase-2 |
| 3 — Durability | Complete, tagged phase-3 |
| 4 — Effects & authority | Complete, tagged phase-4 |
| 5 — Evidence & adversarial | Complete, tagged phase-5 |
| 6 — Topology | Complete, tagged phase-6 |
| 7 — Production hardening | Complete, tagged phase-7 |
| 8 — Publish | This document |
Nine sources audited. Four hold full revision history (EPSS, CVE Records, GHSA, CISA Vulnrichment). NVD and OSV are current-state-only: no revision archive exists, so their past states cannot be reconstructed, only observed prospectively. Daily snapshotting of both began at corpus build time. No EPSS v5-beta dual-model archive was retained, which removed one of the two planned methods for M3A.
EPSS scores arrive within days of publication — upper-bound median (MEASURED) 8.5 days, inflated by biweekly sampling resolution, with the true figure likely lower. This weakened the original argument that EPSS latency justifies durable re-evaluation.
NVD enrichment timing is structurally unmeasurable for the reason M0 found. What is measurable is that records continue being modified at a median of (MEASURED) 490.7 days after publication, which is the durability argument on firmer ground: evidence matures over months, through channels other than EPSS.
The NVD enrichment cliff is real and dated. The "analyzed" rate fell from (MEASURED) ~73% before 2024-12-10 to (MEASURED) ~36–58% after, and sits at (MEASURED) ~42–66% after the 2026-04-15 change. CPE presence tracks that rate almost exactly, so what was lost is machine-readable version identity rather than a label.
CNA-authored CVSS held at (MEASURED) 77–90% throughout. The authority did not disappear; it moved, which is what makes multi-source corroboration a fix rather than a mitigation. See ADR-0005.
Holding the policy fixed and crossing four EPSS model boundaries produces (MEASURED) 15–49% excess decision-flip rate against matched non-boundary controls, at Z-scores of (MEASURED) 3.6 to 3162. The ordinary daily flip rate is (MEASURED) 0.01–1.71%.
A fixed policy changes its behaviour overnight when an upstream model is replaced, with no change to the consumer's code, the consumer's data, or the vulnerabilities themselves. This is the strongest result in the corpus and it earns model-promotion gating outright. See ADR-0004.
EPSS gives (MEASURED) 34.7× lift over random at k=1000 and a median (MEASURED) 244 days of advance warning before a KEV listing. Reported as early-warning utility, deliberately not as calibration: EPSS and KEV measure different things over different windows, and scoring one against the other produces a rigorous-looking number that means nothing. See ADR-0006.
The graded action set did not beat a binary threshold. It matched it on coverage and
cost roughly forty times the effort, and issued (MEASURED) 1346 PIN actions across
(MEASURED) 3991 CVEs — not a defensible policy under any weighting.
The result is published as it stands. The analysis in ADR-0007 argues the measurement is structurally blind to what the action set is for, since KEV cannot evaluate "no fix exists yet", and pre-commits to cutting the action set if M5 also fails to show value.
(MEASURED) 0.0% change-failure rate across (MEASURED) 93 green-baseline npm packages
under a require() smoke test. This is a floor, not a ceiling: the test is too weak to
support the conclusion that change-failure is negligible in general. The M5 kill criterion
does not fire, but the result is honestly labelled as a lower bound.
(MEASURED) 98.9% verification pass rate ((MEASURED) 92/93). OSV rescan caught (MEASURED) 1 case that the remediation tests missed. This proves independent verification adds value beyond "tests passed" — the verifier sees evidence the producer does not.
(MEASURED) 3991/3991 decisions replay bit-for-bit from their checkpoints. Kill-at-boundary test passes: simulating process death after every checkpoint pair, all decisions are intact on resume. Evidence and config digests verified.
(MEASURED) 0 duplicate effects across (MEASURED) 1364 submitted, (MEASURED) 682 timeouts
injected, and (MEASURED) 1364 retries. The idempotency key — a pure SHA256 of
(run_id, cve_id, action, evidence_digest, policy_version) — guarantees exactly-once
execution under crash recovery. Hash-chain ledger integrity intact after all chaos.
(MEASURED) 0/31,928 decision changes across (MEASURED) 8 injection payloads × (MEASURED)
3991 CVEs. Advisory text has no code path into the decision plane: the policy operates on
structured Evidence fields only. Injection resistance is architectural, not prompt-based.
Three of four specialist-agent arguments fail under measurement. Credential isolation is provided by adapters (Phase 5). Specialisation divergence from restricted evidence is information loss, not quality gain. Parallelism is irrelevant for a pure function. Only independent verification survives (M6: (MEASURED) 1/93 catch rate). Decision: collapse to single reasoning agent + isolated adapters + independent verifier. See ADR-0012.
| Component | Earned by |
|---|---|
| External-model promotion gating | M3A — 15–49% excess flip rate at every boundary |
| Multi-source corroboration and authority ranking | M2 — the NVD enrichment cliff |
| Durable re-evaluation | M1 — evidence matures over months, though not via EPSS |
| Daily NVD/OSV snapshotting | M0 — neither source is retrospectively reconstructable |
| Sandboxed untrusted-code execution | Phase 2 — Docker escape suite passes |
| Append-only checkpoint store | Phase 3 — 3991/3991 bit-for-bit replay |
| Idempotent effects ledger | Phase 4 — 0 duplicates under 682 injected timeouts |
| Injection-resistant decision plane | Phase 5 — 0/31,928 decision changes |
| Single-agent + adapters + verifier | M7 — 3/4 topology arguments fail under measurement |
- The prior claim that ~80% of new CVEs lose identity resolution. The measured drop is roughly 30 percentage points, and CNA-supplied data compensates for much of it.
- That EPSS latency forces durable re-evaluation. EPSS arrives in days.
- That the graded action set beats a threshold. It does not, on the evidence available.
- That multi-agent topology earns its complexity. It does not — 3/4 arguments fail under measurement. Only independent verification survives, and it is an information-restriction property, not a topology property.
No exploit code, no proof-of-concept development, no weaponisation research, and no scanning of systems not owned by the operator. The system reasons about the response to public advisories.
Phase 2 executes untrusted third-party code — test suites and package lifecycle hooks — and
runs only inside the sandbox specified in docs/SANDBOX.md: no host
socket, no credentials, ephemeral filesystem, enforced resource limits, default-deny network
with explicit registry egress. If the sandbox is not ready, the experiment does not run.
make setup Resolve and lock the toolchain (uv; uv.lock is committed)
make fetch Download all sources, verify digests, write corpus/manifest.json
make m0 Reconstructability audit
make m1 m2 m3a Phase 0 measurements
make m3b m4 Phase 1 measurements
make m5 m6 Phase 2 measurements (requires Docker)
make durability Phase 3 durability replay
make effects Phase 4 effects idempotency
make injection Phase 5 injection resistance
make m7 Phase 6 topology value
make verify lint + typecheck + tests(+coverage) + claims + engineering <- the gate
Raw corpus data is never committed. corpus/manifest.json carries the retrieval URL,
timestamp, schema version, digest and licence for every source, so the corpus is
reconstructable rather than vendored. The same discipline applies to the toolchain, which
is why uv.lock is in the repository.
Every quantitative claim carries exactly one label: MEASURED, DERIVED, SOURCE_FACT,
LITERATURE, ASSUMED, SIMULATED, or COUNTERFACTUAL. No ASSUMED or SIMULATED
quantity may support a headline result — the loss weights in config.py are assumptions,
so every figure derived from them is labelled and none of them appears above.
Measurement code and the results JSON it produced land in the same commit, so any number traces to the run that made it.
Full rationale, phase plan and kill criteria: docs/DECISIONS.md.
Engineering standards: docs/ENGINEERING.md.
Phase-by-phase findings: docs/findings/STATUS.md.
Seven hypotheses were stated before measurement and published regardless of outcome:
| Hypothesis | Status |
|---|---|
| H-A: Evidence arrives after deadline | Partially confirmed |
| H-B: Source-completeness shift | Confirmed |
| H-C: EPSS early-warning utility | Supported |
| H-D: Action-set beats threshold | Not confirmed (negative result published) |
| H-E: Model transitions flip decisions | Strongly confirmed |
| H-F: Verification adds value | Partially supported |
| H-G: Topology earns its complexity | Not supported (collapsed to single agent) |
An architecture document that names the measurements that would have falsified it is doing something almost nothing in this space does.