Skip to content

0002-measurement-before-architecture.md

Latest commit

 

History

History
64 lines (49 loc) · 3.21 KB

0002-measurement-before-architecture.md

File metadata and controls

64 lines (49 loc) · 3.21 KB
Raw

ADR-0002: Measure before building, and let a gate kill the design

Status: Accepted Date: Phase 0

Context

An earlier draft of this system was designed the usual way: architecture first, with the supporting figures filled in from plausible estimates and industry rules of thumb. Every number in it was confident and none of it was computed. The design that resulted was internally coherent and unfalsifiable, which is the worst combination available — it could not be wrong in a way that would show up before production.

Two specific claims from that draft are worth naming because both turned out to be wrong in the measurable direction:

  • That the December 2024 CISA enrichment change caused roughly 80% of new CVEs to lose identity resolution. M2 measures the drop in NVD "analyzed" rate as (MEASURED) ~73% falling to (MEASURED) ~36–58%. A materially narrower failure than claimed, and one with a compensating mechanism the draft did not account for.
  • That EPSS latency is what forces durable re-evaluation. M1 measures first-EPSS at an upper-bound median of (MEASURED) 8.5 days, with the true figure likely lower. The durability argument survives, but on entirely different grounds (see ADR-0005).

Had either claim gone unmeasured, the architecture built on it would have been wrong in a way no test would have caught.

Decision

Measurement precedes architecture, and a measurement is permitted to kill the design.

  1. Every quantitative claim carries one label: MEASURED, DERIVED, SOURCE_FACT, LITERATURE, ASSUMED, SIMULATED, COUNTERFACTUAL. No ASSUMED or SIMULATED quantity may support a headline result.
  2. tools/check_claims.py enforces this mechanically against results/*.json and runs in make verify. It is a gate, not a linter.
  3. Measurement code and the results JSON it produced land in the same commit, so any number traces to the run that made it.
  4. Findings documents are written after the results exist. Never scaffolded with expected values to be filled in later, which is the specific mechanism by which invented numbers survive into a final draft.
  5. Each phase carries a stated kill criterion. When one fires, work stops and is reported; it is not designed around.
  6. No component enters the system without a sentence naming the measurement that forces it. Queues, caches, workflow engines, additional agents and databases are all subject to this. Kubernetes, Kafka, multi-region and graph databases are refused by default.

Consequences

Accepted: the project is slower to a demo. Phase 0 produced no running system at all, only a corpus and four measurements.

Accepted: some measurements return negative results and are published as such. M4 did exactly this, and the finding stands in docs/findings/M4.md rather than being quietly re-scoped. ADR-0007 records what was done about it.

Gained: every architectural claim in this repository is falsifiable by a reader with the committed corpus manifest and a network connection. That is the only property that distinguishes this from a well-written proposal.

Evidence

Retrospective on the prior draft, plus M1 and M2, both of which contradicted a specific prior claim in a direction that changed the design.