Skip to content

0007-action-set-over-binary-triage.md

Latest commit

 

History

History
85 lines (65 loc) · 4.35 KB

0007-action-set-over-binary-triage.md

File metadata and controls

85 lines (65 loc) · 4.35 KB
Raw

ADR-0007: Retain the graded action set despite a negative result

Status: Accepted, provisional. Revisit when M5 lands. Date: Phase 1, following M4

Context

The design assumption entering Phase 1 was that vulnerability response is badly served by a binary act/do-not-act decision, and that a graded action set — UPGRADE_DIRECT, UPGRADE_TRANSITIVE, PIN, WAIT_FOR_FIX, REQUEST_EXCEPTION, NO_ACTION_REQUIRED — would produce a better coverage-for-effort frontier than a threshold. M4 was built to test that at matched effort, and carried a stated kill criterion.

Measurement

Five policies, evaluated over (MEASURED) 3,991 CVEs holding EPSS and CVSS data, of which (MEASURED) 16 are KEV-positive.

Policy Coverage Efficiency Effort
full_action (MEASURED) 100.0% (MEASURED) 0.9% (ASSUMED) 7379
threshold_only (MEASURED) 100.0% (MEASURED) 88.9% (ASSUMED) 180
kev_only (MEASURED) 100.0% (MEASURED) 100.0% (ASSUMED) 160
cvss_ge7 (MEASURED) 81.2% (MEASURED) 1.0% (ASSUMED) 13580
epss_ge01 (MEASURED) 18.8% (MEASURED) 50.0% (ASSUMED) 60

Effort figures depend on the loss weights in config.py and are labelled ASSUMED accordingly. Coverage and efficiency are measured.

The hypothesis failed. The full action set matched threshold_only on coverage and cost roughly forty times the effort to do it. This is published in docs/findings/M4.md as a negative result.

Analysis

The result is real, and the measurement is nonetheless structurally unable to see the thing the action set exists for.

Both policies check KEV membership first and upgrade every KEV member, so both reach (MEASURED) 100% coverage of the positives by construction. PIN and WAIT_FOR_FIX therefore only ever apply to non-KEV CVEs — which, under a proxy where KEV = 0 is scored as a negative, are definitionally not worth acting on. Every graded action is counted as pure waste. ADR-0006 explains why that scoring is not a statement about the real world.

The value proposition of WAIT_FOR_FIX is that no fix exists yet, and of PIN that an upgrade is unsafe or unavailable. Neither of those is a claim about exploitation, so neither can be confirmed or refuted by an exploitation proxy. M4 measured the right thing against the only label available and the label cannot answer the question.

There is also a genuine policy defect visible in the numbers, independent of the proxy: full_action issued (MEASURED) 1,346 PIN actions across (MEASURED) 3,991 CVEs. Pinning a third of the dependency graph is not a defensible policy under any loss weighting. The PIN threshold is too broad.

Decision

  1. The graded action set is retained. WAIT_FOR_FIX and PIN correspond to real engineering outcomes that a binary policy cannot express, and M4 does not refute that; it demonstrates that a KEV-scored retrospective cannot evaluate it.
  2. The negative result stands as published. It is not re-scoped, and the loss weights are not tuned until the frontier reverses. Tuning ASSUMED weights to produce a preferred result is the failure mode this project's discipline exists to prevent.
  3. The action-set claim is removed from the headline results and marked unproven. It may not support an architecture argument until it is earned.
  4. The PIN threshold is treated as a defect and narrowed in Phase 2.
  5. The claim moves to M5, which measures change-failure rate on real upgrades against real test suites in the Phase 2 sandbox. RESOLUTION_FAILED, INSTALL_FAILED and TEST_REGRESSION outcomes are direct evidence for whether PIN and WAIT_FOR_FIX describe real states — and unlike KEV, that measurement has a ground truth we can observe rather than proxy.

Consequences

Accepted: the project carries an unproven design element into Phase 2, explicitly labelled as such. That is preferable to either deleting a mechanism on the strength of a measurement known to be blind to it, or asserting value the data does not support.

Accepted: if M5 also fails to show value, the action set is cut. That is the pre-committed condition, recorded here before the measurement runs so it cannot be reinterpreted afterwards.

Evidence

M4 (results/m4.json, docs/findings/M4.md), a negative result. The proxy limitation is ADR-0006. The retention decision is judgement, and is labelled provisional because of it.