Status: Accepted Date: Phase 1
CISA's Known Exploited Vulnerabilities catalogue is the only public, curated, dated record of vulnerabilities confirmed to be exploited in the wild. It is the obvious label for any supervised evaluation of a prioritisation policy, and using it that way is close to standard practice.
It is also the wrong shape for that job, in a way that is easy to miss and produces confidently wrong results.
KEV = 1 is strong evidence of exploitation. KEV = 0 proves nothing at all: it may mean
the vulnerability was never exploited, or that exploitation was not observed, or not
reported, or not yet catalogued. This is textbook positive-unlabeled data. Treating the
unlabelled majority as confirmed negatives converts an unknown into a false negative and
produces a false-negative rate that is arithmetic rather than empirical.
There is a second, sharper trap. EPSS targets roughly 2.5–3% observed exploitation activity per 30-day window; KEV covers on the order of 0.5% of CVEs cumulatively. Scoring EPSS against KEV as if KEV were the outcome EPSS predicts makes EPSS appear about five times overconfident purely because the two are measuring different things over different windows. A calibration curve drawn this way looks rigorous, is reproducible, and is meaningless.
M3B measured the KEV base rate in the relevant population directly: (MEASURED) 116 future KEV entries among (MEASURED) 268,639 scored CVEs, a rate of (MEASURED) 0.043%.
- KEV is used only as a positive-unlabeled proxy, and every result computed against it states that bound in the same paragraph as the number.
- No false-negative rate is computed against KEV. No recall figure is reported without the positive-unlabeled caveat attached.
- The word calibration is not used for any consumer-side evaluation of EPSS, and no reliability curve is produced. The available and honest question is early-warning utility: did the score rise before the KEV listing? M3B answers that, and reports lift over random rather than a calibration statistic.
- Every results file computed against KEV carries the caveat in its own JSON, so the qualification travels with the number instead of living only in prose.
Accepted: no clean supervised benchmark, and no accuracy headline. Policies are compared on coverage, effort and lift, all of which are harder to summarise in one figure.
Accepted: the M4 result is bounded by this and cannot be resolved by better analysis of the same data. ADR-0007 records the consequence.
Gained: the results survive scrutiny. A reviewer who knows this literature will check for exactly this error first, and finding it handled is worth more than a stronger number that is not.
M3B (results/m3b.json) established the base rate and the early-warning framing. The
constraint itself is a property of the data source rather than a measurement, and is
recorded as a standing rule in CLAUDE.md.