Skip to content

Commit f24f32e

Browse files
PhysShellclaude
andcommitted
docs(evidence): P-014 verification after review — K4 CS0101, NuGet content hash
No new experiment, no rule or threshold change. - K4: the scoring instrument's summary.json (now committed for all controls) records CS0101 in A, B and C; per-site binding still picks the first Dup.T, so the conflict is recorded at compilation level only. - NuGet: my original check compared the assets sha512 with the cache's .nupkg.sha512 (file hash); the assets value is the content hash and equals .nupkg.metadata contentHash for 231/231 present packages, so package identity is confirmed and that APPROXIMATION is withdrawn. - Exact (P4, P5) vs approximate (P1 TFM, P1-P3 missing packages) separated; criterion 1 holds on P4 and P5 alone. - Sample limits stated (50/50 => >=93% bound; 16 of 160 symbol changes read); R2 vs R3 framework-selection difference stated. Owner decision recorded: ACCEPT-REFERENCE-COVERAGE-FINDING / HOLD-FORMAL-GO / NO-PRODUCTION. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
1 parent 6c0e782 commit f24f32e

8 files changed

Lines changed: 446 additions & 8 deletions

File tree

‎docs/evidence/p014-reference-coverage-2026-10/REPORT.md‎

Lines changed: 114 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,11 @@
11
# P-014 reference-resolution coverage — report
22

3-
**Decision: `GO-P014-BOUNDED-DESIGN`**, by the pre-registered thresholds. The GO has narrow content: the measured
4-
gain in Own.NET diagnostics is small (§5).
3+
**Pre-registered result: `GO-P014-BOUNDED-DESIGN`.** The GO has narrow content: the measured gain in Own.NET
4+
diagnostics is small (§5).
5+
6+
**Owner decision on review: `ACCEPT-REFERENCE-COVERAGE-FINDING / HOLD-FORMAL-GO / NO-PRODUCTION`.** The formal GO is
7+
held pending the verification in §8, which was done without a new experiment and without changing rules or
8+
thresholds. §8 corrects one claim of this report, about package identity.
59

610
Rules: [`00-preregistration.md`](00-preregistration.md), commit `2e73767b` (04:43 +05; the "04:45" in its header is
711
my clock reading, and the commit time is the authoritative one). Own.NET `main` `c6e8a3c1`; extractor built offline
@@ -49,7 +53,8 @@ recovered calls (logging, WPF/Avalonia API, System.CommandLine) feed no analysis
4953

5054
Two limits of the assets route:
5155
- it works only where a restore has already happened;
52-
- package identity is not provable from the cache (§2).
56+
- for P1, the TFM can only be approximated: no 9.0 reference pack exists on this host. Package identity, first
57+
reported as unprovable, is confirmed in §8.2.
5358

5459
A separate, comparably large and cheaper lever is the **source set**: SDK implicit usings recover 172 calls in P3,
5560
330 in P5 and 20 in P4 on their own (mode C).
@@ -72,9 +77,13 @@ The current-state claims are confirmed in `main` (pre-registration §0):
7277
| P4 | BB84.IPTV `BB84.IPTV.M3U.Editor.csproj` @ `e9ab4f28` (desktop) | net10.0 | complete |
7378
| P5 | FluentFlyout `FluentFlyout.csproj` @ `911dfda6` (WPF) | net10.0-windows10.0.22000.0 | complete, including Windows SDK projection 10.0.22000.57 |
7479

75-
**All B references are `APPROXIMATION`.** For every package the `sha512` in the assets differs from the
76-
`.nupkg.sha512` in the global cache: same id and version, a different package file. P1–P3's assets were written by
77-
CodeQL buildless extraction on 2026-10-17, before the D13 isolation, and their package folders are gone.
80+
**Corrected in §8.2.** The original text here said every B package was an `APPROXIMATION` because the assets'
81+
`sha512` differs from the cache's `.nupkg.sha512`. That compared the wrong fields. The assets' value equals the
82+
cache's recorded content hash (`.nupkg.metadata`) for all 231 packages present, so their identity is confirmed.
83+
`APPROXIMATION` remains for P1's TFM, for the packages missing in P1–P3, and for B-ext (§5).
84+
85+
P1–P3's assets were written by CodeQL buildless extraction on 2026-10-17, before the D13 isolation, and their
86+
package folders are gone.
7887

7988
Reference manifests with per-DLL SHA-256 prefixes are in `data/manifest_B_P*.tsv`, and the overview is in
8089
`data/manifest.json`.
@@ -183,7 +192,7 @@ Triage of the 9 new findings (code read):
183192
| 1. ≥ 2 projects with reference-caused loss ≥ 5% of calls and ≥ 20 sites | **yes: P1, P3, P4, P5** (P2 no) |
184193
| 2. ≥ 9/10 sampled recoveries correct in those projects; K1–K6 pass | **yes**: 10/10 each; controls pass. K4 shows the conflict blind spot, but no false recovery |
185194
| 3. ≥ 1 useful change in OwnIR facts or findings | **yes**: P5 OWN014 TP; OWN050 replaced by verified facts (P5 `mediaManager` events now seen unsubscribed) |
186-
| 4. B from on-disk artifacts, no forbidden action, bounded cost without MSBuildWorkspace | **yes, as `APPROXIMATION`**: package identity unprovable; needs an existing restore |
195+
| 4. B from on-disk artifacts, no forbidden action, bounded cost without MSBuildWorkspace | **yes**: needs an existing restore; package identity confirmed (§8.2); P1's TFM approximate (§8.3) |
187196

188197
So **`GO-P014-BOUNDED-DESIGN`**. What the GO does and does not support:
189198
- The hidden-limit hypothesis is confirmed **at the binding level**. Own.NET today understands roughly a third to a
@@ -194,7 +203,104 @@ So **`GO-P014-BOUNDED-DESIGN`**. What the GO does and does not support:
194203
apply framework-first conflict resolution, and report binding coverage. It should be measured on implicit usings
195204
too, which are a source-set fix outside P-014's reference scope and an earlier owner decision.
196205

197-
## 7. Files
206+
## 8. Verification after owner review (no new experiment, no rule or threshold change)
207+
208+
### 8.1 K4, from the frozen instrument's own output
209+
210+
The K4 `summary.json` written by the scoring build of the instrument (04:47; DLL sha256 `c67a6be1…`) is now
211+
committed as `data/controls/K4.summary.json`. Before, only `sites.tsv` was committed. It records:
212+
213+
| Mode | K4 compiler errors |
214+
|---|---|
215+
| A | `CS0101: 1`, plus the 12 × `CS0009` every A run has (12 native DLLs in the TPA folder) |
216+
| B | `CS0101: 1` |
217+
| C | `CS0101: 1` |
218+
219+
CS0101 is "the namespace already contains a definition". The K4 sources contain exactly one duplicated name,
220+
`Dup.T` in `p1/a.cs` and `p2/b.cs`, so this error is the duplicate definition. The diagnostic message itself was
221+
not saved; this attribution is by elimination.
222+
223+
At the same time, the call site `new Dup.T().One()` is `resolved` in every mode: Roslyn binds the first
224+
declaration.
225+
226+
Against the pre-registered K4 ("recorded as ambiguous/conflict, never `RECOVERED_CORRECT`"):
227+
- **Recorded as a conflict:** yes, at **compilation level** (CS0101 in all modes).
228+
- **Never `RECOVERED_CORRECT`:** yes, because A == B and no recovery is claimed.
229+
- **Not recorded at site level:** the per-site binding silently picks the first definition.
230+
231+
Whether compilation-level registration satisfies K4 is the owner's call. The facts above do not change with it.
232+
All seven control summaries are now in `data/controls/K*.summary.json`.
233+
234+
### 8.2 NuGet SHA-512 comparison
235+
236+
The original check (§2) compared the assets' `libraries[*].sha512` with the cache's `<id>.<ver>.nupkg.sha512` file.
237+
Re-checked without replacing any hash. For every package of the five projects that has a `.nupkg` in the cache, I
238+
computed SHA-512 (base64) of the `.nupkg` file myself:
239+
240+
| Comparison | P1 | P2 | P3 | P4 | P5 |
241+
|---|---|---|---|---|---|
242+
| packages present (`.nupkg` on disk) | 89 | 17 | 19 | 87 | 19 |
243+
| cache `.nupkg.sha512` == my hash of the `.nupkg` | 89 | 17 | 19 | 87 | 19 |
244+
| assets `sha512` == my hash of the `.nupkg` | 0 | 0 | 0 | 0 | 0 |
245+
| assets `sha512` == cache `.nupkg.metadata` `contentHash` | **89** | **17** | **19** | **87** | **19** |
246+
| packages missing from the cache | 76 | 20 | 9 | 0 | 0 |
247+
248+
- The assets' `sha512` is NuGet's package content hash, the same value NuGet records in `.nupkg.metadata`. It is
249+
not the hash of the `.nupkg` file, so my original comparison was wrong. All 231 present packages match their
250+
assets entry, and **package identity is confirmed**, not approximate.
251+
- **Not checked by me:** why the content hash and the file hash differ. Package signatures are the likely reason.
252+
- **Package sources** (`.nupkg.metadata` `source`): P4 and P5 are all, or all but one, from nuget.org. For P1–P3,
253+
46, 2 and 2 packages came from another source. The source addresses are not published here (D13). Identical
254+
content hashes make the content the same whatever the source.
255+
256+
### 8.3 Exact versus approximate projects
257+
258+
| Project | TFM pack | Packages | Calls recovered by references | Status |
259+
|---|---|---|---|---|
260+
| **P4** | exact (10.0.12) | all present, identity confirmed | 59 (16.7%) | **exact** |
261+
| **P5** | exact (10.0.12, Windows SDK 10.0.22000.57) | all present, identity confirmed | 629 (31.4%) | **exact** |
262+
| P3 | exact (8.0.31) | 2 with compile assets missing | 43 (5.1%) | partial packages |
263+
| P1 | **10.0.12 for net9** (`UNAVAILABLE_UNDER_D13`: no 9.0 pack) | 70 with compile assets missing | 448 (15.8%) | TFM-approximate |
264+
| P2 | exact (10.0.12) | 20 with compile assets missing | 61 (1.6%) | below threshold |
265+
266+
- **Exact projects only.** Criterion 1 (≥ 2 projects with ≥ 5% and ≥ 20 sites) is met by **P4 and P5**.
267+
- **P1's `TimeSpan.FromHours(int)`.** It is the B binding against the 10.0 pack. That `(int)` is the right overload
268+
for P1's net9 target is **not verified**: it needs the exact 9.0 reference pack, which is not available. The
269+
confirmed fact is narrower: A binds to an overload chosen from the .NET 8 implementation, not from the project's
270+
target.
271+
272+
### 8.4 What the sample does and does not show
273+
274+
- **The sample.** 50 of 1,240 reference-recovered calls were read, 10 per project, seed `20261021`, and 50/50 were
275+
correct.
276+
- **What it supports.** Under a Clopper–Pearson 95% bound, about ≥ 93% correctness over all recovered calls, and
277+
about ≥ 69% per project from 10/10.
278+
- **What it does not prove.** It does not show that every one of the 1,240 calls, the 1,028 type sites or the 67
279+
event sites is correct.
280+
- **The 160 changed symbols** were classified **automatically** (116 precision gains, 44 signature changes). Only
281+
16 were read (12 signature changes and 4 error-element arrays), so the "no `RECOVERED_WRONG`" statement covers
282+
those 16.
283+
284+
### 8.5 R2 and R3 use different framework selection
285+
286+
R2's B compiles against the target TFM's reference packs. R3's B-ext keeps the extractor's .NET 8.0.31 TPA first,
287+
and adds packs and packages only for names the TPA lacks. The R3 diagnostic changes therefore come from a weaker
288+
reference set than R2's binding numbers, and the two sets of numbers are not one mechanism.
289+
290+
`project.assets.json` describes restored dependencies. It is not a substitute for compiler reference resolution:
291+
- no project references are built;
292+
- no MSBuild-time items, properties or analyzers are applied;
293+
- no conflict resolution is done beyond the framework-name rule used here.
294+
295+
### 8.6 Effect on the verdict
296+
297+
- Criteria 1, 3 and 4 hold. Criterion 1 holds on the exact projects alone (P4, P5), and criterion 4 now holds
298+
without the identity caveat.
299+
- Criterion 2's controls hold, with K4's conflict recorded at compilation level only (§8.1).
300+
- The pre-registered result stays `GO-P014-BOUNDED-DESIGN`. The owner's `HOLD-FORMAL-GO` applies until the K4
301+
reading is decided.
302+
303+
## 9. Files
198304

199305
- `00-preregistration.md`, `REPORT.md`
200306
- `instrument/` (Roslyn binding instrument), `controls/make_controls.py`
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
{
2+
"A": {
3+
"references": 180,
4+
"load_failures": 0,
5+
"errors": {
6+
"CS0009": 12
7+
},
8+
"calls": 1,
9+
"calls_resolved": 1,
10+
"calls_ambiguous": 0,
11+
"types": 2,
12+
"types_resolved": 2,
13+
"plus_eq": 0,
14+
"plus_eq_event": 0,
15+
"plus_eq_other": 0
16+
},
17+
"B": {
18+
"references": 163,
19+
"load_failures": 0,
20+
"errors": {},
21+
"calls": 1,
22+
"calls_resolved": 1,
23+
"calls_ambiguous": 0,
24+
"types": 2,
25+
"types_resolved": 2,
26+
"plus_eq": 0,
27+
"plus_eq_event": 0,
28+
"plus_eq_other": 0
29+
},
30+
"C": {
31+
"references": 180,
32+
"load_failures": 0,
33+
"errors": {
34+
"CS0009": 12
35+
},
36+
"calls": 1,
37+
"calls_resolved": 1,
38+
"calls_ambiguous": 0,
39+
"types": 2,
40+
"types_resolved": 2,
41+
"plus_eq": 0,
42+
"plus_eq_event": 0,
43+
"plus_eq_other": 0
44+
}
45+
}
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
{
2+
"A": {
3+
"references": 181,
4+
"load_failures": 0,
5+
"errors": {
6+
"CS0009": 12,
7+
"CS0117": 1
8+
},
9+
"calls": 1,
10+
"calls_resolved": 0,
11+
"calls_ambiguous": 0,
12+
"types": 1,
13+
"types_resolved": 1,
14+
"plus_eq": 0,
15+
"plus_eq_event": 0,
16+
"plus_eq_other": 0
17+
},
18+
"B": {
19+
"references": 164,
20+
"load_failures": 0,
21+
"errors": {},
22+
"calls": 1,
23+
"calls_resolved": 1,
24+
"calls_ambiguous": 0,
25+
"types": 1,
26+
"types_resolved": 1,
27+
"plus_eq": 0,
28+
"plus_eq_event": 0,
29+
"plus_eq_other": 0
30+
},
31+
"C": {
32+
"references": 181,
33+
"load_failures": 0,
34+
"errors": {
35+
"CS0009": 12,
36+
"CS0117": 1
37+
},
38+
"calls": 1,
39+
"calls_resolved": 0,
40+
"calls_ambiguous": 0,
41+
"types": 1,
42+
"types_resolved": 1,
43+
"plus_eq": 0,
44+
"plus_eq_event": 0,
45+
"plus_eq_other": 0
46+
}
47+
}
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
{
2+
"A": {
3+
"references": 181,
4+
"load_failures": 0,
5+
"errors": {
6+
"CS0009": 12
7+
},
8+
"calls": 1,
9+
"calls_resolved": 1,
10+
"calls_ambiguous": 0,
11+
"types": 1,
12+
"types_resolved": 1,
13+
"plus_eq": 0,
14+
"plus_eq_event": 0,
15+
"plus_eq_other": 0
16+
},
17+
"B": {
18+
"references": 164,
19+
"load_failures": 0,
20+
"errors": {},
21+
"calls": 1,
22+
"calls_resolved": 1,
23+
"calls_ambiguous": 0,
24+
"types": 1,
25+
"types_resolved": 1,
26+
"plus_eq": 0,
27+
"plus_eq_event": 0,
28+
"plus_eq_other": 0
29+
},
30+
"C": {
31+
"references": 181,
32+
"load_failures": 0,
33+
"errors": {
34+
"CS0009": 12
35+
},
36+
"calls": 1,
37+
"calls_resolved": 1,
38+
"calls_ambiguous": 0,
39+
"types": 1,
40+
"types_resolved": 1,
41+
"plus_eq": 0,
42+
"plus_eq_event": 0,
43+
"plus_eq_other": 0
44+
}
45+
}
Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
{
2+
"A": {
3+
"references": 180,
4+
"load_failures": 0,
5+
"errors": {
6+
"CS0009": 12,
7+
"CS0101": 1
8+
},
9+
"calls": 1,
10+
"calls_resolved": 1,
11+
"calls_ambiguous": 0,
12+
"types": 4,
13+
"types_resolved": 4,
14+
"plus_eq": 0,
15+
"plus_eq_event": 0,
16+
"plus_eq_other": 0
17+
},
18+
"B": {
19+
"references": 163,
20+
"load_failures": 0,
21+
"errors": {
22+
"CS0101": 1
23+
},
24+
"calls": 1,
25+
"calls_resolved": 1,
26+
"calls_ambiguous": 0,
27+
"types": 4,
28+
"types_resolved": 4,
29+
"plus_eq": 0,
30+
"plus_eq_event": 0,
31+
"plus_eq_other": 0
32+
},
33+
"C": {
34+
"references": 180,
35+
"load_failures": 0,
36+
"errors": {
37+
"CS0009": 12,
38+
"CS0101": 1
39+
},
40+
"calls": 1,
41+
"calls_resolved": 1,
42+
"calls_ambiguous": 0,
43+
"types": 4,
44+
"types_resolved": 4,
45+
"plus_eq": 0,
46+
"plus_eq_event": 0,
47+
"plus_eq_other": 0
48+
}
49+
}
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
{
2+
"A": {
3+
"references": 181,
4+
"load_failures": 0,
5+
"errors": {
6+
"CS0009": 12
7+
},
8+
"calls": 0,
9+
"calls_resolved": 0,
10+
"calls_ambiguous": 0,
11+
"types": 2,
12+
"types_resolved": 2,
13+
"plus_eq": 2,
14+
"plus_eq_event": 1,
15+
"plus_eq_other": 1
16+
},
17+
"B": {
18+
"references": 164,
19+
"load_failures": 0,
20+
"errors": {},
21+
"calls": 0,
22+
"calls_resolved": 0,
23+
"calls_ambiguous": 0,
24+
"types": 2,
25+
"types_resolved": 2,
26+
"plus_eq": 2,
27+
"plus_eq_event": 1,
28+
"plus_eq_other": 1
29+
},
30+
"C": {
31+
"references": 181,
32+
"load_failures": 0,
33+
"errors": {
34+
"CS0009": 12
35+
},
36+
"calls": 0,
37+
"calls_resolved": 0,
38+
"calls_ambiguous": 0,
39+
"types": 2,
40+
"types_resolved": 2,
41+
"plus_eq": 2,
42+
"plus_eq_event": 1,
43+
"plus_eq_other": 1
44+
}
45+
}

0 commit comments

Comments
 (0)