Skip to content

Commit cb9519d

Browse files
PhysShellclaude
andcommitted
docs(evidence): P-014 Tiny PoC — PASS-P014-TINY-POC (metrics only)
Per-project compilations from existing assets/packs: foreign_binding 0 on all controls and on P4 (4 units) and P5; K8a/K8b/K8c/K9/K3 manifests equal the SDK oracle; invocations resolved P4 62.0% vs 25.1% legacy, P5 64.1% vs 32.1%; 22 P4 legacy-only sites all read: implicit usings not applied surface at the use site across project boundaries (legacy over-count), no wrong binding; P5 warm cost 1.43x legacy, peak ~208 MB. No production, OwnIR or verdict change. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
1 parent 35e2c8c commit cb9519d

26 files changed

Lines changed: 10729 additions & 0 deletions
Lines changed: 220 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,220 @@
1+
# P-014 Tiny PoC — report
2+
3+
**Verdict: `PASS-P014-TINY-POC`.** All six pre-registered criteria hold (§6).
4+
- This is a PASS for a **research instrument** that measures binding. It is not a PASS for any change to the
5+
extractor, OwnIR or public diagnostics, and it produces no findings.
6+
- Production stays NO-GO.
7+
8+
Rules: [`00-preregistration.md`](00-preregistration.md).
9+
- Commit `5a9bb32`, 11:04:52 +05. The "11:10" in its header is my clock reading; the commit time is
10+
authoritative.
11+
- Amendment 1 is commit `207ef84`, 11:06:54. Both were committed before the instrument existed.
12+
13+
Base `main` `228e38d7`. Work: 11:00 to about 11:25 +05 on 2026-10-21, inside the 0.25 agent-day budget.
14+
15+
Labels used below:
16+
- **[run]** measured here;
17+
- **[read]** checked by reading code or diagnostics;
18+
- **[doc]** dotnet/sdk source;
19+
- **[inference]** not verified.
20+
21+
## 1. What was built
22+
23+
`instrument/` contains a research program (Roslyn 4.9.2, the extractor's version), built offline. For one root
24+
`.csproj` it does the following:
25+
- **Graph.** It reads the existing `obj/project.assets.json` of the root and of every project-reference member,
26+
and builds **one `CSharpCompilation` per project**. Project references become `CompilationReference`s.
27+
- **References.** Each unit's references are selected by the pre-registered contract (§1 of the
28+
pre-registration):
29+
- targeting-pack `FrameworkList.xml` with profiles;
30+
- package `compile` assets with the content-hash identity check;
31+
- conflicts keyed by file name, decided in the dotnet/sdk order: overrides → AssemblyVersion → FileVersion →
32+
(rank/platform).
33+
- **Output.** It writes the deterministic coverage sidecar (DESIGN §4), with a built-in isolation guard.
34+
- **Source set.** It uses the extractor's own text rules, with default parse options. **No implicit usings and no
35+
defines are applied to any real project.** They are only reported.
36+
37+
No restore or build of third-party code, no network, no MSBuild evaluation of P4/P5. The SDK was run only as an
38+
**oracle on my own synthetic fixtures**, restored offline with the global packages folder as the only source.
39+
40+
## 2. Environment change (recorded before the first run)
41+
42+
The host's .NET was replaced at about 11:00, and I did not make the change:
43+
- SDK 10.0.401 became 11.0.100-rc.2;
44+
- packs 8.0.31/10.0.12 became 8.0.32/10.0.13.
45+
46+
P4/P5 therefore resolve as `PACK_MAJOR_MINOR` against 10.0.13. Whether 10.0.12 and 10.0.13 have the same API
47+
cannot be checked any more.
48+
49+
## 3. Controls
50+
51+
**In memory** [run, `data/fixtures/controls.json`]: K1, K2, K4, K5, K6, K7, F2.3, F2.4, F2.5, K11, K12 and the guard
52+
self-test all **PASS**.
53+
54+
| Control | Observed |
55+
|---|---|
56+
| K2 | `Lib` 1.0 / 2.0: each unit binds its own version; no CS1704 |
57+
| K4 | merged: CS0101. Per unit: U2's `One()` unresolved + CS1061. U3 referencing U1 and U2: type site **ambiguous via CS0433** |
58+
| F2.3 | U2's `File.Exists` unresolved (no leaked implicit usings) |
59+
| F2.4 | U2 binds `System.Timers.Timer.Elapsed` (event) |
60+
| F2.5 | U3 without `Dep`: unresolved, foreign = 0 |
61+
| GUARD | the merged shape's binding of U3's event, judged against U3's own manifest: flagged (3 foreign symbols) |
62+
| K11 / K12 | `unresolved_no_marker = 2` (lambda, `new EventHandler`); `name_heuristic = 1`, not counted resolved |
63+
64+
**On disk, against the SDK oracle** [run, `data/fixtures/*.compare.json`]. The SDK oracle is
65+
`FindReferenceAssembliesForReferences` → `ReferencePathWithRefAssemblies`. Both sides are compared by path and
66+
assembly version.
67+
68+
| Control | Instrument vs oracle | Conflicts decided |
69+
|---|---|---|
70+
| K8a (`net8.0` + SCI/SRM 9.0.0) | **163 = 163**, identical | both packages win, rule `assembly-version` (9.0.0.0 > 8.0.0.0) |
71+
| K3 (same fixture) | compile asset `lib/net8.0`, same as oracle | — |
72+
| K8b (`net8.0` + System.Text.Json 8.0.5) | **163 = 163** | framework wins, rule `file-version` (equal AssemblyVersion 8.0.0.0) |
73+
| K8c (`net10.0-windows`, WPF only) | **206 = 206**; no `System.Windows.Forms.dll` | `WindowsBase.dll` pack copy by `assembly-version` |
74+
| K9 (`net8.0;net10.0`) | without `--tfm`: `UNSUPPORTED(multi-tfm)`; with `--tfm`: 163 = 163 and 167 = 167 | — |
75+
| K10 (project reference) | with child assets: two `COMPLETE` units, event resolved; child `obj` removed: root `SOURCE_SET_INCOMPLETE(p2p-no-assets:Child)`, event unresolved, foreign = 0 | — |
76+
77+
`TIE_FRAMEWORK_ASSUMED` was never needed.
78+
79+
**Not covered by the oracle:** a **package-vs-package** conflict. P5 has one (`Wpf.Ui.dll`, decided by
80+
`assembly-version`), and it was not checked against a real build.
81+
82+
## 4. Real projects P4, P5 (#419 anchors)
83+
84+
**Status** [run, `data/real/P*.exact.sidecar.json`]. All 5 units (P4 root + 3 project-reference members, P5)
85+
have:
86+
- reference status `COMPLETE`;
87+
- `PACK_MAJOR_MINOR`;
88+
- `foreign_binding = 0`;
89+
- no failed invariants;
90+
- no CS0433/CS1704/CS0012/CS0518.
91+
92+
All are `SOURCE_SET_INCOMPLETE`, with named reasons:
93+
94+
| Reason | Units |
95+
|---|---|
96+
| `package-analyzers:*` (generators from packages, e.g. `bb84.sourcegenerators`, `communitytoolkit.mvvm`) | all 5 |
97+
| `directory-build:*` | all 5 (P4 sets `ImplicitUsings` in its own `Directory.Build.props`; P5's only hit is a **host** file above all repositories, `source/repos/Directory.Build.targets`, so the detector is conservative) |
98+
| `wpf-xaml-codegen`, `define-constants`, `implicit-usings-not-applied`, `framework-generator:LibraryImport` | P5 |
99+
| `framework-generator:GeneratedRegex` | P4 Application |
100+
101+
**Binding coverage, `exact-project` vs `legacy-merged`** (today's shape inside the same instrument) [run]:
102+
103+
| | P4 (4 units) | P5 |
104+
|---|---|---|
105+
| invocations resolved | **1798 / 2901 (62.0%)** vs 729 / 2901 (25.1%) | **1284 / 2003 (64.1%)** vs 643 / 2003 (32.1%) |
106+
| type sites resolved | 1597 / 2341 vs 1183 / 2341 | 775 / 1049 vs 299 / 1049 |
107+
| `+=` bound to an event | 23 of 34 (1 unresolved) vs 7 (17 unresolved, 13 with no OWN050 marker) | 33 of 65 (8 unresolved, all without marker) vs 7 (34 unresolved, 19 without marker) |
108+
| disposable-type sites | 24 resolved, 0 by name vs 0 | 13 resolved + 3 by name vs 3 + 3 |
109+
| methods whose relevant sites all resolve | 20 / 23 | 23 / 37 |
110+
| error diagnostics | 1667 vs 2595 | 1352 vs 2741 |
111+
112+
**Comparison with #419 B** (counts only, per Amendment 1):
113+
- **P5.** Same denominator, 2003 calls. #419 B resolved 1272; exact-project resolves 1284, under the stricter rule
114+
that no error may sit on the site. #419 B had 59 × CS0433 "between two framework references"; exact-project has
115+
0. The cause is probably the WPF profile filter [inference].
116+
- **P4 root.** #419 counted 353 calls over 23 files. The extractor's rules, and so this instrument, exclude
117+
`Properties/Resources.Designer.cs`, which holds 129 `GetString`/`GetObject` calls [read]. Hence 223. Type-site
118+
definitions differ between the instruments, so type counts are not compared.
119+
120+
**Sites resolved in legacy and not in exact (`lost`)** [run, `data/real/P*.lost.json`, `P4.lost.diag.txt`]:
121+
- **Counts.** P5: 0. P4: 22. All 22 were read, more than the 10 required.
122+
- **Cause.**
123+
- `ImplicitUsings=enable` is set in P4's `Directory.Build.props` and is not applied (owner rule). So `Task`,
124+
`Exception`, `IDisposable` and `IHttpClientFactory` are error types **in the Application unit's own
125+
declarations**.
126+
- Across the `CompilationReference`, the root reports them **at the use site**: CS0246 at 15 call and type
127+
sites. In Infrastructure, the interface signatures no longer match their implementations: CS0535 at 7 base-list
128+
sites.
129+
- The legacy merged compilation binds the same methods and types, but reports the error only once, at the
130+
declaration, so it counts a call whose signature contains error types as "resolved".
131+
- **Verdict.** In every case exact-project's result is **unresolved, not a different symbol**. None is a wrong
132+
binding. The difference is an **over-count in legacy**: a source-set limit shows up honestly per project instead
133+
of being hidden.
134+
135+
**The 7 #419 R3 sites** [run, `data/real/R3.sites.txt`] are all `+=` sites bound to an `IEventSymbol` in
136+
exact-project:
137+
138+
| Site | Bound event |
139+
|---|---|
140+
| P5 actionable OWN014 | `CommunityToolkit.Mvvm.ComponentModel.ObservableObject.PropertyChanged` |
141+
| P5 FP | `Window.SourceInitialized` |
142+
| P4 FPs | `IControlledApplicationLifetime.Startup`/`.Exit`, `TopLevel.Closed` |
143+
| P5 TP-benign `mediaManager` field | type `WindowsMediaController.MediaManager`, resolved |
144+
145+
The six FPs are therefore correctly bound events, which confirms DESIGN §5: their precision problem is the
146+
lifetime model, not references. No finding was computed and none changed: the extractor was not run with this
147+
mechanism.
148+
149+
## 5. Cost
150+
151+
[run, `data/real/*.timing.json`]. Each figure is the median of 3 separate processes; each process runs a cold
152+
iteration (first build in the process) and a warm iteration (second full build, metadata cache kept). Peak is the
153+
highest process peak working set.
154+
155+
| | legacy cold / warm | exact cold / warm | exact / legacy warm | peak (legacy / exact) |
156+
|---|---|---|---|---|
157+
| P4 (4 units) | 2260 / 1611 ms | 2031 / 1642 ms | 1.02× | 232 / 202 MB |
158+
| **P5** | 1548 / 969 ms | 1776 / 1386 ms | **1.43×** (limit 3×) | 209 / 208 MB (limit 2.0 GB, 3×) |
159+
160+
**Context only** [run, `data/real/extractor_context.log`]. Today's extractor, built offline from `228e38d7`, with
161+
`--flow-locals --no-project-refs`:
162+
- P4: about 1.0 s, about 171 MB;
163+
- P5: about 2.5 s, about 202 MB.
164+
165+
**Its facts are byte-identical across 3 runs** (P4 `AD528B8A1E92F4C8…`, P5 `5F0EA76DA86C623A…`). The instrument never
166+
touches them.
167+
168+
## 6. Verdict against the pre-registered criteria
169+
170+
| # | Criterion | Result |
171+
|---|---|---|
172+
| 1 | Isolation: `foreign_binding = 0` everywhere; K2, K4, F2.3, F2.4, F2.5 | **holds**; guard self-test catches the merged-shape binding |
173+
| 2 | References: K8a/b/c, K9, K3 = oracle; no tie assumption; K10 | **holds** |
174+
| 3 | Honest coverage: K5, K6, K7, K11, K12; invariants; byte-identical sidecars on rerun | **holds**: sidecars identical across 3 processes for every mode × project |
175+
| 4 | Real projects: complete without crash; named statuses; every lost site read has a cause, none a wrong binding | **holds**: 22/22 read, one cause |
176+
| 5 | Cost: P5 warm ≤ 3× legacy; peak ≤ 2.0 GB and ≤ 3× legacy | **holds**: 1.43×; about 208 MB |
177+
| 6 | Boundary: nothing outside `docs/evidence/p014-tiny-poc-2026-10/` | **holds** (`git diff --stat 228e38d7`) |
178+
179+
**`PASS-P014-TINY-POC`.**
180+
181+
## 7. Deviations and defects (disclosed)
182+
183+
1. **Amendment 1.** The #419 per-site B lists were never committed. Per-site comparison was replaced by legacy →
184+
exact lost sites, before any run.
185+
2. **Instrument defects, fixed before any real-project run:**
186+
- a type node inside `new T()` has no `TypeInfo`, so `GetSymbolInfo` is used;
187+
- the K1 check depended on probe order (the observation was already correct);
188+
- the comparator read the oracle's package version from `AssemblyVersion` metadata, which is empty for
189+
packages; `FusionName` is used instead.
190+
3. **A first fixture run used a stale binary.** The fixtures had been copied inside the instrument's build folder,
191+
so the build picked up their `.cs` files and failed. That run was discarded. Every reported fixture result comes
192+
from `scripts/run_fixtures.ps1` on a clean folder.
193+
4. **The `diag` mode** (errors and sites on one line) was added after the real runs, for cause analysis only. The
194+
measured `project`/`lost` code is unchanged from commit `35e2c8c`.
195+
5. **Source-set detection is conservative and coarse:**
196+
- any `Directory.Build.*` above the project counts, including a host file;
197+
- `ImplicitUsings` set in `Directory.Build.props` is reported only as `directory-build`.
198+
199+
## 8. What this does not show
200+
201+
- Any effect on findings, precision or OwnIR: nothing was emitted or judged.
202+
- Package-vs-package conflict correctness, the rank/platform steps, and RID-specific or `netstandard`/.NET Framework
203+
targets.
204+
- That `PACK_MAJOR_MINOR` (10.0.13) equals the pack P4/P5 were built with.
205+
- Generality beyond P4, P5 and the fixtures. P1–P3 were not run.
206+
- The cost of very large closures. P4 has 4 units.
207+
208+
## 9. Files
209+
210+
- `00-preregistration.md`, `REPORT.md`
211+
- `instrument/`: `Units.cs` (assets, packs, conflicts, source set), `Coverage.cs`, `Controls.cs`, `Program.cs`,
212+
`P014Poc.csproj`
213+
- `fixtures/`: K8a, K8b, K8c, K9, K10, my own synthetic projects
214+
- `scripts/`: `run_fixtures.ps1`, `run_real.ps1`, `run_extractor_context.ps1`, `compare_oracle.py`
215+
- `data/fixtures/`: controls, sidecars, oracle comparisons, log
216+
- `data/real/`: sidecars (run 1 of each mode), all timing files, lost sites, diagnostics at lost sites, R3 sites,
217+
logs
218+
219+
Local paths are replaced by `<nuget>`, `<dotnet>`, `<research>` and `<work>`. The oracle JSON files themselves are
220+
not committed; their comparison results are.
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"mode": "exact",
3+
"project": "evidence/pairs/BoBoBaSs84_BB84.IPTV__36/base/src/BB84.IPTV.M3U.Editor/BB84.IPTV.M3U.Editor.csproj",
4+
"timing": [
5+
{
6+
"iteration": 0,
7+
"kind": "cold",
8+
"wall_ms": 2160,
9+
"peak_working_set_mb": 166.0390625
10+
},
11+
{
12+
"iteration": 1,
13+
"kind": "warm",
14+
"wall_ms": 1642,
15+
"peak_working_set_mb": 194.54296875
16+
}
17+
]
18+
}
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"mode": "exact",
3+
"project": "evidence/pairs/BoBoBaSs84_BB84.IPTV__36/base/src/BB84.IPTV.M3U.Editor/BB84.IPTV.M3U.Editor.csproj",
4+
"timing": [
5+
{
6+
"iteration": 0,
7+
"kind": "cold",
8+
"wall_ms": 2031,
9+
"peak_working_set_mb": 165.0859375
10+
},
11+
{
12+
"iteration": 1,
13+
"kind": "warm",
14+
"wall_ms": 1670,
15+
"peak_working_set_mb": 199.8046875
16+
}
17+
]
18+
}
Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
{
2+
"mode": "exact",
3+
"project": "evidence/pairs/BoBoBaSs84_BB84.IPTV__36/base/src/BB84.IPTV.M3U.Editor/BB84.IPTV.M3U.Editor.csproj",
4+
"timing": [
5+
{
6+
"iteration": 0,
7+
"kind": "cold",
8+
"wall_ms": 2015,
9+
"peak_working_set_mb": 165.72265625
10+
},
11+
{
12+
"iteration": 1,
13+
"kind": "warm",
14+
"wall_ms": 1608,
15+
"peak_working_set_mb": 201.79296875
16+
}
17+
]
18+
}

0 commit comments

Comments
 (0)