Skip to content

Commit 2e73767

Browse files
PhysShellclaude
andcommitted
docs(evidence): P-014 reference coverage — pre-registration, controls, instrument
Current-state claims checked in main c6e8a3c (single compilation, TPA of the net8 extractor process, recursive first-filename-wins ref-dir, no MSBuild evaluation, no binding metric; default parse options, no implicit usings). Five projects chosen by rule (3 sweep repos with assets, 2 desktop pairs), modes A (current) / B (TFM ref packs + assets compile assets, APPROXIMATION: package sha512 differs from the cache) / C (source-set probe), controls K1-K7, GO/KILL thresholds. Fixed before any measuring run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
1 parent c6e8a3c commit 2e73767

4 files changed

Lines changed: 421 additions & 0 deletions

File tree

Lines changed: 147 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,147 @@
1+
# P-014 reference-resolution coverage — pre-registration
2+
3+
Written 2026-10-21 04:45 +05, before any measuring run. Budget: 0.25 agent-day to the first GO/KILL decision,
4+
counted from 04:37. Not extended.
5+
6+
## 0. Current state, checked in code (`main` `c6e8a3c1`, `frontend/roslyn/OwnSharp.Extractor/Program.cs`)
7+
8+
| Claim | Verdict | Where |
9+
|---|---|---|
10+
| All inputs go into one `CSharpCompilation.Create("own", …)` with default options | confirmed | `:5983-5985` |
11+
| The framework set is the extractor **process's** `TRUSTED_PLATFORM_ASSEMBLIES`; the extractor targets `net8.0`, so on this host that is the Microsoft.NETCore.App **8.0.31 implementation**, not the target TFM's reference pack | confirmed | `:5889-5894`; `OwnSharp.Extractor.csproj:5` |
12+
| `--ref-dir` and the auto-added project `bin/` are loaded recursively, **first simple filename wins**, and any name already in TPA is skipped | confirmed | `:5911-5933`, `:536-547` |
13+
| No MSBuild evaluation: a `.csproj` is resolved by text to a source set | confirmed | `:276-292` |
14+
| No quantitative semantic-binding report; `--stats` counts flow-locals coverage only | confirmed | `:5993-5997` |
15+
| Also seen: trees are parsed with default `CSharpParseOptions`, with no preprocessor symbols; SDK implicit/global usings are not generated | confirmed | `:5867` |
16+
17+
The last row is a **source-set** effect, not a reference effect. It is measured separately (mode C) and never
18+
credited to references.
19+
20+
## 1. Inputs (fixed now; chosen by rule, not by expected gain)
21+
22+
**Rule.**
23+
1. Take every corpus repository that has a `project.assets.json` already on disk and whose sources are present:
24+
the 3 design-B sweep repositories that have one, plus the direction-research pairs whose assets list packages
25+
that all exist by id/version in the local global cache.
26+
2. From the sweep repositories take all 3. From the qualifying pairs take the 2 that are WPF/desktop
27+
applications (BB84.IPTV, FluentFlyout); the other qualifying ones are libraries (LiteDB, MarkView).
28+
3. In each repository take the largest non-test, non-sample project by `.cs` count.
29+
30+
| # | Project | SHA | Kind | TFM measured | Assets packages / present by id in global cache |
31+
|---|---|---|---|---|---|
32+
| P1 | Flow-Launcher/Flow.Launcher — `Flow.Launcher/Flow.Launcher.csproj` | `2efbf865` | WPF app, 3 project references | `net9.0-windows10.0.19041.0` | 165 / 89 |
33+
| P2 | git-ecosystem/git-credential-manager — `src/Core/Core.csproj` | `e66693fa` | library | `net10.0` | 37 / 17 |
34+
| P3 | nats-io/nats.net — `src/NATS.Client.JetStream/NATS.Client.JetStream.csproj` | `176e52bc` | multi-targeted library, 1 project reference | **`net8.0`** (fixed; of netstandard2.0/2.1/net8.0/net10.0) | 28 / 19 |
35+
| P4 | BoBoBaSs84/BB84.IPTV — `src/BB84.IPTV.M3U.Editor/BB84.IPTV.M3U.Editor.csproj` (pair #36 base) | `e9ab4f28` | desktop app, 3 project references | `net10.0` | 87 / 87 |
36+
| P5 | MERURUXD/FluentFlyout — `FluentFlyoutWPF/FluentFlyout.csproj` (pair #3 base) | `911dfda6` | WPF app | `net10.0-windows10.0.22000.0` (first listed) | 19 / 19 |
37+
38+
**Provenance of the artifacts** (recorded, not trusted):
39+
- **P1–P3.** The assets were written on 2026-10-17 between 17:22 and 17:30 by CodeQL buildless extraction. That
40+
is before the D13 isolation (2026-10-18). Their `packageFolders` (`codeqldb/working/packages`, the VS shared
41+
folder) no longer hold the packages.
42+
- **P4–P5.** The assets come from the isolated pair restores of 2026-10-18.
43+
- **Package hashes.** For **every** package of all five, the `sha512` in the assets differs from the
44+
`.nupkg.sha512` in the global cache. Same id and version, different package file. Package identity is therefore
45+
**not provable**, and every mode-B reference is labelled `APPROXIMATION`.
46+
47+
**Local SDK/TFM.**
48+
- Runtimes: NETCore.App 6.0.36 / 8.0.31 / 10.0.12; AspNetCore.App and WindowsDesktop.App 8.0.31 / 10.0.12.
49+
- Reference packs: Microsoft.NETCore.App.Ref, AspNetCore.App.Ref and WindowsDesktop.App.Ref 8.0.31 and 10.0.12;
50+
NETStandard.Library.Ref 2.1.0.
51+
- **There is no 9.0 reference pack**, so P1's exact TFM is `UNAVAILABLE_UNDER_D13`. P1's mode B uses the 10.0.12
52+
packs and is labelled `TFM-APPROX`.
53+
54+
## 2. Modes (one instrument, identical sources and options in every mode)
55+
56+
**Common to all modes.**
57+
- **Sources.** All `.cs` files under the project directory, excluding `bin`/`obj`, as the extractor's `.csproj`
58+
path resolves them. The list is written to disk once and used by every mode.
59+
- **Parse options.** `CSharpParseOptions.Default` (no defines), as the extractor uses.
60+
- **Compilation options.** `new CSharpCompilationOptions(DynamicallyLinkedLibrary)`.
61+
62+
| Mode | References |
63+
|---|---|
64+
| **A, current** | every DLL of `shared/Microsoft.NETCore.App/8.0.31`, i.e. the extractor process's TPA. Plus the project's `bin/` (absent in all five), first filename wins |
65+
| **B, project-specific (`APPROXIMATION`)** | the target TFM's reference pack(s): NETCore.App.Ref, plus AspNetCore/WindowsDesktop.App.Ref when the assets' `frameworkReferences` name them. Plus, for every package in `targets[<TFM>]`, its `compile` assets (excluding `_._`) from the global cache. A package missing from the cache is recorded, never substituted. No DLL from another TFM; no name-based replacement of a framework assembly |
66+
| **C, source-set probe (not part of A/B)** | A plus one generated file of the SDK default global usings for the project's SDK. It measures how much binding loss is source-set rather than reference |
67+
68+
Project-to-project references are not built, so in every mode their types bind only if the project's own sources
69+
declare them. This is recorded as a `source set` cause.
70+
71+
**R3 (extractor chain).** The unchanged extractor, built offline from `main` with packages from the local cache
72+
only. It runs with `--flow-locals`, as in `own-check`:
73+
74+
- **A-ext:** as shipped.
75+
- **B-ext:** `OWN_EXTRA_REF_DIRS` = the same framework reference packs as B, plus `--ref-dir` for each package
76+
`compile` folder of B.
77+
78+
The extractor keeps its TPA first, so B-ext is the best that **today's** extractor can be given, and it is not
79+
identical to B. Both are reported.
80+
- **Core:** the Python reference core (`python -m ownlang ownir`, the `own-check` default). No Rust core is built
81+
on this host, and building one needs cargo dependencies, so that is skipped.
82+
83+
## 3. Metrics (all with numerators and denominators)
84+
85+
Per project and mode:
86+
- **Method binding.** `InvocationExpressionSyntax` sites counted as resolved (`Symbol` non-null),
87+
ambiguous/candidates (`CandidateSymbols` > 0, `Symbol` null), or unresolved.
88+
- **Type binding.** `TypeSyntax` in declarations, `new`, base lists, parameters and fields, counted as resolved or
89+
`IErrorTypeSymbol`.
90+
- **Event binding.** `+=`/`-=` whose left side is a member access, counted as `IEventSymbol`, other symbol, or
91+
unresolved.
92+
- **Compiler errors** by code, with CS0012, CS0246, CS0234, CS1061, CS0103 and CS0117 broken out.
93+
- **R2, per site.** Every site whose status differs between A and B, recorded with file:line, expression, both
94+
statuses, `SymbolKind`, containing type, signature and containing assembly.
95+
96+
Classification:
97+
- `RECOVERED_CORRECT`;
98+
- `RECOVERED_WRONG`: B binds but to a symbol whose containing assembly is not the package or framework assembly
99+
that declares the named type, or A had a different symbol;
100+
- `STILL_UNRESOLVED`;
101+
- `AMBIGUOUS`;
102+
- `NOT_COMPARABLE`.
103+
104+
A fixed random sample of 10 `RECOVERED` sites per project (seed `20261021`) is checked by reading the code.
105+
- **Cause categories** for sites unresolved in A: fixed by B → `missing/mismatched reference`; fixed by C only →
106+
`source set (implicit usings)`; type declared in a referenced project → `source set (project reference)`;
107+
unresolved in all modes → `still unresolved` (unattributed). The B-vs-A difference attributable to TFM (8.0
108+
implementation vs target reference pack) is reported where B binds a framework-only member.
109+
- **R3.** Extractor stats, OwnIR fact counts by kind (`functions`, `subscriptions`, `services`, flow `acquire`,
110+
OWN050), and core findings by rule, for A-ext vs B-ext. Every finding change is listed and triaged as TP / FP /
111+
UNCLEAR by reading the code. A chain `reference → symbol → fact → diagnostic` is written for each change, and
112+
where it breaks, the break point is named, including #382-type lowering.
113+
114+
## 4. Controls (synthetic, in `controls/`, run through the same instrument; written before the first run)
115+
116+
| # | Control | Expected |
117+
|---|---|---|
118+
| K1 | sources and references already consistent (BCL only) | A == B, no site changes |
119+
| K2 | two emitted assemblies with the same filename `Lib.dll`, different identity/version, in two folders | A's first-filename-wins is recorded; B takes only the one its manifest names |
120+
| K3 | one package with `lib/net8.0` and `lib/netstandard2.0` | B uses only the TFM the manifest selects |
121+
| K4 | two source "projects" declaring the same FQN `Dup.T` | recorded as ambiguous/conflict, never `RECOVERED_CORRECT` |
122+
| K5 | an external `event` and an external `int` field, both used with `+=` | B binds the event as `IEventSymbol` and the field not as an event |
123+
| K6 | a type from an assembly that is not provided | unresolved in A and B; nothing invented |
124+
| K7 | `#if NET8_0 … #else … #endif` | both modes see the same branch (no defines); recorded as a source-set limit |
125+
126+
## 5. Decision (thresholds fixed now)
127+
128+
**`GO-P014-BOUNDED-DESIGN` only if all of the following hold:**
129+
1. **N1.** At least 2 independent projects show reference-caused binding loss in A, meaning sites fixed by B and
130+
not by C. The loss must be at least 5% of that project's invocation sites **and** at least 20 sites.
131+
2. **N3.** In each of those projects, at least 9 of the 10 sampled `RECOVERED` sites are `RECOVERED_CORRECT`.
132+
K1–K6 pass.
133+
3. **N4.** At least one triaged useful change in OwnIR facts or core findings, A-ext vs B-ext. That is: a new TP,
134+
or a removed FP, or an OWN050 replaced by a correct fact. A change in compiler diagnostics alone does not count.
135+
4. **N2 and cost.** B's references come from on-disk artifacts without any build, restore or MSBuild evaluation,
136+
and the cost of shipping them is bounded without `MSBuildWorkspace`.
137+
138+
**`KILL-P014-REFERENCE-INVESTMENT`** if any of:
139+
- fewer than 2 projects meet item 1;
140+
- recovered sites are wrong (item 2 fails), or a control fails;
141+
- the gain stops at compiler diagnostics (item 3 fails);
142+
- B needs a forbidden action;
143+
- the budget runs out.
144+
145+
**`INCONCLUSIVE-P014`** if item 1 holds for exactly one project and the reason the others cannot be measured is
146+
input availability (packages missing from the cache, or the TFM pack absent), not a measured null effect. The
147+
sample is not widened after the first run.
Lines changed: 114 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
# ruff: noqa: E501, RUF005 (control sources are inline C# one-liners)
2+
"""Build the synthetic controls K1-K7 (00-preregistration.md §4) into <out>/K<n>/ as
3+
{sources/, refs_A.txt, refs_B.txt, refs_C.txt}. Library DLLs are emitted by the instrument (`emit`).
4+
5+
usage: python -I make_controls.py --tool <p014bind.dll> --tpa <NETCore.App impl dir> --refpack <ref dir> --out <dir>
6+
"""
7+
8+
from __future__ import annotations
9+
10+
import argparse
11+
import glob
12+
import os
13+
import subprocess
14+
15+
LIB_V1 = "namespace Lib { public class Api { public static int V1Only() => 1; public event System.EventHandler? Changed; public int Count; } }"
16+
LIB_V2 = "namespace Lib { public class Api { public static int V2Only() => 2; public event System.EventHandler? Changed; public int Count; } }"
17+
PKG_NET8 = (
18+
'namespace Pkg { public static class Tfm { public static string Net8Only() => "net8"; } }'
19+
)
20+
PKG_NS20 = (
21+
'namespace Pkg { public static class Tfm { public static string Ns20Only() => "ns20"; } }'
22+
)
23+
24+
SOURCES = {
25+
"K1": {
26+
"a.cs": "using System.Collections.Generic; class K1 { int M() { var l = new List<int>(); l.Add(1); return l.Count; } }"
27+
},
28+
"K2": {"a.cs": "class K2 { int M() => Lib.Api.V2Only(); }"},
29+
"K3": {"a.cs": "class K3 { string M() => Pkg.Tfm.Net8Only(); }"},
30+
"K4": {
31+
"p1/a.cs": "namespace Dup { public class T { public void One() {} } }",
32+
"p2/b.cs": "namespace Dup { public class T { public void Two() {} } }",
33+
"use.cs": "class K4 { void M() { new Dup.T().One(); } }",
34+
},
35+
"K5": {"a.cs": "class K5 { void M(Lib.Api a) { a.Changed += (s, e) => { }; a.Count += 1; } }"},
36+
"K6": {"a.cs": "class K6 { void M() { Missing.Thing.Do(); var x = new Missing.Thing(); } }"},
37+
"K7": {
38+
"a.cs": "class K7 { void M() {\n#if NET8_0\n System.Console.WriteLine(1);\n#else\n NotDefinedWithoutDefines();\n#endif\n} }"
39+
},
40+
}
41+
42+
43+
def emit(tool: str, refpack: str, name: str, src: str, out_dll: str, version: str) -> None:
44+
os.makedirs(os.path.dirname(out_dll), exist_ok=True)
45+
srcf = out_dll + ".src.cs"
46+
open(srcf, "w", encoding="utf-8").write(src)
47+
subprocess.run(["dotnet", tool, "emit", name, version, srcf, out_dll, refpack], check=True)
48+
49+
50+
def main() -> int:
51+
ap = argparse.ArgumentParser()
52+
ap.add_argument("--tool", required=True)
53+
ap.add_argument("--tpa", required=True)
54+
ap.add_argument("--refpack", required=True)
55+
ap.add_argument("--out", required=True)
56+
a = ap.parse_args()
57+
tpa = sorted(glob.glob(os.path.join(a.tpa, "*.dll")))
58+
rp = sorted(glob.glob(os.path.join(a.refpack, "*.dll")))
59+
libs = os.path.join(a.out, "_libs")
60+
emit(a.tool, a.refpack, "Lib", LIB_V1, os.path.join(libs, "v1", "Lib.dll"), "1.0.0.0")
61+
emit(a.tool, a.refpack, "Lib", LIB_V2, os.path.join(libs, "v2", "Lib.dll"), "2.0.0.0")
62+
emit(
63+
a.tool,
64+
a.refpack,
65+
"Pkg",
66+
PKG_NET8,
67+
os.path.join(libs, "pkg", "lib", "net8.0", "Pkg.dll"),
68+
"1.0.0.0",
69+
)
70+
emit(
71+
a.tool,
72+
a.refpack,
73+
"Pkg",
74+
PKG_NS20,
75+
os.path.join(libs, "pkg", "lib", "netstandard2.0", "Pkg.dll"),
76+
"1.0.0.0",
77+
)
78+
v1, v2 = os.path.join(libs, "v1", "Lib.dll"), os.path.join(libs, "v2", "Lib.dll")
79+
p8 = os.path.join(libs, "pkg", "lib", "net8.0", "Pkg.dll")
80+
p20 = os.path.join(libs, "pkg", "lib", "netstandard2.0", "Pkg.dll")
81+
# mode A mimics the extractor: TPA + recursive first-filename-wins over the extra folders (ordinal sort);
82+
# mode B is the manifest-selected set: reference pack + exactly the assets the manifest names.
83+
refs = {
84+
"K1": (tpa, rp),
85+
"K2": (
86+
tpa + [v1],
87+
rp + [v2],
88+
), # ordinal sort puts v1/ before v2/: A gets v1; manifest names v2
89+
"K3": (tpa + [p8 if p8 < p20 else p20], rp + [p8]),
90+
"K4": (tpa, rp),
91+
"K5": (tpa + [v2], rp + [v2]),
92+
"K6": (tpa, rp),
93+
"K7": (tpa, rp),
94+
}
95+
for k, files in SOURCES.items():
96+
d = os.path.join(a.out, k)
97+
for rel, text in files.items():
98+
p = os.path.join(d, "sources", rel)
99+
os.makedirs(os.path.dirname(p), exist_ok=True)
100+
open(p, "w", encoding="utf-8").write(text)
101+
ra, rb = refs[k]
102+
for mode, lst in (("A", ra), ("B", rb), ("C", ra)):
103+
open(os.path.join(d, f"refs_{mode}.txt"), "w", encoding="utf-8").write(
104+
"\n".join(lst) + "\n"
105+
)
106+
open(os.path.join(d, "sources.txt"), "w", encoding="utf-8").write(
107+
"\n".join(sorted(glob.glob(os.path.join(d, "sources", "**", "*.cs"), recursive=True)))
108+
+ "\n"
109+
)
110+
return 0
111+
112+
113+
if __name__ == "__main__":
114+
raise SystemExit(main())
Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
<Project Sdk="Microsoft.NET.Sdk">
2+
<!-- P-014 reference-coverage instrument (research only). No PackageReference: Roslyn comes from the e7d33e33 extractor build
3+
(pass -p:RoslynDir=<dir with Microsoft.CodeAnalysis*.dll>), so restore needs no package feed. -->
4+
<PropertyGroup>
5+
<OutputType>Exe</OutputType>
6+
<TargetFramework>net8.0</TargetFramework><LangVersion>latest</LangVersion>
7+
<Nullable>enable</Nullable>
8+
<ImplicitUsings>enable</ImplicitUsings>
9+
<AssemblyName>p014bind</AssemblyName>
10+
</PropertyGroup>
11+
<ItemGroup>
12+
<Reference Include="Microsoft.CodeAnalysis"><HintPath>$(RoslynDir)/Microsoft.CodeAnalysis.dll</HintPath></Reference>
13+
<Reference Include="Microsoft.CodeAnalysis.CSharp"><HintPath>$(RoslynDir)/Microsoft.CodeAnalysis.CSharp.dll</HintPath></Reference>
14+
</ItemGroup>
15+
</Project>

0 commit comments

Comments
 (0)