Skip to content

Commit b5d6f18

Browse files
PhysShellclaude
PhysShell
and
claude
committed
docs(evidence): N1 competitor gate — GO-CANDIDATE-N1 (3/4 missed by CA2000 AllPaths)
Scored run (stubs as a separate assembly): CA2000 with AllPaths catches whizbang#1204 (`new ServiceBusClient`) and misses rockbot#638 (awaited external async factory), NexusKit#34 (BuildServiceProvider) and BareWire#73 (awaited interface factory) in K1-K5; IDisposableAnalyzers and Sonar report nothing on any acquisition line. Probes with opaque throwing steps pin the limit to the acquisition form; ablation finds no other silencing feature. Disclosed: SARIF 1.0 reader bug (run 0) and in-compilation stubs (run 1, same verdicts). Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01TFkNAFFQz9KoTzpPws8vYG
1 parent d06caf7 commit b5d6f18

13 files changed

Lines changed: 495 additions & 4 deletions

File tree

‎docs/evidence/n1-competitor-gate-2026-10/REPORT.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# N1 competitor gate — report
2+
3+
**Result: `GO-CANDIDATE-N1`.** This is a decision for the owner, not a GO on an N1 PoC.
4+
5+
| Pair | Verdict | What CA2000 (`AllPaths`) does |
6+
|---|---|---|
7+
| W, whizbang#1204 | CAUGHT | reports the buggy `new ServiceBusClient(…)` in K2–K4; silent on the fix |
8+
| R, rockbot#638 | MISSED | does not track `await McpClient.CreateAsync(…)` |
9+
| X, NexusKit#34 | MISSED | does not track `services.BuildServiceProvider(…)` |
10+
| B, BareWire#73 | MISSED | does not track `await _connection.CreateChannelAsync(…)` |
11+
12+
So 3 of 4 pairs are missed by every configuration (K1–K5) of all three analyzers: CA2000, IDisposableAnalyzers
13+
4.0.8 and Sonar 10.35. In run 2, none of the three analyzers reports anything on the R, X or B acquisition lines.
14+
15+
Rules: [`00-preregistration.md`](00-preregistration.md), committed in `d06caf75` before any analyzer run. Time used:
16+
21:11–21:20 +05 on 2026-10-20 for the scored work, inside the 0.25 agent-day budget.
17+
18+
No third-party project was built and no package was downloaded. The analyzers came from the SDK and the local NuGet
19+
cache. Every build ran from a copy in the session scratch directory, with a nuget.org-only config and no
20+
`PackageReference`.
21+
22+
## 1. Runs, in order (runs 1 and 2 kept in `data/`; run 0's output was overwritten and is not kept)
23+
24+
| Run | What changed | Result |
25+
|---|---|---|
26+
| 0 | first build | 0 diagnostics matched, because of a bug in my SARIF reader: csc wrote SARIF 1.0 and the reader expected 2.1. Not a result; the reader was fixed. No rule changed |
27+
| 1 (`run1_stubs_in_compilation.tsv`) | reader fixed | control valid; W CAUGHT; R, X, B silent |
28+
| 2 (`run2_scored.tsv`) | **scored run**: the stubs moved into a separate referenced assembly (§2) | control valid; W CAUGHT; R, X, B MISSED |
29+
30+
The harness control was valid in both runs 1 and 2:
31+
- `new StreamReader` followed by an opaque throwing call and `Dispose` is reported under K2–K5 and silent under K1;
32+
- the same with `using` is silent everywhere.
33+
34+
## 2. Fidelity correction before scoring (disclosed)
35+
36+
The pre-registration requires every MISSED to be checked for a reproduction deviation. The first check found one.
37+
38+
In run 1, the stubs were in the analysed compilation. `McpClient.CreateAsync` is a stub whose body is `throw`, and
39+
CA2000's interprocedural analysis can read that body.
40+
41+
The real types (`ServiceBusClient`, `McpClient`, `IChannel`) live in external packages whose bodies the analyzer
42+
does not see. Run 2 therefore compiles the stubs as their own assembly (`stubs/`), referenced by the reproductions.
43+
The reproductions and the rules did not change. The verdicts were the same in both runs.
44+
45+
## 3. Why R, X and B are missed: analyzer limits, not reproduction artifacts
46+
47+
The probes are not scored (`probes/`, `data/probes_opaque_step.tsv`, `data/ablation_and_probes.tsv`). Each one has
48+
the same leak: acquire a disposable, make an **opaque** throwing call through an interface, then dispose or hand it
49+
over. Only the acquisition form changes.
50+
51+
A first probe set used empty local `MayThrow` methods. CA2000 proved that those cannot throw and stayed silent, so
52+
the set was invalid; its output was overwritten and is not kept. It was redone with interface calls, which is how
53+
the reproductions throw.
54+
55+
| Acquisition form (K2, `AllPaths`) | Tracked? |
56+
|---|---|
57+
| `new T(…)`; an in-source sync or async factory; `File.OpenText`; an external static sync factory (P10, P12) | yes |
58+
| `new T(…)` then store in a dictionary, wrap and return, return, or inside an `async` iterator (P5–P8) | yes |
59+
| **await of an external async factory** (P11), the shape of R and B | **no** |
60+
| **result of an interface method** (P13), the shape of B (`IConnection.CreateChannelAsync`) | **no** |
61+
| **`ServiceCollection.BuildServiceProvider()`** (P9), the shape of X; the real M.E.DI type | **no**, even in a three-line method; the reason was not established |
62+
63+
The ablation of each MISSED reproduction (`probes/Ablation.cs`) found no other feature that silences the analyzer:
64+
- removing the catch-all retry loop (R) leaves it silent;
65+
- removing the resolve calls (X) leaves it silent;
66+
- removing the iterator, or swapping the interface for a class (B), leaves it silent.
67+
68+
The acquisition form alone decides. In all three pairs that form is the real code's own API, so the MISSEDs stand
69+
and none is `UNRESOLVED`.
70+
71+
## 4. What this does and does not show
72+
73+
Confirmed:
74+
- CA2000 with `AllPaths` is the documented tool for this class, and it does catch the `new`-acquired case (W).
75+
- For the acquisition forms that are common in modern async .NET, it reports nothing on these reproductions:
76+
awaited async factories, interface factory methods and `BuildServiceProvider`.
77+
- IDisposableAnalyzers and Sonar report nothing on any of the four acquisition lines. Inference, not measured: they
78+
accept a return or store as a hand-over and do not model the exception path before it.
79+
80+
Not shown:
81+
- **No real build was run.** The verdicts come from faithful reproductions of the methods with stub third-party
82+
types, not from building the four repositories. CA2000 on the real repositories could differ if a real package
83+
type carries attributes or annotations that the stubs lack.
84+
- **Only the options in the pre-registration were tested.** `interprocedural_analysis_kind` and
85+
`excluded_symbol_names` were not varied.
86+
- W's fixed version is silent partly because the fix acquires through a `Func<…>` seam that CA2000 does not
87+
track. A `new` + `try`/`catch` fix was not tested.
88+
- That Own.NET can do better is not shown. Phase 0 recorded that Own.NET tracks only non-escaping `new` locals and
89+
missed 3/3 here. A detector for factory-acquired and awaited acquisitions would be new work.
90+
- Mature-code yield and precision of such a detector are unknown. This gate measured only competitor coverage.
91+
92+
## 5. Files
93+
94+
- [`00-preregistration.md`](00-preregistration.md)
95+
- [`repro/`](repro/): W, R, X, B buggy/fixed reproductions, `Control.cs` and `N1Repro.csproj`.
96+
- [`stubs/`](stubs/): the third-party stand-ins, built as a separate assembly.
97+
- [`probes/`](probes/): explanatory probes and ablation. `ProbeExternals.cs` goes into the stub assembly for the
98+
ablation run only.
99+
- [`scripts/run_gate.py`](scripts/run_gate.py): builds K1–K5 and collects the diagnostics on each acquisition line.
100+
- `data/`: outputs of run 1, run 2 and the probes.

‎docs/evidence/n1-competitor-gate-2026-10/data/ablation_and_probes.tsv‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
config id class file line rule message
2+
K1 _build rc=0 0 errors=0 results=27
3+
K2 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
4+
K2 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
5+
K2 W WBuggy W.cs 23 CA2000 Use recommended dispose pattern to ensure that object created by 'new ServiceBusClient(connectionString)' is disposed on all exception paths. If possible, wrap the creation within a 'using' statement or a 'using' declaration. Otherwise, use a try-finally pattern, with a dedicated local variable declared before the try region and an unconditional Dispose invocation on non-null value in the 'finally' region, say 'x?.Dispose()'. If the object is explicitly disposed within the try region or the dispose ownership is transfered to another object or method, assign 'null' to the local variable just after such an operation to prevent double dispose in 'finally'.
6+
K2 _build rc=0 0 errors=0 results=32
7+
K2 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
8+
K3 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
9+
K3 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
10+
K3 W WBuggy W.cs 23 CA2000 Use recommended dispose pattern to ensure that object created by 'new ServiceBusClient(connectionString)' is disposed on all exception paths. If possible, wrap the creation within a 'using' statement or a 'using' declaration. Otherwise, use a try-finally pattern, with a dedicated local variable declared before the try region and an unconditional Dispose invocation on non-null value in the 'finally' region, say 'x?.Dispose()'. If the object is explicitly disposed within the try region or the dispose ownership is transfered to another object or method, assign 'null' to the local variable just after such an operation to prevent double dispose in 'finally'.
11+
K3 _build rc=0 0 errors=0 results=32
12+
K3 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
13+
K4 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
14+
K4 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
15+
K4 W WBuggy W.cs 23 CA2000 Use recommended dispose pattern to ensure that object created by 'new ServiceBusClient(connectionString)' is disposed on all exception paths. If possible, wrap the creation within a 'using' statement or a 'using' declaration. Otherwise, use a try-finally pattern, with a dedicated local variable declared before the try region and an unconditional Dispose invocation on non-null value in the 'finally' region, say 'x?.Dispose()'. If the object is explicitly disposed within the try region or the dispose ownership is transfered to another object or method, assign 'null' to the local variable just after such an operation to prevent double dispose in 'finally'.
16+
K4 _build rc=0 0 errors=0 results=32
17+
K4 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
18+
K5 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
19+
K5 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
20+
K5 _build rc=0 0 errors=0 results=28
21+
K5 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.

‎docs/evidence/n1-competitor-gate-2026-10/data/probes_opaque_step.tsv‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,57 @@
1+
config id class file line rule message
2+
K1 P1 Probes Probes.cs 26 IDISP017 Prefer using
3+
K1 P2 Probes Probes.cs 29 IDISP017 Prefer using
4+
K1 P3 Probes Probes.cs 32 IDISP017 Prefer using
5+
K1 P4 Probes Probes.cs 35 IDISP017 Prefer using
6+
K1 _build rc=0 0 errors=0 results=6
7+
K2 P1 Probes Probes.cs 26 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
8+
K2 P1 Probes Probes.cs 26 IDISP017 Prefer using
9+
K2 P2 Probes Probes.cs 29 CA2000 Object created by 'Open(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
10+
K2 P2 Probes Probes.cs 29 IDISP017 Prefer using
11+
K2 P3 Probes Probes.cs 32 CA2000 Object created by 'File.OpenText(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
12+
K2 P3 Probes Probes.cs 32 IDISP017 Prefer using
13+
K2 P4 Probes Probes.cs 35 CA2000 Object created by 'OpenAsync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
14+
K2 P4 Probes Probes.cs 35 IDISP017 Prefer using
15+
K2 P5 Probes Probes.cs 38 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
16+
K2 P6 Probes Probes.cs 41 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
17+
K2 P7 Probes Probes.cs 46 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
18+
K2 P8 Probes Probes.cs 53 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
19+
K2 _build rc=0 0 errors=0 results=15
20+
K3 P1 Probes Probes.cs 26 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
21+
K3 P1 Probes Probes.cs 26 IDISP017 Prefer using
22+
K3 P2 Probes Probes.cs 29 CA2000 Object created by 'Open(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
23+
K3 P2 Probes Probes.cs 29 IDISP017 Prefer using
24+
K3 P3 Probes Probes.cs 32 CA2000 Object created by 'File.OpenText(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
25+
K3 P3 Probes Probes.cs 32 IDISP017 Prefer using
26+
K3 P4 Probes Probes.cs 35 CA2000 Object created by 'OpenAsync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
27+
K3 P4 Probes Probes.cs 35 IDISP017 Prefer using
28+
K3 P5 Probes Probes.cs 38 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
29+
K3 P6 Probes Probes.cs 41 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
30+
K3 P7 Probes Probes.cs 46 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
31+
K3 P8 Probes Probes.cs 53 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
32+
K3 _build rc=0 0 errors=0 results=15
33+
K4 P1 Probes Probes.cs 26 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
34+
K4 P1 Probes Probes.cs 26 IDISP017 Prefer using
35+
K4 P2 Probes Probes.cs 29 CA2000 Object created by 'Open(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
36+
K4 P2 Probes Probes.cs 29 IDISP017 Prefer using
37+
K4 P3 Probes Probes.cs 32 CA2000 Object created by 'File.OpenText(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
38+
K4 P3 Probes Probes.cs 32 IDISP017 Prefer using
39+
K4 P4 Probes Probes.cs 35 IDISP017 Prefer using
40+
K4 P5 Probes Probes.cs 38 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
41+
K4 P6 Probes Probes.cs 41 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
42+
K4 P7 Probes Probes.cs 46 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
43+
K4 P8 Probes Probes.cs 53 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
44+
K4 _build rc=0 0 errors=0 results=14
45+
K5 P1 Probes Probes.cs 26 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
46+
K5 P1 Probes Probes.cs 26 IDISP017 Prefer using
47+
K5 P2 Probes Probes.cs 29 CA2000 Object created by 'Open(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
48+
K5 P2 Probes Probes.cs 29 IDISP017 Prefer using
49+
K5 P3 Probes Probes.cs 32 CA2000 Object created by 'File.OpenText(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
50+
K5 P3 Probes Probes.cs 32 IDISP017 Prefer using
51+
K5 P4 Probes Probes.cs 35 CA2000 Object created by 'OpenAsync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
52+
K5 P4 Probes Probes.cs 35 IDISP017 Prefer using
53+
K5 P5 Probes Probes.cs 38 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
54+
K5 P6 Probes Probes.cs 41 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
55+
K5 P7 Probes Probes.cs 46 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
56+
K5 P8 Probes Probes.cs 53 CA2000 Object created by 'new StreamReader(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
57+
K5 _build rc=0 0 errors=0 results=15

0 commit comments

Comments
 (0)