Skip to content

Commit 38fb952

Browse files
PhysShellPhysShell
authored
Merge pull request #417 from PhysShell/research/n1-competitor-gate
docs(evidence): N1 competitor gate — GO-CANDIDATE-N1 (3/4 missed by CA2000 AllPaths)
2 parents d947db8 + b5d6f18 commit 38fb952

19 files changed

Lines changed: 1144 additions & 0 deletions

‎docs/evidence/n1-competitor-gate-2026-10/00-preregistration.md‎

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
# N1 competitor gate — pre-registration
2+
3+
Written 2026-10-20, 21:11 +05. The four fix diffs have been read in order to build the reproductions. No analyzer
4+
has been run on any reproduction.
5+
6+
Authority: owner decision `GO-N1-COMPETITOR-GATE`, given after `KILL-N3-PILOT` was accepted. This gate is not a GO
7+
on an N1 PoC.
8+
9+
- **Budget.** 0.25 agent-day.
10+
- **Rules of the work.** No third-party project is built and no package is downloaded. The analyzers come from the
11+
.NET SDK and from the local NuGet cache, the same versions as direction-research Phase 5.
12+
13+
## Question
14+
15+
N1: a disposable is acquired, a step that can throw runs before ownership is handed over (return, field or
16+
collection store, wrapping), and no dispose runs on the throw path.
17+
18+
Do existing analyzers miss the four confirmed N1 defects of Phase 0 once they are configured to look? The question
19+
matters most for CA2000 with exception paths on (`dispose_analysis_kind = AllPaths`), because that is the
20+
documented setting for exactly this case.
21+
22+
## Why the earlier "not detected" does not answer it
23+
24+
Phase 5 ran the analyzers over loose source trees. Those runs had about 35,000 compile errors (whizbang: 34,859 on
25+
base), so the disposable types were mostly unresolved, and CA2000 ran with its default kind. Those misses therefore
26+
say little about what CA2000 can do.
27+
28+
## Pairs (fixed now)
29+
30+
| Id | Fix | Buggy method | Leaked resource and acquisition | Throwing step before hand-over | Hand-over on success |
31+
|---|---|---|---|---|---|
32+
| W | whizbang-lib/whizbang#1204 | `AzureServiceBusConnectionRetry.CreateClientWithRetryAsync` | `new ServiceBusClient(cs)` (`IAsyncDisposable`), inside a retry loop's `try` | `await VerifyNamespaceReachableAsync(…)` | `return client;` |
33+
| R | MarimerLLC/rockbot#638 (record 638-2) | `McpBridgeService.ConnectServerAsync` | `await McpClient.CreateAsync(transport, …)`, an async factory (`IAsyncDisposable`) | `await newClient.ListToolsAsync(…)`, prompt listing | `_clients[name] = newClient;` (dictionary field) |
34+
| X | NexusFFXIV/NexusKit#34 (record 34-2) | `PluginHostBuilder.BuildAsync` | `services.BuildServiceProvider(validateScopes: false)`, a factory (`ServiceProvider`, `IDisposable` + `IAsyncDisposable`) | `await DbInitializer.InitializeAsync(…)`, service resolution | `return new PluginHost(provider);` (wrapping) |
35+
| B | Wizard-Software/BareWire#73 | `RabbitMqTransportAdapter.ConsumeAsync` (an `async` iterator) | `await _connection.CreateChannelAsync(…)`, an async factory (`IChannel`, `IAsyncDisposable` + `IDisposable`) | `await consumerChannel.BasicQosAsync(…)` | `_activeConsumerChannels[id] = consumerChannel;` (dictionary field) |
36+
37+
The four fixes are by different owners.
38+
39+
## Reproductions
40+
41+
For each pair: one file `repro/<Id>.cs` with two self-contained classes, `<Id>Buggy` and `<Id>Fixed`. The
42+
acquisition line carries the marker comment `// ACQ:<Id>`.
43+
44+
- **Kept from the real code.**
45+
- the acquisition expression form: `new`, factory call, or awaited async factory;
46+
- the throwing step, as an awaited call that can throw;
47+
- the success hand-over (return, wrap, or dictionary store);
48+
- the enclosing control flow: the retry loop with `try`/`catch` in W, the `try` around the attempt in R, and the
49+
`async` iterator in B;
50+
- each fix's exact cleanup shape: W `try`/`catch` + `DisposeAsync` + rethrow, with acquisition through a
51+
`Func<…>` seam; R a nullable local, a `published` flag and cleanup on the failure path; X `try`/`catch` + wrap +
52+
`DisposeAsync` + rethrow; B early registration into the dictionary, then `try`/`catch` + release.
53+
- **Third-party types are stubs** with the real disposable interfaces and the members used: `ServiceBusClient`,
54+
`McpClient`, `IChannel`/`IConnection`. `ServiceCollection`/`ServiceProvider` are the real types from the local
55+
ASP.NET Core reference pack.
56+
- **Lines not on the resource's path are dropped**: logging, metrics, unrelated locals.
57+
- The reproductions must compile with **0 errors**. An analyzer that sees a compile error proves nothing.
58+
59+
## Analyzers and configurations
60+
61+
One SDK-style project, `net10.0`, with no `PackageReference`. Analyzers are referenced by path:
62+
63+
- the .NET SDK `Microsoft.CodeAnalysis.NetAnalyzers` (CA2000);
64+
- `IDisposableAnalyzers` 4.0.8 (IDISP001–IDISP026);
65+
- `SonarAnalyzer.CSharp` 10.35.0.4138 (S2930 and others).
66+
67+
Every rule of every analyzer is set to warning (`dotnet_analyzer_diagnostic.severity = warning`). Configurations:
68+
69+
| K | `dispose_analysis_kind` | ownership transfer at constructor | ownership transfer at method call |
70+
|---|---|---|---|
71+
| K1 | default (`NonExceptionPaths`) | default (false) | default (false) |
72+
| K2 | `AllPaths` | false | false |
73+
| K3 | `AllPaths` | **true** | false |
74+
| K4 | `AllPaths` | false | **true** |
75+
| K5 | `AllPathsOnlyNotDisposed` | false | false |
76+
77+
**Harness control.** A two-method file, `Control.cs`:
78+
79+
- `new StreamReader(path)`, then a call that can throw, then `Dispose()`. CA2000 must report it under K2 and must
80+
not report it under K1.
81+
- the same with `using`: silent in every configuration.
82+
83+
If this control fails, the harness is invalid and the gate stops as `INVALID`, not as KILL.
84+
85+
## Scoring (fixed now)
86+
87+
- **Relevant diagnostic.** A diagnostic whose message is about disposal or ownership, located on the leaked
88+
resource's acquisition statement or its declared variable, in the buggy method.
89+
- **Pair verdict.**
90+
- **`CAUGHT`:** at least one configuration gives a relevant diagnostic on the buggy reproduction. This holds even
91+
when the fixed reproduction also gets one: a flag on the buggy site counts against N1's novelty. Whether the
92+
flag also distinguishes the fix is recorded separately.
93+
- **`MISSED`:** no relevant diagnostic on the buggy reproduction in any of K1–K5, from any of the three analyzers.
94+
95+
**Decision.**
96+
- **`GO-CANDIDATE-N1`** (for an owner decision, not a PoC GO) if at least **2** pairs are `MISSED`.
97+
- **`KILL-N1`** otherwise.
98+
99+
Fidelity risk: a stub or a dropped line can change an analyzer's result. Every `MISSED` is therefore re-checked by
100+
listing the analyzer's documented limits for that shape: a factory, an `async` iterator or a dictionary store. A
101+
`MISSED` that is explained only by a reproduction deviation from the real code becomes `UNRESOLVED` and does not
102+
count toward GO.

‎docs/evidence/n1-competitor-gate-2026-10/REPORT.md‎

Lines changed: 100 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,100 @@
1+
# N1 competitor gate — report
2+
3+
**Result: `GO-CANDIDATE-N1`.** This is a decision for the owner, not a GO on an N1 PoC.
4+
5+
| Pair | Verdict | What CA2000 (`AllPaths`) does |
6+
|---|---|---|
7+
| W, whizbang#1204 | CAUGHT | reports the buggy `new ServiceBusClient(…)` in K2–K4; silent on the fix |
8+
| R, rockbot#638 | MISSED | does not track `await McpClient.CreateAsync(…)` |
9+
| X, NexusKit#34 | MISSED | does not track `services.BuildServiceProvider(…)` |
10+
| B, BareWire#73 | MISSED | does not track `await _connection.CreateChannelAsync(…)` |
11+
12+
So 3 of 4 pairs are missed by every configuration (K1–K5) of all three analyzers: CA2000, IDisposableAnalyzers
13+
4.0.8 and Sonar 10.35. In run 2, none of the three analyzers reports anything on the R, X or B acquisition lines.
14+
15+
Rules: [`00-preregistration.md`](00-preregistration.md), committed in `d06caf75` before any analyzer run. Time used:
16+
21:11–21:20 +05 on 2026-10-20 for the scored work, inside the 0.25 agent-day budget.
17+
18+
No third-party project was built and no package was downloaded. The analyzers came from the SDK and the local NuGet
19+
cache. Every build ran from a copy in the session scratch directory, with a nuget.org-only config and no
20+
`PackageReference`.
21+
22+
## 1. Runs, in order (runs 1 and 2 kept in `data/`; run 0's output was overwritten and is not kept)
23+
24+
| Run | What changed | Result |
25+
|---|---|---|
26+
| 0 | first build | 0 diagnostics matched, because of a bug in my SARIF reader: csc wrote SARIF 1.0 and the reader expected 2.1. Not a result; the reader was fixed. No rule changed |
27+
| 1 (`run1_stubs_in_compilation.tsv`) | reader fixed | control valid; W CAUGHT; R, X, B silent |
28+
| 2 (`run2_scored.tsv`) | **scored run**: the stubs moved into a separate referenced assembly (§2) | control valid; W CAUGHT; R, X, B MISSED |
29+
30+
The harness control was valid in both runs 1 and 2:
31+
- `new StreamReader` followed by an opaque throwing call and `Dispose` is reported under K2–K5 and silent under K1;
32+
- the same with `using` is silent everywhere.
33+
34+
## 2. Fidelity correction before scoring (disclosed)
35+
36+
The pre-registration requires every MISSED to be checked for a reproduction deviation. The first check found one.
37+
38+
In run 1, the stubs were in the analysed compilation. `McpClient.CreateAsync` is a stub whose body is `throw`, and
39+
CA2000's interprocedural analysis can read that body.
40+
41+
The real types (`ServiceBusClient`, `McpClient`, `IChannel`) live in external packages whose bodies the analyzer
42+
does not see. Run 2 therefore compiles the stubs as their own assembly (`stubs/`), referenced by the reproductions.
43+
The reproductions and the rules did not change. The verdicts were the same in both runs.
44+
45+
## 3. Why R, X and B are missed: analyzer limits, not reproduction artifacts
46+
47+
The probes are not scored (`probes/`, `data/probes_opaque_step.tsv`, `data/ablation_and_probes.tsv`). Each one has
48+
the same leak: acquire a disposable, make an **opaque** throwing call through an interface, then dispose or hand it
49+
over. Only the acquisition form changes.
50+
51+
A first probe set used empty local `MayThrow` methods. CA2000 proved that those cannot throw and stayed silent, so
52+
the set was invalid; its output was overwritten and is not kept. It was redone with interface calls, which is how
53+
the reproductions throw.
54+
55+
| Acquisition form (K2, `AllPaths`) | Tracked? |
56+
|---|---|
57+
| `new T(…)`; an in-source sync or async factory; `File.OpenText`; an external static sync factory (P10, P12) | yes |
58+
| `new T(…)` then store in a dictionary, wrap and return, return, or inside an `async` iterator (P5–P8) | yes |
59+
| **await of an external async factory** (P11), the shape of R and B | **no** |
60+
| **result of an interface method** (P13), the shape of B (`IConnection.CreateChannelAsync`) | **no** |
61+
| **`ServiceCollection.BuildServiceProvider()`** (P9), the shape of X; the real M.E.DI type | **no**, even in a three-line method; the reason was not established |
62+
63+
The ablation of each MISSED reproduction (`probes/Ablation.cs`) found no other feature that silences the analyzer:
64+
- removing the catch-all retry loop (R) leaves it silent;
65+
- removing the resolve calls (X) leaves it silent;
66+
- removing the iterator, or swapping the interface for a class (B), leaves it silent.
67+
68+
The acquisition form alone decides. In all three pairs that form is the real code's own API, so the MISSEDs stand
69+
and none is `UNRESOLVED`.
70+
71+
## 4. What this does and does not show
72+
73+
Confirmed:
74+
- CA2000 with `AllPaths` is the documented tool for this class, and it does catch the `new`-acquired case (W).
75+
- For the acquisition forms that are common in modern async .NET, it reports nothing on these reproductions:
76+
awaited async factories, interface factory methods and `BuildServiceProvider`.
77+
- IDisposableAnalyzers and Sonar report nothing on any of the four acquisition lines. Inference, not measured: they
78+
accept a return or store as a hand-over and do not model the exception path before it.
79+
80+
Not shown:
81+
- **No real build was run.** The verdicts come from faithful reproductions of the methods with stub third-party
82+
types, not from building the four repositories. CA2000 on the real repositories could differ if a real package
83+
type carries attributes or annotations that the stubs lack.
84+
- **Only the options in the pre-registration were tested.** `interprocedural_analysis_kind` and
85+
`excluded_symbol_names` were not varied.
86+
- W's fixed version is silent partly because the fix acquires through a `Func<…>` seam that CA2000 does not
87+
track. A `new` + `try`/`catch` fix was not tested.
88+
- That Own.NET can do better is not shown. Phase 0 recorded that Own.NET tracks only non-escaping `new` locals and
89+
missed 3/3 here. A detector for factory-acquired and awaited acquisitions would be new work.
90+
- Mature-code yield and precision of such a detector are unknown. This gate measured only competitor coverage.
91+
92+
## 5. Files
93+
94+
- [`00-preregistration.md`](00-preregistration.md)
95+
- [`repro/`](repro/): W, R, X, B buggy/fixed reproductions, `Control.cs` and `N1Repro.csproj`.
96+
- [`stubs/`](stubs/): the third-party stand-ins, built as a separate assembly.
97+
- [`probes/`](probes/): explanatory probes and ablation. `ProbeExternals.cs` goes into the stub assembly for the
98+
ablation run only.
99+
- [`scripts/run_gate.py`](scripts/run_gate.py): builds K1–K5 and collects the diagnostics on each acquisition line.
100+
- `data/`: outputs of run 1, run 2 and the probes.

‎docs/evidence/n1-competitor-gate-2026-10/data/ablation_and_probes.tsv‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
config id class file line rule message
2+
K1 _build rc=0 0 errors=0 results=27
3+
K2 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
4+
K2 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
5+
K2 W WBuggy W.cs 23 CA2000 Use recommended dispose pattern to ensure that object created by 'new ServiceBusClient(connectionString)' is disposed on all exception paths. If possible, wrap the creation within a 'using' statement or a 'using' declaration. Otherwise, use a try-finally pattern, with a dedicated local variable declared before the try region and an unconditional Dispose invocation on non-null value in the 'finally' region, say 'x?.Dispose()'. If the object is explicitly disposed within the try region or the dispose ownership is transfered to another object or method, assign 'null' to the local variable just after such an operation to prevent double dispose in 'finally'.
6+
K2 _build rc=0 0 errors=0 results=32
7+
K2 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
8+
K3 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
9+
K3 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
10+
K3 W WBuggy W.cs 23 CA2000 Use recommended dispose pattern to ensure that object created by 'new ServiceBusClient(connectionString)' is disposed on all exception paths. If possible, wrap the creation within a 'using' statement or a 'using' declaration. Otherwise, use a try-finally pattern, with a dedicated local variable declared before the try region and an unconditional Dispose invocation on non-null value in the 'finally' region, say 'x?.Dispose()'. If the object is explicitly disposed within the try region or the dispose ownership is transfered to another object or method, assign 'null' to the local variable just after such an operation to prevent double dispose in 'finally'.
11+
K3 _build rc=0 0 errors=0 results=32
12+
K3 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
13+
K4 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
14+
K4 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
15+
K4 W WBuggy W.cs 23 CA2000 Use recommended dispose pattern to ensure that object created by 'new ServiceBusClient(connectionString)' is disposed on all exception paths. If possible, wrap the creation within a 'using' statement or a 'using' declaration. Otherwise, use a try-finally pattern, with a dedicated local variable declared before the try region and an unconditional Dispose invocation on non-null value in the 'finally' region, say 'x?.Dispose()'. If the object is explicitly disposed within the try region or the dispose ownership is transfered to another object or method, assign 'null' to the local variable just after such an operation to prevent double dispose in 'finally'.
16+
K4 _build rc=0 0 errors=0 results=32
17+
K4 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
18+
K5 P10 ExternalProbes ExternalProbes.cs 15 CA2000 Object created by 'ExternalFactories.OpenSync(p)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
19+
K5 P12 ExternalProbes ExternalProbes.cs 29 CA2000 Object created by 'ExternalFactories.CreateClient(cs)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.
20+
K5 _build rc=0 0 errors=0 results=28
21+
K5 control Control Control.cs 11 CA2000 Object created by 'new StreamReader(path)' is not disposed along all exception paths. Call System.IDisposable.Dispose on the object before all references to it are out of scope.

0 commit comments

Comments
 (0)