Skip to content

Commit cc267cb

Browse files
authored
Merge pull request #328 from PhysShell/claude/runtime-evidence-contract-6147se
docs(runtime): assess evidence boundaries and specify artifact identity validation
2 parents 7c10f37 + cd0d771 commit cc267cb

2 files changed

Lines changed: 472 additions & 0 deletions

File tree

‎docs/notes/leaf-dmir-assessment.md‎

Lines changed: 262 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,262 @@
1+
# Leaf / DMIR — assessment against what OwnAudit actually has
2+
3+
Working note. **Trigger:** an outside reading of *"Leaf: An Instrumentation-based
4+
Dynamic Analysis Framework for Rust"*
5+
([arXiv:2607.15025](https://arxiv.org/abs/2607.15025), Omidvar Tehrani, Gaboardi,
6+
Sumner, Ko) proposed it as "a concrete building block for OwnAudit" at 8.5/10,
7+
with a separate Rust-analysis direction at 9/10, and listed five ideas to adopt.
8+
9+
The paper is real and summarised accurately: Leaf instruments **Rust MIR** via
10+
`rustc` internals and delivers **DMIR**, an event-driven stream carrying static
11+
MIR identity plus runtime facts, with a concolic executor / sanitizer / tracer
12+
built on top. The architecture is genuinely good.
13+
14+
The proposal does not survive contact with the repositories. Recorded here so
15+
nobody re-derives it — same purpose as
16+
[P-034](../proposals/P-034-runtime-lifetime-guard.md)'s "do not re-derive" table.
17+
18+
## The blocking fact: wrong runtime
19+
20+
**Leaf instruments Rust MIR. OwnAudit audits .NET.**
21+
22+
Verified against `PhysShell/OwnAudit` at clone time:
23+
24+
- README: an orchestrator over "a legacy **.NET 4.7.2 / WPF / DevExpress** app".
25+
- File census: 98 `.sarif`, 70 `.py`, 46 `.json`, 38 `.md`, 31 `.cs` — and
26+
**zero `.rs`, no `Cargo.toml`**.
27+
- The runtime half is a **ClrMD heap walk** over a CLR process, emitting
28+
`runtime.json` (`schema: "ownAudit/runtime/v1"`).
29+
30+
There is no MIR in .NET and Leaf cannot observe a CLR process. The proposed
31+
`Leaf → adapter → RuntimeEvidenceEvent → OwnAudit rules` pipeline is not
32+
buildable at any effort level. The 8.5/10 is for a wire that has no endpoints.
33+
34+
**The 9/10 is further off still.** [P-017](../proposals/P-017-multi-stack-frontends.md)
35+
takes the core beyond .NET to **OwnTS** and **OwnJVM (Java/Kotlin)** — Rust is
36+
not a target and structurally should not be: this project's entire thesis is
37+
bringing ownership discipline to a **GC language that lacks it**, and the
38+
ROADMAP carries a standing section on *why not a Rust-style borrow checker for
39+
C#*. Rust already has the borrow checker; there is no gap to sell into.
40+
41+
The likely source of the error is real and easy to trip on: **Own.NET's core is
42+
being migrated *to* Rust** ([P-022](../proposals/P-022-rust-core-migration.md)).
43+
That is the analyzer *written in* Rust, not the analyzer *analyzing* Rust.
44+
45+
## The five "adopt these", scored against the tree
46+
47+
| # | Proposed idea | Reality |
48+
|---|---|---|
49+
| 1 | Provider-agnostic Runtime Evidence IR, so we don't become a shell over Leaf | **Seam already exists.** `runtime.json` is a versioned schema consumed by `runtime/cli.py`; the collector is already swappable. What differs is snapshot vs event stream — a real question, but not this one |
50+
| 2 | Correlate static site with runtime fact | **Already shipped.** Member-aware matching (2026-07-19): a `subscription-leak` finding's canonical `event 'A.B.Holder.Member'` identity must equal a retention root's `(short(holder), member)`; the matched root is reported as `root_holder`/`root_member`. Evidence in `docs/evidence/gtd-runtime-transition.md` |
51+
| 3 | Static pass drives targeted instrumentation | **Not built — and the motivation does not transfer.** Leaf needs targeting because per-event MIR instrumentation is reportedly 19×–396×. A post-scenario heap snapshot has no per-event cost to target down. Only becomes relevant *if* OwnAudit moves to event-stream instrumentation, which is a decision on its own merits |
52+
| 4 | Explicit evidence gaps: `COMPLETE / PARTIAL / OPAQUE / UNSUPPORTED` | **Already a machine contract, and stronger.** [`runtime-witness-operations.md`](../runtime-witness-operations.md): exit 0 = heap read, nothing retained; exit 1 = read, retained; **exit 2 = the heap was not read**. A refused attach writes **no** `runtime.json` — there is no verdict to record. *"Proven by CI: a denied attach exits 2, names the policy that refused, and leaves no artifact behind."* Plus `NO-TOOL: skipped` coverage maps, the OWN050/051/052 advisories, and P-036's rule that a check "may not pretend the call was proven harmless" |
53+
| 5 | Use it as an oracle for static rules | **Pattern already shipped**, minus Leaf. The 3-way oracle (Own.NET vs Infer# vs CodeQL) plus the runtime buckets: **confirmed** (gate on these), **static-only** (suspect FP or unexercised path), **runtime-only** = *"the analyzer's blind spot — candidate for a new rule"* |
54+
55+
So: two already shipped, one shipped as a pattern, one already has its seam, and
56+
one solves a cost problem this stack does not have.
57+
58+
The sharpest inversion is #4. The proposal offered explicit evidence gaps as
59+
something to import — *"сделанная частью машинного контракта, а не красивой
60+
фразой в README"*. It is already the machine contract, it is CI-proven, and
61+
distinguishing *not looking* from *looking and finding nothing* is one of the
62+
oldest rules in this codebase.
63+
64+
## What is actually worth taking
65+
66+
Two things, both concept-level, neither depending on Leaf.
67+
68+
**(a) One runtime substrate, many analyzers.** Leaf's real contribution is
69+
decoupling: instrumentation exists separately from any particular analysis, so
70+
one semantic layer (50 DMIR callbacks) serves a concolic executor, a sanitizer
71+
and a tracer. The boundary transfers; the MIR does not.
72+
73+
**(b) Separate the *collection* substrate from the *analysis-facing*
74+
representation.** Subtler than "one shared schema", and the part worth stealing
75+
outright. Leaf does not hand probes to analyzers: probes are shaped for cheap
76+
instrumentation, and a Probe-to-DMIR adapter reconstructs structured events for
77+
analysis. The two formats are deliberately different. The failure this avoids is
78+
one universal envelope that everything gets poured into *because the schema
79+
already exists* — a snapshot and an event stream should not be forced to
80+
pretend they are one datatype.
81+
82+
### Measured against the tree, the question is already overdue
83+
84+
The natural framing — "decide before the second dynamic consumer" — is **too
85+
late by two consumers**. There are **five runtime-evidence producer
86+
implementations across the two repositories**: four under
87+
`Own.NET/audit/runtime/`, plus the lift-out producer in
88+
`OwnAudit/src/OwnAudit.Runtime`.
89+
90+
"Collector" is the wrong word for them, and the distinction is load-bearing
91+
rather than pedantic — it decides which layer a new contract belongs to.
92+
Verified by the APIs each one actually calls:
93+
94+
| runtime evidence producer | acquisition it performs | material it interprets | output | schema field |
95+
|---|---|---|---|---|
96+
| `RetentionPath` | `AttachToProcess`, `procdump` | live CLR **or** `.dmp` | `runtime.json` | `"own-runtime/1"` |
97+
| `OwnAudit.Runtime` (`RuntimeReport.cs`) | stand-side collector | live heap | `runtime.json` | `"ownAudit/runtime/v1"` |
98+
| `LeakHarness` | `procdump` (orchestrates the scenario) | `.dmp` | `leak-harness.json` | none |
99+
| `DuplicateDetector` | `procdump` (optional) | `.dmp` | `duplicate-detector.json` | none |
100+
| `PropertyChangedStorm` | **none** | `.etl` captured externally | `propertychanged-storm.json` | none |
101+
102+
`PropertyChangedStorm` is the clean case: it opens an `ETWTraceEventSource` over
103+
an `.etl` that PerfView/xperf/logman captured against the target's
104+
`OwnNet-Sematix-INPC` EventSource. It acquires nothing. So the real layering is
105+
**acquisition** (live attach / `.dmp` / `.etl`) → **interpretation** →
106+
**analysis-facing artifact** → `correlate.py` / `ingest.py` → SARIF.
107+
108+
That reframes takeaway (b) more sharply than "one substrate, many analyzers":
109+
*the acquisition representation need not — and should not — be the evidence
110+
representation.* A `.dmp` and an `.etl` are already good acquisition substrates
111+
for their jobs; turning them into a universal runtime-event JSON would unify the
112+
wrong layer. The thing worth sharing is a **runtime artifact contract**, not a
113+
collection substrate.
114+
115+
So [`Plan.md`](../../Plan.md) categories 6 (PropertyChanged storms) and 11
116+
(duplicate immutable heap data, flagged there as *"the project's gold"*) are not
117+
future work to design around — they exist as code, each with its own bespoke
118+
JSON. Three findings follow, all checkable:
119+
120+
1. **Two producers of `runtime.json` emit two different schema strings.**
121+
`RetentionPath` (both of its writers) says `own-runtime/1`; the newer
122+
stand-side collector, `docs/runtime-contract.md`, and every fixture say
123+
`ownAudit/runtime/v1`. This is a lift-out artifact — the collector plan
124+
(2026-07-18) ports `stackpeek` into `OwnAudit.Runtime` while the older
125+
`RetentionPath` stays canonical in Own.NET — not sloppiness. But it is live.
126+
2. **Nothing enforces either.** `runtime/correlate.py` consumes exactly
127+
`retained / type / count / expected / bytes / roots / kind / holder /
128+
member`. It never reads `schema`, `scenario`, `iterations`, `collector`, or
129+
`verdict`. The only assertion on the string is
130+
`HeapCollectorContractTests.cs:83`, and it checks the producer against
131+
itself. A `runtime.json` with the wrong version — or none — correlates
132+
silently.
133+
3. **The envelope fields are decorative.** `scenario` and `iterations` are
134+
documented and present in fixtures but unread; `collector` (a genuine
135+
provenance fingerprint) and `verdict` are emitted by `RetentionPath` but
136+
appear in neither the documented schema nor the consumer.
137+
138+
**No impact today** — the correlator is deliberately tolerant of unknown fields,
139+
which is a reasonable forward-compat choice. The risk is latent and the fix is
140+
cheap *now*, which is exactly the argument for settling it before the runtime
141+
artifact surface expands further.
142+
143+
### Counted by artifact family, which is what would get versioned
144+
145+
Producers are the wrong unit — five implementations emit **four** analysis-facing
146+
artifact families, and it is the families that need identity:
147+
148+
| artifact family | producer(s) | identity status |
149+
|---|---|---|
150+
| heap-retention | `RetentionPath` + `OwnAudit.Runtime` | **two conflicting identities** (`own-runtime/1` vs `ownAudit/runtime/v1`) |
151+
| leak-growth | `LeakHarness` | **none** |
152+
| duplicate-immutable | `DuplicateDetector` | **none** |
153+
| propertychanged-storm | `PropertyChangedStorm` | **none** |
154+
155+
> **All four shipped runtime artifact families have an identity problem:
156+
> heap-retention has two conflicting schema identities; the other three have
157+
> none.**
158+
159+
### The cheap next step is versioning, not unification
160+
161+
A shared envelope carrying capture/process/scenario identity, producer
162+
fingerprint, config digest and provenance, with typed-distinct payloads, is the
163+
right *destination*. It is the wrong *first move*: it designs commonality up
164+
front, which is precisely the mistake Leaf avoids. Leaf's lesson is **define the
165+
boundary first, derive the representation second**.
166+
167+
So the minimal work that closes the real hole found above, in order:
168+
169+
1. **Every analysis-facing artifact gets its own mandatory schema identity** —
170+
one per family, e.g. `heap-retention`, `leak-growth`,
171+
`duplicate-immutable`, `propertychanged-storm` (names illustrative, not
172+
normative). Today three of the five producers emit no version at all, and
173+
the two that do disagree with each other — see the family table above.
174+
2. **Readers must validate family and major version.** Tolerant of unknown
175+
*fields*, intolerant of unknown *identity*:
176+
177+
| input | verdict |
178+
|---|---|
179+
| known family + known major + unknown extra fields | accept |
180+
| missing schema | reject |
181+
| wrong artifact family | reject |
182+
| unknown major | reject |
183+
184+
This is the actual gap. `correlate()` currently takes a `dict` and
185+
structurally duck-types it on `retained`/`type`/`count`/… — so *any* JSON
186+
containing a `retained` array is treated as the contract.
187+
3. **Only then extract the envelope**, from commonality *proven* by four
188+
shipped schemas rather than drawn in advance. Fields have to earn the right
189+
to be called common.
190+
191+
This closes the found defect now, changes no fail-closed semantics, and commits
192+
to no universal-envelope architecture.
193+
194+
### Split `completeness` into three orthogonal things
195+
196+
The caution raised earlier resolves cleanly once the word is broken up. These
197+
are three different states and are currently drifting toward one label:
198+
199+
- **Collection outcome.** Success → an artifact may exist. Refusal or failure →
200+
exit 2 and the artifact **must not** exist. *Unchanged where it already
201+
holds* — and it does **not** hold everywhere: `RetentionPath` has the
202+
documented contract, `DuplicateDetector` and `PropertyChangedStorm` catch
203+
broadly and return 2, but `LeakHarness` catches only `ScenarioException`, so
204+
a missing scenario file or a failed dump escapes `Main` with a different exit
205+
code. Normalizing that is a behaviour change, not a clean-up; see the task
206+
spec's caveat.
207+
- **Artifact validity.** Given an artifact: known schema, required fields,
208+
well-formed, producer contract satisfied → valid, else reject. *This is the
209+
layer that is missing today* (step 2 above).
210+
- **Evidence coverage.** Belongs to the **artifact family**, not to a global
211+
enum. For heap retention the family contract can simply be
212+
total-or-no-artifact — no `PARTIAL` ever. For an ETW trace the physics differ
213+
and coverage is real data: capture interval, events lost, buffers lost, trace
214+
truncated, provider-enabled interval. A valid storm artifact can legitimately
215+
rest on a bounded trace if the claim it makes is bounded accordingly.
216+
217+
That keeps "refusal → no artifact" intact, because *collector refused* and
218+
*collector succeeded over bounded observation* are different states, not two
219+
shades of one. The invariant to write down:
220+
221+
> **Artifact existence proves successful execution of its producer contract,
222+
> not completeness of every possible observation. Coverage semantics belong to
223+
> the artifact family and must never be inferred from artifact existence
224+
> alone.**
225+
226+
This is strictly better than a global `"completeness": "PARTIAL"`, which within
227+
a year means five different things per producer and then grows a
228+
`PARTIAL_BUT_USABLE`.
229+
230+
## The invariant worth stating separately
231+
232+
The error that produced the original proposal is natural, will recur, and is
233+
cheap to inoculate against:
234+
235+
> **Do not infer analyzed-language support from the implementation language of
236+
> the Own.NET core.** P-022 moves the *analyzer's implementation* to Rust. It
237+
> does not change the audited runtime, does not introduce Rust MIR, and does not
238+
> make Rust an analysis target — P-017's targets are OwnTS and OwnJVM.
239+
240+
Implementation language and analyzed language are orthogonal axes. Conflating
241+
them is what turned a Rust-only instrumentation framework into an 8.5/10
242+
recommendation for a .NET auditor.
243+
244+
## Provenance and limits
245+
246+
- The paper's existence, title, authors and subject were verified against arXiv.
247+
- The **overhead figures** (19×–396×, ~3.5× for concrete types, ~2.8× for raw
248+
addresses, ~50 DMIR callback types, the `nightly-2026-07-01` pin, and the
249+
three demo analyses) were **not** checked here against the paper. The author
250+
of the original summary re-verified them against it afterwards and reports
251+
them correct; that is second-hand for this note's purposes, and they are
252+
quoted only to show that the motivation for idea #3 is cost, not to rely on
253+
the magnitudes.
254+
- OwnAudit was read at a shallow clone of `main`; the collector inventory and
255+
schema-string findings are about that tree plus `Own.NET/audit/runtime/` at
256+
this branch's base commit.
257+
- **Recorded, not scheduled** — with one exception. The identity/validation
258+
defect measured above is no longer an idea from a paper but a demonstrated
259+
contract hole, so it has a written spec at
260+
[`docs/tasks/runtime-artifact-identity.md`](../tasks/runtime-artifact-identity.md)
261+
(status: *spec, ready to implement*, explicitly **not** inserted into the
262+
current sequence). Everything else here remains recorded only.

0 commit comments

Comments
 (0)