Skip to content

Commit 39d55dc

Browse files
MrScriptyMrScripty
committed
test(standards): strengthen targeted regression checks
1 parent 91ceb0f commit 39d55dc

11 files changed

Lines changed: 593 additions & 15 deletions

File tree

‎docs/plans/README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,3 +131,9 @@ packaging and broader test strengthening are later, separately admitted work.
131131
[Review-evidence packaging](review-evidence-packaging/plan.md) owns the local
132132
review packet builder. Its implementation is in progress. Developer handoff ZIPs
133133
are external artifacts and are not repository content.
134+
135+
## Targeted Test Strengthening
136+
137+
[Targeted Test Strengthening](targeted-test-strengthening/plan.md) prepares the five
138+
selected regression-oracle improvements without product changes. Its integration
139+
follows packaging acceptance; neither earlier accepted sequence is reopened.

‎docs/plans/targeted-test-strengthening/execution-ledger.md‎

Lines changed: 58 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,58 @@
1+
# Targeted Test Strengthening — Execution Ledger
2+
3+
## 2026-09-29 — Admission and source inspection
4+
5+
The user authorized implementation of the next step. An isolated task branch uses
6+
GitHub main `91ceb0fe`, with the mounted Actions source archive verified against
7+
SHA-256 `3e7c09713cde735ca525464872d8e9367b5b853b939635adcfbf790198d24af1`.
8+
The checkout began clean; no user working tree, local operator configuration or
9+
remote ref is a write target. No repository AGENTS.md/CONTEXT.md is present.
10+
Core, Router, current implementation/verification/oracle guidance and the selected
11+
20-standard route were read. Explicit route facts had no unanswered questions.
12+
13+
Packaging CI run `36509340996` now reports completed/success. Its plan still says
14+
Verifying. The source preparation for priority 3 does not assume or grant packaging
15+
acceptance; integration follows the recorded priority order. Earlier acceptance
16+
and deferred advisories retain their original owners.
17+
18+
The five reported test gaps are present in this exact tree. Source inspection of
19+
Applicability and Analysis establishes the three intended diagnostics. A baseline
20+
compiler probe confirmed their code, outcome, message, path and differing field/
21+
observed data. Existing aggregate equality and SQLite error adaptation are correct.
22+
23+
Task setup corrections: store_path belongs to StandardsEngine.open_repository,
24+
not AgentToolFacade.open_repository; the corrected explicit temporary-store path
25+
was used. A display-only attempt to index an already formatted route target as a
26+
mapping failed after the successful route/readback. Neither is a product failure
27+
or acceptance test. The optional task-owned Python 3.12 download failed on DNS.
28+
29+
## T1 — Implementation
30+
31+
Modified the five existing test owners. Added a private static-import predicate
32+
and its allow/deny fixtures in the existing fact-ownership test file; used Python's
33+
own relative-name resolver instead of a second import parser. Four new test methods
34+
cover that predicate, changed aggregate identity and real exclusive-lock probing.
35+
Existing projection and record-independence methods gain stronger observations.
36+
The first seven-test smoke selection passed with no production repair.
37+
38+
## 2026-09-29 — Local verification complete
39+
40+
T1 is Implemented; the plan transitions to Verifying with T2 as its sole next
41+
slice. The complete affected Engine selection passed 97 methods, Analysis passed
42+
125, Snapshots passed 41 and Verifier passed 168: 431 normal executions, no skips
43+
or failures. The full Engine and all other packages were not rerun. Existing real
44+
typed authoring/replay and cold storage-transport tests are in the selected modules.
45+
46+
The separate in-memory fault experiment rejected 42 sampled faults through precise
47+
assertions and no incidental errors. 41 have a recorded counterpart original test
48+
that accepts the fault; this is not a whole-suite mutation score. See the report
49+
for the storage comparison limitation. All substitutions were restored; no
50+
production source change was warranted or made.
51+
52+
Structural verification passed 73 suites/121 checks, generated contracts are fresh,
53+
and the exact staged test diff was reviewed. All 1,498 unaffected tracked paths
54+
were byte-verified unchanged. After final records, focused tests, structural and
55+
generated-input checks are rerun; the delivery contains their actual results and
56+
fresh-base reconstruction. No source commit, remote push or acceptance rewrite is
57+
part of this delivery. Packaging's successful base CI remains distinct from the
58+
unexecuted candidate locked CI and its existing independent acceptance gate.

‎docs/plans/targeted-test-strengthening/issues.md‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
# Targeted Test Strengthening — Issues
2+
3+
## T-01 — Packaging acceptance is a separate prerequisite
4+
5+
The current packaging source is implemented and its exact-source CI is successful,
6+
but its acceptance record remains Verifying. Preserve that record; deliver this
7+
isolated test-only preparation without treating it as packaging acceptance. The
8+
integration/acceptance owner reconciles the preceding gate before integrating this
9+
next slice. No new packaging repair is called for by this work.
10+
11+
## T-02 — Supported candidate environment and independent review
12+
13+
Local Python 3.13.5 with rpds-py 2026.5.1 is not the hash-locked Python 3.12 CI
14+
qualification. The task-owned toolchain download failed on DNS. Keep locks unchanged;
15+
run the existing workflow on the integrated candidate and obtain independent review.
16+
Current baseline CI is not evidence for the added tests.
17+
18+
## T-03 — Scope limits
19+
20+
The equality/hash and float-zero observations, dynamic import analysis, broader
21+
storage tuning and additional typed-edit families remain deferred. Tests here
22+
establish only their named finite mutation population and actual SQLite case.
23+
No new production defect has been demonstrated by the initial checks.
24+
25+
## Final source disposition
26+
27+
All five selected gaps have implemented tests and sampled-fault detection evidence.
28+
No production defect was demonstrated and no production repair is included. T-01
29+
and T-02 remain integration/acceptance conditions, not failing local tests. T-03
30+
remains an explicit scope limit. Source is ready for the separate candidate CI
31+
and reviewer/owner decision once the preceding packaging gate is reconciled.

‎docs/plans/targeted-test-strengthening/plan.md‎

Lines changed: 149 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,149 @@
1+
# Targeted Test Strengthening
2+
3+
**Plan status:** `Verifying`
4+
5+
**Current phase:** T1 implemented and locally verified; candidate CI/review and preceding packaging acceptance remain separate.
6+
7+
**Next slice:** **T2 — Candidate qualification and acceptance**
8+
9+
**Acceptance status:** `partial`
10+
11+
**Composed-design review:** `not-applicable`; tests and isolated fault examples
12+
change only. Existing product owners, public interfaces, state and dependency
13+
contracts remain unchanged. The small AST helper is private to its existing test.
14+
15+
**Execution ledger:** [execution-ledger.md](execution-ledger.md)
16+
17+
**Issues:** [issues.md](issues.md)
18+
19+
## Objective
20+
21+
Make the five previously deferred test weaknesses detect their named regressions,
22+
without changing production behavior or treating an accepted earlier slice as
23+
incomplete. This is priority 3 of the selected 2 → 4 → 3 sequence.
24+
25+
The user authorized implementation of the next step. Source is prepared in an
26+
isolated branch against `91ceb0fedcd6bb23513cf414a60b1e506d7de893`, tree
27+
`83b59dffdb8312867a65582d53c55edf2576604a`. Packaging's exact-source workflow
28+
`36509340996` now reports success. Its current plan still records Verifying;
29+
external review and acceptance-owner disposition remain separate. This delivery
30+
neither edits that plan nor declares it Accepted. Integrate this prepared next
31+
slice after the preceding packaging acceptance is resolved.
32+
33+
## Scope
34+
35+
T1's exact write set comprises the following existing tests, this plan directory,
36+
a link-only update to `docs/plans/README.md`, and the producer-generated
37+
`evaluation/standards-effectiveness/generated/suite-inputs.json`:
38+
39+
- `tools/standards_engine/tests/test_typed_routing_projection.py`
40+
- `tools/standards_analysis/tests/test_routing_records.py`
41+
- `tools/standards_engine/tests/test_fact_material_ownership.py`
42+
- `tools/standards_engine/tests/test_immutable_work_reuse.py`
43+
- `tools/standards_snapshots/tests/test_storage_lifecycle.py`
44+
45+
No production repair, dependency, CI configuration, new framework, schema,
46+
interface version, persistent format, acceptance-history rewrite or provider
47+
operation is included. A demonstrated product defect is recorded before admitting
48+
any production repair. Hash/equality and float-zero advisories remain deferred.
49+
50+
## Objective Acceptance
51+
52+
| ID | Observable criterion | Kind | Environment | Mode | Status | Evidence |
53+
| --- | --- | --- | --- | --- | --- | --- |
54+
| T-A1 | The three routing negatives distinguish the specified projection failure, not merely AnalysisError. | contract-negative | existing real compiler fixture | automated | satisfied | [Verification](reports/verification.md) |
55+
| T-A2 | The record-independence assertion uses a detached oracle and detects shared outer and nested containers. | focused + mutation | local Python | automated | satisfied | [Verification](reports/verification.md) |
56+
| T-A3 | The material-owner import guard detects declared absolute, relative and aliased static navigation imports while accepting unrelated imports/text. | structural + negative | Python AST | automated | satisfied | [Verification](reports/verification.md) |
57+
| T-A4 | A changed aggregate_id cannot hit exact-record decoding and cannot replace its previously admitted entry. | focused + mutation | real snapshot/authoring fixture | automated | satisfied | [Verification](reports/verification.md) |
58+
| T-A5 | A real EXCLUSIVE writer blocks the no-work expiry probe with BUSY/unavailable, without maintenance writes or a stranded transaction. | persistence integration | Linux SQLite DELETE journal | automated | satisfied | [Verification](reports/verification.md) |
59+
| T-A6 | Existing affected suites, generated freshness and structural checks pass; delivered changes preserve all production files. | integration | supported Python 3.12 locked CI | automated | pending | [Verification](reports/verification.md) |
60+
| T-A7 | Independent review and owner disposition accept this bounded test slice without reopening previous work. | review | reviewer and acceptance owner | manual | pending | [Verification](reports/verification.md) |
61+
62+
## Binding Decisions
63+
64+
Use literal AnalysisFailure expectations tied to the existing applicability and
65+
Analysis diagnostic owners. Parsing stays outside the projection exception scope;
66+
changing an error class's fields in a controlled example must invalidate the test.
67+
The unchanged real compiler remains the ordinary execution path.
68+
69+
Retain the ordinary canonical-record fixture and make its expected containers
70+
independent before mutation. Verify freshness of the outer list, records, values
71+
and aliases. Never patch production merely to pass a newly copied expectation.
72+
73+
The dependency guard parses static imports with Python's AST and resolves relative
74+
module names with `importlib.util.resolve_name`. It guards only supporting's
75+
navigation dependencies; it is not a dynamic import detector or a repository-wide
76+
architecture framework. Fixture module strings are parsed, never imported.
77+
78+
Construct a valid second revision through Authoring, then substitute only that
79+
revision's ID in the first aggregate record. Require the original stored-revision
80+
identity diagnostic and canonical decoder invocation, followed by an intact old
81+
cache entry. The test must distinguish identity-blind reuse from normal behavior.
82+
83+
For the SQLite case, use two admitted real connections and BEGIN EXCLUSIVE under
84+
DELETE journal mode. Shorten only the test connection's busy timeout. Observe
85+
actual probe SQL, the underlying SQLITE_BUSY cause, no maintenance transaction,
86+
and unchanged counts/material after rollback. No sleeps, latency threshold,
87+
fabricated SQLite error, or production timeout changes are part of acceptance.
88+
89+
## Evidence And Oracle Plan
90+
91+
The source-backed advisory population is the preserved typed-routing external
92+
review (broad projection errors), fact-owner review (mutable expected containers
93+
and incomplete imports), reuse review (aggregate identity), and storage review
94+
(exclusive-read contention). Their prior acceptance dispositions remain intact.
95+
Use the supplied review's exact framing: these were test gaps, not established
96+
production failures.
97+
98+
Controlled in-memory substitutes prove detection of named faults only. Record
99+
baseline-test survival versus strengthened-test rejection where comparable.
100+
Required normal tests run without substitutes. Raw logs, exact mutation functions,
101+
environment, candidate identities and results are retained in the delivery;
102+
no mutated production file is distributed or committed. No mutation score or
103+
complete fault-coverage claim is inferred from the sampled cases.
104+
105+
## Milestones
106+
107+
### T1 — Five stronger test boundaries
108+
109+
**Status:** `Implemented`
110+
111+
Work: inspect original diagnostics/fixtures; update only the exact test owners;
112+
prove sampled regression detection; run complete affected suites and the normal
113+
structural/generated checks; review the diff and package against the exact base.
114+
115+
Gate: T-A1–T-A5 have direct positive and negative evidence. Local test evidence
116+
remains separate from the supported-runtime qualification required by T-A6.
117+
118+
### T2 — Candidate qualification and acceptance
119+
120+
**Status:** `Planned`
121+
122+
Write set: this plan's records and generated inputs; necessary T1 repairs only.
123+
Obtain or reuse exact-material locked CI, independent test review and owner
124+
acceptance. Bind to the reviewed candidate and preserve packaging's separate gate.
125+
No retrospective PR or repeated full campaign is required without changed claims.
126+
127+
## Blockers
128+
129+
Handoff preparation had only Python 3.13.5/rpds-py 2026.5.1. Its task-owned Python
130+
3.12 installation attempt failed on DNS; no lock or environment guarantee was changed.
131+
The integrated candidate later passed the affected local selections on Python 3.12.3,
132+
as recorded in the verification report. Required hosted candidate CI and independent
133+
review cannot be claimed from these local runs.
134+
These do not prevent isolated source/test work; keep acceptance Verifying when
135+
those are the remaining claims. Packaging acceptance remains an integration gate,
136+
not an assumed fact or a reason to modify already-correct product code.
137+
138+
## Re-Plan Triggers
139+
140+
Record/reassess a demonstrated product defect, a changed public failure contract,
141+
a need for a general import/test framework, fixture failure before the intended
142+
boundary, or unsupported required SQLite behavior. A narrow direct fixture repair
143+
within these test owners does not expand the product scope.
144+
145+
## Final Acceptance
146+
147+
T-A1–T-A5 have local passing evidence. T-A6/T-A7 still require the named
148+
supported candidate environment and independent review/owner disposition. Close
149+
only this slice after all required claims are satisfied. Earlier accepted work and the separate packaging records are unchanged.

0 commit comments

Comments
 (0)