fix(emit): a contracted floor that ran nothing is not a pass - #16
Closed
khymerao wants to merge 2 commits into
Closed
fix(emit): a contracted floor that ran nothing is not a pass#16khymerao wants to merge 2 commits into
khymerao wants to merge 2 commits into
Conversation
Dogfood 14 closed the RED floor: a scope-clean job whose floor exited non-zero is
`blocked`, not `success`. The EMPTY floor was left open, and on a real run it was
the more dangerous half.
`_tests_block_from_floor` returns None when the floor executed no command ("the
schema says to omit the object entirely rather than report a fabricated zero"), so
tests_block = _tests_block_from_floor(...) if tests else None
if status == "success" and isinstance(tests_block, dict):
never fires and the job keeps `success` with NO test evidence at all.
Observed on two sibling worktree jobs in one wave. One installed the project's
dependencies so its floor could run, and was BLOCKED for the install (node_modules
is gitignored, and the gate counts ignored writes with no exceptions). The other
installed nothing, so its floor could not run at all - and PASSED. The job that
made its floor runnable was refused; the job whose floor never ran was merged.
Absent evidence was being read as absence of objection.
The rule is narrow on purpose: it fires only when the manifest actually contracted
a floor. A job with no `test_contract` has nothing to run and stays `success`, so
"no floor at all is not a failure" is unchanged. `blocked`, not `error`: the
machinery is fine, the job simply has no evidence it may merge on.
This does not fix why the floor could not run - a worktree is unprovisioned and
provisioning it is charged to the job, which is an architecture question filed
separately. It stops that condition from being reported as a pass.
Selftest: 529/529 (was 526). Three new rows; mutation-checked - disabling the rule
fails "a CONTRACTED floor that ran nothing is blocked, not success".
… no command
Reworked after review. The previous commit's justification was an anecdote that
turns out to be FALSE, and its predicate was the wrong field. Both corrected.
THE ARGUMENT, from the invariants rather than a story:
`run_test_floor` initialises every result {"passed": False, "merge_blocked": True} -
refusal is the floor's default state - and ADR 0003 relies on it: "already
merge-blocking and already fail-closed on an empty command", Consequences: "A job
reporting no test command at all is a FAIL, not a pass." spec-reviewer.md enforces
the same in prose (ISSUE: NO_TEST_EVIDENCE, "Silence is not success").
The fast path's own consumer honours it: build_review_spec gate 1 refuses on
`not passed or merge_blocked`. This consumer did not. It reads the floor only
through `_tests_block_from_floor`, which returns None when no check carries a
`checker`, so `merge_blocked: True` was never consulted and the job kept the scope
verdict's `success`. One floor document, two consumers, opposite conclusions.
Dogfood 14 moved the RED half of that rule from prose into this function; this is
the empty half, same layer, same reason.
CORRECTIONS TO THE PREVIOUS COMMIT:
- The predicate is now the MANIFEST's declaration, not the resolved slice.
`contract` is read from jobs/<id>.test-contract.json, which is absent exactly
when resolution FAILED - the reachable case. Keying on it left the hole open.
- `tests is None` (no floor document at all: fastpath-run.py missing or dead
before writing JSON) is `error`, not `blocked`. The machinery IS broken there.
- The refusal now carries the floor's own reasons into `summary`; it previously
fell back to the job title, refusing for an unstated reason.
- The sibling-jobs anecdote is removed. Checking the run record, `task-4-wire` was
status `blocked` with tests.exit_code 1 - its floor DID run and went red, and the
dogfood-14 rule caught it. The `tests_block is None` path was never exercised
there. This is a latent fail-open found by reading, not a reproduced defect, and
the comment now says so by not claiming otherwise.
Unchanged: an uncontracted manifest still records `success` with no `tests` object
("no floor at all is not a failure").
Selftest: 532/532 (was 526). Six rows; mutation-checked - disabling the rule fails
four of them.
Contributor
Author
|
Folded into #13, which now carries both halves of the same story: what a Nothing is dropped - the empty-floor rule, its six selftest rows and the corrections from review (predicate on the manifest's declaration rather than the resolved slice; |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes defect 3a of #15.
The defect, from the invariants
recorddiscards the floor's own fail-closed verdict when the floor resolved to no command.run_test_floorinitialises every result as{"passed": false, "merge_blocked": true}and documents "merge_blockedis True on any floor FAILURE (Iron-Invariant #6)"._cmd_test_flooremits exactly that document onTestContractError. ADR 0003 relies on it: "run_test_flooris already merge-blocking and already fail-closed on an empty command."The fast path's own consumer honours it -
build_review_specgate 1 refuses onnot passed or merge_blocked. Engine C's consumer does not:_job_result_fromreads the floor only through_tests_block_from_floor, which returnsNonewhenchecks[]carries nochecker, somerge_blocked: trueis never consulted and the job keeps the scope verdict'ssuccess. Same floor document, two consumers, opposite conclusions.ADR 0003 Consequences: "A job reporting no test command at all is a FAIL, not a pass."
agents/spec-reviewer.mdenforces that in prose (ISSUE: NO_TEST_EVIDENCE- "Silence is not success; an absent record is exactly what a worker that skipped the step looks like"). Dogfood 14 moved the red half of the rule from prose intorecord, because "thetestsblock is the fourteenth mechanism this project built and left without a consumer". This closes the empty half at the same layer, for the same reason.Frequency, stated honestly
Never observed. 0 of 198 recorded results in this repository and 0 of 17 in the reporting project show
successwith notestsblock under a contracted manifest. The run in #15 was caught by the red-floor rule:task-4-wirewasstatus: blocked,tests.exit_code: 1- its floor did run and went red.This is a latent fail-open found by reading the code, not a reproduced defect. An earlier revision of this PR claimed otherwise on the strength of a narrative I had not checked against the run record; that claim is withdrawn.
The rule
contractis read fromjobs/<id>.test-contract.json, which does not exist precisely when resolution failed - the reachable case (a manifest declaringimpacted_mapwith nofull_commandpasses the validator, then the resolver raises and no slice is written). Keying on the slice would leave that hole open.tests is Noneiserror, notblocked. No floor document at all meansfastpath-run.pywas absent or died before writing JSON - the machinery is broken, which is a different taxonomy from a refused job.summary. It previously fell back to the job title, which is a refusal for an unstated reason - the failure mode the gate-receipt code itself warns about.Not changed
An uncontracted manifest still records
successwith notestsobject - "no floor at all is not a failure" (dogfood 14), and the emitter's own note that "the worker runs no tests and reports notestsobject, which is the documented honest outcome, not a silent zero".No false positives for docs-only jobs.
resolve_test_commandsputsfloor_commandfirst at every tier and raises on an empty set, so a docs-only change at SCOPED/DIRECT/FULL still resolves to[floor_command]and produces achecks[]entry. A job whose floor legitimately has nothing impacted to run still has a floor, sotests_blockis non-None and this rule cannot fire.Tests
532/532 (was 526/526 on
main). Six rows: a contracted floor that resolved to no command is blocked; it carries no fabricated tests block; the refusal states the floor's own reason; a declared contract whose slice is missing is still caught; no floor document at all iserror; an uncontracted job still passes.Mutation-checked - disabling the rule fails four of the six.
Full
scripts/*.py --selftestsweep,lint-frontmatter,rules-lintgreen locally.Note on CI
The red
Full test suiteistests/test-native-points.sh, which fails 2 PRECOMPACT rows on a cleanmaintoo - the hardcoded-date time bomb that #11 fixes. Not from this branch.