Skip to content

fix(emit): a contracted floor that ran nothing is not a pass - #16

Closed
khymerao wants to merge 2 commits into
procoders:mainfrom
khymerao:fix/empty-floor-not-success
Closed

fix(emit): a contracted floor that ran nothing is not a pass#16
khymerao wants to merge 2 commits into
procoders:mainfrom
khymerao:fix/empty-floor-not-success

Conversation

@khymerao

@khymerao khymerao commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Fixes defect 3a of #15.

The defect, from the invariants

record discards the floor's own fail-closed verdict when the floor resolved to no command.

run_test_floor initialises every result as {"passed": false, "merge_blocked": true} and documents "merge_blocked is True on any floor FAILURE (Iron-Invariant #6)". _cmd_test_floor emits exactly that document on TestContractError. ADR 0003 relies on it: "run_test_floor is already merge-blocking and already fail-closed on an empty command."

The fast path's own consumer honours it - build_review_spec gate 1 refuses on not passed or merge_blocked. Engine C's consumer does not: _job_result_from reads the floor only through _tests_block_from_floor, which returns None when checks[] carries no checker, so merge_blocked: true is never consulted and the job keeps the scope verdict's success. Same floor document, two consumers, opposite conclusions.

ADR 0003 Consequences: "A job reporting no test command at all is a FAIL, not a pass." agents/spec-reviewer.md enforces that in prose (ISSUE: NO_TEST_EVIDENCE - "Silence is not success; an absent record is exactly what a worker that skipped the step looks like"). Dogfood 14 moved the red half of the rule from prose into record, because "the tests block is the fourteenth mechanism this project built and left without a consumer". This closes the empty half at the same layer, for the same reason.

Frequency, stated honestly

Never observed. 0 of 198 recorded results in this repository and 0 of 17 in the reporting project show success with no tests block under a contracted manifest. The run in #15 was caught by the red-floor rule: task-4-wire was status: blocked, tests.exit_code: 1 - its floor did run and went red.

This is a latent fail-open found by reading the code, not a reproduced defect. An earlier revision of this PR claimed otherwise on the strength of a narrative I had not checked against the run record; that claim is withdrawn.

The rule

if status == "success" and tests_block is None and contract_declared:
    status = "error" if tests is None else "blocked"
  • The predicate is the manifest's declaration, not the resolved slice. contract is read from jobs/<id>.test-contract.json, which does not exist precisely when resolution failed - the reachable case (a manifest declaring impacted_map with no full_command passes the validator, then the resolver raises and no slice is written). Keying on the slice would leave that hole open.
  • tests is None is error, not blocked. No floor document at all means fastpath-run.py was absent or died before writing JSON - the machinery is broken, which is a different taxonomy from a refused job.
  • The refusal carries the floor's own reasons into summary. It previously fell back to the job title, which is a refusal for an unstated reason - the failure mode the gate-receipt code itself warns about.

Not changed

An uncontracted manifest still records success with no tests object - "no floor at all is not a failure" (dogfood 14), and the emitter's own note that "the worker runs no tests and reports no tests object, which is the documented honest outcome, not a silent zero".

No false positives for docs-only jobs. resolve_test_commands puts floor_command first at every tier and raises on an empty set, so a docs-only change at SCOPED/DIRECT/FULL still resolves to [floor_command] and produces a checks[] entry. A job whose floor legitimately has nothing impacted to run still has a floor, so tests_block is non-None and this rule cannot fire.

Tests

532/532 (was 526/526 on main). Six rows: a contracted floor that resolved to no command is blocked; it carries no fabricated tests block; the refusal states the floor's own reason; a declared contract whose slice is missing is still caught; no floor document at all is error; an uncontracted job still passes.

Mutation-checked - disabling the rule fails four of the six.

Full scripts/*.py --selftest sweep, lint-frontmatter, rules-lint green locally.

Note on CI

The red Full test suite is tests/test-native-points.sh, which fails 2 PRECOMPACT rows on a clean main too - the hardcoded-date time bomb that #11 fixes. Not from this branch.

Dogfood 14 closed the RED floor: a scope-clean job whose floor exited non-zero is
`blocked`, not `success`. The EMPTY floor was left open, and on a real run it was
the more dangerous half.

`_tests_block_from_floor` returns None when the floor executed no command ("the
schema says to omit the object entirely rather than report a fabricated zero"), so

    tests_block = _tests_block_from_floor(...) if tests else None
    if status == "success" and isinstance(tests_block, dict):

never fires and the job keeps `success` with NO test evidence at all.

Observed on two sibling worktree jobs in one wave. One installed the project's
dependencies so its floor could run, and was BLOCKED for the install (node_modules
is gitignored, and the gate counts ignored writes with no exceptions). The other
installed nothing, so its floor could not run at all - and PASSED. The job that
made its floor runnable was refused; the job whose floor never ran was merged.
Absent evidence was being read as absence of objection.

The rule is narrow on purpose: it fires only when the manifest actually contracted
a floor. A job with no `test_contract` has nothing to run and stays `success`, so
"no floor at all is not a failure" is unchanged. `blocked`, not `error`: the
machinery is fine, the job simply has no evidence it may merge on.

This does not fix why the floor could not run - a worktree is unprovisioned and
provisioning it is charged to the job, which is an architecture question filed
separately. It stops that condition from being reported as a pass.

Selftest: 529/529 (was 526). Three new rows; mutation-checked - disabling the rule
fails "a CONTRACTED floor that ran nothing is blocked, not success".
… no command

Reworked after review. The previous commit's justification was an anecdote that
turns out to be FALSE, and its predicate was the wrong field. Both corrected.

THE ARGUMENT, from the invariants rather than a story:

`run_test_floor` initialises every result {"passed": False, "merge_blocked": True} -
refusal is the floor's default state - and ADR 0003 relies on it: "already
merge-blocking and already fail-closed on an empty command", Consequences: "A job
reporting no test command at all is a FAIL, not a pass." spec-reviewer.md enforces
the same in prose (ISSUE: NO_TEST_EVIDENCE, "Silence is not success").

The fast path's own consumer honours it: build_review_spec gate 1 refuses on
`not passed or merge_blocked`. This consumer did not. It reads the floor only
through `_tests_block_from_floor`, which returns None when no check carries a
`checker`, so `merge_blocked: True` was never consulted and the job kept the scope
verdict's `success`. One floor document, two consumers, opposite conclusions.
Dogfood 14 moved the RED half of that rule from prose into this function; this is
the empty half, same layer, same reason.

CORRECTIONS TO THE PREVIOUS COMMIT:

- The predicate is now the MANIFEST's declaration, not the resolved slice.
  `contract` is read from jobs/<id>.test-contract.json, which is absent exactly
  when resolution FAILED - the reachable case. Keying on it left the hole open.
- `tests is None` (no floor document at all: fastpath-run.py missing or dead
  before writing JSON) is `error`, not `blocked`. The machinery IS broken there.
- The refusal now carries the floor's own reasons into `summary`; it previously
  fell back to the job title, refusing for an unstated reason.
- The sibling-jobs anecdote is removed. Checking the run record, `task-4-wire` was
  status `blocked` with tests.exit_code 1 - its floor DID run and went red, and the
  dogfood-14 rule caught it. The `tests_block is None` path was never exercised
  there. This is a latent fail-open found by reading, not a reproduced defect, and
  the comment now says so by not claiming otherwise.

Unchanged: an uncontracted manifest still records `success` with no `tests` object
("no floor at all is not a failure").

Selftest: 532/532 (was 526). Six rows; mutation-checked - disabling the rule fails
four of them.
@khymerao

Copy link
Copy Markdown
Contributor Author

Folded into #13, which now carries both halves of the same story: what a test_contract command means to its executor, and what the floor document means to its consumer. They key on the same document and should be reviewed and bisected together.

Nothing is dropped - the empty-floor rule, its six selftest rows and the corrections from review (predicate on the manifest's declaration rather than the resolved slice; error when no floor document exists; the floor's own reasons carried into summary) are all in #13 unchanged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant