Skip to content

fix(modelchecker): evaluate exists assertions without the retained graph - #376

Merged
jp-fizzbee merged 1 commit into
mainfrom
user/jp/fix-exists-no-graph
Aug 25, 2026
Merged

fix(modelchecker): evaluate exists assertions without the retained graph#376
jp-fizzbee merged 1 commit into
mainfrom
user/jp/fix-exists-no-graph

Conversation

@jp-fizzbee

@jp-fizzbee jp-fizzbee commented Aug 24, 2026

Copy link
Copy Markdown
Collaborator

Summary

Fixes the silent false-negative for exists assertions under --experimental_no_graph: they were never evaluated (the post-run CheckSimpleExistsWitness pass needs the retained node list, and the no_graph path returns before it), so an unsatisfiable exists reported PASSED.

Repro (from the bug report):

exists assertion ImpossibleProbe:
    return len(db.appointments) > 999
Command Before After
fizz --experimental_no_graph spec.fizz PASSED ❌ FAILED: Expected states never reached ✅
fizz spec.fizz FAILED ✅ unchanged

Approach

exists needs a single witness state, ever — not the graph. The processor now OR-accumulates the per-state Witness flags into a run-level matrix as states are discarded, hooked at all four evaluation sites (init, yield, both crash paths). The accumulated signal is the exact same flag the full-graph CheckSimpleExistsWitness pass readsCheckInvariants sets Witness[i][j] for exists assertions; if this signal were wrong, default mode would have been broken all along.

Mode-appropriate semantics:

  • no_graph (exhaustive): missing witness = proof of unreachability → same FAILED verdict + format as full-graph. Skipped for guided traces (same rationale as full-graph path).
  • simulation (non-exhaustive): missing witness across all runs = coverage observation → WARNING naming the assertions, with a pointer to model-check for certainty. Intersected across runs. (Explicitly requested behavior; previously exists was silently ignored in simulation.)
  • Default full-graph mode untouched.

Regression tests (added per review)

test_references.py now supports an optional flags.txt sidecar per example — read only by the test harness (no CLI or user-facing change; examples without it run exactly as before). This closes a standing harness gap: flagged modes previously could not have automated baselines.

Three new reference examples pin the verdicts:

  • 09-10: unsatisfiable exists + --experimental_no_graph → FAILED. Verified to fail against pre-fix code (reports PASSED there — the bug).
  • 09-11: satisfiable exists + --experimental_no_graph → PASSED (accumulator detects witnesses, not just failures).
  • 09-12: unsatisfiable exists + simulation with fixed seed (deterministic single run) → named WARNING, no failure.

Reference suite: 101/101.

🤖 Generated with Claude Code

@jp-fizzbee
jp-fizzbee force-pushed the user/jp/fix-exists-no-graph branch from 739078d to d60d79e Compare August 25, 2026 00:01
Under --experimental_no_graph, `exists` assertions were silently never
evaluated: the post-run CheckSimpleExistsWitness pass walks the retained
node list, and the no_graph summary path returns before reaching it. A
spec with an unsatisfiable exists reported PASSED — a silent false
negative that flips real reachability conclusions. (Inconsistently,
liveness in no_graph mode gets a loud startup refusal.)

The fix exploits what exists actually needs: a SINGLE witness state,
ever — not the graph. CheckInvariants already computes per-state Witness
flags (the exact signal the full-graph CheckSimpleExistsWitness pass
reads); the processor now ORs them into a run-level accumulator
(accumulateExistsWitness, hooked at all four evaluation sites: init,
yield, and both crash paths) as states are discarded.
UnsatisfiedExists() then reports the exists positions that never found
a witness.

Consumers, with mode-appropriate semantics:

- no_graph model checking (exhaustive): a missing witness is PROOF of
  unreachability -> same "FAILED: Expected states never reached"
  verdict and format as the full-graph path. Skipped under guided
  traces, matching the full-graph path's rationale.
- simulation (non-exhaustive): a missing witness across all runs is a
  coverage observation, not proof -> prints a WARNING naming the
  never-satisfied assertions after "Stopped after N runs", with a
  pointer to model-check for a definitive answer. Witnesses are
  intersected across runs (each run uses a fresh Processor), so an
  exists satisfied in ANY run does not warn. Previously exists was
  silently ignored in simulation entirely.
- Default full-graph mode is untouched (still CheckSimpleExistsWitness).

Regression tests: test_references.py now supports per-example flags —
an optional flags.txt sidecar next to the spec, read only by the test
harness (no CLI/binary/user-facing change; examples without flags.txt
run exactly as before). This enables reference baselines for
non-default modes. Three new examples pin the verdicts:

- 09-10: unsatisfiable exists + --experimental_no_graph -> FAILED.
  Verified to fail against pre-fix code (reports PASSED there).
- 09-11: satisfiable exists + --experimental_no_graph -> PASSED (the
  accumulator detects witnesses, not just failures).
- 09-12: unsatisfiable exists + simulation (fixed seed for a
  deterministic single run) -> named WARNING, no failure.

Reference suite: 101/101.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@jp-fizzbee
jp-fizzbee force-pushed the user/jp/fix-exists-no-graph branch from d60d79e to 625f07b Compare August 25, 2026 23:50
@jp-fizzbee
jp-fizzbee merged commit 0b6ad9d into main Aug 25, 2026
1 check passed
@jp-fizzbee
jp-fizzbee deleted the user/jp/fix-exists-no-graph branch August 25, 2026 23:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants