fix(modelchecker): evaluate exists assertions without the retained graph - #376
Merged
Conversation
jp-fizzbee
force-pushed
the
user/jp/fix-exists-no-graph
branch
from
August 25, 2026 00:01
739078d to
d60d79e
Compare
Under --experimental_no_graph, `exists` assertions were silently never evaluated: the post-run CheckSimpleExistsWitness pass walks the retained node list, and the no_graph summary path returns before reaching it. A spec with an unsatisfiable exists reported PASSED — a silent false negative that flips real reachability conclusions. (Inconsistently, liveness in no_graph mode gets a loud startup refusal.) The fix exploits what exists actually needs: a SINGLE witness state, ever — not the graph. CheckInvariants already computes per-state Witness flags (the exact signal the full-graph CheckSimpleExistsWitness pass reads); the processor now ORs them into a run-level accumulator (accumulateExistsWitness, hooked at all four evaluation sites: init, yield, and both crash paths) as states are discarded. UnsatisfiedExists() then reports the exists positions that never found a witness. Consumers, with mode-appropriate semantics: - no_graph model checking (exhaustive): a missing witness is PROOF of unreachability -> same "FAILED: Expected states never reached" verdict and format as the full-graph path. Skipped under guided traces, matching the full-graph path's rationale. - simulation (non-exhaustive): a missing witness across all runs is a coverage observation, not proof -> prints a WARNING naming the never-satisfied assertions after "Stopped after N runs", with a pointer to model-check for a definitive answer. Witnesses are intersected across runs (each run uses a fresh Processor), so an exists satisfied in ANY run does not warn. Previously exists was silently ignored in simulation entirely. - Default full-graph mode is untouched (still CheckSimpleExistsWitness). Regression tests: test_references.py now supports per-example flags — an optional flags.txt sidecar next to the spec, read only by the test harness (no CLI/binary/user-facing change; examples without flags.txt run exactly as before). This enables reference baselines for non-default modes. Three new examples pin the verdicts: - 09-10: unsatisfiable exists + --experimental_no_graph -> FAILED. Verified to fail against pre-fix code (reports PASSED there). - 09-11: satisfiable exists + --experimental_no_graph -> PASSED (the accumulator detects witnesses, not just failures). - 09-12: unsatisfiable exists + simulation (fixed seed for a deterministic single run) -> named WARNING, no failure. Reference suite: 101/101. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
jp-fizzbee
force-pushed
the
user/jp/fix-exists-no-graph
branch
from
August 25, 2026 23:50
d60d79e to
625f07b
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes the silent false-negative for
existsassertions under--experimental_no_graph: they were never evaluated (the post-runCheckSimpleExistsWitnesspass needs the retained node list, and the no_graph path returns before it), so an unsatisfiable exists reported PASSED.Repro (from the bug report):
fizz --experimental_no_graph spec.fizzfizz spec.fizzApproach
existsneeds a single witness state, ever — not the graph. The processor now OR-accumulates the per-stateWitnessflags into a run-level matrix as states are discarded, hooked at all four evaluation sites (init, yield, both crash paths). The accumulated signal is the exact same flag the full-graphCheckSimpleExistsWitnesspass reads —CheckInvariantssetsWitness[i][j]for exists assertions; if this signal were wrong, default mode would have been broken all along.Mode-appropriate semantics:
Regression tests (added per review)
test_references.pynow supports an optionalflags.txtsidecar per example — read only by the test harness (no CLI or user-facing change; examples without it run exactly as before). This closes a standing harness gap: flagged modes previously could not have automated baselines.Three new reference examples pin the verdicts:
--experimental_no_graph→ FAILED. Verified to fail against pre-fix code (reports PASSED there — the bug).--experimental_no_graph→ PASSED (accumulator detects witnesses, not just failures).Reference suite: 101/101.
🤖 Generated with Claude Code