Repository navigation
fix: comprehension-first guard + reuse rung (#245, #217) - #253
Merged
Merged
Conversation
#245 "Dangerously lazy": add an operational "fix the root cause, not the symptom" directive — grep every caller of the function you touch and fix the shared function once (the smaller diff). Validated on the agentic benchmark: on a shared-helper bug-fix trap, baseline fixes the root cause 1/6 while ponytail does 6/6 on both Sonnet 4.6 (the model the issue was filed on) and Opus 4.8, verified by reading the produced code. Plain prose ("trace the flow") did not move it; the actionable, lazy-framed directive did. #217 "Missing rung": add ladder rung 2 "Already in this codebase? Reuse it, don't re-write it." Propagated across SKILL.md, AGENTS.md, all agent mirror copies, the hook fallback, and both READMEs (check-rule-copies passes). Benchmark: 4 new deterministic quality-tier tasks (reuse-slug, reuse-money, trace-transfer, trace-amount) with selftest-proven good/bad refs; harness gains multi-file seed support in --selftest, distinctive-behaviour reuse detection, and counts in-file __main__/demo() self-checks as test LOC (not source bloat) for surgical tasks. Full writeup in benchmarks/results/2026-06-22-issue-245-217-comprehension.md. Also carries the in-progress todo-null benchmark task already present in the working tree. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This was referenced Jun 22, 2026
Closed
DietrichGebert
pushed a commit
that referenced
this pull request
Jul 9, 2026
The reuse rung added in #253 ("Already in this codebase? Reuse it, don't re-write it.") is load-bearing now, but it was not in the INVARIANTS list, so a future reword could silently drop it from SKILL.md or AGENTS.md without tripping the check. Pin it with the substring "in this codebase", present in both files' rung 2, the same way the safety carve-outs are pinned. Co-authored-by: Rafaelpta <15382305+Rafaelpta@users.noreply.github.com>
Afristrat
pushed a commit
to Afristrat/ponytail
that referenced
this pull request
Sep 29, 2026
…richGebert#217) (DietrichGebert#253) DietrichGebert#245 "Dangerously lazy": add an operational "fix the root cause, not the symptom" directive — grep every caller of the function you touch and fix the shared function once (the smaller diff). Validated on the agentic benchmark: on a shared-helper bug-fix trap, baseline fixes the root cause 1/6 while ponytail does 6/6 on both Sonnet 4.6 (the model the issue was filed on) and Opus 4.8, verified by reading the produced code. Plain prose ("trace the flow") did not move it; the actionable, lazy-framed directive did. DietrichGebert#217 "Missing rung": add ladder rung 2 "Already in this codebase? Reuse it, don't re-write it." Propagated across SKILL.md, AGENTS.md, all agent mirror copies, the hook fallback, and both READMEs (check-rule-copies passes). Benchmark: 4 new deterministic quality-tier tasks (reuse-slug, reuse-money, trace-transfer, trace-amount) with selftest-proven good/bad refs; harness gains multi-file seed support in --selftest, distinctive-behaviour reuse detection, and counts in-file __main__/demo() self-checks as test LOC (not source bloat) for surgical tasks. Full writeup in benchmarks/results/2026-06-22-issue-245-217-comprehension.md. Also carries the in-progress todo-null benchmark task already present in the working tree. Co-authored-by: Dietrich Gebert <dgebert@Dietrichs-MacBook-Pro.local>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes two behavioural issues with the ruleset.
#245 "Dangerously lazy"
The "shortest diff wins" reflex made the agent patch the nearest symptom instead of tracing the problem and shipping a confident wrong fix. The fix adds a comprehension-first guard plus the part that actually changed behaviour — an operational directive:
Framing the root-cause fix as the lazier (smaller) diff makes ponytail's own instinct pull toward it.
Validated on the agentic benchmark. New
trace-transfertask:transfer()andwithdraw()both debit through a shared_debit(); the report names transfers, the scorer exercises an un-named overdrawing withdraw, so only a fix that traces the flow and repairs_debit()passes.All passing cells repair the shared
_debit()(verified by reading the code). A control confirms it's the operational wording: pre-fix ponytail and a plain-prose version both scored 0/3 on Opus; only the grep-the-callers directive reached 6/6.#217 "Missing rung"
Adds ladder rung 2 — "Already in this codebase? Reuse it, don't re-write it." Propagated across
SKILL.md,AGENTS.md, all agent mirror copies, the hook fallback, and both READMEs (check-rule-copiespasses). Note: in benchmarking, both baseline and ponytail already reuse existing helpers, so the rung is correct guidance that regresses nothing, but its behavioural benefit didn't reproduce to measure (see writeup).Benchmark additions
reuse-slug,reuse-money,trace-transfer,trace-amount), each with--selftest-proven good/bad refs.--selftest, distinctive-behaviour reuse detection, and counting in-file__main__/demo()self-checks as test LOC (not source bloat) on surgical tasks.benchmarks/results/2026-06-22-issue-245-217-comprehension.md.Regression check
Pre-fix vs post-fix across the 27-task runnable suite (Haiku, n=3): safety identical (all 7 deterministic safety tasks 1.0 before and after — no guard dropped), LOC cuts preserved, correctness unchanged within noise.
Closes #245
Closes #217