Working log of methodology decisions made during P1 build. For Rizwan's review. Items marked REVIEW need a call before publication.
-
Distractors in seeded drafts. Terms of art retained from base filing text ("comprehensive income", "reasonable assurance") are recorded in the manifest as distractors, never counted in recall. Flags overlapping them feed the FP analysis. This gives term-of-art collision data inside the seeded set, not just the human corpus.
-
Plant origin field. Some pattern instances occur naturally in real filing prose (J&J's actual 10-K says "Leveraging the extensive resources" and "empowered and inspired"). Retained instances are marked
origin: retainedand count in recall the same as inserted ones; the split is reportable if it becomes interesting. -
lex-t3 density plants are sub-threshold by design for some engines. Drafts plant density words at 3 occurrences per ~350 words (~0.9%). That crosses the-antislop's claimed gate (3+ per section) but sits far below avoid-ai-writing's engine gate (3% of word count, min 3, which is 10+ occurrences in a 350-word draft). Result: avoid-ai-writing detector scores 0% on lex-3 while its SKILL.md prose reads as if ~3 repeats matter. This threshold divergence is a finding, not a bug. Documented here so nobody "fixes" the drafts.
-
Prose filter on human corpus. Flattened HTML tables in MD&A sections would dilute flags/1k, so scoring uses
clean/files (lines that are mostly numeric or heading fragments dropped, raw/ kept as provenance). Filter: scripts/clean_corpus.py. -
unslop invocation. intensity=full (maximum deterministic rule coverage), structural=off, soul=off (those are rewrites, not detections). Voice profiles never touch the deterministic layer, so runs are voice-neutral without further steps.
-
Recall matching is span-based, not attribution-based. A plant counts as recalled if any rule fires on its span, even if the tool names it differently than the crosswalk expects. Crosswalk attribution is used for analysis, not for recall credit. Rationale: tools disagree on taxonomy; punishing naming differences would measure vocabulary, not detection.
-
Whitelist addition. "comprehensive income" added as its own term (ASC 220) after observing Tier 1 "comprehensive" firing on real KO and DE audit opinions. Whitelist status remains pending_review.
- Whitelist final review before publication (brief, open item). Note "materiality" is listed but bare "material"/"materially" are excluded from the starter set; they occur constantly in ordinary filing prose and would swamp the ToA metric. Confirmed 2026-07-24: leave them excluded.
- Corpus gaps. KO and DE MD&A extraction failed (unusual heading structure) and WMT/PG 10-Ks fell outside the submissions-API pagination window. Corpus is 18 docs / 111k clean words, comfortably above the 10-doc / 20k-word floor, so not blocking. Could be fixed for completeness.
- PCAOB inspection reports (PDFs) are in the brief's corpus spec but not yet fetched. The EDGAR material alone exceeds targets. Add later if wanted.
All 192 detect-only passes done: 4 prompt tools x 18 human-corpus files (72), plus 4 prompt tools x 10 seeded drafts x 3 independent runs (120). Ran via a Workflow script spawning one Claude Code subagent per (tool, file) or (tool, draft, run), model pinned to sonnet (resolves to claude-sonnet-5), schema-forced structured output. Results ingested by scripts/ingest_prompt_full_run.py into the same raw_*.json schema the code-track runners produce, then scored by scripts/score.py.
Fidelity caveat stands regardless of completeness: this is the subagent path, not the raw-API path. Claude Code injects its own system prompt, so each tool's ruleset rode as agent task instructions rather than a literal system prompt. runners/prompt_track/run_prompt_tool.py remains the protocol-exact API harness for anyone who wants a fully canonical rerun (needs REGISTERBENCH_API_KEY in the environment or the gitignored .env, --approved flag, roughly under $30 for the finance slice at claude-sonnet-5 pricing; until 2026-10-11 it read ANTHROPIC_API_KEY, see below).
Why this took four resume attempts: the fan-out is expensive by volume, not by model tier; 192 full-document subagent passes, several against the largest corpus file (~20k words), repeatedly hit the Claude Max session usage cap before finishing. Each Workflow(..., resumeFromRunId: "wf_9628a91e-89d") call replayed already-completed jobs from cache for free and only retried what had failed, so nothing was wasted across attempts, but it took spanning a full overnight reset cycle (roughly 8:10pm -> 1:50am -> 9:20am -> 9:40pm Toronto resets) to clear the last ~65 jobs. Lesson for any future large fan-out like this: pass effort: 'low' on mechanical detect-and-quote agent calls; none of these calls needed deep reasoning, and the default inherited effort tier likely multiplied token cost substantially, especially on the large documents.
Headline finding (numbers corrected 2026-07-26 after the scoring-defect fixes below; the figures first published on 2026-07-25 were inflated): judgment-based prompt detection beats regex on this taxonomy. avoid-ai-writing's own SKILL.md read by an LLM scores 72.7% recall vs 46.7% for its own regex detector engine on the identical rule catalog, a 26.0-point gap concentrated in structural patterns regex cannot reach. stop-slop is the most aggressive tool on real filing text (6.9 flags/1k). SUPERSEDED 2026-10-11: its ToA rate of 0.243/1k was a containment-rule artifact; under the flag-anchored rule it is 0.072/1k and ranks third. See v0.1.1 free fixes below.
Recall variance across the real N=3 runs was tight (widest spread: the-antislop at 43.4%-46.7%; stop-slop returned an identical 73/122 all three times), suggesting the prompt track is reasonably stable run-to-run on this task at this model tier.
N asymmetry, disclosed: the 192 passes give the seeded drafts N=3 but the human corpus N=1 per tool (72 passes = 4 tools x 18 files, single pass each). So prompt-track recall carries a spread and prompt-track flags/1k and ToA/1k do not. Nothing published is false, since spread is only shown for recall, but the human-corpus columns for the four prompt tools are single-run point estimates. Fix is either a disclosure line (done in README Limitations) or 144 more passes to bring the human corpus to N=3.
Two scoring defects found and fixed 2026-07-26, both surfaced by an adversarial multi-agent review of this repo (60 raw findings, 37 refuted, 23 survived). (1) Recall credited one finding to unlimited plants, and overlap was measured against the shorter span, so a paragraph-long quote scored a perfect match on every plant inside it; a synthetic tool quoting whole drafts scored 122/122. Now maximum bipartite matching, one plant per quoted finding, document-level rules exempt. Moved recall 0.8 to 8.5 points by tool and reordered two rows. (2) Term-of-art matching used plain substring, so "qualified opinion" matched inside "unqualified opinion"; now word-bounded, and index-less flags capped at min(flags, colliding occurrences). Lesson: I validated the scorer by checking whether its output looked plausible, never by attacking the matching mechanism. The whole-draft-quote test takes ten minutes to write and belonged in the repo before any number was published.
Four items deferred from the completeness review, all resolvable without new detection runs.
Crosswalk v0.2.0, blind double-coded. Two coders per tool face (pattern-first and rule-first), reading only that tool's source, never the existing crosswalk, under one written claim standard. 172/174 cells agreed (98.9%), 2 adjudicated. All five errors the critic found were independently confirmed. A pre-commit verification then flagged three cells as inconsistently applied; each went to a fresh 3-adjudicator vote. stop-slop phr-novelty-inflation upheld as unclaimed (1/3), no-ai-slop phr-transition upheld as unclaimed (0/3), the-antislop phr-novelty-inflation overturned from claimed to unclaimed (0/3) so the hidden-truth-header reading matches stop-slop's. Net: 15 claim decisions differ from the hand-authored v0.1.0, in both directions across five tools. No code-track identifier change touched DOC_LEVEL_RULES, so engine recall is unchanged. Evidence per cell: crosswalk/coding_record.json.
Claimed-only recall shipped. Skill 73.1, no-ai-slop 69.3, stop-slop 63.4, detector 56.6, the-antislop 54.3, unslop 37.0. Rows 2 and 3 swap relative to full-taxonomy recall. Scorer reproduced the critic's independent claimed-only numbers exactly for four tools under the old crosswalk before the new one was applied.
Weighting sensitivity shipped. Skill minus detector: 26.0 plant-weighted, 23.1 equal-tier, 10.6 lexical only, 36.2 phrasal+structural, 16.5 claimed-only. Invariant across all five cuts: skill ranks first, and skill beats its own engine. Rows 2-6 reorder in every alternative cut. README now says so explicitly instead of presenting 26.0 as a precise property.
Flag-anchored term-of-art rule. Protected-term text must cover at least half the flagged span (union across terms). Containment counts are still published. stop-slop 0.243 -> 0.072, third. Its adverb ban hits "present fairly, in all material respects" (AS 3101.08) in all 10 opinions; 8 count, 2 are quoted too wide (103, 117 chars) and drop out, a residual width sensitivity disclosed in README Limitations. Of the 19 flags the stricter rule removes, 17 were flagged for something else. Rank history: worst under containment at v0.1.0, second under containment after leverage joined the whitelist, third under the anchored rule.
Pre-commit verification fixes (2026-10-11). A 27-agent adversarial pass confirmed 24 findings, 0 refuted, about 15 distinct. Changes: (1) Per-term ToA counts were read from a 25-item example cap; score.py now emits uncapped toa_by_term. Real unslop split is comprehensive income 19, leverage 17 (21 and 17 after the loss-caption fix below). (2) unslop's higher per-flag rate is reporting granularity: it reports every occurrence, the detector once per document. New granularity-neutral count of distinct (document, term) pairs: detector 13, unslop 13, identical; stop-slop 8; skill 1 (14 and 14 after the loss-caption fix below). 'Heaviest offender' language removed. (3) Indexed flags now used only when the flagged text is actually at the index (some detector offsets were misaligned); index-less flags located case-sensitively first. (4) Containment made a strict superset of anchored. (5) Whitelist occurrence stats now merge nested variants: 168 total, 147 in opinions, comprehensive income 19 (previously 176/150/25; 170/149/21 after the loss-caption fix below). (6) Gap decomposition restated: 11.7 points from 8 patterns with no engine rule, 7.1 from 4 where an engine rule exists but did not fire on the drafts' wording, 2.5 from cluster gating, 0.3 from density. (7) About ten prose overstatements corrected, including stop-slop's vocabulary overlap (one term, not two) and its flag-rate lead (more than 3x, not 5x).
Second adversarial pass (2026-10-11). 14 agents on the delta only: 10 confirmed, all low severity, 2 refuted (first pass: 24 confirmed, several medium), so treated as converged. Fixed: (1) ASC 220 loss-year captions ('Comprehensive (Loss)/Income', 'comprehensive loss') added as variants; Delta's caption had been flagged by both engines but not counted. Detector 0.135 -> 0.144 and 14 pairs, unslop 0.323 -> 0.341 and 14 pairs, comprehensive income 21 occurrences in 12 documents. (2) Per-term attribution now goes to the shortest fully enclosed term, so stop-slop's adverb flags count under 'present fairly' rather than 'in all material respects'. (3) Detector reports once per analyzed chunk, not strictly once per document. (4) Prose corrections: 'only ones that fire on opinion boilerplate' narrowed to 'only rules whose quotes enclose protected wording without targeting it'; wide-quote coverage stated as 37% and 32%; stop-slop's lex-1 figure attributed to 5/5 hollow intensifiers (only 4/18 vocabulary plants); detection dates start 2026-07-24; pervasive's ISA citation acknowledged; em-dash 'only the detector fires' narrowed to threshold-based rules.
Harness prep for the paid rerun (2026-10-11). No calls made. (1) Key now read from REGISTERBENCH_API_KEY (environment, else repo .env, which tolerates the UTF-16 that PowerShell 5.1 redirection writes); ANTHROPIC_API_KEY is deliberately never read, because Claude Code also reads that name and would bill an interactive session to the API instead of the subscription. (2) --smoke mode: one call, prints served model, stop reason and tokens, writes only the raw response. (3) Bug fixed before it cost money: directory inputs did not skip README.md, so a seeded run would have sent the synthetic-draft disclaimer as an 11th draft, three times per tool, and scored it; the harness now uses the code-track skip set. (4) Result files record served_models. (5) Non-retryable HTTP errors exit with the API's error body instead of a traceback. 31 offline tests against a mocked endpoint pass.
All 336 requests through runners/prompt_track/run_prompt_batch.py: 4 prompt tools x (10 seeded drafts + 18 human filings) x 3 runs, Message Batches API, model claude-sonnet-5 (served model recorded in every result file), ruleset system prompt cached (1h), thinking and effort left at the model defaults (adaptive, effort high), max_tokens 64000. Billed $7.04 at batch prices, plus $1.26 at list price for one sync smoke call and the diagnostics below; all from the promotional API credit. One response (the-antislop, MMM_2021_opinion, run 2) ended after 6,930 thinking tokens (6,931 output tokens) with no text block; it was discarded and resent rather than recorded as zero findings. Code-track results are unchanged (no model calls).
Results. Recall: skill 74.9% (73.8-75.4), stop-slop 64.2%, no-ai-slop 62.6%, the-antislop 57.4%, detector 46.7%, unslop 28.7%. Every prompt tool rose against the subagent path (skill +2.2, stop-slop +4.4, no-ai-slop +7.1, the-antislop +12.0); the-antislop moves above the detector. Skill minus detector: 28.1 points published, 29.7 equal-tier, 12.8 lexical only, 37.1 phrasal + structural; the headline holds under every weighting. Claimed-only: no-ai-slop 76.4% now edges the skill's 75.3%.
The model declines to apply prompt rulesets to most genuine filings. Per run, the prompt tools return [] on 13 to 17 of the 18 filings, almost always with zero thinking tokens; the same tools engage fully with every seeded draft. Diagnostics on stop-slop (sync, list price) rule out length. The seeded drafts are adapted from the same kind of filing text, so what separates them is the planted patterns rather than the register; the model appears to triage authentic filing prose as clean before applying any rule, though not absolutely (stop-slop engages Delta's MD&A in every run). A 264-word excerpt of the Ford opinion gets [] with no thinking; six synthetic drafts concatenated to 2,194 words get 59 findings after 18,759 thinking tokens. Neither a uniform scope sentence ("apply each rule as written, whatever its source or genre") nor framing the user turn as "Text to audit:" changes it. effort: max thought for 63,999 tokens and hit max_tokens without answering. The protocol was left unchanged: this is what the pinned model does with these rulesets at its defaults. Consequence: prompt-track flags/1k on the human corpus fall to 0.10-0.39, and term-of-art violations are zero for every prompt tool in every run. The v0.1 finding that stop-slop's adverb ban flags "present fairly, in all material respects" does not reproduce on the API's human corpus, because the model never reads the real opinions; it does reproduce on the API inside seeded draft fin-03, where the same sentence is a retained distractor and the adverb ban flagged fairly in two of three runs. The only term-of-art violations in the canonical run come from the two code engines (detector 0.144/1k, unslop 0.341/1k, 14 document-term pairs each), which are deterministic.
README rewritten to the canonical numbers and tagged v0.1.1 (2026-10-12).
An external review of the public repository (no paid calls) raised three validation issues; all three were confirmed and fixed. (1) parse_findings dropped malformed elements, so [{}], [42] or ["bad response"] scored as an empty result; it now rejects the whole response. 0 of the 1,598 elements in the canonical run were malformed, so no published number depended on it. (2) 11 of 393 prompt-track human-corpus quotes, and 1 seeded quote (stop-slop run 1), occur nowhere in their documents. Provenance checked before labelling: corpus files unchanged since 2026-07-24 (f1ac23d), each quote verbatim in the raw API response, so these are non-verbatim model quotes; three are phrases absent from the document entirely (robust, It's worth noting, ideate, iterate, and deliver; all on DAL_2021_mdna), eight are reworded spans. A whitespace- and punctuation-tolerant search locates none of them. score.py now reports them per tool and excludes them from flags/1k: stop-slop 0.34 -> 0.31, no-ai-slop 0.10 -> 0.09, the-antislop 0.36 -> 0.35, skill unchanged; recall, claimed-only, weightings and term-of-art unchanged. The v0.1.0 subagent-path comparison figure for stop-slop is 6.86 under the same rule (6.90 as published). Raw evidence untouched. (3) No tests were tracked; the 67 offline checks used during v0.1.1 lived outside the repo. tests/ now holds 38 unittest cases (mocked API, stand-in ruleset repos, no key, no network), including reproduction of the committed results from committed raw findings, and GitHub Actions runs them on Python 3.10 and 3.12. Also found: the local working copy stores the corpus with CRLF (core.autocrlf) while the repository stores LF; Python reads both as LF, and the harness sends Python-read text, so no scored number depends on it.
README cut from 11,227 to about 5,000 words in the same template; the full build decisions, limitations, citations, reproduction detail and per-pattern analysis moved verbatim, then updated, into docs/METHODOLOGY.md and docs/FINDINGS.md. Wording changes from the review: empty lists are reported as measured results with the explanation left open (no 'declines' or 'triage' as fact); seeding and crosswalk coding are labelled recorded judgments, not deterministic; the universal tagline and novelty claims were narrowed to what this cohort supports.
Still outstanding: draft the upstream PR. Open design question for a later version: a term-of-art rule that tests the flagged token rather than the quote would remove the residual width sensitivity, but prompt-track tools return quotes, not the token they targeted.
- Uniform detect-only instruction appended to each tool's verbatim skill files; requests JSON
{rule, quote}pairs. the-antislop's own "audit only" mode and no-ai-slop's detect mode align with this; stop-slop has no native detect mode, so the instruction does more work there. - stop-slop needs all four files (SKILL.md + 3 references); no-ai-slop needs SKILL.md + eval.md. Confirmed against each repo's own instructions.