story-131: A filed query answers for a scope of any size an inspection hands it

Implemented by the l5 harness story workflow.

Author: jerodw

.harness/docs/ARCHITECTURE.md

@@ -2114,6 +2114,32 @@ story-130 merges the three tracker commands into `templates/scripts/github.sh`,
 
 **Three questions were deliberately left open.** Whether these values belong in `config.yaml` rather than at the top of the file is a separate decision and was not made here. Whether an installed copy should be held byte-identical to its template is filed on its own brief; this story converted the item assertion only because there is no longer a separate item script. And `.harness/census.py` stays where it is — moving it into `.harness/scripts/` is its own brief.
 
+## A filed query answers for a scope of any size
+
+The tracker tier of the inspector's dedupe had never run. story-131's brief records that every post-story inspection from story-101 onward reported "dedupe did not run" in its run's `events.log`, and the cause was in the script rather than in the harness: the query branch searched once per path in a sequential loop, `.harness/stories/story-131.yaml` measures one search against this repository's tracker at roughly 0.85 seconds, and the query is held to the 30-second default in `orchestration/filed_query.py`. A 60-file scope therefore spent close to a minute against a bound of thirty, was killed partway through, and never answered — which the harness correctly read as dedupe not having run. **The second tier still worked**, because the inspection unions the tracker's answer with a local index of what this machine filed, so what had never functioned was precisely the tier the per-path marker scheme exists to serve: the one that can see a brief filed from another checkout, another developer or the tracker's web UI.
+
+**The repair is in the answering command, and the bound and the scope are deliberately untouched.** `orchestration/filed_query.py` gained one docstring paragraph and no behaviour: the question stays one question, the timeout stays where it is, and no partial or per-path answer is invented. The reason is what an answer means there — `Answer.answered` is a statement about the *whole* question, so a scope trimmed to fit a bound would be a partial answer reported as a complete one, which is the false agreement that flag exists against. Raising the bound only moves the size at which the same failure returns. How a command's work scales with the scope is that command's business, so the fix belongs where the searching is.
+
+**The query branch batches rather than looping.** `BATCH` is declared beside `LIMIT` in the query job's own constants block of `templates/scripts/github.sh`, read from `L5_QUERY_BATCH` the way `LIMIT` is read from its own name, and defaults to 20. `do_query` accumulates path markers — each quoted, joined with ` OR ` — and calls `query_batch` whenever the batch is full and once more for the remainder. The `jq` composition below it is unchanged, because it already unioned pages, deduplicated by URL and attributed per-item paths from the asked list, so the items the batched query answers with are the items the per-path loop answered with. The story artifact records the measurement the constant rests on: a search carrying 5 markers cost 0.83 seconds, 10 cost 0.90 and 20 about 1.0, so the cost is nearly flat in batch size and a 60-file scope becomes three searches rather than sixty.
+
+**Batching is only safe with a fallback, and the fallback is the load-bearing half.** A search is capped at `LIMIT` results. With one path per search that cap is per path; with many paths in one search a filled page could be several paths' worth of issues truncated, and a truncated page read as complete is a duplicate filed. So `query_batch` re-asks a batch's paths one at a time in three cases — the search exited non-zero, its page held `LIMIT` items, or its page could not be counted — and only a page it can trust goes into `found`. That fallback is also what keeps the change safe on a tracker whose limit on query length is tighter than GitHub's: a batch too long to search fails, and failing is what re-asks its paths singly. A per-path search that fails after the fallback still fails the whole answer with the message it always carried, naming the path, on the rule the branch already stated: reporting the paths that did answer would say that nothing is filed against the ones that did not.
+
+**The batch state is file-level rather than `do_query`'s locals** — `found`, `batch_search`, `batch_paths` and `batch_count` — because `query_batch` reads and appends to all four. `fail` inside it still exits the script, since a `while` loop with a heredoc redirection is not a subshell.
+
+**The contract comment at the head of the file is where a target writing its own command meets this**, in the shape "Every contract the three headers stated is stated in the one header" above established. Two paragraphs were added: `ONE SEARCH PER PATH DOES NOT SCALE`, which says that a command whose searches are proportional to the scope is killed and never answers with nothing loud about the failure, and `BATCHING IS ONLY SAFE WITH A FALLBACK`, which states the two re-ask conditions. The harness enforces neither, as it enforces none of the others. `.harness/scripts/github.sh` carries the identical query branch with its configured board constants untouched, so the standing "differs only in constant-assignment lines" comparison still holds — **an edit to a shipped script is still an edit to two files**.
+
+**A failed dedupe is now a failure rather than a clause.** It had been a trailing `; dedupe did not run` on the inspection's summary line and nothing else, which is how it stayed true for dozens of stories without being read as the standing failure it was: at the end of a long line it is indistinguishable from an inspection that found nothing worth filing. A scope whose filed query could not answer now appends **its own line** to the run's `events.log`, naming the scope, the reason and that what was filed may already be filed, said *before* the summary because the summary reports a filed count and this is what a reader needs to know what that count is worth. The summary line is unchanged and still carries its clause. `scripts/l5-inspect` reports the same thing as a block outside the per-scope lines and still exits zero, because a failed query costs dedupe and costs nothing else.
+
+**The new line goes through `_note`, which carries no kind of its own, and that is deliberate.** It reaches `events.log` and the run's structured history and reaches the cross-run inspection log not at all: that log holds one record per inspection, and a second entry carrying `INSPECTION_EVENT` would be read as a second inspection by anything querying it. What the durable record says about dedupe is the field below, not a second line.
+
+**`dedupe_ran` makes it a question one read of a tracked file answers.** `schemas/cross-run-history.schema.json` declares it on `inspection-log.jsonl` with what it means and why an absent tracker answer is not an empty one, and both modes write it on every line — broad mode from `report.dedupe_ran` in `inspection.record`, narrow mode through `_say`. It is written even where dedupe did run, because a boolean has a false to say and absence would otherwise mean either that or a line written before the field existed. An inspection that made no invocation at all — an unusable file cap, nothing the story changed in an inspected scope, an inspection that could not run — records `false`, because no query answered for it, and the vacuous `true` an empty scope list would yield would claim dedupe ran when nothing asked. It is a statement about the filed query alone: the local index tier is not what failed, was not touched, and reading it successfully does not make dedupe complete.
+
+**The bill for a new field on an event came due again, for the third named file and the same cause.** `orchestration/story_coordinator.py`'s `append_event` gained a `dedupe_ran` keyword — the narrow mode's record goes through that one write path, and a second write path for the cross-run log is the drift the single-`append_event` rule exists against — and `schemas/execution-history.schema.json` had to declare it, because `tests/test_execution_history.py` holds the schema's optional properties and `append_event`'s keywords equal in both directions. Neither file was in the story's declared scope; both were edited and reported as deviations rather than worked around, with no `required` list, existing property or enum changed. That is now stories 055, 100, 101, 106 and 131, and the section "An inspection records what it cost" above already says a story adding fields to an event should expect this bill rather than discover it.
+
+**The stub `gh` in `tests/test_filed_query.py` was a forced adaptation and the story declared it as one.** It matched a search by plain substring against an issue body, so it could not answer the OR'd form and every existing query-branch test went red the moment the script batched. It now reads a search as quoted terms joined by `OR` and matches an issue whose body contains any of them, with a search carrying no quotes read as one term — which is what the fetch and the sync branch's own searches look like. Beside that it honours `--limit`, so a page filled to the limit is something it can produce, and records every search in its ledger, so how many searches a scope cost is read rather than inferred. `BATCHED_SEARCH_CALL` joins the failure-injection names and fails only a search carrying more than one term: a tracker that answers a per-path search and refuses a batched one, which is exactly what the fallback exists for.
+
+**What did not change.** The local index tier, how the union of the two sources decides a drop and which source a drop names; the timeout, the scope handed to the query, `inspect_after_story_max_files`, the expansion and `cap_paths`; the sync and item branches of both copies of the script; the single declaration of the marker the sync branch writes and the query branch searches for. A failed dedupe still blocks, delays, refuses and fails nothing, and the post-story inspection stays total.
+
 ## Run directory anatomy
 
     .harness/runs/story-001/

.harness/history/execution-log.jsonl

@@ -60,3 +60,4 @@
 {"story_id": "story-130", "status": "escalated", "retry_count": 0, "timestamp": "2026-09-09 17:59:56"}
 {"story_id": "story-130", "status": "resumed", "retry_count": 0, "timestamp": "2026-09-09 19:40:02"}
 {"story_id": "story-130", "status": "completed", "retry_count": 0, "timestamp": "2026-09-09 20:21:17"}
+{"story_id": "story-131", "status": "completed", "retry_count": 0, "timestamp": "2026-09-09 22:29:26"}

.harness/history/retry-log.jsonl

@@ -49,3 +49,4 @@
 {"story_id": "story-129", "timestamp": "2026-09-09 09:28:14", "stage": "tester", "retry_stage": "tester"}
 {"story_id": "story-127", "timestamp": "2026-09-09 13:34:31", "stage": "tester", "retry_stage": "tester"}
 {"story_id": "story-130", "timestamp": "2026-09-09 17:46:36", "stage": "implementer", "retry_stage": "implementer"}
+{"story_id": "story-131", "timestamp": "2026-09-09 22:12:23", "stage": "tester", "retry_stage": "tester"}

.harness/scripts/github.sh

@@ -155,6 +155,34 @@
 # "suppress nothing closed" errs toward hearing a finding twice, which is the
 # cheaper mistake.
 #
+# ONE SEARCH PER PATH DOES NOT SCALE, and a target writing its own command has
+# to be told so, because nothing about the failure is loud. The harness asks one
+# question carrying the whole scope and holds the answer to one bound, so a
+# command whose searches are proportional to the scope is killed partway through
+# and never answers — which the harness reads as dedupe not having run, on every
+# inspection, for as long as nobody looks. Measured against this tracker one
+# search costs roughly 0.85 seconds, so a 60-file scope spent close to a minute
+# against a 30-second bound. This branch therefore batches: BATCH path markers
+# are quoted and OR'd into one search, and the pages are unioned through the
+# composition below, which already deduplicates by URL. The cost of a search is
+# nearly flat in batch size — 5 markers measured 0.83 seconds, 10 measured 0.90
+# and 20 measured about 1.0 — so a 60-file scope becomes three searches rather
+# than sixty. Neither the harness's bound nor the scope it hands over is what
+# changed: the scope is not capped, because a capped scope means inventing a
+# partial answer and an answer here means the whole question was answered.
+#
+# BATCHING IS ONLY SAFE WITH A FALLBACK, and the fallback is what a target
+# writing its own command must carry too. A search is capped at LIMIT results.
+# With one path per search that cap is per path; with many paths in one search a
+# filled page could be several paths' worth of issues truncated, and a truncated
+# page read as complete is a duplicate filed. So a batch whose page fills to the
+# limit, and a batch whose search fails, are both re-asked one path at a time —
+# which is also what keeps this safe on a tracker whose limit on query length is
+# tighter than this one's, since a batch that is too long to search falls back
+# rather than failing. A per-path search that fails after that still fails the
+# whole answer, on the rule above: reporting the paths that did answer would say
+# that nothing is filed against the ones that did not.
+#
 # THE QUERY BRANCH IS THE SYNC BRANCH'S PAIR, and they are now the same file,
 # which is what removes the way they used to be able to drift: the sync branch
 # writes one searchable marker per path and records the whole payload under a
@@ -325,10 +353,20 @@ IN_PROGRESS_OPTION="${L5_ITEM_IN_PROGRESS_OPTION:-In Progress}"
 READY_TO_MERGE_OPTION="${L5_ITEM_READY_TO_MERGE_OPTION:-Ready to Merge}"
 
 # --- what only the query job uses. Edit these. ---------------------------
-# How many items one path's search may return. The harness bounds what it will
+# How many items one search may return. The harness bounds what it will
 # read as well; this bound is about what the tracker is asked for.
 LIMIT="${L5_QUERY_LIMIT:-50}"
 
+# How many path markers one search carries. It exists because the number of
+# searches, not the cost of one, is what stopped this branch answering a scope
+# of any size: see ONE SEARCH PER PATH DOES NOT SCALE above for the measurement
+# and for what the fallback below guarantees. It bounds two things at once —
+# how long one search's text is, for a tracker whose limit on query length is
+# tighter than this one's, and how many paths one filled page can hide, since a
+# batch whose page fills to LIMIT is re-asked path by path and a smaller batch
+# makes that fallback rarer.
+BATCH="${L5_QUERY_BATCH:-20}"
+
 # --- the failure vocabularies -------------------------------------------
 # fail_transient is the sync branch's alone. Exit 75 means "the entry stays
 # pending and a later sweep tries again", and nothing retries behind the query
@@ -659,10 +697,74 @@ PATHS
 # The query job
 # ==========================================================================
 
+# The batched search's own state, held here rather than in do_query's locals
+# because query_batch reads and appends to all four. `found` accumulates the
+# pages every search returned, in the order they were made; the other three are
+# the batch being assembled.
+found=""
+batch_search=""
+batch_paths=""
+batch_count=0
+
+# Search for one batch's markers at once, and fall back to one search per path
+# where the batch's answer cannot be trusted.
+#
+# Two answers cannot be trusted and both fall back rather than being read. A
+# search that exited non-zero says nothing about what is filed against any of
+# its paths. And a page holding LIMIT items is a page the tracker truncated:
+# with one path per search that cap is per path, but a batch's filled page could
+# be several paths' worth of issues cut off, and a truncated page read as
+# complete is a duplicate filed. The fallback is also what keeps this safe on a
+# tracker whose limit on query length is tighter than this one's — a batch too
+# long to search fails, and failing is what re-asks its paths one at a time.
+#
+# A per-path search that fails after that fails the whole answer, which is the
+# behaviour this branch has always had: reporting the paths that did answer
+# would say that nothing is filed against the ones that did not.
+query_batch() {
+  local page returned fallback one marker
+
+  echo "searching for ${batch_count} path marker(s) in one search" >&2
+  fallback=0
+  if page="$(gh issue list --search "$batch_search" --state all --limit "$LIMIT" \
+               --json number,title,body,url,state,stateReason 2>/dev/null)"; then
+    returned="$(printf '%s' "$page" | jq 'length' 2>/dev/null)" || returned=""
+    if [ -z "$returned" ]; then
+      echo "the batched search's page could not be counted, so it is re-asked one path at a time" >&2
+      fallback=1
+    elif [ "$returned" -ge "$LIMIT" ]; then
+      echo "the batched search filled its page of ${LIMIT}, so it may be truncated and is re-asked one path at a time" >&2
+      fallback=1
+    fi
+  else
+    echo "the batched search failed, so it is re-asked one path at a time" >&2
+    fallback=1
+  fi
+
+  if [ "$fallback" -eq 0 ]; then
+    found="${found}${page}
+"
+    return 0
+  fi
+
+  while IFS= read -r one; do
+    [ -n "$one" ] || continue
+    marker="${PATH_MARKER_PREFIX}${one}"
+    echo "searching for ${marker}" >&2
+    page="$(gh issue list --search "\"${marker}\"" --state all --limit "$LIMIT" \
+              --json number,title,body,url,state,stateReason 2>/dev/null)" \
+      || fail "the search for ${one} failed, so what is filed is not known"
+    found="${found}${page}
+"
+  done <<BATCH_PATHS
+$batch_paths
+BATCH_PATHS
+}
+
 do_query() {
   require_tools fail
 
-  local question key body encoded asked paths found page one marker
+  local question key body encoded asked paths one
 
   question="$(cat)" || fail "the question could not be read from stdin"
 
@@ -707,23 +809,39 @@ do_query() {
     exit 0
   fi
 
-  # One search per path. A search that fails makes the whole answer unreliable —
-  # reporting the paths that did answer would say that nothing is filed against
-  # the ones that did not — so a failure here is a failure to answer.
+  # BATCH markers to a search rather than one search per path, so the number of
+  # searches is proportional to the scope divided by BATCH rather than to the
+  # scope. Each batch's search text is its markers quoted and joined with OR;
+  # every page goes into `found` and the composition below unions them.
   found=""
+  batch_search=""
+  batch_paths=""
+  batch_count=0
   while IFS= read -r one; do
     [ -n "$one" ] || continue
-    marker="${PATH_MARKER_PREFIX}${one}"
-    echo "searching for ${marker}" >&2
-    page="$(gh issue list --search "\"${marker}\"" --state all --limit "$LIMIT" \
-              --json number,title,body,url,state,stateReason 2>/dev/null)" \
-      || fail "the search for ${one} failed, so what is filed is not known"
-    found="${found}${page}
+    if [ "$batch_count" -gt 0 ]; then
+      batch_search="${batch_search} OR "
+    fi
+    batch_search="${batch_search}\"${PATH_MARKER_PREFIX}${one}\""
+    batch_paths="${batch_paths}${one}
 "
+    batch_count=$((batch_count + 1))
+    if [ "$batch_count" -ge "$BATCH" ]; then
+      query_batch
+      batch_search=""
+      batch_paths=""
+      batch_count=0
+    fi
   done <<PATHS
 $paths
 PATHS
 
+  # The last batch, which is short of BATCH whenever the scope does not divide
+  # by it. A scope smaller than one batch is answered by exactly one search.
+  if [ "$batch_count" -gt 0 ]; then
+    query_batch
+  fi
+
   # One document on stdout and nothing else. Every item's fields are what the
   # tracker said; nothing is invented for an item the searches did not return.
   printf '%s' "$found" | jq -s -c \