Skip to content

Commit 42a2e2d

Browse files
committed
Port upstream DeusData#2198: benchmark runtime isolation (upstream DeusData#1696)
Finish the merge of the benchmark harness isolation work onto the CLI-first fork. The auto-merged upstream hunks left both scripts as a mix of fork and upstream invocations (undefined INDEX_RC, an unused INDEX_ERR, MCP `cli index_repository` calls). benchmark-index.sh: run `codebase-memory-cli index REPO --mode full --json` with stderr captured to INDEX_ERR and the exit status kept in INDEX_RC, so a refused index fails the run, prints "--- index stderr ---" and the cause. Dropped the now-unneeded hand-built JSON path (argv carries the path). benchmark-search-graph.sh: index through `index REPO --mode full --json` instead of the MCP `cli index_repository` request; stop unwrapping the MCP content envelope; fix usage comment (arg is a repo path). docs/EVALUATION_PLAN.md: example invocations name codebase-memory-cli. Contract test adaptations (assertions on isolation, stderr surfacing, exit status, handoff and stop behaviour unchanged): - Probe fixture detects `index` (argv 1) instead of `cli index_repository`. - The fixture logs the repo path (argv 2) rather than a JSON body; the quoted/backslash-path check now asserts the path reaches the index command as one intact argument, since the fork's CLI has no JSON request. - "MCP session" comment wording changed to "graph session".
1 parent 3a4fdb9 commit 42a2e2d

4 files changed

Lines changed: 24 additions & 30 deletions

File tree

‎docs/EVALUATION_PLAN.md‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -283,7 +283,7 @@ for lang in $ALL_LANGS; do # ALL_LANGS = full 159-name list
283283
# --- step 2: cold index in the main channel, TIMED (key metric) ---
284284
t0=$(now_ms)
285285
CBM_BENCH_KEEP_RUNTIME=1 \
286-
scripts/benchmark-index.sh ~/.local/bin/codebase-memory-mcp "$lang" /tmp/bench/"$lang" /tmp/eval-results
286+
scripts/benchmark-index.sh ~/.local/bin/codebase-memory-cli "$lang" /tmp/bench/"$lang" /tmp/eval-results
287287
index_ms=$(( $(now_ms) - t0 )) # clone+index wall-clock → manifest + report (§5)
288288
# The harness indexes into a run-private root (#1696). CBM_BENCH_KEEP_RUNTIME leaves that
289289
# root behind on success and records its paths; the graph session in steps 4-7 must be
@@ -878,7 +878,7 @@ scripts/clone-bench-repos.sh /tmp/bench
878878
mkdir -p /tmp/eval-results
879879
for lang in $ALL_LANGS; do
880880
CBM_BENCH_KEEP_RUNTIME=1 \
881-
scripts/benchmark-index.sh ~/.local/bin/codebase-memory-mcp "$lang" /tmp/bench/"$lang" /tmp/eval-results
881+
scripts/benchmark-index.sh ~/.local/bin/codebase-memory-cli "$lang" /tmp/bench/"$lang" /tmp/eval-results
882882
done
883883

884884
# 3. Cross-repo pass for the 9 LSP pairs (index each service dir, then cross-repo-intelligence)

‎scripts/benchmark-index.sh‎

Lines changed: 7 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@
22
set -euo pipefail
33

44
# Index a single benchmark repository and capture metrics.
5-
# Usage: benchmark-index.sh <binary> <lang> <repo_path> <results_dir>
5+
# Usage: benchmark-index.sh <codebase-memory-cli binary> <lang> <repo_path> <results_dir>
66

77
BINARY="${1:?Usage: benchmark-index.sh <binary> <lang> <repo_path> <results_dir>}"
88
LANG="${2:?}"
@@ -66,10 +66,6 @@ DAEMON_START_LOG="$BENCH_TMP/daemon-start.log"
6666

6767
# Resolve symlinks
6868
REPO=$(cd "$REPO" && pwd -P)
69-
# One pre-escaped spelling of the path for every request below, the way the
70-
# soak harness builds its own: a repository path may legitimately contain a
71-
# quote or a backslash, and hand-built JSON turns that into a parse error.
72-
REPO_JSON=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$REPO")
7369

7470
# Elapsed time is read from a monotonic clock, never the wall clock: an NTP
7571
# step mid-run would otherwise skew — or negate — a figure whose whole purpose
@@ -115,7 +111,12 @@ fi
115111
# Index via CLI and capture timing
116112
START_MS=$(bench_now_ms)
117113

118-
INDEX_JSON=$("$BINARY" index "$REPO" --mode full --json 2>/dev/null || echo '{"error":"index failed"}')
114+
# The path is one argv element, so a quote or backslash in it needs no escaping.
115+
# The exit status is kept (a bare `|| echo` would hide it) and stderr goes to
116+
# INDEX_ERR for the failure path below.
117+
INDEX_RC=0
118+
INDEX_JSON=$("$BINARY" index "$REPO" --mode full --json 2>"$INDEX_ERR") || INDEX_RC=$?
119+
[ -n "$INDEX_JSON" ] || INDEX_JSON='{"error":"index failed"}'
119120

120121
END_MS=$(bench_now_ms)
121122
ELAPSED=$((END_MS - START_MS))

‎scripts/benchmark-search-graph.sh‎

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
# codebase-memory-cli binary to measure regex / BM25 search performance.
44
#
55
# Usage:
6-
# scripts/benchmark-search-graph.sh <binary-path> <project-name>
6+
# scripts/benchmark-search-graph.sh <binary-path> <repo-path>
77

88
set -euo pipefail
99

@@ -30,20 +30,14 @@ if ! "$BINARY" daemon start >"$DAEMON_START_LOG" 2>&1; then
3030
fi
3131
exit 1
3232
fi
33-
# One pre-escaped spelling of the path, the way the soak harness builds its
34-
# own: a repository path may legitimately contain a quote or a backslash, and
35-
# hand-built JSON turns that into a parse error.
36-
REPO_JSON=$(python3 -c 'import json,sys; print(json.dumps(sys.argv[1]))' "$REPO")
3733
# Index and parse keep their stderr instead of discarding it: without it the
3834
# failure below names only its symptom, and the cause — an unreadable
39-
# repository, a refused daemon, a malformed envelope — is unrecoverable.
40-
INDEX_JSON=$("$BINARY" cli index_repository "{\"repo_path\":$REPO_JSON,\"mode\":\"full\"}" \
41-
2>"$INDEX_ERR" || echo '{}')
35+
# repository, a refused daemon, a malformed envelope — is unrecoverable. The
36+
# repository path is one argv element, so a quote or backslash needs no escaping.
37+
INDEX_JSON=$("$BINARY" index "$REPO" --mode full --json 2>"$INDEX_ERR" || echo '{}')
4238
PROJECT=$(echo "$INDEX_JSON" | python3 -c "
4339
import json, sys
4440
d = json.load(sys.stdin)
45-
if 'content' in d:
46-
d = json.loads(d['content'][0]['text'])
4741
print(d.get('project', ''))
4842
" 2>>"$INDEX_ERR" || echo "")
4943
if [ -z "$PROJECT" ]; then

‎tests/test_benchmark_runtime_isolation_contract.sh‎

Lines changed: 11 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -36,8 +36,8 @@ ENV_PROBE="$WORKDIR/environment-probe"
3636
cat > "$ENV_PROBE" <<'EOF'
3737
#!/usr/bin/env bash
3838
printf '%s\t%s\n' "${CBM_CACHE_DIR-}" "${CBM_RUNTIME_DIR-}" >> "$CBM_BENCH_ENV_PROBE"
39-
if [[ "${1-} ${2-}" == "cli index_repository" ]]; then
40-
[[ -z "${CBM_BENCH_REQUEST_LOG-}" ]] || printf '%s\n' "${3-}" >> "$CBM_BENCH_REQUEST_LOG"
39+
if [[ "${1-}" == "index" ]]; then
40+
[[ -z "${CBM_BENCH_REQUEST_LOG-}" ]] || printf '%s\n' "${2-}" >> "$CBM_BENCH_REQUEST_LOG"
4141
if [[ -n "${CBM_BENCH_PROBE_INDEX_OK-}" ]]; then
4242
printf '%s\n' '{"project":"probe"}'
4343
exit 0
@@ -117,7 +117,7 @@ grep -q 'probe: index refused' "$WORKDIR/index.out" ||
117117
fail "benchmark-index did not surface the cause of the index failure"
118118

119119
# The evaluation plan indexes a language and then reads that index from its own
120-
# MCP session (docs/EVALUATION_PLAN.md §7). Asked to keep the runtime, a
120+
# graph session (docs/EVALUATION_PLAN.md §7). Asked to keep the runtime, a
121121
# successful run must leave its root behind and record, sourceably, the paths
122122
# that reach it — and they must be the paths the product processes actually
123123
# used. Unasked, the root is gone (asserted above). The fixture answers this
@@ -257,10 +257,11 @@ for harness in benchmark-index benchmark-search-graph; do
257257
fail "$harness did not surface why its private daemon did not start"
258258
done
259259

260-
# The request carrying the repository path has to be built as JSON. A path may
261-
# legitimately contain a quote or a backslash — hand-built JSON turns that into
262-
# a payload the server cannot parse, or one that means something else. NTFS
263-
# rejects both characters in a path component, so this case is POSIX-only.
260+
# The repository path reaches the CLI as a single argv element. A path may
261+
# legitimately contain a quote or a backslash — word-splitting or re-quoting
262+
# would mangle it. NTFS rejects both characters in a path component, so this
263+
# case is POSIX-only. (Adapted: upstream asserted a JSON request body; the
264+
# fork's CLI takes the path as `index <repo>`, so the fixture logs argv.)
264265
case "$(uname -s)" in
265266
MINGW* | MSYS* | CYGWIN*) ;;
266267
*)
@@ -275,8 +276,7 @@ MINGW* | MSYS* | CYGWIN*) ;;
275276
CBM_BENCH_REQUEST_LOG="$REQUEST_LOG" \
276277
"$ROOT/scripts/benchmark-search-graph.sh" "$ENV_PROBE" "$QUIRKY_REPO" \
277278
> "$WORKDIR/quirky.out" 2>&1 || true
278-
python3 - "$REQUEST_LOG" "$QUIRKY_RESOLVED" <<'PY' || fail "benchmark-search-graph built an index request that is not valid JSON for a quoted path"
279-
import json
279+
python3 - "$REQUEST_LOG" "$QUIRKY_RESOLVED" <<'PY' || fail "benchmark-search-graph did not pass a quoted path intact to the index command"
280280
import sys
281281
282282
request_log, expected = sys.argv[1], sys.argv[2]
@@ -286,9 +286,8 @@ except OSError:
286286
sys.exit("the search benchmark sent no index request")
287287
if not lines:
288288
sys.exit("the search benchmark sent no index request")
289-
payload = json.loads(lines[0])
290-
if payload.get("repo_path") != expected:
291-
sys.exit(f"repo_path is {payload.get('repo_path')!r}, expected {expected!r}")
289+
if lines[0] != expected:
290+
sys.exit(f"repo path is {lines[0]!r}, expected {expected!r}")
292291
PY
293292
;;
294293
esac

0 commit comments

Comments
 (0)