Skip to content

Commit 78bf6a2

Browse files
committed
fix(test): hand over a kept benchmark runtime only after its daemon stopped
With CBM_BENCH_KEEP_RUNTIME=1, bench_finish ran `daemon stop` with its output and exit status discarded and wrote runtime-root.txt regardless. `daemon stop` refuses while a committed client still uses the daemon and fails when the daemon does not answer; the handoff then promised a stopped daemon that was still running. A refused stop now prints the explanation of the product, writes no handoff, keeps the root without cleaning it up (the daemon may still be using it) and exits non-zero. A stop with nothing to stop still exits 0, so a clean run is unaffected. Section 7 step 8 of the evaluation plan follows the same rule: the kept root is removed only if its daemon stopped, and is kept with a message otherwise. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent b985170 commit 78bf6a2

2 files changed

Lines changed: 19 additions & 5 deletions

File tree

‎docs/EVALUATION_PLAN.md‎

Lines changed: 7 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -299,9 +299,13 @@ for lang in $ALL_LANGS; do # ALL_LANGS = full 159-name list
299299

300300
# --- step 8: delete THIS language's index so the next is cold, then mark done ---
301301
# The graph session (steps 4-7) may still have a daemon running under the kept
302-
# CBM_RUNTIME_DIR (exported above); stop it before its root goes away.
303-
~/.local/bin/codebase-memory-mcp daemon stop || true
304-
rm -rf -- "$CBM_BENCH_RUNTIME_ROOT"
302+
# CBM_RUNTIME_DIR (exported above); stop it before its root goes away. A refused
303+
# stop (a session still attached) means the root is still in use: keep it.
304+
if ~/.local/bin/codebase-memory-mcp daemon stop; then
305+
rm -rf -- "$CBM_BENCH_RUNTIME_ROOT"
306+
else
307+
echo "$lang: daemon under $CBM_RUNTIME_DIR did not stop; keeping $CBM_BENCH_RUNTIME_ROOT" >&2
308+
fi
305309
manifest_mark_done "$lang" "$index_ms"
306310
done
307311
```

‎scripts/benchmark-index.sh‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -33,11 +33,21 @@ cbm_test_runtime_init
3333
# including its removal — passes to the caller. A failed run cleans up
3434
# regardless: there is no index worth keeping, and nothing must leak. Never on
3535
# by default, or an unattended run accumulates one root per language.
36+
#
37+
# The handoff promises a stopped daemon, so it is written only once
38+
# `daemon stop` succeeded. A refused stop (a client still committed to the
39+
# daemon, or no answer) fails the run with the product's explanation, writes
40+
# no handoff, and keeps the root without cleaning it up: the daemon may still
41+
# be using it, and the operator decides what happens to it.
3642
bench_finish() {
37-
local rc=$?
43+
local rc=$? stop_out=""
3844
[ -z "${BENCH_TMP:-}" ] || rm -rf -- "$BENCH_TMP" || true
3945
if [ "$rc" -eq 0 ] && [ -n "${CBM_BENCH_KEEP_RUNTIME:-}" ] && [ -d "${OUT:-}" ]; then
40-
"$BINARY" daemon stop >/dev/null 2>&1 || true
46+
if ! stop_out=$("$BINARY" daemon stop 2>&1); then
47+
echo " $LANG: private daemon did not stop; runtime kept at $CBM_TEST_RUNTIME_ROOT, no handoff written" >&2
48+
[ -z "$stop_out" ] || printf '%s\n' "$stop_out" >&2
49+
exit 1
50+
fi
4151
printf 'CBM_BENCH_RUNTIME_ROOT=%q\nCBM_RUNTIME_DIR=%q\nCBM_CACHE_DIR=%q\n' \
4252
"$CBM_TEST_RUNTIME_ROOT" "$CBM_RUNTIME_DIR" "$CBM_CACHE_DIR" > "$OUT/runtime-root.txt"
4353
echo " $LANG: runtime kept at $CBM_TEST_RUNTIME_ROOT; paths in $OUT/runtime-root.txt" >&2

0 commit comments

Comments
 (0)