Skip to content

Commit b9a186d

Browse files
committed
test(bench): cover setup failure, refused daemon start and refused stop
Three more cases in the benchmark runtime isolation contract, each RED against the harness without its fix: - a stale runtime-root.txt is gone after a run that fails during setup (a repository path that does not exist); - with the keep flag and a `daemon stop` that refuses, the run fails, writes no handoff, keeps its root and prints the refusal; the test removes that root itself, before any verdict, so a failing assertion leaks nothing; - a refused `daemon start` fails both harnesses, prints the cause and leaves no root behind. The fixture refuses the start on stderr and the stop on stdout, the channel the product uses for a busy refusal. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 32f60a4 commit b9a186d

1 file changed

Lines changed: 83 additions & 1 deletion

File tree

‎tests/test_benchmark_runtime_isolation_contract.sh‎

Lines changed: 83 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -11,7 +11,9 @@ set -euo pipefail
1111
# an environment-probe fixture and require that no product process ever
1212
# receives the caller's runtime or cache, that the index benchmark records
1313
# the setup cost it now pays explicitly, and that a refused index fails the
14-
# run, names its cause, and leaves nothing behind.
14+
# run, names its cause, and leaves nothing behind. A daemon that refuses to
15+
# start or to stop fails the run the same way, and a kept runtime is handed
16+
# over only once its daemon stopped.
1517

1618
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
1719
WORKDIR="$(mktemp -d)"
@@ -43,6 +45,16 @@ if [[ "${1-} ${2-}" == "cli index_repository" ]]; then
4345
echo "probe: index refused" >&2
4446
exit 1
4547
fi
48+
if [[ "${1-} ${2-}" == "daemon start" && -n "${CBM_BENCH_PROBE_START_REFUSED-}" ]]; then
49+
echo "probe: daemon start refused" >&2
50+
exit 1
51+
fi
52+
# The product reports a busy refusal (a committed client still uses the
53+
# daemon) on stdout, so that is where the fixture reports its own.
54+
if [[ "${1-} ${2-}" == "daemon stop" && -n "${CBM_BENCH_PROBE_STOP_REFUSED-}" ]]; then
55+
echo "probe: daemon NOT stopped, a committed client still uses it"
56+
exit 1
57+
fi
4658
[[ "${1-} ${2-}" == "daemon status" ]] && exit 1
4759
exit 0
4860
EOF
@@ -136,6 +148,37 @@ grep -qF -- "${KEPT_CACHE}"$'\t'"${KEPT_RUNTIME}" "$KEEP_LOG" ||
136148
fail "runtime-root.txt does not name the runtime and cache the product processes used"
137149
rm -rf -- "$KEPT_ROOT"
138150

151+
# The handoff promises a stopped daemon, so a kept runtime is handed over only
152+
# once `daemon stop` succeeded. A refused stop fails the run, writes no handoff
153+
# and keeps the root: the daemon may still be using it, so removing it is not
154+
# the harness's call. The refusal itself has to reach the output.
155+
STOP_LOG="$WORKDIR/keep-stop-refused-environment.log"
156+
STOP_HANDOFF="$WORKDIR/results/keep-stop-refused/runtime-root.txt"
157+
stop_rc=0
158+
CBM_CACHE_DIR="$CALLER_CACHE" \
159+
CBM_RUNTIME_DIR="$CALLER_RUNTIME" \
160+
CBM_BENCH_ENV_PROBE="$STOP_LOG" \
161+
CBM_BENCH_PROBE_INDEX_OK=1 \
162+
CBM_BENCH_PROBE_STOP_REFUSED=1 \
163+
CBM_BENCH_KEEP_RUNTIME=1 \
164+
"$ROOT/scripts/benchmark-index.sh" "$ENV_PROBE" keep-stop-refused "$REPO" "$WORKDIR/results" \
165+
> "$WORKDIR/keep-stop-refused.out" 2>&1 || stop_rc=$?
166+
[[ -s "$STOP_LOG" ]] || fail "benchmark-index (keep, refused stop) did not execute the environment-probe fixture"
167+
STOP_RUNTIME=$(normalize_path "$(tail -n 1 "$STOP_LOG" | cut -f2)")
168+
STOP_ROOT="${STOP_RUNTIME%/*}"
169+
stop_root_kept=0
170+
[[ ! -d "$STOP_ROOT/runtime" ]] || stop_root_kept=1
171+
# Removed here, before any verdict, so a failing assertion leaks no root.
172+
case "${STOP_ROOT##*/}" in
173+
cbmrt.?* | cbmrt-?*) rm -rf -- "$STOP_ROOT" ;;
174+
*) fail "benchmark-index (keep, refused stop) used an unexpected private root: $STOP_ROOT" ;;
175+
esac
176+
[[ "$stop_rc" -ne 0 ]] || fail "benchmark-index exited 0 although its daemon refused to stop"
177+
[[ ! -e "$STOP_HANDOFF" ]] || fail "benchmark-index handed over a runtime whose daemon refused to stop"
178+
(( stop_root_kept )) || fail "benchmark-index removed a runtime root whose daemon refused to stop"
179+
grep -q 'probe: daemon NOT stopped' "$WORKDIR/keep-stop-refused.out" ||
180+
fail "benchmark-index did not surface why its daemon refused to stop"
181+
139182
# Asked to keep the runtime and refused the index, the harness has nothing worth
140183
# keeping: the run fails, no root survives, and no handoff is written — nor left
141184
# over from an earlier run, since the evaluation loop reuses the results
@@ -159,6 +202,23 @@ done
159202
assert_isolated "benchmark-index (keep, refused index)" "$REFUSED_LOG"
160203
[[ ! -e "$REFUSED_HANDOFF" ]] || fail "benchmark-index left a handoff for an index that failed"
161204

205+
# The same holds for a run that never reaches the index: one whose repository
206+
# does not exist fails during setup, and an earlier run's handoff in the reused
207+
# results directory must already be gone by then.
208+
SETUP_HANDOFF="$WORKDIR/results/setup-failed/runtime-root.txt"
209+
mkdir -p "${SETUP_HANDOFF%/*}"
210+
echo 'CBM_BENCH_RUNTIME_ROOT=/stale/root/from/an/earlier/run' > "$SETUP_HANDOFF"
211+
setup_rc=0
212+
CBM_CACHE_DIR="$CALLER_CACHE" \
213+
CBM_RUNTIME_DIR="$CALLER_RUNTIME" \
214+
CBM_BENCH_ENV_PROBE="$WORKDIR/setup-failed-environment.log" \
215+
CBM_BENCH_KEEP_RUNTIME=1 \
216+
"$ROOT/scripts/benchmark-index.sh" "$ENV_PROBE" setup-failed "$WORKDIR/no-such-repo" "$WORKDIR/results" \
217+
> "$WORKDIR/setup-failed.out" 2>&1 || setup_rc=$?
218+
[[ "$setup_rc" -ne 0 ]] || fail "benchmark-index exited 0 for a repository that does not exist"
219+
[[ ! -e "$SETUP_HANDOFF" ]] ||
220+
fail "benchmark-index left an earlier handoff behind after a run that failed during setup"
221+
162222
SEARCH_LOG="$WORKDIR/search-environment.log"
163223
CBM_CACHE_DIR="$CALLER_CACHE" \
164224
CBM_RUNTIME_DIR="$CALLER_RUNTIME" \
@@ -175,6 +235,28 @@ grep -q -- '--- index/parse stderr ---' "$WORKDIR/search.out" ||
175235
grep -q 'probe: index refused' "$WORKDIR/search.out" ||
176236
fail "benchmark-search-graph did not surface the cause of the index failure"
177237

238+
# A private daemon that does not start fails either harness, and the product's
239+
# own explanation has to reach the output: "did not start" alone leaves the
240+
# operator with the symptom only.
241+
for harness in benchmark-index benchmark-search-graph; do
242+
if [[ "$harness" == benchmark-index ]]; then
243+
harness_args=("$ENV_PROBE" start-refused "$REPO" "$WORKDIR/results")
244+
else
245+
harness_args=("$ENV_PROBE" "$REPO")
246+
fi
247+
start_rc=0
248+
CBM_CACHE_DIR="$CALLER_CACHE" \
249+
CBM_RUNTIME_DIR="$CALLER_RUNTIME" \
250+
CBM_BENCH_ENV_PROBE="$WORKDIR/$harness-start-refused-environment.log" \
251+
CBM_BENCH_PROBE_START_REFUSED=1 \
252+
"$ROOT/scripts/$harness.sh" "${harness_args[@]}" \
253+
> "$WORKDIR/$harness-start-refused.out" 2>&1 || start_rc=$?
254+
[[ "$start_rc" -ne 0 ]] || fail "$harness exited 0 although its private daemon did not start"
255+
assert_isolated "$harness (daemon start refused)" "$WORKDIR/$harness-start-refused-environment.log"
256+
grep -q 'probe: daemon start refused' "$WORKDIR/$harness-start-refused.out" ||
257+
fail "$harness did not surface why its private daemon did not start"
258+
done
259+
178260
# The request carrying the repository path has to be built as JSON. A path may
179261
# legitimately contain a quote or a backslash — hand-built JSON turns that into
180262
# a payload the server cannot parse, or one that means something else. NTFS

0 commit comments

Comments
 (0)