Skip to content

Commit da4bd25

Browse files
committed
fix(test): the ENOSPC bootstrap test no longer lets a deadline decide its verdict
daemon_bootstrap_fails_fast_when_daemon_dies_at_publication went red on the MSan lane of three PRs that touch no daemon code at all (DeusData#2245, DeusData#2158, DeusData#2140) with tests/test_daemon_bootstrap.c:1116: ASSERT(daemon_named_cause) and green on a rerun of the identical commit. Cause, attributed 2026-09-20. The test forks a real daemon host whose record publication fails with ENOSPC, and asserts that the client names that cause instead of waiting out its 30 s startup deadline. But before it listens, the host SHA-256s its own executable image, and the test runner is ~450 MB: ~3 s natively, 20-32 s under MSan (48 CI samples, median 25.8 s). The spawn override returned as soon as it had forked, so the client's 30 s clock raced the host's hash. On a slow run the deadline fired first, the reaper SIGKILLed a host that had not failed yet, and the log had no listen_failed line to find. Nothing was wrong with the product: with a 28 s artificial delay the client does learn the recorded ENOSPC cause. The spawn override now reaps the forked host before it returns. The host writes its start-failure record and only then releases its lifetime reservation and exits, so "record on disk, reservation released" is a stable state pinned by construction when the client takes its first look -- and that first look is guaranteed: the client's post-spawn wait loop runs at least once even past the deadline, and its first failure check after a spawn is unthrottled. How long the host takes no longer matters. No production code changes; the assertions are unchanged. A regression to the pre-DeusData#1828 behaviour (a client that ignores the record) still fails the unchanged assertions: its message no longer names the failure, the errno and the path. Proof, one build with temporary env-switched instrumentation that is not part of this commit (a sleep in the forked host before it starts, and a switch back to the old fire-and-forget spawn): spawn forked host delayed daemon_bootstrap suite synchronous no 28 passed old (async) 31 s 27 passed, 1 failed: ASSERT(log_read) synchronous 31 s 28 passed The old spawn fails under the delay one assertion earlier than CI did (log_read, not daemon_named_cause): the injected sleep sits before the host opens its log, whereas in CI the host was killed mid-hash with the log already open. Same mechanism -- the client's deadline beating the host -- and the 2026-09-20 attribution reproduced the CI message itself with a delay placed after the log is opened. The final tree, without the instrumentation: daemon_bootstrap 28 passed, three runs in a row. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent bf4c476 commit da4bd25

1 file changed

Lines changed: 30 additions & 7 deletions

File tree

‎tests/test_daemon_bootstrap.c‎

Lines changed: 30 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -893,6 +893,8 @@ typedef struct {
893893
char parent[BOOTSTRAP_TEST_PATH_CAP];
894894
cbm_daemon_build_identity_t identity;
895895
pid_t children[BOOTSTRAP_ENOSPC_MAX_CHILDREN];
896+
int exit_status[BOOTSTRAP_ENOSPC_MAX_CHILDREN];
897+
bool reaped[BOOTSTRAP_ENOSPC_MAX_CHILDREN];
896898
size_t child_count;
897899
size_t spawn_calls;
898900
} bootstrap_enospc_host_t;
@@ -928,19 +930,40 @@ static bool bootstrap_enospc_host_spawn(void *opaque,
928930
int run_result = endpoint ? cbm_daemon_host_run(&config) : 0;
929931
_exit(run_result == -1 ? 0 : 50);
930932
}
931-
state->children[state->child_count++] = child;
933+
/* SYNCHRONOUS on purpose. The daemon host writes its start-failure record
934+
* and only then lets go of its lifetime reservation and exits, so once it
935+
* has been reaped "record on disk, reservation released" is a STABLE state
936+
* pinned by construction. The client's first look after the spawn (the
937+
* wait loop always runs once and its first failure check is unthrottled)
938+
* therefore finds the record however long the host took to get there.
939+
*
940+
* Returning while the host was still running made the verdict ride on the
941+
* client's 30 s startup deadline: before it listens the host SHA-256s its
942+
* own image, which for the ~450 MB test runner is ~3 s natively and ~26 s
943+
* under MSan (48 CI samples: 20-32 s). The deadline won that race on slow
944+
* runs and the reaper then SIGKILLed a host that had not failed yet --
945+
* `ASSERT(daemon_named_cause)` red on PRs that touch no C at all (#2245,
946+
* #2158, #2140). A deadline must never decide a test (O9). */
947+
size_t slot = state->child_count++;
948+
state->children[slot] = child;
949+
int status = 0;
950+
pid_t waited;
951+
do {
952+
waited = waitpid(child, &status, 0);
953+
} while (waited < 0 && errno == EINTR);
954+
state->reaped[slot] = waited == child;
955+
state->exit_status[slot] = status;
932956
return true;
933957
}
934958

935959
static void bootstrap_enospc_reap(bootstrap_enospc_host_t *state, int *nonzero_exits) {
936960
*nonzero_exits = 0;
937961
for (size_t i = 0; i < state->child_count; i++) {
938-
int status = 0;
939-
pid_t waited;
940-
do {
941-
waited = waitpid(state->children[i], &status, WNOHANG);
942-
} while (waited < 0 && errno == EINTR);
943-
if (waited == 0) {
962+
int status = state->exit_status[i];
963+
if (!state->reaped[i]) {
964+
/* Only reachable when the synchronous wait in the spawn itself
965+
* failed; never leave a host behind. */
966+
pid_t waited;
944967
(void)kill(state->children[i], SIGKILL);
945968
do {
946969
waited = waitpid(state->children[i], &status, 0);

0 commit comments

Comments
 (0)