Skip to content

Commit fa99ff6

Browse files
committed
test(cli): stop a negative probe from spending 15s proving silence
The macos-15-intel leg killed the cli suite at its 900s wall clock while reporting 7537 passed, 0 failed. Two separate things were wrong. The suite was misclassified. cli spends 497s of the 900s default on macos-14, the FASTEST macOS runner, while macos-15-intel in the same matrix is 2.4-3.6x slower on comparable suites (daemon_runtime 842s vs 349s, stack_overflow_b 277s vs 76s). 497s at that ratio cannot fit. daemon_runtime, at 842s on that same runner, passes only because it was already in SLOW_SUITES. cli belongs there too -- it is deterministic and simply large, which is the only thing that list is allowed to mean. The drain test was also waiting on a clock instead of on the system. Profiling the suite per test (315 tests, 300s wall / 166s cpu) put cli_install_into_host_namespace_still_drains_host_cohort at 42s wall for 19s of cpu -- 23s idle, the largest single idle block, where the next worst was 11s. The existing cli_install_force_quiesces_active_cohort_before_replacing_binary drains a cohort too and idles 0.5s, which is what pointed at the difference. The cost was the host_serving probe. Its generous budget exists for the POSITIVE question -- is this daemon still up? -- where a slow reply on a loaded runner must not be misread as drained; that fixed a real flake and is kept. Asked in the negative it inverts: no reply is coming, so the full 15s is spent establishing silence and ASSERT_FALSE is decided by the timeout expiring rather than by the daemon's behaviour. The drain is already proven positively in that test -- install returns 0 only after the activation completed, and the host child exits CLI_SCOPE_HOST_DRAINED -- so the probe only has to confirm it. Also shortens the drained child's teardown budget, which retried service_free / lease_release / manager_free against 10s on a path where the activation had already torn the service down. Behaviour at the deadline is unchanged; it is reached sooner when nothing is wedged. Measured at only -3s on its own, kept because it is correct and free. Local, same machine, ASan+UBSan: the drain test 42.0s -> 25.8s (-38%), the suite 300s -> 279s, 315 passed before and after -- no test removed, no assertion weakened. PR DeusData#2188, which runs the same leg without these two tests, passed macos-15-intel in 37m34s, so the suite was not already over budget on main. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent 9a460b3 commit fa99ff6

2 files changed

Lines changed: 58 additions & 8 deletions

File tree

‎scripts/run-test-wave.py‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,19 @@
2626
SUMMARY = re.compile(r"^ (?P<passed>[0-9]+) passed")
2727
FAILED = re.compile(r"(?:^|, )(?P<failed>[0-9]+) failed")
2828
SKIPPED = re.compile(r"(?:^|, )(?P<skipped>[0-9]+) skipped")
29-
SLOW_SUITES = frozenset(("incremental", "store_arch", "daemon_runtime"))
29+
# Suites whose honest runtime does not fit the default per-suite budget, and so
30+
# get --slow-timeout instead. This is a statement about SIZE, never about
31+
# flakiness: every suite here is deterministic and simply long, and a racy suite
32+
# must be made deterministic rather than given more clock.
33+
#
34+
# `cli` joined the list because the classification had gone stale, not because
35+
# anything regressed. It spends 497s of the 900s default on macos-14 -- the
36+
# FASTEST macOS runner -- while the macos-15-intel runner in the same matrix is
37+
# 2.4-3.6x slower on comparable suites (daemon_runtime 842s vs 349s,
38+
# stack_overflow_b 277s vs 76s). 497s at that ratio cannot fit, so the suite was
39+
# killed at 900s and reported as hung. daemon_runtime, at 842s on that same
40+
# runner, survives only because it was already listed here.
41+
SLOW_SUITES = frozenset(("incremental", "store_arch", "daemon_runtime", "cli"))
3042
POLL_SECONDS = 0.05
3143

3244
# WHY: the Windows descendant probe below is a cold `powershell.exe` + CIM

‎tests/test_cli.c‎

Lines changed: 45 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -1522,6 +1522,27 @@ TEST(cli_install_recovers_markerless_stale_rendezvous) {
15221522
#define CLI_SCOPE_READY_TIMEOUT_MS 90000U
15231523
#define CLI_SCOPE_HOST_REAP_TIMEOUT_MS 60000U
15241524
#define CLI_SCOPE_HOST_SERVING_TIMEOUT_MS 15000U
1525+
/* Teardown budget for a child the activation ALREADY drained: its service is
1526+
* gone, so each free/release succeeds immediately and this is only the
1527+
* give-up point if one unexpectedly does not. The undrained path keeps the
1528+
* full 2 x CLI_SCOPE_TIMEOUT_MS, which is where a wedged teardown is real. */
1529+
#define CLI_SCOPE_CLEANUP_DRAINED_MS 1000U
1530+
/* Budget for probing that an ALREADY-drained host has stopped serving.
1531+
*
1532+
* The generous CLI_SCOPE_HOST_SERVING_TIMEOUT_MS exists for the POSITIVE
1533+
* question -- "is this daemon still up?" -- where a slow reply on a loaded
1534+
* runner must not be misread as drained. Asked in the negative it inverts:
1535+
* there is no reply coming, so the whole budget is spent proving silence, and
1536+
* the assertion is decided by a timeout expiring rather than by the system's
1537+
* own behaviour. That was 15 s of the drain test's wall clock and the largest
1538+
* single idle block in the cli suite.
1539+
*
1540+
* The drain is proven POSITIVELY elsewhere in that test: install returns 0
1541+
* only after the activation completed, and the host child exits with
1542+
* CLI_SCOPE_HOST_DRAINED. By the time this probe runs the daemon is already
1543+
* gone, so a short budget confirms the same fact a long one would -- it just
1544+
* stops charging the suite for the wait. */
1545+
#define CLI_SCOPE_HOST_DRAINED_PROBE_MS 2000U
15251546

15261547
typedef struct {
15271548
char tmpdir[256];
@@ -1588,7 +1609,16 @@ static _Noreturn void cli_scope_host_child(const cli_scope_fixture_t *fixture, i
15881609
break;
15891610
}
15901611
}
1591-
uint64_t cleanup_deadline = cbm_now_ms() + 2U * CLI_SCOPE_TIMEOUT_MS;
1612+
/* A DRAINED child has already had its service torn down by the activation,
1613+
* so every teardown below succeeds on the first attempt; the long deadline
1614+
* exists for the wedged case, where we keep retrying before giving up. On
1615+
* the drained path that budget was pure wall clock -- the parent sits in
1616+
* cli_scope_reap_host waiting for this exit, and it was the single largest
1617+
* idle block in the whole cli suite. Behaviour at the deadline is
1618+
* unchanged (give up and _exit); it is simply reached sooner when there is
1619+
* nothing wedged to wait for. */
1620+
uint64_t cleanup_deadline =
1621+
cbm_now_ms() + (drained ? CLI_SCOPE_CLEANUP_DRAINED_MS : 2U * CLI_SCOPE_TIMEOUT_MS);
15921622
if (service) {
15931623
if (!drained) {
15941624
(void)cbm_daemon_runtime_service_stop(service, CLI_SCOPE_TIMEOUT_MS);
@@ -1752,19 +1782,24 @@ static bool cli_scope_fixture_start(cli_scope_fixture_t *fixture, const char *ta
17521782
return fixture->client != NULL;
17531783
}
17541784

1785+
static bool cli_scope_host_serving_within(const cli_scope_fixture_t *fixture,
1786+
uint32_t timeout_ms) {
1787+
cbm_daemon_runtime_status_t status = {0};
1788+
return fixture->endpoint &&
1789+
cbm_daemon_runtime_request_status(fixture->endpoint, &fixture->identity, timeout_ms,
1790+
&status) &&
1791+
!status.stopping && status.committed_clients == 1;
1792+
}
1793+
17551794
/* Ask the host daemon itself: still running, not stopping, and the parent's
17561795
* committed client still admitted. */
17571796
static bool cli_scope_host_serving(const cli_scope_fixture_t *fixture) {
1758-
cbm_daemon_runtime_status_t status = {0};
17591797
/* A generous, bounded status deadline: a foreign-namespace install leaves
17601798
* this daemon serving, so a slow response on a loaded runner must not be
17611799
* misread as "drained" (the flaky failure this fixture showed). The call
17621800
* still fails cleanly — a genuinely drained daemon is unreachable or
17631801
* reports stopping — it just no longer decides survival on a 5 s budget. */
1764-
return fixture->endpoint &&
1765-
cbm_daemon_runtime_request_status(fixture->endpoint, &fixture->identity,
1766-
CLI_SCOPE_HOST_SERVING_TIMEOUT_MS, &status) &&
1767-
!status.stopping && status.committed_clients == 1;
1802+
return cli_scope_host_serving_within(fixture, CLI_SCOPE_HOST_SERVING_TIMEOUT_MS);
17681803
}
17691804

17701805
static int cli_scope_install(cli_scope_fixture_t *fixture, const char *home, const char *cache,
@@ -1902,7 +1937,10 @@ TEST(cli_install_into_host_namespace_still_drains_host_cohort) {
19021937
int install_rc =
19031938
ready ? cli_scope_install(&fixture, fixture.host_home, fixture.host_cache, host_bin, false)
19041939
: -1;
1905-
bool host_serving = ready && cli_scope_host_serving(&fixture);
1940+
/* Negative probe: see CLI_SCOPE_HOST_DRAINED_PROBE_MS. The drain itself is
1941+
* asserted positively below via host_exit == CLI_SCOPE_HOST_DRAINED. */
1942+
bool host_serving = ready && cli_scope_host_serving_within(&fixture,
1943+
CLI_SCOPE_HOST_DRAINED_PROBE_MS);
19061944
const char *events = read_test_file(activation_log);
19071945
bool drained_in_log = events && strstr(events, "cohort drained") != NULL &&
19081946
strstr(events, "\"daemon_active_clients\":1") != NULL;

0 commit comments

Comments
 (0)