Skip to content

Commit dc840ee

Browse files
authored
fix(exec): pause the project watcher during the gate and retry once on an infra exit (#451)
fix(exec): pause the project watcher during the gate and retry once on an infra exit Verdict: APPROVE r3 on head fd90628. The r2 must-fix is done (f4cc5ae): the -p value is no longer cut at the first space; the longest run of words that exists on disk is kept, so a duplicated "<proj> copy" watcher is not matched, and this project's own watcher under a path with a space is. The r2 nice-to-have 1 is also done: the pause record is written before the SIGSTOP loop. Test D now runs a "<proj> copy" sibling with its own .mpr; it fails on 0c14d17 (22/24) and passes with the fix (24/24). lock-watch 84/84, crash-rollback 13/13, check-scripts, check-portability and check-pr-discipline clean. Base release/v1.0 merged (CHANGELOG keep-both). Everything else in r2 stands.
1 parent 618e1b3 commit dc840ee

4 files changed

Lines changed: 312 additions & 5 deletions

File tree

‎CHANGELOG.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -19,6 +19,7 @@ Sections dated before 2026-09-19 predate the cycle and stay as they are. Named m
1919

2020
## Unreleased
2121

22+
- fix(exec): **the gate pauses this project's `mxcli run --local --watch` while mxbuild runs and retries once on an infra exit.** The gate's mxbuild and a live watcher both write `deployment/`; in a field migration project (12 modules) 17 of 27 failed gates were "gate could not run" (mxbuild exit 1, no CE code), against 0 of 22 in watcher-free copies. The watcher is stopped (SIGSTOP) for the gate and resumed (SIGCONT, also in the EXIT trap); an exit 1 or 3 with no CE code and no errors file is retried once and the BUILD-LOG row says `retry after infra exit`; a final failure still takes the #424 rollback. The match is exact (the watcher's project dir must equal this project's root, so a sibling `-copy` or a nested worktree watcher is never paused), and the paused pids are recorded in `.mpr-snapshots/.watcher-paused` so the next exec.sh resumes a watcher left stopped by a SIGKILLed one and says so. Test: `tests/wave2/test-exec-gate-retry.sh` (Refs #441, #444, #171, #285, #424) — field migration project
2223
- learn(load-test): **trap 9, an unlicensed local runtime caps concurrent sessions at 6 ("Maximum number of sessions exceeded!") and stops itself after a random 2 to 4 hours, both reading as the app failing under load; more than 6 users needs a licensed runtime or a cloud sandbox.** Text only, ondemand skill. Source: a personal licence-budget gate written after parallel journey runs hit both — MendixMau
2324
- fix(progress-report): **the full plan tables (one table per phase plus views A and B) now sit in one closed `<details>` card below the phase cards, so a large plan no longer pushes the rest of the report off screen; the phase cards above stay the summary, and nothing is written when the project has no plan fragment.** Test: P6 in `test-progress-report-polish.sh` — MendixMau
2425
- fix(build-plan-status): **every State the build-plan skill lists (built, partial, not built, blocked) and every spelling the phase card already counts (complete, verified, in progress, not needed and the like) is now read the same way by the plan header and the phase badge, so card, header and badge give one count; blocked counts as not built and is named in the header, genuinely unreadable values are still reported as unreadable; the later-stage placeholder FAIL is relabelled PENDING only while that stage is pending in PROJECT.md, with tests that a real FAIL stays FAIL and that FAIL stays when the current stage is unknown** — MendixMau

‎project-bin/_common.sh‎

Lines changed: 132 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -569,6 +569,138 @@ find_java_exe() {
569569
esac
570570
}
571571

572+
# ---------------------------------------------------------------------------
573+
# Watcher pause and infra retry around the exec gate's mxbuild.
574+
#
575+
# Field (a migration project, 12 modules, 2026-10-16/17): the gate's mxbuild and a live
576+
# `mxcli run --local --watch` on the SAME project both write deployment/. One of the two dies
577+
# part-way ("Directory not empty: .../deployment/web", mxbuild exit 1 or 3, no errors file):
578+
# 17 of 27 failed gates, against 0 of 22 gates run in watcher-free copies of the same project.
579+
#
580+
# Chosen fix: pause the watcher for the length of the gate (SIGSTOP, then SIGCONT). mxbuild
581+
# has no option that moves the deploy target's output out of <project>/deployment, so a
582+
# separate output dir is not available; a pause needs no mxbuild support and no change to the
583+
# watcher. Only the watcher process itself is stopped. Its runtime keeps serving, and a rebuild
584+
# the watcher already started before the pause can still collide, which is what the retry below
585+
# is for. exec.sh resumes the watcher in its EXIT trap too, so an interrupted run does not leave
586+
# it stopped. A SIGKILL runs no trap, so the paused pids are also written to
587+
# .mpr-snapshots/.watcher-paused; the next exec.sh resumes them once it holds the lock. The file
588+
# is a dot FILE: the `*/` snapshot globs in restore-mpr.sh and snapshot-mpr.sh never match it.
589+
# ---------------------------------------------------------------------------
590+
MXTK_PAUSED_PIDS=""
591+
MXTK_PAUSED_FILE=""
592+
MXTK_GATE_RETRY_NOTE=""
593+
594+
# mxtk_watcher_pids <project-root> - pids of `mxcli run --local --watch` that belong to this
595+
# project. Exact match only: the watcher's project dir (the directory of its -p argument,
596+
# resolved against its cwd, or its cwd when it has no -p) must equal the root after pwd -P. A
597+
# substring test also hit a sibling `<App>-copy` and a nested `.claude/worktrees/x` watcher.
598+
mxtk_watcher_pids() {
599+
local _root _pid _cmd _cwd _p _dir
600+
_root="$(cd "$1" 2>/dev/null && pwd -P)" || return 0
601+
for _pid in $(pgrep -f 'run --local' 2>/dev/null || true); do # portability-ok: pgrep -f ships with Linux procps and macOS; Windows has no watcher to pause here
602+
[ "$_pid" = "$$" ] && continue
603+
_cmd="$(ps -o command= -p "$_pid" 2>/dev/null)" || continue
604+
case "$_cmd" in *mxcli*"run --local"*--watch*) ;; *) continue ;; esac
605+
_cwd="$(readlink "/proc/$_pid/cwd" 2>/dev/null || lsof -a -p "$_pid" -d cwd -Fn 2>/dev/null | sed -n 's/^n//p')"
606+
[ -n "$_cwd" ] && _cwd="$(cd "$_cwd" 2>/dev/null && pwd -P)"
607+
_p=""
608+
case "$_cmd" in
609+
*" -p "*) _p="${_cmd#* -p }" ;;
610+
*" --project "*) _p="${_cmd#* --project }" ;;
611+
esac
612+
if [ -n "$_p" ]; then
613+
case "$_p" in /*) ;; *) [ -n "$_cwd" ] || continue; _p="$_cwd/$_p" ;; esac
614+
# ps drops the quotes, so "-p /x/App copy/App.mpr" must not read as /x/App: keep the
615+
# longest run of words that exists on disk (it also drops any flags after the path).
616+
while [ ! -e "$_p" ]; do case "$_p" in *" "*) _p="${_p% *}" ;; *) continue 2 ;; esac; done
617+
if [ -d "$_p" ]; then _dir="$_p"; else _dir="$(dirname "$_p")"; fi
618+
_dir="$(cd "$_dir" 2>/dev/null && pwd -P)" || continue
619+
else
620+
_dir="$_cwd"
621+
fi
622+
[ -n "$_dir" ] && [ "$_dir" = "$_root" ] && echo "$_pid"
623+
done
624+
return 0
625+
}
626+
627+
# mxtk_pause_watchers <project-root> - SIGSTOP every watcher of this project, remember the pids
628+
# (in memory, and in <root>/.mpr-snapshots/.watcher-paused for a SIGKILL of this process).
629+
mxtk_pause_watchers() {
630+
local _pid _cands
631+
MXTK_PAUSED_PIDS=""
632+
MXTK_PAUSED_FILE="$1/.mpr-snapshots/.watcher-paused"
633+
_cands="$(mxtk_watcher_pids "$1" | tr '\n' ' ')"
634+
[ -n "$_cands" ] || return 0
635+
# Record before stopping: a SIGKILL between kill -STOP and the record would leave a stopped
636+
# watcher nobody resumes. Resuming a pid that was never stopped is harmless.
637+
{ mkdir -p "$1/.mpr-snapshots" && echo "$$ $_cands" > "$MXTK_PAUSED_FILE"; } 2>/dev/null || true
638+
for _pid in $_cands; do
639+
if kill -STOP "$_pid" 2>/dev/null; then
640+
MXTK_PAUSED_PIDS="$MXTK_PAUSED_PIDS $_pid"
641+
echo " → paused the run --local --watch watcher (pid $_pid) for the gate; it resumes right after" >&2
642+
fi
643+
done
644+
if [ -n "$MXTK_PAUSED_PIDS" ]; then
645+
{ mkdir -p "$1/.mpr-snapshots" && echo "$$$MXTK_PAUSED_PIDS" > "$MXTK_PAUSED_FILE"; } 2>/dev/null || true
646+
fi
647+
return 0
648+
}
649+
650+
# mxtk_resume_watchers - SIGCONT whatever mxtk_pause_watchers stopped. Safe to call twice.
651+
mxtk_resume_watchers() {
652+
local _pid
653+
for _pid in $MXTK_PAUSED_PIDS; do kill -CONT "$_pid" 2>/dev/null || true; done
654+
[ -n "$MXTK_PAUSED_FILE" ] && rm -f "$MXTK_PAUSED_FILE"
655+
MXTK_PAUSED_PIDS=""
656+
return 0
657+
}
658+
659+
# mxtk_resume_leftover <project-root> - call once the exec lock is held. A watcher left stopped
660+
# by an exec.sh that was SIGKILLed (its record is still on disk) is resumed when its pid still
661+
# exists and is still a watcher of this project; the record is then cleared.
662+
mxtk_resume_leftover() {
663+
local _f="$1/.mpr-snapshots/.watcher-paused" _rec _pid _live _n=0
664+
[ -f "$_f" ] || return 0
665+
_rec="$(head -1 "$_f" 2>/dev/null || true)"
666+
_live="$(mxtk_watcher_pids "$1")"
667+
for _pid in ${_rec#* }; do
668+
case "$_pid" in ''|*[!0-9]*) continue ;; esac
669+
if grep -Fxq -- "$_pid" <<<"$_live" && kill -CONT "$_pid" 2>/dev/null; then _n=$((_n+1)); fi
670+
done
671+
rm -f "$_f"
672+
[ "$_n" -gt 0 ] && echo " → resumed $_n watcher(s) left stopped by an exec.sh that was killed during the gate"
673+
return 0
674+
}
675+
676+
# mxtk_gate_mxbuild <mpr> <timeout-seconds> <errors-file> - the exec gate's way to run
677+
# mxbuild: mxtk_mxbuild_error_count with this project's watcher paused around it, and ONE
678+
# retry when mxbuild exits 1 or 3 and left no sign of a model problem (no CE code in its
679+
# output, nothing in the errors file). That shape is an infrastructure failure, not a verdict
680+
# on the model; a real model error always names a CE code. An errors file with content (for
681+
# example a version mismatch in errors[]) is deterministic and is not retried.
682+
# Prints nothing on stdout (callers discard the count); sets the same MXTK_MXBUILD_* globals as
683+
# mxtk_mxbuild_error_count, plus MXTK_GATE_RETRY_NOTE ("retry after infra exit" or empty).
684+
# Returns mxtk_mxbuild_error_count's code for the final attempt.
685+
mxtk_gate_mxbuild() {
686+
local _mpr="$1" _timeout="${2:-0}" _ef="$3" _rc=0
687+
MXTK_GATE_RETRY_NOTE=""
688+
mxtk_pause_watchers "${PROJECT_ROOT:-$(dirname "$_mpr")}"
689+
mxtk_mxbuild_error_count "$_mpr" "$_timeout" "$_ef" >/dev/null || _rc=$?
690+
case "${MXTK_MXBUILD_EXIT:-}" in
691+
1|3)
692+
if [ ! -s "$_ef" ] && ! grep -Eq -- 'CE[0-9]{4}' <<<"${MXTK_MXBUILD_OUT:-}"; then
693+
echo " ⚠ retry after infra exit: mxbuild exited $MXTK_MXBUILD_EXIT with no CE code and no errors file; building once more" >&2
694+
MXTK_GATE_RETRY_NOTE="retry after infra exit"
695+
: > "$_ef"
696+
_rc=0
697+
mxtk_mxbuild_error_count "$_mpr" "$_timeout" "$_ef" >/dev/null || _rc=$?
698+
fi ;;
699+
esac
700+
mxtk_resume_watchers
701+
return "$_rc"
702+
}
703+
572704
# ---------------------------------------------------------------------------
573705
# mxtk_mxbuild_error_count — run mxbuild once, print its Error-severity count.
574706
#

‎project-bin/exec.sh‎

Lines changed: 9 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -437,6 +437,7 @@ BUILD_LOG="$PROJECT_ROOT/docs/BUILD-LOG.md"
437437
GATE_STATE="not-run" # not-run | skipped | unverified | pass | fail
438438
GATE_ERRORS="?" # mxbuild error count behind a pass (0 = clean); recorded in the stamp
439439
MXB_WHY="" # mxbuild's errors[] reason when it exited without checking the model
440+
GATE_NOTE="" # "retry after infra exit" when a gate mxbuild was run twice
440441

441442
# ── Exec approval (auto records itself) ──────────────────────────────────────
442443
# bin/exec-approval.sh decides ask vs auto; the ASKING happens before exec.sh ever runs (in
@@ -490,7 +491,7 @@ HDR2
490491
# The gate cell is read from GATE_STATE rather than passed in, so no caller
491492
# can omit it — a blank cell in this table reads as "fine", which is the exact
492493
# false-green the gate exists to prevent.
493-
_detail="$2${CHANGE_CLASS:+ · change: $CHANGE_CLASS}${CR_ID:+ · cr: $CR_ID}${SCAN_NOTE:+ · $SCAN_NOTE}${HV_NOTE:+ · $HV_NOTE}$_EXEC_APPROVAL_SUFFIX${STEP_TIMES:+ · steps: $STEP_TIMES}"
494+
_detail="$2${CHANGE_CLASS:+ · change: $CHANGE_CLASS}${CR_ID:+ · cr: $CR_ID}${SCAN_NOTE:+ · $SCAN_NOTE}${GATE_NOTE:+ · $GATE_NOTE}${HV_NOTE:+ · $HV_NOTE}$_EXEC_APPROVAL_SUFFIX${STEP_TIMES:+ · steps: $STEP_TIMES}"
494495
printf '| %s | `%s` | %s | %s | %s |\n' \
495496
"$(date '+%Y-%m-%dT%H:%M:%S%z')" "$LOG_NAME" \
496497
"${GATE_STATE:-not-run}" "$1" "$_detail" >> "$BUILD_LOG"
@@ -708,7 +709,8 @@ if [ "$LOCK_TAKEN" != 1 ]; then
708709
echo " → Wait for the other exec to finish, then re-run. Nothing was written."
709710
exit 1
710711
fi
711-
trap 'exec_lock_release' EXIT
712+
trap 'mxtk_resume_watchers; exec_lock_release' EXIT
713+
mxtk_resume_leftover "$PROJECT_ROOT" # a watcher an earlier, SIGKILLed exec.sh left stopped
712714

713715
# 3. A raw `mxcli exec` elsewhere on the machine is NOT this project's business. The old guard
714716
# here (`pgrep -fl "mxcli exec"`) matched every exec on the machine, so a test run on a
@@ -1353,7 +1355,8 @@ if [ "${SKIP_BASELINE:-0}" != "1" ] && [ -x "$MXBUILD" ] && [ -x "$JAVA_EXE" ];
13531355
# can drift apart. Its own printed count is discarded here: this call site
13541356
# already has err_set/err_count/err_codes on hand and needs BASELINE_SET
13551357
# (the message set, not just a count) for the delta-gate comparison below.
1356-
mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$_BF" >/dev/null || true
1358+
mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$_BF" >/dev/null || true
1359+
[ -n "$MXTK_GATE_RETRY_NOTE" ] && GATE_NOTE="$MXTK_GATE_RETRY_NOTE"
13571360
# Studio Pro 11's mxbuild (its --help says so) writes --write-errors "only if the
13581361
# project has errors": on a clean model the mktemp'd file stays EMPTY, json.load
13591362
# fails, and this read "Model ALREADY has ? error(s) [?]" on every clean run
@@ -1633,7 +1636,8 @@ if [ -x "$MXBUILD" ] && [ -x "$JAVA_EXE" ]; then
16331636
# discarded here: this call site keeps its own file-presence/exit-code
16341637
# branching below unchanged, computing CE_COUNT via err_count once it knows
16351638
# which branch applies.
1636-
mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$ERRORS_FILE" >/dev/null || true
1639+
mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$ERRORS_FILE" >/dev/null || true
1640+
[ -n "$MXTK_GATE_RETRY_NOTE" ] && GATE_NOTE="$MXTK_GATE_RETRY_NOTE"
16371641
MXBUILD_EXIT="$MXTK_MXBUILD_EXIT"
16381642
MXBUILD_OUT="$MXTK_MXBUILD_OUT"
16391643
MXB_WHY="${MXTK_MXBUILD_WHY:-}"
@@ -1751,7 +1755,7 @@ EOF
17511755
BASE_ERRS=$(mktemp /tmp/mxbuild-base.XXXXXX)
17521756
# Same shared run+capture as the pre-flight baseline and the main gate
17531757
# above (mxtk_mxbuild_error_count, _common.sh) — one implementation.
1754-
mxtk_mxbuild_error_count "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$BASE_ERRS" >/dev/null || true
1758+
mxtk_gate_mxbuild "$MPR" "${MXTK_GATE_TIMEOUT:-0}" "$BASE_ERRS" >/dev/null || true
17551759
# Same empty-file rule as the pre-flight baseline: SP 11 mxbuild writes no
17561760
# errors file on a clean build, and that plus exit 0 is a verified 0.
17571761
if [ ! -s "$BASE_ERRS" ] && [ "$MXTK_MXBUILD_EXIT" -eq 0 ]; then

0 commit comments

Comments
 (0)