[MendixMau]

Direct lane

What changed and why (one paragraph):
Before this change, a stale lock was taken over under one <lock>.takeover file. That file went stale by pid or by age, and it was moved aside and put back with ln. If a third contender created a new takeover file in that gap, the ln failed and two contenders were inside the takeover at once. One contender's compare-then-rm could then delete the other's fresh lock (r-eng-backlog item 1). An earlier fix that re-checked our own line just before the rm was rejected, because it only narrowed the window. This PR replaces the takeover file with reap tickets:

• To reap a dead holder's line X, a contender creates <lock>.reap.<cksum X>.<N>. The ticket is written to <lock>.tmp.<pid> and then lned into place, so it is exclusive and never empty.
• Ticket N+1 is allowed only when ticket N's pid is dead, never by age.
• Only the holder of the highest ticket re-reads the lock and runs rm, and only if the lock still holds exactly X. It then creates its own lock exclusively.
• A new holder deletes the tickets of every other key.
• Lock lines gain a nonce= field, so a line can never come back. This is why the compare-then-rm cannot hit a fresh lock: while the lock holds X, only one live process can be past the ticket check for X.

The code now lives once, in project-bin/_common.sh, as mxtk_lock_take / mxtk_lock_release / mxtk_lock_live (+ mxtk_lock_fields, mxtk_lock_blocked_text). It is pure bash 3.2 with POSIX tools and no flock. It replaces exec.sh's exec_lock_* and test-scratch.sh's take_over/holder_live. The exec.sh diff is only the call sites plus the older-_common.sh guard (the app-restart.sh pattern), so it should not collide much with #355, the exec-on-copy PR or the baseline-lock PR.

Review fixes (second commit, 81981f2):

1. An empty stale line is not unique. The old code, and the no-hard-link fallback, create the lock empty and write it afterwards. A fresh empty lock could therefore replace the stale empty one while a reaper sat between compare and rm, and the reaper would delete it. Now, before the rm, an empty X must still be more than a minute old (find -mmin +1). The new case in test-lock-reap.sh swaps in a fresh empty lock at the compare anchor. It fails on a677671 and passes now.
2. Naming the blocker. When the take gives up (rc 2) because a live reaper's ticket, or a live old-style .takeover, never finished, MXTK_LOCK_BLOCKER names the pid and file. That covers a SIGSTOPped reaper, or a reused pid where ps -o args= cannot tell processes apart. mxtk_lock_blocked_text then says to resume the process (kill -CONT <pid> / fg) or end it (kill <pid>). exec.sh and test-scratch.sh print this on rc 2.
3. mxtk_pid_args not used. Reusing mxtk_pid_args in the liveness check was left out on purpose. On Windows it runs PowerShell, about a second per call, inside a retry loop. On Linux it gives the same answer as ps -o args= for this match. So it does not come out clean.

Mixed versions:

• The lock path, the line format (the nonce is a trailing field) and the liveness rule are unchanged, so old and new code each respect the other's live holders.
• The new code also waits while an old-style .takeover is live, and clears a stale one once it holds the lock.
• Neither version can deadlock the other: test-lock-reap.sh runs 9 old-block and 9 new contenders against one dead holder, and all 18 acquire.
• An old and a new reaper of the same line are still not serialised against each other. That was already the case on master, and it lasts only while both versions are installed.

Test anchors: MXTK_LOCK_TEST_PAUSE names a command that runs as <cmd> <anchor> <pid> at four points: ticket-taken, compare, rm and created. When it is unset it does nothing. MXTK_LOCK_RETRY_SLEEP sets the pause between attempts (default 1 s).

Regression harness: tests/wave2/bench-lock-race.sh.

• Each round starts 10 contenders together on a lock whose holder is dead.
• Contenders pause 0–39 ms at every anchor, are SIGKILLed at ticket-taken/compare/rm (25% each), and are also SIGKILLed at random 0–50 ms offsets (10%).
• Every holder checks two things: that no other live holder has a mark down, and that the lock file still names it.
• After each round, a probe with no hooks must acquire the lock within 15 s.
• --old runs master's block, the verbatim copy in fixtures/mxtest-guard/exec-lock-block.sh, identical to d16f578. Anchors are sed-injected at the same points, plus judged and aside (the old move-aside step).

Measured, Linux, 200 rounds × 10 contenders, same settings (first commit):

subject	overlapping holders	locks taken from a live holder	starved	probe failures	SIGKILLs	acquisitions
master's block (--old)	13 events in 6 rounds	12	0	0	664	1538
reap tickets	0	0	0	0	547	1653

The reap-ticket run left 0 files beside the lock at the end.

After the review fixes, 60 rounds × 10 contenders:

SIGKILL anchors	overlaps	stolen	starved	probe failures	SIGKILLs	acquisitions	files left
default	0	0	0	0	164	496	0
all four (including created)	0	0	0	0	328	333	—

Fixtures:

• tests/wave2/test-lock-reap.sh (45/45) covers each rule separately: the line format and nonce; release removes only our own line; a live holder refuses; every old and new line format is reaped; a live ticket dated 2000 still blocks (no age rule) and names its pid, file and remedy; dead tickets 1 and 2 lead to ticket 3; the line changing at the compare anchor; a higher ticket appearing stops the rm; a reaper SIGKILLed at rm does not block the next; GC; an empty lock; an empty lock swapped in mid-reap; legacy .takeover interop; mixed-version no-deadlock; and a 15-round harness run.
• test-shared-lock-takeover.sh and test-test-scratch.sh now source _common.sh instead of awk-lifting the lock code.

Field runs

Platform	Result	New code, 200 × 10	--old (master's block)	test-lock-reap	Source
Linux	PASS	overlaps 0, stolen 0, starved 0, probe failures 0	13 overlaps in 6 rounds, 12 stolen	45/45	this PR (tables above)
macOS 26.6 arm64, bash 3.2.57	PASS	overlaps 0, stolen 0, starved 0, probe failures 0	21 overlaps in 9 rounds, 18 stolen	45/45	issue #390
Git Bash (Windows)	owed	—	—	—	—

Field evidence: no consuming-project run; this is concurrency code, and the proof is the harness. The macOS run covers APFS hard links, ps -o args= on a macOS zombie and fractional sleep. Still owed before merge: one Git Bash run of bash tests/wave2/bench-lock-race.sh (and --old) to cover NTFS hard links and kill -0 alone deciding liveness. That is why this PR stays a draft.

Merge with master (7f4169d): origin/master (24fe336) is merged in with a merge commit. The end-of-file conflict in project-bin/_common.sh with #370's mxtest helpers was resolved by keeping both sections. After that, c932fa1 clears MXTK_LOCK_BLOCKER at the start of every take attempt (review nit), so it describes only the last attempt.

• No client data anywhere in the diff
• Size cap: 9 files, about 400 non-fixture lines
• Test tier: T1, fixtures plus the race harness on Linux and macOS. Fixtures run on c932fa1 (after the merge): test-lock-reap 45/45, test-shared-lock-takeover 11/11, test-test-scratch 52/52, test-exec-lock-watch 84/84, test-exec-change-class 21/21, test-bug07-08 43/43, test-app-restart 69/69; bench 30 rounds × 10: 0 overlaps, 0 stolen, 0 starved, 0 probe failures; check-portability clean; check-scripts 114/114
• Both platforms: macOS PASS (Field runs 2026-10-15: #379 PASS, #384 PASS (sync --upgrade-bin not scoped; copy 87-134 s), #374 PASS (.bak in mprcontents) #390); Git Bash run still owed
• CHANGELOG.md line appended, credited
• Baseline words: 73,522 before and after on current master (no skill text changed)

Impact

Row	Effect	Evidence
Process	None new. When a take gives up on a live reaper ticket (rc 2), exec.sh and test-scratch.sh now name the blocking pid/file and the resume-or-kill remedy; the same condition blocked before. A pid that is not an exec.sh / test-scratch.sh (reused) is never told to be killed: its remedy is rm -- "<ticket>" (79850b9). An empty lock under a minute old now says so instead of "another session keeps re-taking it". Draft until the macOS + Git Bash bench runs exist.	PR text
Dependencies	Shared files: project-bin/_common.sh (new mxtk_lock_*), project-bin/exec.sh (call sites), project-bin/test-scratch.sh. The end-of-file conflict with master's mxtest-identity section in project-bin/_common.sh is resolved by merge 7f4169d (both sections kept; fixtures green on c932fa1). Also conflicts with #384 in _common.sh; exec.sh merges clean with #374 and #375. Platform: bash 3.2 + POSIX; relies on ln hard links, ps -o args=, find -mmin, fractional sleep; macOS + Git Bash runs owed. Reads only its own lock and ticket files.	git merge-tree vs 868b3cb (CONFLICT in _common.sh) and pairwise
Context cost	Unchanged: baseline 73,522 → 73,522 words (budget 80,000); Stage 5 mdl 30,185 → 30,185, lead 25,820 → 25,820	bin/render-routing.sh --check on master 868b3cb vs this head (81981f2) merged onto 868b3cb via git merge-tree in a temporary worktree (conflict markers left in project-bin/_common.sh, which render-routing does not read)
Behaviour	n/a (scripts only).	diff touches no skill or agent stub
Runtime	Lock bench, 200 rounds × 10 contenders: 0 overlapping holders on the new code vs 13 on master's block (Linux) and vs 21 (macOS, #390). Uncontended take cost: NOT RUN (not measured). Contended retry pause MXTK_LOCK_RETRY_SLEEP default 1 s.	tests/wave2/bench-lock-race.sh results in the PR

Update 2026-11-03

Merge: origin/master (dbd9b3e, 21 commits past the last merge base 6615869, incl. #587/#590 exec speed work, #596/#597/#603/#606) merged into this branch as 91d08ff. No rebase, no force-push. The PR's diff against master is still the same 9 files.

Conflicts and how they were resolved:

• project-bin/exec.sh, lock block: master had edited the old exec_lock_* block this PR removes, switching $(date +%s) to ${EPOCHSECONDS:-$(date +%s)} (exec: ~60% fewer process starts per exec (Windows overhead, #584/#585) #590, fewer process starts) and mkdir -p "${LOCK%/*}" (merged cleanly). Resolution: the PR's mxtk_lock_take call sites stay; master's intent is carried over by using ${EPOCHSECONDS:-$(date +%s)} for the held-for age in exec.sh and in the shared lock in _common.sh (_mxtk_lock_legacy_busy, the nonce, the lock line). Under bash 3.2, which has no EPOCHSECONDS, it falls back to date.
• project-bin/exec.sh, EXIT trap: master added the MXTK_PROFILE report. The trap is now mxtk_resume_watchers; mxtk_lock_release "$LOCK"; [ "$_PROFILING" != 1 ] || _prof_report.
• CHANGELOG.md: both sides' Unreleased lines kept. This PR's line stays at the top and now notes the merge.
• project-bin/_common.sh and project-bin/test-scratch.sh merged without conflicts. Master did not change the lock logic itself, so nothing in this PR was dropped as already on master. I diffed master's current exec.sh lock block against fixtures/mxtest-guard/exec-lock-block.sh: the only differences are the EPOCHSECONDS spellings and comments.

Harness fix: bench-lock-race.sh now makes a relative --common path absolute. Before, the contenders, which run from $WORK, could not source it, and every probe failed.

Test evidence, Linux, bash 5.2, on the merged tree (scoped fixtures only, no full suites):

• test-lock-reap 45/45
• test-shared-lock-takeover 11/11
• test-test-scratch 52/52
• test-exec-lock-watch 84/84 (includes the two-tree app/ layout)
• test-bug07-08 48/48
• test-app-restart 96/96
• test-mdl-scan 53/53
• test-exec-gate-kind 32/32
• test-exec-hash-verify 49/49
• test-model-scope 50/51: the one failure, "partial restore", fails the same way on unmodified origin/master in this root container (a chmod-based read-only case). It is not caused by this PR.
• bench-lock-race.sh with the new code, 200 rounds × 10 contenders: 0 overlaps, 0 stolen, 0 starved, 0 probe failures, 517 SIGKILLs, 1684 acquisitions, 0 files left.
• bench-lock-race.sh --old against master's current exec.sh lock block (extracted from dbd9b3e), 40 × 10: 9 overlaps in 4 rounds, 6 stolen, exit 1. The race reproduces on master and is gone with this PR.
• check-portability: clean (339 files). check-scripts: 126/126 parse.
• render-routing --check: baseline 64,252 words, unchanged (no skill text touched).

Still owed: the Git Bash run. This environment has no bash 3.2, so the 3.2 evidence is still the macOS run in #390; the EPOCHSECONDS fallback is a plain ${var:-default}.

Review fixes (79850b9):

1. The bench could not see overlaps on Git Bash. The contender's alive() in bench-lock-race.sh treated empty ps -o args= output as dead, so where ps -o does not exist every overlap was invisible. It now uses the lock's own rule: case "$args" in ""|*contender-exec.sh*) return 0 ;;. Proof with a fake ps on PATH that prints nothing and exits 1 (Git Bash simulated): --old 40 × 10 reported overlaps=0 (stolen=5) before the fix and overlaps=10 in 5 rounds (stolen=9) after; two more --old runs gave 5 and 2 overlaps. The new code under the same fake ps, 100 × 10: 0 overlaps, 0 stolen.
2. No kill for a reused pid. mxtk_lock_blocked_text told the user to kill $p when the pid was "not an exec.sh / test-scratch.sh at all (a reused pid)", which can kill an unrelated process. The message now has two separate cases, after a ps -o args= -p <pid> check: (a) it IS a stopped or hung exec.sh / test-scratch.sh reaper — resume it (kill -CONT) or end it (kill), and leave its ticket alone; (b) it is NOT one — do not kill it, delete the dead reaper's ticket: rm -- "<ticket file>". A live reaper's ticket is never offered for deletion.
3. Empty crashed lock. When the lock file is empty and under a minute old for every attempt, mxtk_lock_take sets MXTK_LOCK_BLOCKER="empty <lock>". exec.sh and test-scratch.sh then say the lock file is empty (a crashed take) and to re-run in a minute or delete it (rm -- "<lock>"), instead of exec.sh's "another session keeps re-taking it".
4. GC cost. _mxtk_lock_gc tests the <lock>.reap.* glob first and computes the cksum/awk key only when a ticket exists, so an uncontended take starts no extra process for it.

New fixture assertions (all FAIL on 91d08ff, PASS on 79850b9): test-lock-reap.sh +6 (the ticket rm -- remedy, kill and rm in separate cases, no kill for a reused pid; blocker=empty, the empty-lock remedy), test-exec-lock-watch.sh +5 (exec.sh on a fresh empty lock: refuses, says "lock file is empty", names rm -- "<lock>", no "keeps re-taking it", nothing executed). On 91d08ff with the new fixtures: test-lock-reap 44/49, test-exec-lock-watch 86/89.

Test evidence on 79850b9, Linux, bash 5.2 (scoped fixtures only):

fixture / run	normal	fake ps on PATH (Git Bash simulated)
test-lock-reap (incl. 15 × 10 harness)	50/50	50/50
test-shared-lock-takeover	11/11	11/11
test-test-scratch	52/52	52/52
test-exec-lock-watch	89/89	89/89 (reused-pid case skips, as on Git Bash)
test-bug07-08	48/48	48/48
bench, new code, 100 × 10	overlaps 0, stolen 0, starved 0, probe_fail 0, 278 kills, 822 acquired, 0 files left	overlaps 0, stolen 0, starved 0, probe_fail 0, 194 kills, 906 acquired, 0 files left
bench --old, 40 × 10	overlaps 5 (3 rounds), stolen 6	overlaps 10 (5 rounds), stolen 9; reruns 5 and 2 overlaps (before the fix: 0)

check-portability: clean (339 files). check-scripts: 126/126 shell, 27/27 Node parse.

Git Bash run (owed before merge)

Run on Windows in Git Bash, from a checkout of this branch (79850b9 or later), with a real project at hand:

cd <toolkit-checkout>          # on fix/lock-reap-tickets
git log -1 --format=%h         # 79850b9 or later

# (a) new code: overlaps=0 stolen=0 starved=0 probe_fail=0, "lock-side files left at the end: 0", exit 0
bash tests/wave2/bench-lock-race.sh --rounds 200 --contenders 10; echo "exit=$?"
#     if that is too slow, at least:
bash tests/wave2/bench-lock-race.sh --rounds 100 --contenders 10; echo "exit=$?"

# (b) old code on the same machine: overlaps>=1 or stolen>=1 (exit 1)
bash tests/wave2/bench-lock-race.sh --old --rounds 40 --contenders 10; echo "exit=$?"

# (c) hard links in a real .mpr-snapshots dir: prints 2
cd <project-root>/.mpr-snapshots && echo probe > lnprobe.a && ln lnprobe.a lnprobe.b && stat -c %h lnprobe.a; rm -f lnprobe.a lnprobe.b; cd -

# (d) the lock fixtures: each ends PASS=<n> FAIL=0
bash tests/wave2/test-lock-reap.sh
bash tests/wave2/test-shared-lock-takeover.sh
bash tests/wave2/test-test-scratch.sh project-bin/test-scratch.sh
bash tests/wave2/test-exec-lock-watch.sh

• (a) new code ≥200 × 10 (≥100 × 10 if slow): overlaps=0 stolen=0 starved=0 probe_fail=0, 0 files left
• (b) --old on the same machine: ≥1 overlap or stolen lock
• (c) ln a b in a real .mpr-snapshots dir: link count 2
• (d) test-lock-reap, test-shared-lock-takeover, test-test-scratch, test-exec-lock-watch: FAIL=0

Review: pending independent review

---

Generated by Claude Code

[MendixMau]

After 1.0: master still uses the takeover-file lock from #304, so the race this closes is still there. This needs a rebase onto the current exec.sh.