Skip to content

Commit 5689a30

Browse files
cknittclaude
andauthored
Fix Windows rewatch CI flakes from files opened without delete sharing (#9034)
* Open files with delete sharing on Windows Unix.openfile opens files on Windows without FILE_SHARE_DELETE unless asked, so while the watcher read lib/watch.lock (every 100 ms) or hashed a source, other processes could not delete or rename it. Removing the lock then failed with "Device or resource busy" and the watcher kept running; Rust's File::open shares delete by default. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GtJW7McE7BZqNZGt5VULiK Signed-off-by: Christoph Knittel <ck@cca.io> * Retry removing watch.lock and stop a watcher that ignores it A single failed rm left the watcher running into later tests, where the runner later killed it as an orphan. Retry for five seconds, and kill the watcher when it still does not exit so the failure stays in its test. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GtJW7McE7BZqNZGt5VULiK Signed-off-by: Christoph Knittel <ck@cca.io> * Diagnose Windows rewatch CI failures that leave no evidence The retained-parse test now prints watch.log and calls.log when the edit of B does not re-parse A. A watchdog in the parity step reports and stops runaway rescript/bsc processes: four Windows runners lost contact with GitHub during that step, past its timeout, without any log. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GtJW7McE7BZqNZGt5VULiK Signed-off-by: Christoph Knittel <ck@cca.io> * Serialize writes to the compiler call log Parallel domains appended to REWATCH_COMPILER_CALL_LOG through separate handles, and Windows emulates append as seek-then-write, so one domain's line could overwrite another's. The retained-parse test then missed A's parse line (and the initial build's implementation line for A) although A was parsed: the Windows "flake" in that test was a lost log line. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01GtJW7McE7BZqNZGt5VULiK Signed-off-by: Christoph Knittel <ck@cca.io> --------- Signed-off-by: Christoph Knittel <ck@cca.io> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
1 parent 096af6d commit 5689a30

12 files changed

Lines changed: 160 additions & 22 deletions

File tree

‎.github/workflows/ci.yml‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -225,6 +225,8 @@ jobs:
225225
if: ${{ !contains(matrix.ocaml_compiler, '5.0.0') }}
226226
timeout-minutes: 20
227227
run: |
228+
. rewatch-ocaml/tests/process_watchdog.sh
229+
start_process_watchdog
228230
bash rewatch-ocaml/tests/check_canonical_test_coverage.sh
229231
bash rewatch-ocaml/tests/check_config_acceptance.sh \
230232
packages/@rescript/${{ matrix.node-target }}/bin/rescript-rust.exe \
Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
- On Windows, let other programs delete or rename files while the build system reads them, so stopping a watcher by removing `lib/watch.lock` no longer fails intermittently.

‎rewatch-ocaml/ast_header.ml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
type t = {dependencies: string list; source: string option}
22

33
let read path =
4-
let descriptor = Unix.openfile path [Unix.O_RDONLY; Unix.O_CLOEXEC] 0 in
4+
let descriptor = File_util.open_for_reading path in
55
let channel = Unix.in_channel_of_descr descriptor in
66
Fun.protect
77
~finally:(fun () -> close_in_noerr channel)

‎rewatch-ocaml/build_lock.ml‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,9 @@ let publish ~candidate ~path ~contents =
3535
| Unix.Unix_error (Unix.EEXIST, _, _) -> false
3636
| Unix.Unix_error _ -> (
3737
match
38-
Unix.openfile path [Unix.O_WRONLY; Unix.O_CREAT; Unix.O_EXCL] 0o644
38+
Unix.openfile path
39+
[Unix.O_WRONLY; Unix.O_CREAT; Unix.O_EXCL; Unix.O_SHARE_DELETE]
40+
0o644
3941
with
4042
| exception Unix.Unix_error (Unix.EEXIST, _, _) -> false
4143
| descriptor ->

‎rewatch-ocaml/compiler_process.ml‎

Lines changed: 13 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -26,17 +26,24 @@ let compiler_phase args =
2626
in
2727
(phase, input)
2828

29+
(* Requests run in parallel domains. Windows emulates append mode as a seek to
30+
the end followed by a write, so two domains appending at once could write
31+
at the same offset and lose one of the lines the tests count. *)
32+
let call_log_mutex = Mutex.create ()
33+
2934
let log_compiler_request (request : request) =
3035
match Sys.getenv_opt "REWATCH_COMPILER_CALL_LOG" with
3136
| None -> ()
3237
| Some path ->
3338
let phase, input = compiler_phase request.args in
34-
let channel =
35-
open_out_gen [Open_creat; Open_append; Open_text] 0o644 path
36-
in
37-
Fun.protect
38-
~finally:(fun () -> close_out_noerr channel)
39-
(fun () -> Printf.fprintf channel "%s\t%s\t%s\n" phase request.cwd input)
39+
Process_child.with_lock call_log_mutex (fun () ->
40+
let channel =
41+
open_out_gen [Open_creat; Open_append; Open_text] 0o644 path
42+
in
43+
Fun.protect
44+
~finally:(fun () -> close_out_noerr channel)
45+
(fun () ->
46+
Printf.fprintf channel "%s\t%s\t%s\n" phase request.cwd input))
4047

4148
let exit_code = function
4249
| Unix.WEXITED code -> code

‎rewatch-ocaml/file_util.ml‎

Lines changed: 12 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -27,8 +27,15 @@ let ensure_dir path =
2727
in
2828
loop path
2929

30+
let open_for_reading path =
31+
(* Windows denies deleting or renaming a file while any handle lacks
32+
FILE_SHARE_DELETE, and Unix.openfile only adds it on request. Without it
33+
the watcher's periodic watch.lock read can make the shutdown protocol's
34+
rm fail, and hashing a source can block an editor's atomic rename. *)
35+
Unix.openfile path [Unix.O_RDONLY; Unix.O_CLOEXEC; Unix.O_SHARE_DELETE] 0
36+
3037
let read_file path =
31-
let descriptor = Unix.openfile path [Unix.O_RDONLY; Unix.O_CLOEXEC] 0 in
38+
let descriptor = open_for_reading path in
3239
let channel = Unix.in_channel_of_descr descriptor in
3340
Fun.protect
3441
~finally:(fun () -> close_in_noerr channel)
@@ -54,7 +61,7 @@ let is_regular_file path =
5461
| exception Unix.Unix_error ((Unix.ENOENT | Unix.ENOTDIR), _, _) -> false
5562

5663
let digest_file path =
57-
let descriptor = Unix.openfile path [Unix.O_RDONLY; Unix.O_CLOEXEC] 0 in
64+
let descriptor = open_for_reading path in
5865
let channel = Unix.in_channel_of_descr descriptor in
5966
Fun.protect
6067
~finally:(fun () -> close_in_noerr channel)
@@ -117,9 +124,7 @@ let copy_buffer = Domain.DLS.new_key (fun () -> Bytes.create 65_536)
117124
let compare_buffer = Domain.DLS.new_key (fun () -> Bytes.create 65_536)
118125

119126
let copy_existing_file source destination =
120-
let input_descriptor =
121-
Unix.openfile source [Unix.O_RDONLY; Unix.O_CLOEXEC] 0
122-
in
127+
let input_descriptor = open_for_reading source in
123128
let input_channel = Unix.in_channel_of_descr input_descriptor in
124129
Fun.protect
125130
~finally:(fun () -> close_in_noerr input_channel)
@@ -166,16 +171,12 @@ let files_equal first second =
166171
| Some second_stat ->
167172
first_stat.Unix.st_size = second_stat.Unix.st_size
168173
&&
169-
let first_descriptor =
170-
Unix.openfile first [Unix.O_RDONLY; Unix.O_CLOEXEC] 0
171-
in
174+
let first_descriptor = open_for_reading first in
172175
let first_channel = Unix.in_channel_of_descr first_descriptor in
173176
Fun.protect
174177
~finally:(fun () -> close_in_noerr first_channel)
175178
(fun () ->
176-
let second_descriptor =
177-
Unix.openfile second [Unix.O_RDONLY; Unix.O_CLOEXEC] 0
178-
in
179+
let second_descriptor = open_for_reading second in
179180
let second_channel = Unix.in_channel_of_descr second_descriptor in
180181
Fun.protect
181182
~finally:(fun () -> close_in_noerr second_channel)

‎rewatch-ocaml/file_util.mli‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
val path_of_parts : string -> string list -> string
22
val is_directory : string -> bool
33
val ensure_dir : string -> unit
4+
val open_for_reading : string -> Unix.file_descr
45
val read_file : string -> string
56
val is_regular_file : string -> bool
67
val digest_file : string -> Digest.t
Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
# Source this file and call start_process_watchdog to run the rest of the
2+
# shell under windows_process_watchdog.ps1. It does nothing on other systems.
3+
# The shell exits with status 1 when the watchdog had to stop processes.
4+
5+
start_process_watchdog() {
6+
case "${RUNNER_OS:-}" in
7+
Windows) ;;
8+
*) return 0 ;;
9+
esac
10+
watchdog_dir=$(mktemp -d)
11+
export WATCHDOG_RUNNING="$watchdog_dir/running"
12+
export WATCHDOG_REPORT="$watchdog_dir/report"
13+
: >"$WATCHDOG_RUNNING"
14+
powershell.exe -NoProfile -NonInteractive -ExecutionPolicy Bypass \
15+
-File "$(dirname "${BASH_SOURCE[0]}")/windows_process_watchdog.ps1" &
16+
trap stop_process_watchdog EXIT
17+
}
18+
19+
stop_process_watchdog() {
20+
local status=$?
21+
rm -f "$WATCHDOG_RUNNING"
22+
if [ -s "$WATCHDOG_REPORT" ]; then
23+
echo "::error::The process watchdog stopped runaway test processes:"
24+
cat "$WATCHDOG_REPORT"
25+
status=1
26+
fi
27+
rm -rf "$watchdog_dir"
28+
exit "$status"
29+
}

‎rewatch-ocaml/tests/run.sh‎

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1940,8 +1940,15 @@ while [ "$quiet_polls" -lt 10 ]; do
19401940
done
19411941
reported_errors=$(grep -c 'This let-binding misses an expression' "$retained_parse/watch.log")
19421942
printf 'let other = 2\n' >"$retained_parse/src/B.res"
1943-
wait_for_count "$retained_parse/calls.log" '^parse.*A\.res' $((failed_parses + 1))
1944-
wait_for_text "$retained_parse/watch.log" 'This let-binding misses an expression' $((reported_errors + 1))
1943+
if ! wait_for_count "$retained_parse/calls.log" '^parse.*A\.res' $((failed_parses + 1)) ||
1944+
! wait_for_text "$retained_parse/watch.log" 'This let-binding misses an expression' $((reported_errors + 1)); then
1945+
echo "--- watch.log" >&2
1946+
cat "$retained_parse/watch.log" >&2
1947+
echo "--- calls.log" >&2
1948+
cat "$retained_parse/calls.log" >&2
1949+
echo "the edit of B did not parse and report the pending A again (expected $((failed_parses + 1)) parses, $((reported_errors + 1)) reports)" >&2
1950+
exit 1
1951+
fi
19451952
grep 'value = 1' "$retained_parse/src/A.js" >/dev/null
19461953
printf 'let value = 2\n' >"$retained_parse/src/A.res"
19471954
wait_for_text "$retained_parse/src/A.js" 'value = 2'
Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
# Stops runaway test processes before they starve the CI runner. Hosted
2+
# Windows runners that run out of memory lose contact with GitHub and leave no
3+
# log at all, so report the largest processes and stop the test processes
4+
# while the runner can still upload the log.
5+
#
6+
# Runs until the file named by WATCHDOG_RUNNING is removed. Writes its report
7+
# to the file named by WATCHDOG_REPORT.
8+
9+
$names = 'rescript', 'rescript-rust', 'bsc'
10+
$perProcessLimit = 3GB
11+
$countLimit = 64
12+
$minimumFree = 1.5GB
13+
14+
function Describe-Processes($processes) {
15+
$processes |
16+
Sort-Object WorkingSet64 -Descending |
17+
Select-Object -First 15 |
18+
ForEach-Object {
19+
$process = Get-CimInstance Win32_Process -Filter "ProcessId=$($_.Id)" -ErrorAction SilentlyContinue
20+
'{0,7} parent {1,7} {2,8:N0} MB {3,9:N1} s CPU {4}' -f `
21+
$_.Id, $process.ParentProcessId, ($_.WorkingSet64 / 1MB), $_.CPU, `
22+
$(if ($process.CommandLine) { $process.CommandLine } else { $_.ProcessName })
23+
}
24+
}
25+
26+
while (Test-Path -LiteralPath $env:WATCHDOG_RUNNING) {
27+
Start-Sleep -Seconds 5
28+
$tested = @(Get-Process -Name $names -ErrorAction SilentlyContinue)
29+
if ($tested.Count -eq 0) { continue }
30+
$free = (Get-CimInstance Win32_OperatingSystem).FreePhysicalMemory * 1KB
31+
$reason = $null
32+
$large = @($tested | Where-Object { $_.WorkingSet64 -gt $perProcessLimit })
33+
if ($large.Count -gt 0) {
34+
$reason = "process $($large[0].Id) uses $([int]($large[0].WorkingSet64 / 1MB)) MB"
35+
} elseif ($tested.Count -gt $countLimit) {
36+
$reason = "$($tested.Count) rescript/bsc processes are running"
37+
} elseif ($free -lt $minimumFree) {
38+
$reason = "free memory is down to $([int]($free / 1MB)) MB"
39+
}
40+
if ($reason) {
41+
$report = @("Process watchdog: $reason; stopping rescript/bsc processes.") +
42+
'Largest processes:' + (Describe-Processes (Get-Process)) +
43+
'Stopped test processes:' + (Describe-Processes $tested)
44+
$report | Add-Content -LiteralPath $env:WATCHDOG_REPORT
45+
$report | Write-Host
46+
$tested | Stop-Process -Force -ErrorAction SilentlyContinue
47+
}
48+
}

0 commit comments

Comments
 (0)