fix: retry atomic journal opens and preserve supervisor diagnostics

Author: Sunrisepeak

.agents/docs/2026-11-06-compiler-attempt-recovery.md

@@ -245,6 +245,20 @@ in-range ticks. A direct exact-toolchain control compiled the failing expression
 and the corrected one, including a real file identity. Native Clang, ARM and
 bootstrap jobs still require the next full CI run.
 
+The next cold GCC run failed at the owner witness without a branch-specific
+diagnostic. Code review found a separate record-publication observation race:
+a reader's first open can report absence while an atomic publisher installs
+the record before the following existence probe. Treating this pair as a
+corrupt record can falsely fail the owner witness. A libc open-boundary control
+reproduced the old failure and the corrected retry with identical published
+bytes. The reader retries one open; a persistent refusal still fails. Witness
+diagnostics now distinguish an invalid record, an unreadable attempt and a
+controller without retirement so the next CI can test this attribution. The
+native Windows suite also showed that
+`file_size` can report a directory's size; output identity now requires a
+regular leaf before reading size and timestamp. The Windows negative test and
+the cold GCC build remain final-CI gates.
+
 An actual split-compiler control held a previously absent `.rc` observation
 until the controller had published status and exited. The earlier engine failed
 with `compiler controller ended without a status`; the corrected engine rereads
@@ -265,6 +279,15 @@ OS 0.3.0. The older consumer platform selected by the application's own
 target remains independent of the launcher's build platform. The actual
 cross-platform packing control and complete CI remain release gates.
 
+The cross-host Windows→Linux job still reached a missing final status after
+the controller exited. Because detached supervisor stdio was discarded, the
+first CI did not retain whether unit retirement, certificate publication or
+status publication failed. The supervisor now appends a bounded reason to its
+existing compiler log at each of those exits; a waiter replays that log only
+when the controller is gone and status remains absent. This changes the
+diagnostic boundary, not the success criteria. The next Windows cross job must
+identify the cause before any protocol or OS change is attributed to it.
+
 ## Separate Windows build-program defect
 
 The released 2026.11.5.1 engine can try to install `msvc@system` as an indexed

CHANGELOG.md

@@ -14,6 +14,10 @@
   platform, so their generated dependency resolves alongside openkal-platform 0.24.
 - Compiler attempt journals preserve file timestamps on Clang/libc++ when the
   file clock uses a wider integer representation.
+- Compiler journals retry an open that raced with atomic record publication,
+  and BMI/object identity checks require a regular output file on every host.
+  A witness failure identifies its failed observation boundary, and a detached
+  supervisor's failure is available to the waiter even though its stdio is closed.
 - Launch descriptions include the selected target, profile, resolved platform
   and available named runners, including the reachable leg of a target set.
   Editor debug launches can retain the configuration that actually built the program.

src/build/schedule/compiler_transaction.cppm

@@ -44,7 +44,12 @@ Record read(const fs::path& path, std::string_view tag) {
     if (!input) {
         std::error_code ec;
         if (!fs::exists(path,ec) && !ec) return std::optional<Fields>{};
-        return std::unexpected("cannot read compiler transaction record");
+        if(ec)return std::unexpected("cannot read compiler transaction record");
+        // An atomic publisher may install the record between the failed open
+        // and the existence probe. Retry once; a persistent refusal still
+        // fails rather than being mistaken for an absent record.
+        input.open(path,std::ios::binary);
+        if(!input)return std::unexpected("cannot read compiler transaction record");
     }
     std::string bytes(65537,'\0');
     input.read(bytes.data(),bytes.size());bytes.resize(static_cast<std::size_t>(input.gcount()));
@@ -99,7 +104,10 @@ std::optional<long long> journal_time(fs::file_time_type time) {
 }
 std::optional<Fields> identity(const fs::path& path) {
     if(path.empty())return Fields{"", ""};
-    std::error_code ec;const auto size=fs::file_size(path,ec);if(ec)return {};
+    std::error_code ec;
+    const auto status=fs::symlink_status(path,ec);
+    if(ec||status.type()!=fs::file_type::regular)return {};
+    const auto size=fs::file_size(path,ec);if(ec)return {};
     const auto time=fs::last_write_time(path,ec);if(ec)return {};
     const auto ticks=journal_time(time);if(!ticks)return {};
     return Fields{std::to_string(size),std::to_string(*ticks)};
@@ -377,21 +385,27 @@ private:
     std::atomic<bool> stopping_{false},failed_{false};
     std::mutex admission_;
     ObserverMap held_;
+    void fail(std::string_view reason) {
+        if(!failed_.exchange(true))
+            std::println(std::cerr,"mcpp: compiler witness: {}",reason);
+    }
     void audit() {
         for(auto it=held_.begin();it!=held_.end();) {
             const auto directory=named(it->first);
             auto retired=read(directory/"retired","mcpp-retired-v1");
-            if(!retired||(*retired&&(**retired).size()!=3)) {failed_.store(true);++it;continue;}
+            if(!retired) {fail(retired.error());++it;continue;}
+            if(*retired&&(**retired).size()!=3) {fail("invalid retirement certificate");++it;continue;}
             if(*retired) {it=held_.erase(it);continue;}
             auto attempt=Attempt::load(directory);
-            if(!attempt) {failed_.store(true);++it;continue;}
+            if(!attempt) {fail(attempt.error());++it;continue;}
             if(!controller_alive(*attempt)) {
                 // Retirement can be published between the first read and the
                 // controller's exit. Observe it again before reporting loss.
                 retired=read(directory/"retired","mcpp-retired-v1");
-                if(!retired||(*retired&&(**retired).size()!=3))failed_.store(true);
+                if(!retired)fail(retired.error());
+                else if(*retired&&(**retired).size()!=3)fail("invalid retirement certificate");
                 else if(*retired) {it=held_.erase(it);continue;}
-                else failed_.store(true);
+                else fail("controller ended without retirement certificate");
             }
             ++it;
         }

src/build/schedule/detach_codegen.cppm

@@ -533,6 +533,18 @@ int run_to_completion(std::string_view command,
     return owner.active() ? r.exit_code : 130;
 }
 
+void record_supervisor_error(const std::filesystem::path& slot,std::string_view reason) {
+    // Detached supervisors have null stdio. Keep the failure beside the
+    // compiler's ordinary log so either waiter can report why status is absent.
+    std::ofstream log(suffixed(slot,".log"),std::ios::app);
+    if(log)log << "mcpp: " << reason << '\n';
+}
+
+void replay_supervisor_log(const std::filesystem::path& slot) {
+    std::ifstream log(suffixed(slot,".log"));
+    if(log)std::cerr << log.rdbuf();
+}
+
 std::expected<void,std::string> retire_owned(mcpplibs::os::process::owned_unit& unit,const Owner& owner) {
     const auto limit=std::chrono::steady_clock::now()+std::chrono::seconds(5);
     for(;;) {
@@ -812,6 +824,7 @@ int compile_release_at_bmi(const CompileRequest& req) {
                 // The final status can be published between the first read and
                 // this liveness observation. The next iteration validates it.
                 if(read_rc(actual.slot))continue;
+                replay_supervisor_log(actual.slot);
                 std::println(std::cerr,"mcpp: compiler controller ended without a final status; retaining its outputs until unit recovery");
                 return 1;
             }
@@ -872,10 +885,13 @@ int supervise(const std::filesystem::path& slot,
         }
         const int rc=run_to_completion(command,suffixed(slot,".log"),actor->owner,unit.get());
         if(auto retired=retire_owned(*unit,actor->owner);!retired) {
-            std::println(std::cerr,"mcpp: {}",retired.error());return 1;
+            record_supervisor_error(slot,retired.error());return 1;
+        }
+        if(!attempt.retired(rc)) {record_supervisor_error(slot,"cannot publish compiler retirement certificate");return 1;}
+        if(!write_rc(slot,rc)) {record_supervisor_error(slot,"cannot publish compiler final status");return 1;}
+        if(!transactions::release_reservation(permit->reservation,attempt.directory)) {
+            record_supervisor_error(slot,"cannot release compiler reservation");return 1;
         }
-        if(!attempt.retired(rc)||!write_rc(slot,rc))return 1;
-        if(!transactions::release_reservation(permit->reservation,attempt.directory))return 1;
         return 0;
     }
     auto settled = std::make_shared<std::atomic<bool>>(false);
@@ -956,6 +972,7 @@ int await_unit(const std::filesystem::path& slot, const std::filesystem::path& o
                 // A controller may publish its status and exit after the read
                 // in the loop condition. Let the normal result path inspect it.
                 if((rc=read_rc(slot)))break;
+                replay_supervisor_log(slot);
                 std::println(std::cerr,"mcpp: compiler controller ended without a status; unit recovery will retain or invalidate its outputs");
                 return 1;
             }

tests/e2e/helpers/record_open_appearance.c

@@ -0,0 +1,24 @@
+#define _GNU_SOURCE
+#include <dlfcn.h>
+#include <errno.h>
+#include <stdio.h>
+#include <string.h>
+
+/* Simulate an atomic publish between the reader's first open and its
+ * existence probe. The real completed record remains present throughout. */
+static FILE *(*native_open)(const char *, const char *);
+static int injected;
+
+FILE *fopen64(const char *path, const char *mode) {
+    if (!native_open) native_open = dlsym(RTLD_NEXT, "fopen64");
+    if (!injected && strcmp(path, "record") == 0 && mode[0] == 'r') {
+        injected = 1;
+        FILE *marker = native_open("record-open-interposed", "wb");
+        if (marker) fclose(marker);
+        errno = ENOENT;
+        return NULL;
+    }
+    return native_open(path, mode);
+}
+
+FILE *fopen(const char *path, const char *mode) { return fopen64(path, mode); }

tests/unit/test_compiler_transaction.cpp

@@ -227,6 +227,32 @@ TEST(CompilerTransaction, RecordCloseFailureDriver) {
     EXPECT_EQ(Transaction::contents("record"),"previous");
     EXPECT_TRUE(fs::exists("record-close-refused"));
 }
+TEST(CompilerTransaction, RecordOpenAppearanceDriver) {
+    if(!mcpp::invocation::get("MCPP_TEST_RECORD_OPEN_DRIVER"))GTEST_SKIP()<<"subprocess driver";
+    auto record=tx::read("record","schema");
+    EXPECT_TRUE(fs::exists("record-open-interposed"));
+    ASSERT_TRUE(record&&*record);
+    EXPECT_EQ((**record)[0],"published");
+}
+TEST(CompilerTransaction, RecordAppearanceBetweenOpenAndExistsIsNotCorruption) {
+    if constexpr (!mcpplibs::os::is_linux)GTEST_SKIP()<<"Linux libc open interposition";
+    const auto runtime=mcpp::runtime::elf::inspect_elf_runtime(mcpp::invocation::self_exe_path());
+    if(!runtime || runtime->interp.find("ld-linux")==std::string::npos)
+        GTEST_SKIP()<<"open interposition requires the dynamic glibc loader";
+    Transaction t;
+    const auto library=t.root/"record-open-appearance.so";
+    const auto source=fs::current_path()/"tests/e2e/helpers/record_open_appearance.c";
+    auto compiled=mcpp::run::execute({"/usr/bin/cc","-shared","-fPIC","-Wall","-Wextra","-Werror",
+        source.string(),"-o",library.string(),"-ldl"},{.total=std::chrono::seconds(10)});
+    ASSERT_EQ(compiled.exit_code,0)<<compiled.output;
+    ASSERT_TRUE(tx::publish(t.root/"record","schema",{"published"}));
+    const auto result=mcpp::run::execute({mcpp::invocation::self_exe_path().string(),
+        "--gtest_filter=CompilerTransaction.RecordOpenAppearanceDriver"},
+        {.env={{"MCPP_TEST_RECORD_OPEN_DRIVER","1"},{"LD_PRELOAD",library.string()}},
+         .cwd=t.root.string(),.total=std::chrono::seconds(10)});
+    EXPECT_EQ(result.exit_code,0)<<result.output;
+    EXPECT_TRUE(fs::exists(t.root/"record-open-interposed"));
+}
 TEST(CompilerTransaction, FailedCloseCannotPublishOrAcknowledgeARecord) {
     if constexpr (!mcpplibs::os::is_linux)GTEST_SKIP()<<"Linux libc close failure control";
     const auto runtime=mcpp::runtime::elf::inspect_elf_runtime(mcpp::invocation::self_exe_path());