Skip to content

Commit dc5e75e

Browse files
SunrisepeakSunrisepeak
committed
fix: recover compiler attempts only after all writers retire
1 parent 728db27 commit dc5e75e

23 files changed

Lines changed: 1995 additions & 154 deletions

‎.agents/docs/2026-11-06-compiler-attempt-recovery.md‎

Lines changed: 189 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,189 @@
1+
# Compiler attempt lifetime and output recovery
2+
3+
Status: source implementation and focused acceptance in progress. This record
4+
supplements the release candidate's compiler scheduling corrections. Native
5+
source-group API checks do not qualify core or the released dependency group.
6+
7+
## Problem and responsibility
8+
9+
A compiler driver may start a descendant that continues writing after the
10+
supervisor and directly bound driver have ended. Releasing a kernel permit
11+
when its holder dies therefore does not prove that its compiler unit is quiet.
12+
A separate failure occurs after early BMI publication: Ninja has recorded the
13+
BMI edge while the object/status never completes, so a later build can enter
14+
an orphan await instead of recompiling.
15+
16+
Core owns the attempt journal, concurrency reservation and recovery of its
17+
outputs. OS owns declared units, sealed admission and observation through an
18+
original native reference. OpenKal and the native backends own the kernel
19+
identity and completion semantics. Plugins do not manage process units. The
20+
core implementation uses those generic capabilities and does not inspect
21+
native process identifiers, process groups or platform job handles.
22+
23+
## Runtime protocol
24+
25+
One build drive owns a generation and an actor barrier. Generation identity is
26+
runtime state; it changes neither build identity nor Ninja command hashes.
27+
Phase one, ordinary wrappers, supervisors and await helpers acquire actor
28+
leases before their final owner check. Recovery waits for every actor, including
29+
phase-one BMI settling and status writers, rather than only compiler permits.
30+
31+
Each attempt publishes a checksummed prepared record before modifying outputs.
32+
It resolves parent directories in the original execution context and records
33+
absolute BMI, object, dependency and status identities. A leaf symlink is not
34+
resolved into an unrelated output target. A previous BMI is copied into a
35+
protected baseline; its content identity is certified after the copy completes.
36+
Separate immutable stage records avoid competing writers replacing one record.
37+
38+
A helper acquires a kernel slot, declares its execution unit, publishes that
39+
unit and its logical slot reservation, and registers with the live owner. The
40+
owner retains the original native observation reference before acknowledging
41+
registration over existing directory-local blocking IPC. Only then may the
42+
helper publish admission and start the compiler. Record, registration or ACK
43+
failure does not start an unregistered compiler.
44+
45+
An unlocked kernel slot whose reservation remains occupied or unknown is not
46+
available. A hard-killed supervisor therefore cannot admit a replacement while
47+
its compiler descendant continues writing. The live owner's listener also
48+
notices controller loss, so an orphan descendant holding Ninja's output pipe
49+
does not leave the build waiting indefinitely for EOF.
50+
51+
After the direct compiler exits, retirement seals admission and establishes
52+
that no external writer remains. On systems that permit leaving a unit, the
53+
helper uses the build owner's full program identity as a continuation; the
54+
short-lived BMI publisher is insufficient. A dead, reused or inaccessible
55+
continuation, including a different native session, yields unknown. Retirement
56+
does not grant termination or admission authority over the continuation unit.
57+
The helper publishes its quiet-unit certificate before atomic status and
58+
releases its logical reservation only after this barrier.
59+
60+
## Recovery and boundaries
61+
62+
Recovery follows the actor barrier and requires either a verified retirement
63+
certificate, positive native emptiness, or proof that admission never occurred.
64+
A successful status also needs matching output identities. An incomplete early
65+
BMI publication is removed to force a new BMI edge; its verified prior baseline
66+
is retained with a certificate for that retry's equivalence/failure handling.
67+
A successful cached object carries a quiet completion proof so a later missing
68+
object can invalidate the BMI without inferring unit death from leader absence.
69+
70+
Malformed records, invalid stage order, write refusal, occupied units and
71+
unknown observation retain reservations and evidence. Recovery is repeatable
72+
between file operations. If every original native reference holder is forcibly
73+
lost before retirement can be certified, Windows may no longer identify the
74+
original job. That case remains a bounded refusal and is not automatic recovery.
75+
A C observation job has an observe/close caller precondition; the OS observer
76+
class hides every native control path.
77+
78+
## Validation layers
79+
80+
The OpenKal actual-source API workflow
81+
[42110324914](https://github.com/mcpplibs/openkal/actions/runs/42110324914)
82+
and OS actual-source lifecycle workflow
83+
[42110397318](https://github.com/mcpplibs/os/actions/runs/42110397318)
84+
passed on Linux, macOS and Windows. They cover declared-unit registration,
85+
blocking ACK failure/timeout, driver descendants, sealed retirement, retained
86+
observation, an expired publisher, forged continuation birth and observer
87+
control rejection. Linux additionally kills the continuation at native migration
88+
entry and immediately after successful migration; both report unknown.
89+
90+
Each OS native run recorded two batches of eight paired startup samples. The
91+
median paired ACK increments were 0.175 ms on Linux, 0.409 ms on macOS and
92+
2.508 ms on Windows. Listener start/wake/close medians were 0.177, 0.289 and
93+
0.721 ms respectively. All samples, including negative deltas and tail values,
94+
are retained in the run artifacts. These measurements describe the source
95+
consumer, rather than a complete build or core no-op benchmark.
96+
97+
Core acceptance includes checksummed records, baseline staging/corruption,
98+
atomic status gaps, late failures, missing cached objects, repeated recovery,
99+
write refusal and changed directory symlinks. The actual helper matrix also
100+
covers supervisor-only hard loss with a still-writing descendant, a free kernel
101+
slot with a retained logical reservation, owner cancellation and immediate
102+
restart, phase-one state writers, and normal/quiet cached replay. Syscall-boundary
103+
fault controls belong to the test driver and add no product fault hooks.
104+
105+
Final acceptance still requires the completed core fault matrix, actual GCC
106+
mixed-cap/replay controls, updated user/spec documentation, the coherent public
107+
OpenKal/backend/runtime/platform/OS version group and full core CI. Core receives
108+
an independent ecosystem review after those checks and before merge/release.
109+
110+
## 2026-11-06 cached object admission closeout
111+
112+
A completed generation already leaves a quiet completion certificate beside
113+
each module's object slot. A later missing object originally made cached await
114+
remove its BMI and fail, requiring a second invocation. The corrected boundary
115+
reuses those existing certificates under the configuration's object tree after
116+
old owners, actors and units are quiet and before any new Ninja reader starts.
117+
Only a certificate's own declared object is checked. A missing object invokes
118+
the existing certified-baseline invalidation; Ninja then rebuilds in its normal
119+
graph on that same invocation. There is no additional success index, source
120+
dirty model, dry run or alternate compiler entry.
121+
122+
Certificate names are bound to their slot/object paths. Absolute BMI scope,
123+
field values, checksums, regular certificate files and unchanged BMI parents
124+
are checked before invalidation. Symlinked state does not authorize recovery.
125+
The checksum detects accidental corruption; it is not authentication against
126+
a user who can already replace the build tool and its outputs.
127+
128+
The transaction suite passes 19 cases. The actual fresh engine recovers one
129+
and two deleted completed module objects in a single build and preserves the
130+
following no-op. Actual CLI controls reject a corrupt certificate before Ninja
131+
starts and reject a directory symlink without changing an unrelated artifact.
132+
The implementation's 30 warm admission samples have medians of 0.057 ms for
133+
two real certificates, 2.200 ms for the core object tree's 1908 entries without
134+
these new certificates, and 1.089 ms for 200 format-correct synthetic records
135+
with real object existence checks. The latter is a scale fixture, not evidence
136+
that those synthetic records came from real retirements.
137+
138+
A separate paired protocol control preserves the actual GCC argument multiset,
139+
cap two and Ninja twelve. Eight no-op pairs have a median paired difference of
140+
-0.041 ms. Five small cold-build pairs have a median new/old ratio of 1.101 and
141+
a median paired increase of 250 ms. The comparison includes the complete old
142+
protocol/OS and the current source group, and its controlled compiler driver
143+
sleeps 150 ms after real compilation. It does not isolate IPC, establish an
144+
absence of build overhead, or extrapolate to large module workloads.
145+
146+
## 2026-11-06 process-visible publication scope
147+
148+
The recovery contract covers owner, supervisor and driver process loss while
149+
the filesystem continues operating. It requires complete atomic visibility,
150+
checked write/close outcomes and rejection of damaged records. It does not
151+
promise persistence across power loss or a filesystem/storage service failure.
152+
The OS library's existing durable `write_atomic` operation remains unchanged.
153+
154+
The runtime publisher creates an exclusive same-directory temporary, writes
155+
and flushes the user-space stream, checks its explicit close, then uses the
156+
existing generic replacing rename. Failed write/close/rename does not publish
157+
the final name or acknowledge admission. It removes only the temporary it
158+
created. This composition preserves process-loss visibility without flushing
159+
each short-lived runtime record to storage.
160+
161+
A controlled libc close returns EIO after closing the actual temporary; the
162+
publisher rejects the replacement, preserves the previous record and removes
163+
its temporary. The transaction suite passes 20 functional cases plus one
164+
subprocess-driver skip. The same candidate passes the actual lifecycle and
165+
syscall-boundary fault controls, including occupied descendants, reservation
166+
retention and restart.
167+
168+
A frozen current-protocol durable binary/driver is the comparison baseline.
169+
Five paired small cold builds preserve source, actual GCC argument multiset,
170+
cap two and Ninja twelve; the candidate/durable median ratio is 0.900, with a
171+
median paired reduction of 232 ms. Eight no-op pairs have a median paired
172+
reduction of 1.639 ms. These samples use blocking child waits, a timeout guard
173+
and alternating order. The earlier timeout-based sampler rounded short waits
174+
and its small no-op delta was within that measurement's noise.
175+
176+
A corresponding syscall control records 200 fsync calls and 554 ms of summed
177+
call wall-time for the durable protocol, and zero fsync calls for the candidate.
178+
The sampled process wall-times are 2.868 and 2.668 seconds. Summed syscall
179+
wall-time includes concurrent threads and is not the build wall-time. These
180+
small-workload results support this publication choice; they are not an
181+
extrapolation to large module builds or a general benchmark of IPC.
182+
183+
The capacity counter also distinguishes a forcibly ended driver from live
184+
activity. It records each original pid/birth pair, preserves entries whose
185+
finally handler did not execute, and counts only the original live processes.
186+
A deterministic cancellation control kills a real wrapper, retains that
187+
history, then starts a command returning seven without inflating peak one.
188+
The prior false failure and its dead-process/empty-reservation evidence remain
189+
part of the experiment record.

‎CHANGELOG.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -26,6 +26,14 @@
2626
- Cancelled detached builds stop their compilers and finish state cleanup before
2727
another generation starts. Stable kernel permits retain live ownership across
2828
immediate rebuilds; runtime generations leave no-op build identities unchanged.
29+
- Compiler units retain their concurrency reservation after a supervisor is
30+
forcibly ended. Recovery waits for descendants and final state writers;
31+
occupied or unknown units retain their outputs and fail within a bound.
32+
- Missing completed module objects are recovered in the same build before Ninja
33+
reads their BMIs. Corrupt completion evidence and redirected cache state refuse
34+
recovery without authorizing changes to unrelated outputs.
35+
- Captured idle bounds consume queued progress from both streams before
36+
classifying silence, including progress buffered during a slow line callback.
2937
- Child stop predicates apply with inherited and discarded streams, including
3038
executions without an absolute deadline. Captured stream EOF retains stop,
3139
absolute and observed-output idle bounds until the child exits.

‎docs/04-mcpp-toml.md‎

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1053,7 +1053,11 @@ generation share the same total `jobs` limit (2026.11.5.2+). Additional
10531053
scheduling and join tasks do not increase the number of compiler processes.
10541054
Cancelling the build ends its ordinary and detached compilers. An immediately
10551055
following build waits for their cleanup before starting another compiler;
1056-
partial outputs are rebuilt. An unchanged build still performs no compilation.
1056+
partial outputs are rebuilt. A missing object from a completed module is
1057+
recreated by that build. An unchanged build still performs no compilation.
1058+
If a forcibly ended build leaves a compiler unit whose completion cannot be
1059+
verified, the next build reports an occupied or unknown unit and retains its
1060+
outputs; it does not start a replacement compiler in that slot.
10571061

10581062
| value | |
10591063
|---|---|

‎docs/specs/concurrency.md‎

Lines changed: 36 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -4,10 +4,10 @@
44
|---|---|
55
| 规范编号 | SPEC-011 |
66
| 标题 | 同一工程上的并发运行:构建目录锁、原子发布、乐观写入与取消 |
7-
| 状态 | 评审中 v1.0 |
8-
| 版本 | 1.0 |
9-
| 最后修改 | 2026-10-26 |
10-
| 对应实现 | mcpp >= 2026.10.26.1 |
7+
| 状态 | 评审中 v1.1 |
8+
| 版本 | 1.1 |
9+
| 最后修改 | 2026-11-06 |
10+
| 对应实现 | 基础保证: mcpp >= 2026.10.26.1; §5.1: mcpp >= 2026.11.5.2 |
1111
| 相关设计文档 | `.agents/docs/2026-10-26-control-contract-runtime-and-ide-ecosystem-design.md` §3、§4、§5.4 |
1212
| 相关 issue | #839 |
1313
| 使用文档 | [50 — 机器可读输出](../zh/50-machine-output.md) §9、§11 |
@@ -79,13 +79,41 @@ CI。在 2026.10.24.1 上,同一配置的两次冷构建相隔 0.1–0.5 s 启
7979

8080
1. SIGINT、SIGTERM、SIGHUP 结束修改段中的运行时,mcpp **必须**先结束它启动的进程组,
8181
再写下 `cancelled` 的结果记录,然后按信号的默认动作结束(退出状态见
82-
[SPEC-003](exit-codes.md) §3.7)。记录的这一形态在修改段开始时预先生成,信号处理器
83-
只调用异步信号安全的 `open`、`write`、`close`、`rename`。
82+
[SPEC-003](exit-codes.md) §3.7)。记录的这一形态在修改段开始时预先生成。
83+
结束请求由 OS 库接收;运行时的等待线程结束执行单元并发布取消记录。
8484
2. 锁由进程结束释放。构建目录保持一致:ninja 的日志只记录完成的步骤,被中断的步骤在
8585
下一次构建中重做。
8686
3. SIGKILL 不经过处理器;`status` 为 `running` 而 `pid` 已不存在的记录,读者**应当**
8787
读作"被强制结束"。
8888

89+
### 5.1 分离编译的执行单元与恢复
90+
91+
以下保证适用于 GCC 的 `bmi_schedule = "on"`（2026.11.5.2+）:
92+
93+
1. 普通编译与分离代码生成**必须**共用实际的 `jobs` 总上限。等待任务与调度任务
94+
**禁止**作为编译名额的替代。内核锁已释放而原编译单元仍被占用或状态未知时,
95+
对应名额**禁止**重新分配。
96+
2. 真实编译器启动前,其控制程序**必须**发布输出身份、执行单元及名额保留记录,
97+
并取得活所有者已保留原生观察引用的确认。登记、写入或确认失败时**禁止**启动编译器。
98+
3. 直接编译器或控制程序结束**禁止**被解释为所有后代写入者已经结束。正常状态发布
99+
**必须**在封闭执行单元的接纳并确认外部成员已结束之后进行。
100+
4. 下一次构建恢复输出前**必须**等待旧代所有状态写入者及原编译单元。恢复使用
101+
原执行上下文中确定的绝对输出身份,**禁止**依据恢复时的 cwd/home 或重定向后的
102+
目标目录符号链接重新解释路径。运行代身份**禁止**进入构建指纹或 Ninja 命令身份。
103+
5. 未完成的早发布 BMI **必须**使下一次构建重新执行 BMI 边,同时保留经认证的
104+
先前 BMI 作为恢复基线。成功状态**必须**与已经安静的执行单元及输出身份相符。
105+
重复恢复**必须**保持安全。已认证完成的模块缺 object 时，**必须**在新 Ninja
106+
读取 BMI 之前失效对应 BMI，使该次构建正常重建。损坏证书、目录符号链接或
107+
已重定向的 BMI 父目录**禁止**授权删除其他输出。损坏记录、非法阶段顺序或
108+
写入失败**禁止**猜测成功。
109+
6. 原单元被占用或不能被确认时,恢复**必须**在有限等待后明确失败并保留名额及证据。
110+
所有原生引用持有者在认证完成之前都被强制结束时,平台可能只能报告未知;
111+
此边界**不保证**自动恢复,**禁止**把名字消失或数值标识复用解释为空单元。
112+
113+
7. 本节的恢复范围是文件系统持续工作时的进程异常退出；**不保证**断电或存储服务
114+
故障后的持久性。运行记录**必须**完整写入并检查关闭结果后才原子替换最终名字，
115+
确认消息**禁止**先于该替换。OS 库的通用持久化写入接口保持其既有语义。
116+
89117
## 6. 全局一侧
90118

91119
全局 `MCPP_HOME` 保持既有的细粒度保护:索引刷新使用非阻塞锁,BMI 缓存按条目加锁,
@@ -99,3 +127,5 @@ CI。在 2026.10.24.1 上,同一配置的两次冷构建相隔 0.1–0.5 s 启
99127
| 版本 | 日期 | 变更 |
100128
|---|---|---|
101129
| 1.0 | 2026-10-26 | 首版:三段、构建目录锁、原子发布、`mcpp.lock` 的乐观写入、取消(#839) |
130+
131+
| 1.1 | 2026-11-06 | 分离编译的单元观察、共享名额、绝对输出事务与有限恢复边界;对齐取消请求的 OS 接收方式。 |

‎docs/zh/04-mcpp-toml.md‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -975,7 +975,10 @@ bmi_schedule = "off" # auto (default, = off) | on | off
975975
使用 GCC 且 `bmi_schedule = "on"` 时，普通编译与分离的代码生成共用同一个 `jobs` 总上限
976976
（2026.11.5.2+）。额外的调度和等待任务不增加编译器进程数量。
977977
取消构建会结束本轮普通编译器和分离的编译器。紧接着的构建等待它们完成清理后
978-
才启动新的编译器；未完成的输出会重新构建。输入不变的构建仍不进行编译。
978+
才启动新的编译器；未完成的输出会重新构建。已完成模块的 object 缺失时，
979+
该次构建会重新生成它。输入和输出均不变的构建仍不进行编译。
980+
若被强制结束的构建留下了无法确认已经完成的编译单元，下一次构建报告该单元
981+
仍被占用或状态未知，并保留其输出；该名额不会启动替代编译器。
979982

980983
| 取值 | |
981984
|---|---|

0 commit comments

Comments
 (0)