Skip to content

Commit d4e6801

Browse files
2026.11.7.1: compiler recovery, build generations, platform version groups and minimal header regions (#889)
Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com>
1 parent 1e6d1b3 commit d4e6801

182 files changed

Lines changed: 9135 additions & 26618 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

‎.agents/docs/2026-11-02-ecosystem-part3-ok-image-performance-parity-design.md‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -685,3 +685,23 @@ mcpp 在首次初始化时已把 Ninja 装进自己的 home(`ensure_ninja`,`
685685
(memchr/4096 1.46、memcpy/4096 1.34、strcmp/64 1.29),另有 clock、stat 与线程私有数据;同一份源码以 Clang 编成 hosted 时为
686686
9 行,映像多出的部分大半来自编译器而不是映像。这些差距在 L2 的工作负载中不显现。
687687

688+
## 更正:L3 缺页统计的边界(2026-11-04,Sunrisepeak)
689+
690+
本块更正第 9.4 节及后续记录中的缺页口径。周期进程树采样会漏计两次采样之间
691+
退出并被回收的 worker。冻结构件的 macOS 对照中,原生与映像就绪采样分别为
692+
126581 与 294066,而不含 Python 探针的完整引擎内核总账分别为 506753 与
693+
692680。两条路径漏计程度不同;该对照不证明新引擎达标。
694+
695+
测量版本二将缺页门禁明确为完整引擎生命期的内核总账。每次引擎启动前读取
696+
RUSAGE_CHILDREN,完成 shutdown、exit 与 wait 后再取差;minor 与 major 相加。
697+
范围包含引擎自身、它等待回收的后代及关闭阶段,不含 Python 探针和重挂至其他
698+
父进程的后代。Linux 与 macOS 统一采用该范围,标识为
699+
complete-engine-and-waited-descendants-v1。原 page_faults 字段保留就绪时诊断,
700+
新增 complete_page_faults 承担门禁,阈值仍为原生的两倍。Windows 缺少该
701+
POSIX 接口,完整总账显式为空,继续只作两条不同构建路径的对照记录。
702+
703+
就绪时间与就绪时处理器门禁仍为原生的 1.15 倍,三百零四单元、三个打开文件、
704+
一次冷会话与三次暖会话保持。验收拒绝旧版本、范围不符、原始计数缺失、负数或
705+
与原始计数不一致的合计及暖中位数。旧口径的结果继续保留,不能改名后充作
706+
新口径验收;新版本组的完整 CI 重新取得两边原始记录。
707+

‎.agents/docs/2026-11-03-ecosystem-part4-beyond-native-design.md‎

Lines changed: 1522 additions & 0 deletions
Large diffs are not rendered by default.

‎.agents/docs/2026-11-05-heterogeneous-build-implementation-plan.md‎

Lines changed: 255 additions & 1 deletion
Large diffs are not rendered by default.
Lines changed: 223 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,223 @@
1+
---
2+
subject: build-progress-and-scheduling
3+
status: active
4+
---
5+
6+
# BMI publication, compiler capacity and active download budgets
7+
8+
Audience: maintainers closing the 2026.11.5.2 candidate and reviewing PR #889.
9+
This record separates three failures exposed during candidate validation.
10+
It does not qualify a diagnostic object substitution as a released binary.
11+
12+
## Failure boundaries
13+
14+
| Failure | Confirmed cause | Owner |
15+
|---|---|---|
16+
| An importer cannot find a BMI after its interface is touched | GCC unlinks the old final path before renaming the new BMI; an absent identity was accepted as publication | Core detached compilation |
17+
| A build exceeds its requested compiler count | Extra Ninja slots serve publication and join edges, while ordinary compilers bypass the detached semaphore | Core Ninja scheduling |
18+
| A newly activated download is immediately considered stalled | Its queued state was observed earlier; starting without a source attempt or byte change retained the queue-entry timestamp | Core provider progress tracking |
19+
20+
No compiler toolchain payload, plugin semantic or timeout constant is changed
21+
by these corrections.
22+
23+
## BMI publication
24+
25+
GCC 16.1 tracing shows an `unlink(final)` immediately before
26+
`rename(temporary, final)`. The observed interval was approximately 26
27+
microseconds. A changed identity alone is therefore insufficient: publication
28+
requires a present file with a changed identity. The adapter still preserves
29+
the previous BMI for equivalence checking and failure recovery.
30+
31+
The causal control amplifies only that existing unlink interval to 200
32+
milliseconds. The previous core reproduces the candidate failure: the middle
33+
module cannot read the core module. Replacing only the detached-compilation
34+
implementation makes the same real GCC workload succeed. A separate
35+
deterministic publisher confirms the boundary: the old phase returns zero
36+
while the BMI is absent; the corrected phase returns only after publication.
37+
38+
An exit status of zero without the required BMI is diagnosed as a failure.
39+
The previous BMI is restored. E2E 234 retains its cold execution, graph-shape,
40+
incremental build and stale-token assertions, and adds both publication cases.
41+
42+
## Shared compiler permits
43+
44+
The candidate generated `sched_cap = 8`, while its scheduling policy allows
45+
48 Ninja slots. The BMI supervisor used that semaphore; ordinary C/C++
46+
compilation and compiler-driven scanning did not. A full local candidate
47+
build exceeded eight compiler processes and encountered memory-pressure
48+
kills. Its interrupted output is not build qualification.
49+
50+
The selected correction retains one actual compiler budget. Under the
51+
detached strategy, ordinary built-in compilation and scanning use an internal
52+
argv executor holding the same permit until the real child exits. BMI
53+
supervisors continue to hold their permit across early publication and code
54+
generation. Join edges acquire no permit. A holder never waits for another
55+
permit, so waiting joins cannot form a permit dependency cycle.
56+
57+
The internal executor introduces no user mode and no argument serialization.
58+
It receives the compiler's actual argv, inherits the existing streams and
59+
returns its status. It dispatches before global option scans so child flags
60+
cannot configure mcpp. Existing response files retain their original owner and
61+
encoding. Ordinary completion and interrupt cleanup release a permit once;
62+
release cannot remove a permit already acquired by another compiler.
63+
Failure to create or acquire the semaphore is reported rather than followed
64+
by an unbounded compile. A failed supervisor start releases its permit.
65+
66+
The mixed control uses a real generated Ninja graph and GCC 16.1. A driver
67+
records active invocations and remains alive briefly after BMI publication,
68+
amplifying overlap with ordinary C and C++ units. With a requested cap of
69+
one, the previous graph reaches six active drivers. Corrected controls use
70+
caps of one and two, retain the extra Ninja slots, and observe peaks of
71+
exactly one and two. They also check no-op builds, ordinary compiler failure,
72+
a module failure after BMI publication, subsequent recovery, ordinary
73+
compiler cancellation, child termination and permit cleanup. E2E 234 retains
74+
the full-set stale-token reclamation check.
75+
76+
These local controls use selected recompiled objects and an extracted copy of
77+
the production dispatch branch. They establish causality and small-load
78+
behavior. The complete current sources still require the ordinary CI build
79+
and all required checks before release.
80+
81+
## Download activation and deadline reasons
82+
83+
The pinned xlings 2026.10.26.1 provider reports queued files in each batch
84+
with `started = false`. Once a worker obtains a slot, `started` becomes true
85+
before a source attempt or byte count necessarily changes. That transition
86+
starts the active budget. Waiting in the queue consumes no active budget;
87+
repeated samples of an already active file do not renew it. A byte increase
88+
or source change retains its existing renewal semantics.
89+
90+
An isolated provider invocation retained raw NDJSON from one download stream.
91+
Replaying the same 347 snapshots through the actual C++ provider modules gives:
92+
93+
| Module | Expired snapshots | First expiration |
94+
|---|---:|---:|
95+
| Previous state machine | 11 | 95.751 seconds |
96+
| Corrected activation transition | 0 | none |
97+
98+
The first false expiration belongs to a queued entry activated at 95.751
99+
seconds with source attempt zero. Its inherited age was 95.651 seconds.
100+
With queue time excluded, the maximum active idle interval at the observed
101+
snapshots is 7.735 seconds. The isolated network invocation reached its
102+
150-second observer budget; it was stopped by the experiment, not by a
103+
provider progress deadline, and does not establish a completed graphics run.
104+
105+
The failed candidate CI log contains rendered status rather than raw NDJSON.
106+
Its timing is consistent with activation of the final queued entries. It
107+
does not identify which raw file state first expired, and is not presented as
108+
proof of a particular file's network behavior. Progress on an independent
109+
download remains insufficient to renew a genuinely stalled file.
110+
111+
The provider source was followed through its HTTP implementation: the
112+
connect and idle environment budgets are consumed below the downloader
113+
wrapper. They remain eight and thirty seconds, respectively; the parent's
114+
per-file budget remains forty-five seconds. No budget is increased to hide
115+
the state error. Progress timeouts now identify the file, interval and source
116+
attempt; process deadlines remain distinct. Time-injected unit cases cover
117+
queue waits longer than forty-five seconds, genuine per-file stalls, source
118+
fallback and completion.
119+
120+
## Qualification sequence
121+
122+
The local controls precede the complete PR CI. The graphics job is accepted
123+
only after its shader, rendered-image and pixel assertions actually run.
124+
After every required check succeeds, an independent ecosystem review
125+
precedes merge and release. Published tags remain unchanged.
126+
127+
Plugin PR #52 has independently passed all six jobs against the published
128+
2026.11.5.1 core. Its final public core pin and real subproject-sidecar fixture
129+
follow ordinary installation of the new release, in the same plugin PR.
130+
131+
Public records contain synthetic assertions and aggregate evidence. Raw
132+
provider streams, user logs, usernames and private paths remain outside the
133+
repository and pull-request description.
134+
135+
## Correction, 2026-11-06: ownership across cancellation and generations
136+
137+
The shared-cap change exposed an existing lifetime gap. A real Ninja control
138+
cancelled the top-level build after BMI publication. Its detached compiler
139+
remained alive 200 milliseconds later. Deleting the previous directory tokens
140+
at the next build could therefore admit new compilers over the requested cap.
141+
The correction below supersedes directory-token ownership in the earlier
142+
shared-permit description.
143+
144+
Each detached-schedule backend drive owns a runtime generation. Normal graph
145+
execution and cached replay, including quiet replay, open the same scope. Its
146+
owner has both the operating system's process identity with birth time and an
147+
exclusive kernel lock. A PID alone, a surviving record, or an absent birth time
148+
cannot establish ownership. Closing the lock revokes the drive even while its
149+
process is alive. The generation is passed only in the child environment after
150+
cache decoding; it never enters an action, build fingerprint or saved graph.
151+
152+
Every phase-one helper, supervisor, join helper and ordinary compiler executor
153+
registers a kernel-locked actor before modifying build state and rechecks the
154+
owner after registration. The actor covers status publication, BMI equivalence
155+
settlement and backup restoration, as well as compiler execution. Interrupt
156+
callbacks retain that actor until their final write. A delayed helper from a
157+
closed generation returns cancellation before changing any build state.
158+
159+
A stable gate serializes generations for one canonical output directory. The
160+
next generation waits for all previous actor locks, including actors that have
161+
already released a compiler permit. It retains busy state and refuses admission
162+
if the actors do not drain within the cleanup bound. Runtime locks reside under
163+
the work directory's `.mcpp/compiler-runs`, outside the target output tree and
164+
independently of `MCPP_HOME`. Cleaning output files cannot unlink a live permit.
165+
166+
Actual compiler permits are stable kernel-locked files. A supervisor acquires
167+
its permit itself and holds it until the child has stopped; phase one does not
168+
transfer a lock or consume a second permit. Ordinary compilation uses the same
169+
slots. Files left after process death are reused by locking, never reclaimed by
170+
unlinking. A typed busy result is retried; permission and other failures stop
171+
the invocation. This also replaces the directory-creation contention path that
172+
reported access denied in the Windows-host GCC cross-build.
173+
174+
Child cancellation uses the existing launcher stop predicate. A further causal
175+
control found that inherited or discarded streams skipped that predicate when
176+
no absolute deadline was set. The launcher now polls child completion in that
177+
case, retaining its argv, streams, exit status and existing deadline. No timeout
178+
is increased. Failed or cancelled compiler invocations withdraw partial outputs;
179+
subsequent ordinary builds recover missing objects and BMIs.
180+
181+
The permanent real-Ninja control checks:
182+
183+
- Module and ordinary compilation under top-level SIGINT and SIGKILL, followed
184+
immediately by another generation; the old compiler has stopped before the
185+
new drive is admitted.
186+
- No-op builds across generations, retaining compiler count and object mtime;
187+
normal and quiet cached replay retain their graph, objects and BMIs.
188+
- Changed homes and a symlinked target directory retain the work-directory
189+
runtime barrier, independently of the disposable output location.
190+
- Owner-lock closure while the owner process remains alive.
191+
- Missing and incorrect process birth identities, rejected before state writes.
192+
- Delayed phase-one BMI settlement and backup restoration after compiler exit;
193+
both still block the next generation until their final state write.
194+
- Unregistered old phase-one and supervisor processes delayed until after a
195+
successful new generation; neither can alter its BMI, object or status.
196+
197+
The controls execute the actual helper and launcher implementations with a
198+
small synthetic compiler. The separate GCC mixed workload retains the real
199+
compiler cap, late failure, recovery, argv and no-op assertions. Selected-object
200+
controls establish causality; final qualification remains complete current-source
201+
CI and independent ecosystem review.
202+
203+
### Addendum, 2026-11-06: the final child wait retains its bounds
204+
205+
A final launcher control closes both captured streams before the child sleeps.
206+
The preceding correction still returned success after 2.002 seconds instead of
207+
observing the fifty-millisecond stop request: stream EOF had led to an unbounded
208+
child wait. The same boundary also bypassed an absolute deadline after EOF.
209+
210+
One shared predicate now applies during output reads, remaining error reads and
211+
the final child wait. It retains the original absolute deadline and stop request;
212+
observed output retains its idle budget after EOF. Activity on a separately
213+
captured error stream renews that observed-output budget. Inherited and discarded
214+
streams retain no inferred idle observation. Child completion is checked during
215+
the final bounded wait, preserving already completed child statuses.
216+
217+
A finite matrix covers all six stream modes, ordinary and signalled exits,
218+
completion with bounds configured, stop requests before and after one or both
219+
streams close, absolute deadlines after closure, captured idle expiry and live
220+
stderr after stdout closure. Its fifty-three cases pass, together with the
221+
original launcher tests and the direct EOF regression. Top-level signal behavior
222+
is additionally covered by the real-Ninja lifecycle control. No budget, compiler
223+
argument or output stream is changed to make a cancellation succeed.

0 commit comments

Comments
 (0)