|
| 1 | +--- |
| 2 | +subject: build-progress-and-scheduling |
| 3 | +status: active |
| 4 | +--- |
| 5 | + |
| 6 | +# BMI publication, compiler capacity and active download budgets |
| 7 | + |
| 8 | +Audience: maintainers closing the 2026.11.5.2 candidate and reviewing PR #889. |
| 9 | +This record separates three failures exposed during candidate validation. |
| 10 | +It does not qualify a diagnostic object substitution as a released binary. |
| 11 | + |
| 12 | +## Failure boundaries |
| 13 | + |
| 14 | +| Failure | Confirmed cause | Owner | |
| 15 | +|---|---|---| |
| 16 | +| An importer cannot find a BMI after its interface is touched | GCC unlinks the old final path before renaming the new BMI; an absent identity was accepted as publication | Core detached compilation | |
| 17 | +| A build exceeds its requested compiler count | Extra Ninja slots serve publication and join edges, while ordinary compilers bypass the detached semaphore | Core Ninja scheduling | |
| 18 | +| A newly activated download is immediately considered stalled | Its queued state was observed earlier; starting without a source attempt or byte change retained the queue-entry timestamp | Core provider progress tracking | |
| 19 | + |
| 20 | +No compiler toolchain payload, plugin semantic or timeout constant is changed |
| 21 | +by these corrections. |
| 22 | + |
| 23 | +## BMI publication |
| 24 | + |
| 25 | +GCC 16.1 tracing shows an `unlink(final)` immediately before |
| 26 | +`rename(temporary, final)`. The observed interval was approximately 26 |
| 27 | +microseconds. A changed identity alone is therefore insufficient: publication |
| 28 | +requires a present file with a changed identity. The adapter still preserves |
| 29 | +the previous BMI for equivalence checking and failure recovery. |
| 30 | + |
| 31 | +The causal control amplifies only that existing unlink interval to 200 |
| 32 | +milliseconds. The previous core reproduces the candidate failure: the middle |
| 33 | +module cannot read the core module. Replacing only the detached-compilation |
| 34 | +implementation makes the same real GCC workload succeed. A separate |
| 35 | +deterministic publisher confirms the boundary: the old phase returns zero |
| 36 | +while the BMI is absent; the corrected phase returns only after publication. |
| 37 | + |
| 38 | +An exit status of zero without the required BMI is diagnosed as a failure. |
| 39 | +The previous BMI is restored. E2E 234 retains its cold execution, graph-shape, |
| 40 | +incremental build and stale-token assertions, and adds both publication cases. |
| 41 | + |
| 42 | +## Shared compiler permits |
| 43 | + |
| 44 | +The candidate generated `sched_cap = 8`, while its scheduling policy allows |
| 45 | +48 Ninja slots. The BMI supervisor used that semaphore; ordinary C/C++ |
| 46 | +compilation and compiler-driven scanning did not. A full local candidate |
| 47 | +build exceeded eight compiler processes and encountered memory-pressure |
| 48 | +kills. Its interrupted output is not build qualification. |
| 49 | + |
| 50 | +The selected correction retains one actual compiler budget. Under the |
| 51 | +detached strategy, ordinary built-in compilation and scanning use an internal |
| 52 | +argv executor holding the same permit until the real child exits. BMI |
| 53 | +supervisors continue to hold their permit across early publication and code |
| 54 | +generation. Join edges acquire no permit. A holder never waits for another |
| 55 | +permit, so waiting joins cannot form a permit dependency cycle. |
| 56 | + |
| 57 | +The internal executor introduces no user mode and no argument serialization. |
| 58 | +It receives the compiler's actual argv, inherits the existing streams and |
| 59 | +returns its status. It dispatches before global option scans so child flags |
| 60 | +cannot configure mcpp. Existing response files retain their original owner and |
| 61 | +encoding. Ordinary completion and interrupt cleanup release a permit once; |
| 62 | +release cannot remove a permit already acquired by another compiler. |
| 63 | +Failure to create or acquire the semaphore is reported rather than followed |
| 64 | +by an unbounded compile. A failed supervisor start releases its permit. |
| 65 | + |
| 66 | +The mixed control uses a real generated Ninja graph and GCC 16.1. A driver |
| 67 | +records active invocations and remains alive briefly after BMI publication, |
| 68 | +amplifying overlap with ordinary C and C++ units. With a requested cap of |
| 69 | +one, the previous graph reaches six active drivers. Corrected controls use |
| 70 | +caps of one and two, retain the extra Ninja slots, and observe peaks of |
| 71 | +exactly one and two. They also check no-op builds, ordinary compiler failure, |
| 72 | +a module failure after BMI publication, subsequent recovery, ordinary |
| 73 | +compiler cancellation, child termination and permit cleanup. E2E 234 retains |
| 74 | +the full-set stale-token reclamation check. |
| 75 | + |
| 76 | +These local controls use selected recompiled objects and an extracted copy of |
| 77 | +the production dispatch branch. They establish causality and small-load |
| 78 | +behavior. The complete current sources still require the ordinary CI build |
| 79 | +and all required checks before release. |
| 80 | + |
| 81 | +## Download activation and deadline reasons |
| 82 | + |
| 83 | +The pinned xlings 2026.10.26.1 provider reports queued files in each batch |
| 84 | +with `started = false`. Once a worker obtains a slot, `started` becomes true |
| 85 | +before a source attempt or byte count necessarily changes. That transition |
| 86 | +starts the active budget. Waiting in the queue consumes no active budget; |
| 87 | +repeated samples of an already active file do not renew it. A byte increase |
| 88 | +or source change retains its existing renewal semantics. |
| 89 | + |
| 90 | +An isolated provider invocation retained raw NDJSON from one download stream. |
| 91 | +Replaying the same 347 snapshots through the actual C++ provider modules gives: |
| 92 | + |
| 93 | +| Module | Expired snapshots | First expiration | |
| 94 | +|---|---:|---:| |
| 95 | +| Previous state machine | 11 | 95.751 seconds | |
| 96 | +| Corrected activation transition | 0 | none | |
| 97 | + |
| 98 | +The first false expiration belongs to a queued entry activated at 95.751 |
| 99 | +seconds with source attempt zero. Its inherited age was 95.651 seconds. |
| 100 | +With queue time excluded, the maximum active idle interval at the observed |
| 101 | +snapshots is 7.735 seconds. The isolated network invocation reached its |
| 102 | +150-second observer budget; it was stopped by the experiment, not by a |
| 103 | +provider progress deadline, and does not establish a completed graphics run. |
| 104 | + |
| 105 | +The failed candidate CI log contains rendered status rather than raw NDJSON. |
| 106 | +Its timing is consistent with activation of the final queued entries. It |
| 107 | +does not identify which raw file state first expired, and is not presented as |
| 108 | +proof of a particular file's network behavior. Progress on an independent |
| 109 | +download remains insufficient to renew a genuinely stalled file. |
| 110 | + |
| 111 | +The provider source was followed through its HTTP implementation: the |
| 112 | +connect and idle environment budgets are consumed below the downloader |
| 113 | +wrapper. They remain eight and thirty seconds, respectively; the parent's |
| 114 | +per-file budget remains forty-five seconds. No budget is increased to hide |
| 115 | +the state error. Progress timeouts now identify the file, interval and source |
| 116 | +attempt; process deadlines remain distinct. Time-injected unit cases cover |
| 117 | +queue waits longer than forty-five seconds, genuine per-file stalls, source |
| 118 | +fallback and completion. |
| 119 | + |
| 120 | +## Qualification sequence |
| 121 | + |
| 122 | +The local controls precede the complete PR CI. The graphics job is accepted |
| 123 | +only after its shader, rendered-image and pixel assertions actually run. |
| 124 | +After every required check succeeds, an independent ecosystem review |
| 125 | +precedes merge and release. Published tags remain unchanged. |
| 126 | + |
| 127 | +Plugin PR #52 has independently passed all six jobs against the published |
| 128 | +2026.11.5.1 core. Its final public core pin and real subproject-sidecar fixture |
| 129 | +follow ordinary installation of the new release, in the same plugin PR. |
| 130 | + |
| 131 | +Public records contain synthetic assertions and aggregate evidence. Raw |
| 132 | +provider streams, user logs, usernames and private paths remain outside the |
| 133 | +repository and pull-request description. |
| 134 | + |
| 135 | +## Correction, 2026-11-06: ownership across cancellation and generations |
| 136 | + |
| 137 | +The shared-cap change exposed an existing lifetime gap. A real Ninja control |
| 138 | +cancelled the top-level build after BMI publication. Its detached compiler |
| 139 | +remained alive 200 milliseconds later. Deleting the previous directory tokens |
| 140 | +at the next build could therefore admit new compilers over the requested cap. |
| 141 | +The correction below supersedes directory-token ownership in the earlier |
| 142 | +shared-permit description. |
| 143 | + |
| 144 | +Each detached-schedule backend drive owns a runtime generation. Normal graph |
| 145 | +execution and cached replay, including quiet replay, open the same scope. Its |
| 146 | +owner has both the operating system's process identity with birth time and an |
| 147 | +exclusive kernel lock. A PID alone, a surviving record, or an absent birth time |
| 148 | +cannot establish ownership. Closing the lock revokes the drive even while its |
| 149 | +process is alive. The generation is passed only in the child environment after |
| 150 | +cache decoding; it never enters an action, build fingerprint or saved graph. |
| 151 | + |
| 152 | +Every phase-one helper, supervisor, join helper and ordinary compiler executor |
| 153 | +registers a kernel-locked actor before modifying build state and rechecks the |
| 154 | +owner after registration. The actor covers status publication, BMI equivalence |
| 155 | +settlement and backup restoration, as well as compiler execution. Interrupt |
| 156 | +callbacks retain that actor until their final write. A delayed helper from a |
| 157 | +closed generation returns cancellation before changing any build state. |
| 158 | + |
| 159 | +A stable gate serializes generations for one canonical output directory. The |
| 160 | +next generation waits for all previous actor locks, including actors that have |
| 161 | +already released a compiler permit. It retains busy state and refuses admission |
| 162 | +if the actors do not drain within the cleanup bound. Runtime locks reside under |
| 163 | +the work directory's `.mcpp/compiler-runs`, outside the target output tree and |
| 164 | +independently of `MCPP_HOME`. Cleaning output files cannot unlink a live permit. |
| 165 | + |
| 166 | +Actual compiler permits are stable kernel-locked files. A supervisor acquires |
| 167 | +its permit itself and holds it until the child has stopped; phase one does not |
| 168 | +transfer a lock or consume a second permit. Ordinary compilation uses the same |
| 169 | +slots. Files left after process death are reused by locking, never reclaimed by |
| 170 | +unlinking. A typed busy result is retried; permission and other failures stop |
| 171 | +the invocation. This also replaces the directory-creation contention path that |
| 172 | +reported access denied in the Windows-host GCC cross-build. |
| 173 | + |
| 174 | +Child cancellation uses the existing launcher stop predicate. A further causal |
| 175 | +control found that inherited or discarded streams skipped that predicate when |
| 176 | +no absolute deadline was set. The launcher now polls child completion in that |
| 177 | +case, retaining its argv, streams, exit status and existing deadline. No timeout |
| 178 | +is increased. Failed or cancelled compiler invocations withdraw partial outputs; |
| 179 | +subsequent ordinary builds recover missing objects and BMIs. |
| 180 | + |
| 181 | +The permanent real-Ninja control checks: |
| 182 | + |
| 183 | +- Module and ordinary compilation under top-level SIGINT and SIGKILL, followed |
| 184 | + immediately by another generation; the old compiler has stopped before the |
| 185 | + new drive is admitted. |
| 186 | +- No-op builds across generations, retaining compiler count and object mtime; |
| 187 | + normal and quiet cached replay retain their graph, objects and BMIs. |
| 188 | +- Changed homes and a symlinked target directory retain the work-directory |
| 189 | + runtime barrier, independently of the disposable output location. |
| 190 | +- Owner-lock closure while the owner process remains alive. |
| 191 | +- Missing and incorrect process birth identities, rejected before state writes. |
| 192 | +- Delayed phase-one BMI settlement and backup restoration after compiler exit; |
| 193 | + both still block the next generation until their final state write. |
| 194 | +- Unregistered old phase-one and supervisor processes delayed until after a |
| 195 | + successful new generation; neither can alter its BMI, object or status. |
| 196 | + |
| 197 | +The controls execute the actual helper and launcher implementations with a |
| 198 | +small synthetic compiler. The separate GCC mixed workload retains the real |
| 199 | +compiler cap, late failure, recovery, argv and no-op assertions. Selected-object |
| 200 | +controls establish causality; final qualification remains complete current-source |
| 201 | +CI and independent ecosystem review. |
| 202 | + |
| 203 | +### Addendum, 2026-11-06: the final child wait retains its bounds |
| 204 | + |
| 205 | +A final launcher control closes both captured streams before the child sleeps. |
| 206 | +The preceding correction still returned success after 2.002 seconds instead of |
| 207 | +observing the fifty-millisecond stop request: stream EOF had led to an unbounded |
| 208 | +child wait. The same boundary also bypassed an absolute deadline after EOF. |
| 209 | + |
| 210 | +One shared predicate now applies during output reads, remaining error reads and |
| 211 | +the final child wait. It retains the original absolute deadline and stop request; |
| 212 | +observed output retains its idle budget after EOF. Activity on a separately |
| 213 | +captured error stream renews that observed-output budget. Inherited and discarded |
| 214 | +streams retain no inferred idle observation. Child completion is checked during |
| 215 | +the final bounded wait, preserving already completed child statuses. |
| 216 | + |
| 217 | +A finite matrix covers all six stream modes, ordinary and signalled exits, |
| 218 | +completion with bounds configured, stop requests before and after one or both |
| 219 | +streams close, absolute deadlines after closure, captured idle expiry and live |
| 220 | +stderr after stdout closure. Its fifty-three cases pass, together with the |
| 221 | +original launcher tests and the direct EOF regression. Top-level signal behavior |
| 222 | +is additionally covered by the real-Ninja lifecycle control. No budget, compiler |
| 223 | +argument or output stream is changed to make a cancellation succeed. |
0 commit comments