Skip to content

Commit 435a88b

Browse files
committed
fix(index): coordinate publication between the worker and its supervisor
A worker resource limit is enforced by the supervisor, but only the worker replaces the serving database, and it does so before the persistence artifact export. A limit observed in that window stopped a worker that had already published, and the response still said serving_index_preserved=true while the previous index had been replaced. Publication and resource termination now coordinate through a per-attempt publication fence: a never-created path next to a reserved unique file in the private worker directory, which both sides claim by creating it exclusively (cbm_fopen "wbx": O_EXCL on POSIX, CREATE_NEW on Windows), so exactly one of them can win. - The worker claims it through a new pipeline publication gate that cbm_pipeline_run calls once, after the last pre-publication check and before the serving database is touched. Losing means the supervisor already stopped the attempt for a limit: the run aborts before publication and the previous generation stays in place. - The supervisor claims it before it terminates the worker for a limit or a failed RSS probe. Losing means the worker already committed to publishing: the limit is no longer enforced for that attempt, which runs to completion, and index.supervisor.resource_limit_after_publication is logged instead. A claim that can be neither won nor lost is retried on the next poll rather than risk stopping a committed worker. - At terminal the supervisor settles the fence into result.publication_committed. An unquiesced tree keeps the fence, so a surviving member can never publish after the report. serving_index_preserved now describes the request, not only its last attempt: a servable index existed when the request began, no attempt of the request committed to publishing (an earlier attempt can publish and then crash into recovery), and that index is still servable. The direct path and the daemon job both fold every attempt into these request facts; cbm_mcp_index_worker_resource_response takes them as a third argument, and NULL claims nothing. The fence path travels as an optional last pair of the exact worker grammar (--index-worker-publish-fence), accepted by the worker argv parser and by the daemon bootstrap role check, stripped by the CLI and installed by both worker entries. Limits stay opt-in: without index_max_rss_mb or index_max_duration_seconds there is no fence, no extra argv, no claim and no extra database read. Graph content is unchanged either way. RED (c78c077 plus only the new end-to-end test): a FIFO at the artifact temp path holds the real worker after the rename, and the duration clock crosses its limit only once the serving database has been replaced. The supervisor stops the worker there and answers resource_limit_exceeded with serving_index_preserved=true over a replaced database. GREEN: the worker claimed first, the supervisor logs the limit as not enforced, and the released worker finishes and reports the published index. Signed-off-by: Martin Vogel <martin.vogel.tech@gmail.com>
1 parent c78c077 commit 435a88b

15 files changed

Lines changed: 992 additions & 20 deletions

‎docs/CONFIGURATION.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -94,7 +94,7 @@ Current keys:
9494
| `index_max_files` | `off` | Optional maximum number of accepted source files in one discovery run. |
9595
| `index_max_source_mb` | `off` | Optional maximum accepted source size in MiB in one discovery run. |
9696
| `index_max_rss_mb` | `off` | Optional maximum charged memory in MiB for the complete contained index-worker process tree (`64..1048576`). On macOS this is phys_footprint, not resident size. |
97-
| `index_max_duration_seconds` | `off` | Optional maximum wall-clock duration in seconds for the whole index request, including crash/hang recovery (`1..86400`). |
97+
| `index_max_duration_seconds` | `off` | Optional maximum wall-clock duration in seconds for the whole index request, including crash/hang recovery (`1..86400`). Like `index_max_rss_mb`, it stops a worker only before it starts to publish the new index; see [INDEX_RESOURCE_LIMITS.md](INDEX_RESOURCE_LIMITS.md). |
9898

9999
For a watched project, `index_status` names the polling method in `watch.strategy`:
100100
`git` (HEAD and dirty-state polling), `tree` (a non-git root polled by the

‎docs/INDEX_RESOURCE_LIMITS.md‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,18 @@ terminal only after the tree is quiescent or a bounded containment failure is
5858
explicitly surfaced. Resource termination is not retried and does not
5959
quarantine a source file.
6060

61+
A worker limit never stops a worker after it has started to publish. The
62+
supervisor enforces the limits, but only the worker replaces the serving
63+
database, so the two coordinate through a per-attempt publication claim that
64+
exactly one of them can win. The worker claims it immediately before it
65+
replaces the serving database; the supervisor claims it before it stops the
66+
worker for a limit. If the limit is reached first, the worker can no longer
67+
publish and the previous index stays untouched. If the worker claimed first,
68+
the limits no longer apply to that attempt: the new index is published and the
69+
rest of the attempt, including the persistence artifact export, runs to
70+
completion. The supervisor then logs `index.supervisor.resource_limit_after_publication`
71+
instead of stopping it. With both worker limits `off` there is no claim.
72+
6173
If RSS is enabled and three consecutive probes cannot obtain any trustworthy
6274
tree measurement while the root worker is still running, CBM fails closed with
6375
`code=resource_probe_failed`. A probe is untrustworthy as soon as one member of
@@ -102,7 +114,10 @@ Worker limit failures use the same shape with `stage=worker`,
102114
`resource=rss_bytes` and `unit=bytes`, or `resource=duration_ms` and
103115
`unit=milliseconds`. RSS measurement failures use `code=resource_probe_failed`
104116
and omit `observed`, `limit`, and `unit` because no trustworthy observation was
105-
available.
117+
available. For a worker failure `serving_index_preserved` describes the whole
118+
request: it is true only when a servable index existed when the request began,
119+
no attempt of the request (including an earlier one that crashed and was
120+
recovered) claimed publication, and that index still serves.
106121

107122
## Trust and compatibility
108123

‎src/daemon/application.c‎

Lines changed: 22 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -160,6 +160,10 @@ struct cbm_daemon_application_job {
160160
bool async_owned;
161161
bool worker_started;
162162
uint64_t request_started_ms;
163+
/* What a worker resource failure of this job is reported against; set
164+
* before the first attempt, folded over every attempt (mcp_internal.h). */
165+
bool request_facts_begun;
166+
cbm_mcp_index_request_facts_t request_facts;
163167
cbm_daemon_application_job_t *next;
164168
};
165169

@@ -1346,6 +1350,17 @@ static void application_cancelled_result(cbm_index_worker_result_t *result) {
13461350
result->tree_quiesced = true;
13471351
}
13481352

1353+
/* Only an enabled worker limit can end a job in a worker resource failure, the
1354+
* one response the request facts feed; without one nothing is read. */
1355+
static void application_job_request_facts_begin(cbm_daemon_application_job_t *job) {
1356+
cbm_index_resource_policy_t policy;
1357+
char error[CBM_SZ_256] = {0};
1358+
if (cbm_mcp_index_policy_from_internal_args(job->args_json, &policy, error, sizeof(error)) &&
1359+
cbm_index_policy_worker_enabled(&policy)) {
1360+
cbm_mcp_index_request_facts_begin(job->args_json, &job->request_facts);
1361+
}
1362+
}
1363+
13491364
static application_attempt_status_t application_job_run_attempt(cbm_daemon_application_job_t *job,
13501365
const char *marker_path,
13511366
const char *quarantine_path,
@@ -1382,6 +1397,10 @@ static application_attempt_status_t application_job_run_attempt(cbm_daemon_appli
13821397
if (job->request_started_ms == 0) {
13831398
job->request_started_ms = cbm_index_worker_now_ms();
13841399
}
1400+
if (!job->request_facts_begun) {
1401+
job->request_facts_begun = true;
1402+
application_job_request_facts_begin(job);
1403+
}
13851404
application_tmp_lock();
13861405
int start_result = application->worker_ops.start(
13871406
application->worker_ops.context, job->args_json, memory_budget_bytes, marker_path,
@@ -1573,6 +1592,7 @@ static application_attempt_decision_t application_consume_attempt(
15731592
application_attempt_disposition(job, attempt);
15741593
application_record_attempt(attempt, &execution->last_result, &execution->have_last_result,
15751594
execution->last_log);
1595+
cbm_mcp_index_request_facts_record(&job->request_facts, &attempt->result);
15761596
if (disposition == CBM_MCP_SUPERVISED_RESULT_SUCCESS) {
15771597
execution->response = attempt->result.response;
15781598
attempt->result.response = NULL;
@@ -1581,8 +1601,8 @@ static application_attempt_decision_t application_consume_attempt(
15811601
return APPLICATION_ATTEMPT_DECISION_SUCCESS;
15821602
}
15831603
if (disposition == CBM_MCP_SUPERVISED_RESULT_RESOURCE_FAILURE) {
1584-
execution->response =
1585-
cbm_mcp_index_worker_resource_response(job->args_json, &attempt->result);
1604+
execution->response = cbm_mcp_index_worker_resource_response(
1605+
job->args_json, &attempt->result, &job->request_facts);
15861606
application_attempt_free(attempt);
15871607
return APPLICATION_ATTEMPT_DECISION_STOP;
15881608
}

‎src/daemon/bootstrap.c‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -141,6 +141,12 @@ static bool bootstrap_worker_argv_exact(int argc, char *const argv[]) {
141141
}
142142
next += 2;
143143
}
144+
if (next < argc && bootstrap_arg_is(argv[next], "--index-worker-publish-fence")) {
145+
if (next + 1 >= argc || !argv[next + 1] || !argv[next + 1][0]) {
146+
return false;
147+
}
148+
next += 2;
149+
}
144150
return next == argc;
145151
}
146152

‎src/main.c‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -897,6 +897,8 @@ static int run_cli(int argc, char **argv, cbm_project_lock_manager_t *project_lo
897897
const char *worker_marker = cli_strip_flag_value(&argc, argv, CBM_INDEX_WORKER_MARKER_ARG);
898898
const char *worker_quarantine =
899899
cli_strip_flag_value(&argc, argv, CBM_INDEX_WORKER_QUARANTINE_ARG);
900+
const char *worker_publish_fence =
901+
cli_strip_flag_value(&argc, argv, CBM_INDEX_WORKER_PUBLISH_FENCE_ARG);
900902
if (index_worker) {
901903
/* The graph lives on this process's heaps: SQLite gets heaps of its
902904
* own here, and only here (cbm_sqlite_dedicated_heap). */
@@ -906,6 +908,7 @@ static int run_cli(int argc, char **argv, cbm_project_lock_manager_t *project_lo
906908
cbm_index_set_worker_role_options(index_worker, response_out, worker_single_thread,
907909
worker_marker, worker_quarantine,
908910
cbm_index_worker_memory_budget_bytes());
911+
cbm_index_set_worker_publish_fence(index_worker ? worker_publish_fence : NULL);
909912

910913
if (argc < MAIN_MIN_ARGC) {
911914
(void)fprintf(stderr, CLI_USAGE);
@@ -3188,6 +3191,7 @@ int main(int argc, char **argv) {
31883191
cbm_index_set_worker_role_options(true, invocation.response_out, invocation.single_thread,
31893192
invocation.marker_file, invocation.quarantine_file,
31903193
invocation.memory_budget_bytes);
3194+
cbm_index_set_worker_publish_fence(invocation.publish_fence_file);
31913195
#ifndef _WIN32
31923196
/* Split into three ordered steps rather than one condition, because the
31933197
* ORDER is load-bearing and the middle step only exists in test builds:

0 commit comments

Comments
 (0)