You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit e8da0d1
Browse filesBrowse the repository at this point in the historyBrowse files
The endpoint accepts only cleanup-related manual holds. It preserves the claim token and stored result, clears the persisted hold, and wakes the paused worker without a container restart. It does not override coordinator completion conflicts.
96
+
The endpoint accepts only cleanup-related manual holds. It preserves the claim token and stored result, records the operator-verified cleanup, clears the persisted hold, and wakes the paused worker without a container restart.
97
+
98
+
The dashboard's local worker control remains visible for completion conflicts. It shows whether cleanup is confirmed, a completion payload is persisted, the local claim is held, and Tropibot acknowledged a result. Once cleanup is confirmed, the operator can either retry the exact persisted completion or explicitly accept Tropibot's existing result and release the local claim. The latter is destructive to the saved delivery obligation and requires typing the exact run ID. The worker remains alive and paused while waiting for either action, so no package restart is needed.
97
99
98
100
Heartbeats run independently of package execution. A cancellation request is observed at safe phase boundaries and inside stabilization polling. It prevents a new mutating phase but never bypasses required cleanup. If Tropibot no longer recognizes the claim, the worker finishes the current operation, reconciles cleanup, and releases the obsolete local claim when the target is safe.
`RunRecord.worker` holds the opaque claim token, mutation/cleanup flag, final worker error when no normal result exists, exact pending completion JSON, acknowledgement status, and a manual-recovery reason. The atomic file store writes a temporary file in `/data` and persists it atomically. Only one unacknowledged claimed record is allowed to proceed. Pending completion bytes take precedence over legacy delivery-related recovery markers during restart. Compatibility rejections retry automatically with capped backoff, and an acknowledgement clears any stale marker.
20
20
21
-
On startup the worker finds an unresolved record before it polls. It retries a pending completion first. If an active job might have changed Dappmanager, it checks the target, refuses the harness and core packages, and executes the persisted recovery plan: restore the exact pre-test or intentionally retained baseline, or remove a target that was absent before the run. It verifies the final inventory, then sends an `interrupted` worker error. It never reruns the interrupted test. Successfully reconciled claim loss releases the obsolete local claim and lets Tropibot decide whether another claim is allowed. A cleanup-related manual hold keeps the worker task alive but not ready; `POST /operator/recovery/continue` atomically clears the hold for an exact run and wakes the same task after an operator has fixed the node. The claim loop also remains paused, rather than exiting, when Tropibot reports an unresolved lost job. The UI separately proxies Tropibot's worker-authenticated lost-job lookup and exact-job ready command through `GET /api/coordinator/lost-job` and `POST /operator/coordinator/ready`; the browser never receives the coordinator token. The latter always injects `cleanupConfirmed: true`, lets the operator choose whether Tropibot should requeue the package, and wakes polling only after Tropibot accepts the recovery. Ordinary delivery compatibility failures self-heal when Tropibot becomes compatible.
21
+
On startup the worker finds an unresolved record before it polls. It retries a pending completion first. If an active job might have changed Dappmanager, it checks the target, refuses the harness and core packages, and executes the persisted recovery plan: restore the exact pre-test or intentionally retained baseline, or remove a target that was absent before the run. It verifies the final inventory, then sends an `interrupted` worker error. It never reruns the interrupted test. Successfully reconciled claim loss releases the obsolete local claim and lets Tropibot decide whether another claim is allowed. Cleanup and completion-conflict holds keep the worker task alive but not ready. `POST /operator/recovery/continue` remains the compatible cleanup-only endpoint; the dashboard uses `POST /operator/recovery/action` to record verified cleanup, retry exact persisted completion bytes, or explicitly accept the coordinator's existing result and release local delivery state. Every action is scoped to an exact run, and conflict release is rejected until cleanup is confirmed. The claim loop also remains paused, rather than exiting, when Tropibot reports an unresolved lost job. The UI separately proxies Tropibot's worker-authenticated lost-job lookup and exact-job ready command through `GET /api/coordinator/lost-job` and `POST /operator/coordinator/ready`; the browser never receives the coordinator token. The latter always injects `cleanupConfirmed: true`, lets the operator choose whether Tropibot should requeue the package, and wakes polling only after Tropibot accepts the recovery. Ordinary delivery compatibility failures self-heal when Tropibot becomes compatible.
22
22
23
23
The controller’s baseline and candidate paths share stabilization and capture. It always resolves the requested baseline first; an omitted `baselineRef` means the latest published release and never means “whatever is installed.” An existing target is reused only when its observed version matches the resolved baseline. Outside the retained-baseline allowlist, an original installed version remains the restoration target. For retained packages, the resolved published baseline becomes the restoration target so an unpublished candidate left on the node cannot be mistaken for a baseline. Stabilization requires a non-empty, all-running container list with the same sorted names across consecutive samples. Failed detail calls, empty lists, non-running states, and container-set changes reset the streak. Evidence history, log collection, redaction, and analysis input are bounded.
"Tropibot holds different state, but local cleanup is not recorded as safe. Verify the package state first; completion controls unlock after that confirmation.";
"Tropibot already holds different state for this claim. Retry is non-destructive and resends the exact saved payload. Accept coordinator result permanently discards the local pending payload and releases the claim.";
0 commit comments