Skip to content

Commit e8da0d1

Browse files
committed
Expand operator recovery controls
1 parent 1230454 commit e8da0d1

8 files changed

Lines changed: 472 additions & 56 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -93,7 +93,9 @@ curl --fail-with-body \
9393
http://HARNESS_HOST:8080/operator/recovery/continue
9494
```
9595

96-
The endpoint accepts only cleanup-related manual holds. It preserves the claim token and stored result, clears the persisted hold, and wakes the paused worker without a container restart. It does not override coordinator completion conflicts.
96+
The endpoint accepts only cleanup-related manual holds. It preserves the claim token and stored result, records the operator-verified cleanup, clears the persisted hold, and wakes the paused worker without a container restart.
97+
98+
The dashboard's local worker control remains visible for completion conflicts. It shows whether cleanup is confirmed, a completion payload is persisted, the local claim is held, and Tropibot acknowledged a result. Once cleanup is confirmed, the operator can either retry the exact persisted completion or explicitly accept Tropibot's existing result and release the local claim. The latter is destructive to the saved delivery obligation and requires typing the exact run ID. The worker remains alive and paused while waiting for either action, so no package restart is needed.
9799

98100
Heartbeats run independently of package execution. A cancellation request is observed at safe phase boundaries and inside stabilization polling. It prevents a new mutating phase but never bypasses required cleanup. If Tropibot no longer recognizes the claim, the worker finishes the current operation, reconciles cleanup, and releases the obsolete local claim when the target is safe.
99101

‎docs/architecture.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@ Tropibot <--------------------------------------------> Package harness
1818

1919
`RunRecord.worker` holds the opaque claim token, mutation/cleanup flag, final worker error when no normal result exists, exact pending completion JSON, acknowledgement status, and a manual-recovery reason. The atomic file store writes a temporary file in `/data` and persists it atomically. Only one unacknowledged claimed record is allowed to proceed. Pending completion bytes take precedence over legacy delivery-related recovery markers during restart. Compatibility rejections retry automatically with capped backoff, and an acknowledgement clears any stale marker.
2020

21-
On startup the worker finds an unresolved record before it polls. It retries a pending completion first. If an active job might have changed Dappmanager, it checks the target, refuses the harness and core packages, and executes the persisted recovery plan: restore the exact pre-test or intentionally retained baseline, or remove a target that was absent before the run. It verifies the final inventory, then sends an `interrupted` worker error. It never reruns the interrupted test. Successfully reconciled claim loss releases the obsolete local claim and lets Tropibot decide whether another claim is allowed. A cleanup-related manual hold keeps the worker task alive but not ready; `POST /operator/recovery/continue` atomically clears the hold for an exact run and wakes the same task after an operator has fixed the node. The claim loop also remains paused, rather than exiting, when Tropibot reports an unresolved lost job. The UI separately proxies Tropibot's worker-authenticated lost-job lookup and exact-job ready command through `GET /api/coordinator/lost-job` and `POST /operator/coordinator/ready`; the browser never receives the coordinator token. The latter always injects `cleanupConfirmed: true`, lets the operator choose whether Tropibot should requeue the package, and wakes polling only after Tropibot accepts the recovery. Ordinary delivery compatibility failures self-heal when Tropibot becomes compatible.
21+
On startup the worker finds an unresolved record before it polls. It retries a pending completion first. If an active job might have changed Dappmanager, it checks the target, refuses the harness and core packages, and executes the persisted recovery plan: restore the exact pre-test or intentionally retained baseline, or remove a target that was absent before the run. It verifies the final inventory, then sends an `interrupted` worker error. It never reruns the interrupted test. Successfully reconciled claim loss releases the obsolete local claim and lets Tropibot decide whether another claim is allowed. Cleanup and completion-conflict holds keep the worker task alive but not ready. `POST /operator/recovery/continue` remains the compatible cleanup-only endpoint; the dashboard uses `POST /operator/recovery/action` to record verified cleanup, retry exact persisted completion bytes, or explicitly accept the coordinator's existing result and release local delivery state. Every action is scoped to an exact run, and conflict release is rejected until cleanup is confirmed. The claim loop also remains paused, rather than exiting, when Tropibot reports an unresolved lost job. The UI separately proxies Tropibot's worker-authenticated lost-job lookup and exact-job ready command through `GET /api/coordinator/lost-job` and `POST /operator/coordinator/ready`; the browser never receives the coordinator token. The latter always injects `cleanupConfirmed: true`, lets the operator choose whether Tropibot should requeue the package, and wakes polling only after Tropibot accepts the recovery. Ordinary delivery compatibility failures self-heal when Tropibot becomes compatible.
2222

2323
The controller’s baseline and candidate paths share stabilization and capture. It always resolves the requested baseline first; an omitted `baselineRef` means the latest published release and never means “whatever is installed.” An existing target is reused only when its observed version matches the resolved baseline. Outside the retained-baseline allowlist, an original installed version remains the restoration target. For retained packages, the resolved published baseline becomes the restoration target so an unpublished candidate left on the node cannot be mistaken for a baseline. Stabilization requires a non-empty, all-running container list with the same sorted names across consecutive samples. Failed detail calls, empty lists, non-running states, and container-set changes reset the streak. Evidence history, log collection, redaction, and analysis input are bounded.
2424

‎src/api/dashboard/dashboard.css‎

Lines changed: 45 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -243,6 +243,12 @@ summary:focus-visible {
243243
background: var(--orange);
244244
}
245245

246+
.button-danger {
247+
color: #ffd7d5;
248+
border-color: rgba(255, 107, 104, 0.4);
249+
background: var(--red-soft);
250+
}
251+
246252
.updated {
247253
color: var(--subtle);
248254
font-size: 11px;
@@ -288,7 +294,8 @@ summary:focus-visible {
288294
color: var(--orange);
289295
}
290296

291-
.coordinator-recovery {
297+
.coordinator-recovery,
298+
.local-recovery {
292299
margin-top: 22px;
293300
padding: 26px 28px;
294301
border: 1px solid rgba(255, 107, 104, 0.42);
@@ -299,6 +306,13 @@ summary:focus-visible {
299306
box-shadow: var(--shadow);
300307
}
301308

309+
.local-recovery {
310+
border-color: rgba(255, 122, 26, 0.5);
311+
background:
312+
linear-gradient(120deg, rgba(255, 122, 26, 0.12), transparent 55%),
313+
var(--panel);
314+
}
315+
302316
.recovery-heading,
303317
.recovery-actions {
304318
display: flex;
@@ -311,14 +325,24 @@ summary:focus-visible {
311325
margin-bottom: 0;
312326
}
313327

314-
.coordinator-recovery-copy {
328+
.coordinator-recovery-copy,
329+
.local-recovery-copy,
330+
.recovery-guidance {
315331
max-width: 850px;
316332
margin: 16px 0 22px;
317333
color: var(--muted);
318334
font-size: 13px;
319335
line-height: 1.6;
320336
}
321337

338+
.local-recovery-copy {
339+
color: #f0d6c3;
340+
}
341+
342+
.recovery-explanation {
343+
max-width: 960px;
344+
}
345+
322346
.recovery-context {
323347
display: grid;
324348
grid-template-columns: 1.25fr 2fr 0.8fr 1.25fr;
@@ -330,6 +354,10 @@ summary:focus-visible {
330354
background: var(--line);
331355
}
332356

357+
.local-recovery .recovery-context {
358+
grid-template-columns: repeat(3, minmax(0, 1fr));
359+
}
360+
333361
.recovery-context > div {
334362
min-width: 0;
335363
padding: 15px 17px;
@@ -628,6 +656,11 @@ h2 {
628656
width: 100%;
629657
}
630658

659+
.stacked-actions {
660+
display: grid;
661+
gap: 8px;
662+
}
663+
631664
.notice {
632665
margin-bottom: 14px;
633666
padding: 12px 14px;
@@ -752,6 +785,10 @@ h2 {
752785
grid-template-columns: 1fr 1fr;
753786
}
754787

788+
.local-recovery .recovery-context {
789+
grid-template-columns: 1fr 1fr;
790+
}
791+
755792
.search input {
756793
width: 100%;
757794
}
@@ -785,7 +822,8 @@ h2 {
785822
grid-template-columns: 1fr 1fr;
786823
}
787824

788-
.coordinator-recovery {
825+
.coordinator-recovery,
826+
.local-recovery {
789827
padding: 21px 18px;
790828
}
791829

@@ -799,6 +837,10 @@ h2 {
799837
grid-template-columns: 1fr;
800838
}
801839

840+
.local-recovery .recovery-context {
841+
grid-template-columns: 1fr;
842+
}
843+
802844
.metric {
803845
min-height: 120px;
804846
padding: 20px;

‎src/api/dashboard/dashboard.js‎

Lines changed: 139 additions & 29 deletions
Original file line numberDiff line numberDiff line change
@@ -32,6 +32,13 @@ const elements = {
3232
lostJobHeartbeat: document.querySelector("#lost-job-heartbeat"),
3333
readyAndRetry: document.querySelector("#ready-and-retry"),
3434
readyWithoutRetry: document.querySelector("#ready-without-retry"),
35+
localRecovery: document.querySelector("#local-recovery"),
36+
localRecoveryTitle: document.querySelector("#local-recovery-title"),
37+
localRecoveryKind: document.querySelector("#local-recovery-kind"),
38+
localRecoveryReason: document.querySelector("#local-recovery-reason"),
39+
localRecoveryContext: document.querySelector("#local-recovery-context"),
40+
localRecoveryExplanation: document.querySelector("#local-recovery-explanation"),
41+
localRecoveryActions: document.querySelector("#local-recovery-actions"),
3542
};
3643

3744
function humanize(value) {
@@ -153,10 +160,106 @@ function renderCoordinatorRecovery() {
153160
: "Not reported";
154161
}
155162

163+
function recoveryContextItem(label, value) {
164+
const item = node("div");
165+
item.append(node("span", null, label), node("strong", null, value));
166+
return item;
167+
}
168+
169+
function recoveryActionButton(job, action, label, className = "button-primary") {
170+
const button = node("button", `button ${className}`, label);
171+
button.type = "button";
172+
button.addEventListener("click", (event) => {
173+
event.preventDefault();
174+
event.stopPropagation();
175+
runRecoveryAction(job, action, button);
176+
});
177+
return button;
178+
}
179+
180+
function recoveryPanelContent(job, compact = false) {
181+
const fragment = document.createDocumentFragment();
182+
const copy = node("p", compact ? null : "recovery-guidance");
183+
const actions = node("div", compact ? "stacked-actions" : "recovery-actions");
184+
185+
if (job.recoveryKind === "cleanup") {
186+
copy.textContent =
187+
"Verify the target package is restored or removed as intended. Confirming records the manual cleanup and asks the live worker to reconcile again.";
188+
actions.append(
189+
recoveryActionButton(
190+
job,
191+
"continueAfterCleanup",
192+
"Cleanup verified — continue",
193+
),
194+
);
195+
} else if (job.recoveryKind === "completion_conflict") {
196+
const cleanupConfirmed = ["passed", "skipped"].includes(job.cleanupStatus);
197+
if (!cleanupConfirmed) {
198+
copy.textContent =
199+
"Tropibot holds different state, but local cleanup is not recorded as safe. Verify the package state first; completion controls unlock after that confirmation.";
200+
actions.append(
201+
recoveryActionButton(job, "confirmCleanup", "Cleanup verified — unlock resolution"),
202+
);
203+
} else {
204+
copy.textContent =
205+
"Tropibot already holds different state for this claim. Retry is non-destructive and resends the exact saved payload. Accept coordinator result permanently discards the local pending payload and releases the claim.";
206+
actions.append(
207+
recoveryActionButton(job, "retryCompletion", "Retry exact completion"),
208+
recoveryActionButton(
209+
job,
210+
"acceptCoordinatorResult",
211+
"Accept Tropibot result & release",
212+
"button-danger",
213+
),
214+
);
215+
}
216+
} else {
217+
copy.textContent =
218+
"This hold has no safe automated override. Inspect the reason and local record, then refresh after correcting the underlying state.";
219+
}
220+
fragment.append(copy, actions);
221+
return fragment;
222+
}
223+
224+
function renderLocalRecovery() {
225+
const job = state.jobs.find((candidate) => candidate.manualRecoveryReason);
226+
elements.localRecovery.classList.toggle("is-hidden", !job);
227+
if (!job) return;
228+
229+
const titles = {
230+
cleanup: "Manual cleanup must be verified",
231+
completion_conflict: "Completion conflicts with Tropibot",
232+
manual: "Manual recovery required",
233+
};
234+
elements.localRecoveryTitle.textContent = titles[job.recoveryKind] || titles.manual;
235+
elements.localRecoveryKind.textContent = humanize(job.recoveryKind || "manual");
236+
elements.localRecoveryReason.textContent = job.manualRecoveryReason;
237+
elements.localRecoveryContext.replaceChildren(
238+
recoveryContextItem("Package", job.dnpName),
239+
recoveryContextItem("Run", job.runId),
240+
recoveryContextItem("Cleanup", humanize(job.cleanupStatus)),
241+
recoveryContextItem(
242+
"Saved delivery",
243+
job.pendingCompletion ? "Pending" : "None",
244+
),
245+
recoveryContextItem("Local claim", job.hasClaim ? "Held" : "Released"),
246+
recoveryContextItem(
247+
"Tropibot acknowledgement",
248+
job.completionAcknowledged ? "Recorded" : "Missing",
249+
),
250+
);
251+
elements.localRecoveryExplanation.replaceChildren();
252+
elements.localRecoveryActions.replaceChildren();
253+
const content = recoveryPanelContent(job);
254+
const children = [...content.childNodes];
255+
elements.localRecoveryExplanation.append(children[0]);
256+
if (children[1]) elements.localRecoveryActions.append(...children[1].childNodes);
257+
}
258+
156259
function renderJob(job) {
157260
const card = node("details", `job${job.requiresAttention ? " needs-attention" : ""}`);
158261
card.dataset.runId = job.runId;
159-
card.open = state.expandedJobs.has(job.runId);
262+
card.open = job.requiresAttention || state.expandedJobs.has(job.runId);
160263
card.addEventListener("toggle", () => {
161264
if (card.open) {
162265
state.expandedJobs.add(job.runId);
@@ -215,25 +318,21 @@ function renderJob(job) {
215318
}
216319
body.append(information);
217320

218-
if (job.canContinueAfterCleanup) {
321+
if (job.manualRecoveryReason) {
219322
const recovery = node("aside", "recovery-panel");
220-
recovery.append(node("h3", null, "Manual cleanup required"));
221323
recovery.append(
222324
node(
223-
"p",
325+
"h3",
224326
null,
225-
job.manualRecoveryReason ||
226-
"The worker is paused until the target has been cleaned up manually.",
327+
job.recoveryKind === "completion_conflict"
328+
? "Completion conflict"
329+
: "Manual cleanup required",
227330
),
228331
);
229-
const action = node("button", "button button-primary", "Cleanup done — continue");
230-
action.type = "button";
231-
action.addEventListener("click", (event) => {
232-
event.preventDefault();
233-
event.stopPropagation();
234-
continueRecovery(job, action);
235-
});
236-
recovery.append(action);
332+
recovery.append(
333+
node("p", null, job.manualRecoveryReason),
334+
);
335+
recovery.append(recoveryPanelContent(job, true));
237336
body.append(recovery);
238337
}
239338

@@ -274,6 +373,7 @@ async function loadJobs({ quiet = false } = {}) {
274373
if (jobsChanged) {
275374
renderMetrics();
276375
renderJobs();
376+
renderLocalRecovery();
277377
}
278378
elements.updated.textContent = `Updated ${new Intl.DateTimeFormat(undefined, {
279379
hour: "2-digit",
@@ -354,11 +454,10 @@ async function markCoordinatorReady(retry, button) {
354454
const blocked = payload.blockingJobIds?.length
355455
? ` Blocking jobs: ${payload.blockingJobIds.join(", ")}.`
356456
: "";
357-
toast(`Worker released. Retry: ${retryResult}.${blocked}`, "success");
358-
state.lostJob = null;
359-
state.lostJobFingerprint = "null";
360-
renderCoordinatorRecovery();
361-
renderMetrics();
457+
toast(
458+
`Tropibot accepted the command. Retry: ${retryResult}.${blocked} Checking the resulting worker state…`,
459+
"success",
460+
);
362461
await Promise.all([loadJobs(), loadCoordinatorRecovery()]);
363462
} catch (error) {
364463
toast(error instanceof Error ? error.message : String(error), "error");
@@ -369,30 +468,41 @@ async function markCoordinatorReady(retry, button) {
369468
}
370469
}
371470

372-
async function continueRecovery(job, button) {
373-
const confirmed = window.confirm(
374-
`Confirm that manual cleanup for ${job.dnpName} is complete and allow the worker to continue?`,
375-
);
376-
if (!confirmed) return;
471+
async function runRecoveryAction(job, action, button) {
472+
const acceptingCoordinator = action === "acceptCoordinatorResult";
473+
const confirmation = acceptingCoordinator
474+
? window.prompt(
475+
`This discards the saved local completion and trusts Tropibot's existing result.\n\nType the run ID to continue:\n${job.runId}`,
476+
) === job.runId
477+
: window.confirm(
478+
["continueAfterCleanup", "confirmCleanup"].includes(action)
479+
? `Confirm manual cleanup for ${job.dnpName} is complete and verified?`
480+
: `Retry the exact persisted completion for ${job.runId}?`,
481+
);
482+
if (!confirmation) return;
483+
const originalLabel = button.textContent;
377484
button.disabled = true;
378-
button.textContent = "Continuing…";
485+
button.textContent = acceptingCoordinator ? "Releasing…" : "Working…";
379486
try {
380-
const response = await fetch("/operator/recovery/continue", {
487+
const response = await fetch("/operator/recovery/action", {
381488
method: "POST",
382489
headers: {
383490
accept: "application/json",
384491
"content-type": "application/json",
385492
},
386-
body: JSON.stringify({ runId: job.runId }),
493+
body: JSON.stringify({ runId: job.runId, action }),
387494
});
388495
const payload = await response.json();
389496
if (!response.ok) throw new Error(payload.error || `Request failed (${response.status})`);
390-
toast(payload.message || "Worker recovery resumed", "success");
497+
toast(payload.message || "Worker recovery action accepted", "success");
498+
state.jobsFingerprint = "";
391499
await loadJobs();
500+
window.setTimeout(() => loadJobs({ quiet: true }), 750);
392501
} catch (error) {
393502
toast(error instanceof Error ? error.message : String(error), "error");
503+
} finally {
394504
button.disabled = false;
395-
button.textContent = "Cleanup done — continue";
505+
button.textContent = originalLabel;
396506
}
397507
}
398508

‎src/api/dashboard/index.html‎

Lines changed: 14 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,20 @@ <h1>Jobs, recovery, and delivery<br>in one place.</h1>
6666
</article>
6767
</section>
6868

69+
<section class="local-recovery is-hidden" id="local-recovery" aria-live="polite">
70+
<div class="recovery-heading">
71+
<div>
72+
<p class="eyebrow">Local worker control</p>
73+
<h2 id="local-recovery-title">Worker recovery required</h2>
74+
</div>
75+
<span class="badge failed" id="local-recovery-kind">Paused</span>
76+
</div>
77+
<p class="local-recovery-copy" id="local-recovery-reason"></p>
78+
<div class="recovery-context" id="local-recovery-context"></div>
79+
<div class="recovery-explanation" id="local-recovery-explanation"></div>
80+
<div class="recovery-actions" id="local-recovery-actions"></div>
81+
</section>
82+
6983
<section class="coordinator-recovery is-hidden" id="coordinator-recovery" aria-live="polite">
7084
<div class="recovery-heading">
7185
<div>

0 commit comments

Comments
 (0)