Skip to content

Commit bf0b94c

Browse files
committed
drain
1 parent a9a367f commit bf0b94c

10 files changed

Lines changed: 564 additions & 28 deletions

File tree

‎README.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,8 @@ The endpoint accepts only cleanup-related manual holds. It preserves the claim t
9797

9898
The dashboard's local worker control remains visible for completion conflicts. It shows whether cleanup is confirmed, a completion payload is persisted, the local claim is held, and Tropibot acknowledged a result. Once cleanup is confirmed, the operator can either retry the exact persisted completion or explicitly accept Tropibot's existing result and release the local claim. The latter is destructive to the saved delivery obligation and requires typing the exact run ID. The worker remains alive and paused while waiting for either action, so no package restart is needed.
9999

100+
For planned package updates or host maintenance, choose **Drain worker** in the dashboard first. The lifecycle moves from `accepting` to `draining`; no new claim request is started, while an already claimed job is allowed to finish mutation, mandatory cleanup, and persisted completion delivery. Restart only after the dashboard reports `drained` and **Safe to restart**. **Resume accepting jobs** cancels or exits a drain. A claim response that crosses the drain boundary is completed safely because the v1 coordinator protocol has no claim-release operation. `POST /operator/worker/drain` and `POST /operator/worker/resume` expose the same controls. `/readyz` returns unavailable while draining or drained, but `/healthz` remains a process liveness check. SIGTERM also requests a drain, and the Compose service grants up to two hours for active safe work to finish.
101+
100102
Heartbeats run independently of package execution. A cancellation request is observed at safe phase boundaries and inside stabilization polling. It prevents a new mutating phase but never bypasses required cleanup. If Tropibot no longer recognizes the claim, the worker finishes the current operation, reconciles cleanup, and releases the obsolete local claim when the target is safe.
101103

102104
## Dappmanager MCP

‎docker-compose.yml‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,9 @@ services:
66
dockerfile: Dockerfile
77
image: package-harness.dnp.dappnode.eth:0.1.1
88
restart: unless-stopped
9+
# Planned drains normally finish before shutdown. If SIGTERM arrives first,
10+
# keep Docker from killing an in-flight mutation or mandatory cleanup.
11+
stop_grace_period: 2h
912
environment:
1013
PORT: "8080"
1114
DATA_DIR: /data

‎docs/architecture.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,8 @@ Tropibot <--------------------------------------------> Package harness
1414

1515
`CoordinatorClient` owns the three normative worker endpoints and their DTOs. It adds bearer authentication and the package version user-agent, bounds request timeouts and response bytes, previews errors, and classifies retryable failures. Golden fixtures make the camelCase JSON contract explicit.
1616

17+
Planned restart control is an in-process `accepting → draining → drained` state machine. Draining wakes an idle poll immediately and prevents another claim request. Work already claimed—including a response crossing the drain boundary—continues through cleanup and durable completion before the worker publishes `drained`. The supervision server stays available for status and resume operations; readiness is false while draining or drained. SIGTERM requests the same drain and stops future loop iterations, while Docker's stop grace period leaves enough time for long Dappmanager mutations and mandatory cleanup.
18+
1719
`PackageHarnessWorker` persists a full claim before execution, then starts a heartbeat task beside the controller. The controller depends only on the narrow `RunProgress` port: it publishes its already-persisted phase and reads cancellation/claim-loss signals. No coordinator HTTP call runs in the controller or can block cleanup.
1820

1921
`RunRecord.worker` holds the opaque claim token, mutation/cleanup flag, final worker error when no normal result exists, exact pending completion JSON, acknowledgement status, and a manual-recovery reason. The atomic file store writes a temporary file in `/data` and persists it atomically. Only one unacknowledged claimed record is allowed to proceed. Pending completion bytes take precedence over legacy delivery-related recovery markers during restart. Compatibility rejections retry automatically with capped backoff, and an acknowledgement clears any stale marker.

‎src/api/dashboard/dashboard.css‎

Lines changed: 73 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,16 @@ button {
152152
box-shadow: 0 0 0 4px var(--red-soft);
153153
}
154154

155+
.worker-state[data-status="draining"] .state-dot {
156+
background: var(--yellow);
157+
box-shadow: 0 0 0 4px var(--yellow-soft);
158+
}
159+
160+
.worker-state[data-status="drained"] .state-dot {
161+
background: var(--green);
162+
box-shadow: 0 0 0 4px var(--green-soft);
163+
}
164+
155165
.hero {
156166
display: flex;
157167
gap: 48px;
@@ -295,7 +305,8 @@ summary:focus-visible {
295305
}
296306

297307
.coordinator-recovery,
298-
.local-recovery {
308+
.local-recovery,
309+
.drain-control {
299310
margin-top: 22px;
300311
padding: 26px 28px;
301312
border: 1px solid rgba(255, 107, 104, 0.42);
@@ -306,6 +317,13 @@ summary:focus-visible {
306317
box-shadow: var(--shadow);
307318
}
308319

320+
.drain-control {
321+
border-color: rgba(112, 167, 255, 0.3);
322+
background:
323+
linear-gradient(120deg, rgba(112, 167, 255, 0.08), transparent 55%),
324+
var(--panel);
325+
}
326+
309327
.local-recovery {
310328
border-color: rgba(255, 122, 26, 0.5);
311329
background:
@@ -327,6 +345,7 @@ summary:focus-visible {
327345

328346
.coordinator-recovery-copy,
329347
.local-recovery-copy,
348+
.drain-copy,
330349
.recovery-guidance {
331350
max-width: 850px;
332351
margin: 16px 0 22px;
@@ -335,6 +354,49 @@ summary:focus-visible {
335354
line-height: 1.6;
336355
}
337356

357+
.drain-status {
358+
display: grid;
359+
grid-template-columns: repeat(2, minmax(0, 1fr));
360+
gap: 1px;
361+
overflow: hidden;
362+
margin-bottom: 22px;
363+
border: 1px solid var(--line);
364+
border-radius: 12px;
365+
background: var(--line);
366+
}
367+
368+
.drain-status > div {
369+
padding: 15px 17px;
370+
background: rgba(17, 16, 15, 0.72);
371+
}
372+
373+
.drain-status span,
374+
.drain-status strong {
375+
display: block;
376+
}
377+
378+
.drain-status span {
379+
margin-bottom: 7px;
380+
color: var(--subtle);
381+
font-size: 10px;
382+
font-weight: 700;
383+
letter-spacing: 0.08em;
384+
text-transform: uppercase;
385+
}
386+
387+
.drain-status strong {
388+
color: var(--muted);
389+
font-size: 12px;
390+
}
391+
392+
.drain-status.is-safe {
393+
border-color: rgba(80, 216, 144, 0.32);
394+
}
395+
396+
.drain-status.is-safe > div:last-child strong {
397+
color: var(--green);
398+
}
399+
338400
.local-recovery-copy {
339401
color: #f0d6c3;
340402
}
@@ -556,7 +618,8 @@ h2 {
556618
.badge.warning,
557619
.badge.pending,
558620
.badge.running,
559-
.badge.queued {
621+
.badge.queued,
622+
.badge.draining {
560623
color: var(--yellow);
561624
border-color: rgba(245, 196, 81, 0.25);
562625
background: var(--yellow-soft);
@@ -573,7 +636,8 @@ h2 {
573636
}
574637

575638
.badge.completed,
576-
.badge.skipped {
639+
.badge.skipped,
640+
.badge.drained {
577641
color: var(--blue);
578642
border-color: rgba(112, 167, 255, 0.24);
579643
background: rgba(112, 167, 255, 0.1);
@@ -1007,7 +1071,8 @@ h2 {
10071071
}
10081072

10091073
.coordinator-recovery,
1010-
.local-recovery {
1074+
.local-recovery,
1075+
.drain-control {
10111076
padding: 21px 18px;
10121077
}
10131078

@@ -1025,6 +1090,10 @@ h2 {
10251090
grid-template-columns: 1fr;
10261091
}
10271092

1093+
.drain-status {
1094+
grid-template-columns: 1fr;
1095+
}
1096+
10281097
.metric {
10291098
min-height: 120px;
10301099
padding: 20px;

‎src/api/dashboard/dashboard.js‎

Lines changed: 57 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@ const state = {
77
jobsFingerprint: "",
88
lostJob: null,
99
lostJobFingerprint: "",
10+
worker: null,
1011
expandedJobs: loadExpandedJobs(),
1112
};
1213

@@ -39,6 +40,13 @@ const elements = {
3940
localRecoveryContext: document.querySelector("#local-recovery-context"),
4041
localRecoveryExplanation: document.querySelector("#local-recovery-explanation"),
4142
localRecoveryActions: document.querySelector("#local-recovery-actions"),
43+
drainLifecycle: document.querySelector("#drain-lifecycle"),
44+
drainMessage: document.querySelector("#drain-message"),
45+
drainCurrentJob: document.querySelector("#drain-current-job"),
46+
drainSafety: document.querySelector("#drain-safety"),
47+
drainStatus: document.querySelector("#drain-status"),
48+
drainWorker: document.querySelector("#drain-worker"),
49+
resumeWorker: document.querySelector("#resume-worker"),
4250
};
4351

4452
function humanize(value) {
@@ -675,9 +683,56 @@ function renderJobs() {
675683
}
676684

677685
function renderWorker(worker) {
686+
state.worker = worker;
678687
elements.workerState.dataset.status = worker.status;
679688
elements.workerLabel.textContent = humanize(worker.status);
680689
elements.workerMessage.textContent = worker.message;
690+
elements.drainLifecycle.textContent = humanize(worker.lifecycle);
691+
elements.drainLifecycle.className = `badge ${worker.safeToRestart ? "ready" : worker.lifecycle}`;
692+
elements.drainMessage.textContent = worker.message;
693+
const active = state.jobs.find(
694+
(job) => !job.completionAcknowledged && job.hasClaim,
695+
);
696+
elements.drainCurrentJob.textContent = active
697+
? `${active.dnpName} · ${humanize(active.phase)}`
698+
: "No active job";
699+
elements.drainSafety.textContent = worker.safeToRestart
700+
? "Safe to restart"
701+
: worker.lifecycle === "draining"
702+
? "Waiting for safe completion"
703+
: "Drain required";
704+
elements.drainStatus.classList.toggle("is-safe", worker.safeToRestart);
705+
elements.drainWorker.disabled = worker.lifecycle !== "accepting";
706+
elements.resumeWorker.disabled = worker.lifecycle === "accepting";
707+
}
708+
709+
async function setWorkerLifecycle(action) {
710+
const draining = action === "drain";
711+
if (
712+
draining &&
713+
!window.confirm(
714+
"Stop claiming new jobs and finish all current mutation, cleanup, and delivery work before restart?",
715+
)
716+
) {
717+
return;
718+
}
719+
elements.drainWorker.disabled = true;
720+
elements.resumeWorker.disabled = true;
721+
try {
722+
const response = await fetch(`/operator/worker/${action}`, {
723+
method: "POST",
724+
headers: { accept: "application/json" },
725+
});
726+
const payload = await response.json();
727+
if (!response.ok) throw new Error(payload.error || `Request failed (${response.status})`);
728+
toast(payload.message, "success");
729+
state.jobsFingerprint = "";
730+
await loadJobs();
731+
window.setTimeout(() => loadJobs({ quiet: true }), 500);
732+
} catch (error) {
733+
toast(error instanceof Error ? error.message : String(error), "error");
734+
if (state.worker) renderWorker(state.worker);
735+
}
681736
}
682737

683738
async function loadJobs({ quiet = false } = {}) {
@@ -849,6 +904,8 @@ elements.readyAndRetry.addEventListener("click", () =>
849904
elements.readyWithoutRetry.addEventListener("click", () =>
850905
markCoordinatorReady(false, elements.readyWithoutRetry),
851906
);
907+
elements.drainWorker.addEventListener("click", () => setWorkerLifecycle("drain"));
908+
elements.resumeWorker.addEventListener("click", () => setWorkerLifecycle("resume"));
852909
elements.search.addEventListener("input", (event) => {
853910
state.query = event.target.value.trim().toLowerCase();
854911
renderJobs();

‎src/api/dashboard/index.html‎

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -66,6 +66,37 @@ <h1>Jobs, recovery, and delivery<br>in one place.</h1>
6666
</article>
6767
</section>
6868

69+
<section class="drain-control" id="drain-control" aria-live="polite">
70+
<div class="recovery-heading">
71+
<div>
72+
<p class="eyebrow">Planned restart control</p>
73+
<h2>Worker lifecycle</h2>
74+
</div>
75+
<span class="badge ready" id="drain-lifecycle">Accepting</span>
76+
</div>
77+
<p class="drain-copy" id="drain-message">
78+
The worker is accepting package jobs.
79+
</p>
80+
<div class="drain-status" id="drain-status">
81+
<div>
82+
<span>Current work</span>
83+
<strong id="drain-current-job">Checking…</strong>
84+
</div>
85+
<div>
86+
<span>Restart safety</span>
87+
<strong id="drain-safety">Not drained</strong>
88+
</div>
89+
</div>
90+
<div class="recovery-actions">
91+
<button class="button button-primary" id="drain-worker" type="button">
92+
Drain worker
93+
</button>
94+
<button class="button button-secondary" id="resume-worker" type="button" disabled>
95+
Resume accepting jobs
96+
</button>
97+
</div>
98+
</section>
99+
69100
<section class="local-recovery is-hidden" id="local-recovery" aria-live="polite">
70101
<div class="recovery-heading">
71102
<div>

0 commit comments

Comments
 (0)