The harness is a one-at-a-time polling worker. The local Axum server is supervision only; package jobs and result delivery are outbound to Tropibot.
claim / heartbeat / complete over HTTPS
Tropibot <--------------------------------------------> Package harness
|
+-- RunStore → /data
+-- RunController → Dappmanager MCP
+-- LogAnalyzer → heuristic / Nexus
+-- local /healthz and /readyz
CoordinatorClient owns the three normative worker endpoints and their DTOs. It adds bearer authentication and the package version user-agent, bounds request timeouts and response bytes, previews errors, and classifies retryable failures. Golden fixtures make the camelCase JSON contract explicit.
Planned restart control is an in-process accepting → draining → drained state machine. Draining wakes an idle poll immediately and prevents another claim request. Work already claimed—including a response crossing the drain boundary—continues through cleanup and durable completion before the worker publishes drained. The supervision server stays available for status and resume operations; readiness is false while draining or drained. SIGTERM requests the same drain and stops future loop iterations. Docker's default 10-second stop timeout does not cover a long Dappmanager mutation, so a stop during a job relies on restart recovery; operators drain before restarting.
PackageHarnessWorker persists a full claim before execution, then starts a heartbeat task beside the controller. The controller depends only on the narrow RunProgress port: it publishes its already-persisted phase and reads cancellation/claim-loss signals. No coordinator HTTP call runs in the controller or can block cleanup.
RunRecord.worker holds the opaque claim token, mutation/cleanup flag, final worker error when no normal result exists, exact pending completion JSON, acknowledgement status, and any manual-recovery hold (its kind and reason). The atomic file store writes a temporary file in /data and persists it atomically. Only one unacknowledged claimed record is allowed to proceed. On restart, pending completion bytes are redelivered even under a completion-conflict hold. Compatibility rejections retry automatically with capped backoff, and an acknowledgement clears the hold.
On startup the worker finds an unresolved record before it polls. It retries a pending completion first. If an active job might have changed Dappmanager, it checks the target, refuses the harness and core packages, and executes the persisted recovery plan: reinstall the explicit IPFS baseline (when supplied) or latest published release with volumes preserved when the target existed before the run, or remove a target that was absent before the run. The restore_baseline plan takes its immutable IPFS reference from the persisted request and verifies the restored origin; missing or mutable references are rejected before removal. A cleanup that already passed, in the controller or by operator confirmation, is not repeated. Heartbeats run throughout restart cleanup and delivery, so a long reinstall is not mistaken for a lost worker. It verifies the final inventory, then sends an interrupted worker error. It never reruns the interrupted test. Successfully reconciled claim loss releases the obsolete local claim and lets Tropibot decide whether another claim is allowed. Cleanup and completion-conflict holds keep the worker task alive but not ready. POST /operator/recovery/action continues after verified cleanup, records verified cleanup, retry exact persisted completion bytes, or explicitly accept the coordinator's existing result and release local delivery state. Every action is scoped to an exact run, and conflict release is rejected until cleanup is confirmed. The claim loop also remains paused, rather than exiting, when Tropibot reports an unresolved lost job. The UI separately proxies Tropibot's worker-authenticated lost-job lookup and exact-job ready command through GET /api/coordinator/lost-job and POST /operator/coordinator/ready; the browser never receives the coordinator token. The latter always injects cleanupConfirmed: true, lets the operator choose whether Tropibot should requeue the package, and wakes polling only after Tropibot accepts the recovery. It refuses a job this worker still holds locally, because that worker is recovering rather than lost. Ordinary delivery compatibility failures self-heal when Tropibot becomes compatible.
The controller’s baseline and candidate paths share stabilization and capture. It always resolves the requested baseline first; an omitted baselineRef means the latest published release and never means “whatever is installed.” An existing target is reused only when its observed version and install origin match the resolved baseline (Dappmanager records an origin only for IPFS installs, so an earlier IPFS build with the released version is not reused), and cleanup reinstalls the explicit IPFS baseline when supplied, or the latest published release otherwise, regardless of the pre-run version. A deterministic ENS resolution failure for a registry baseline completes as inconclusive / baseline_unresolvable before any mutation; network failures and unrelated reverts remain infrastructure errors. The skipped comparison is delivered as a normal terminal result and does not pause the worker or retry the registry lookup within that job. After each install or update, the package must report the version and install origin its preview resolved; a reported success that left another version or build running is an infrastructure error, never a pass. The origin check catches a candidate build that keeps the baseline version. The same version-and-origin check confirms an install or update whose MCP call timed out, instead of resending it. If the target was absent before the run, cleanup removes it. Stabilization requires a non-empty, all-running container list with stable reported identity, acceptable Docker health, and passing application probes across consecutive samples. Failed detail calls, empty lists, non-running states, failed probes, and container identity or restart changes reset the streak. Evidence history, log collection, redaction, and analysis input are bounded.
The Dappmanager adapter uses separate read and mutation timeouts. Install, update, and removal calls retry only failures classified as transient transport or package-download errors, with bounded exponential backoff. Before reissuing a timed-out operation, it reconciles observable state so an installed package, reached version (compared with the version the preview resolves, since candidate refs are IPFS hashes), or completed removal is accepted without a duplicate mutation. When Dappmanager reports that an earlier attempt is still running, the adapter waits up to the mutation timeout for the requested state instead of assuming success. Authentication, validation, required-setup, and malformed-response failures are terminal. The baseline preview determines the version used for testing; registry recovery resolves the latest release again before removing the target, and the pre-run installed version is ignored. Cleanup verifies restored identity and application readiness, followed by the final package inventory within the same post-install deadline. Failed verification, unexpected packages, or changed/missing unrelated packages make cleanup fail and hold the worker. Volume contents are preserved, not rolled back.
Readiness profiles are parsed and validated at startup from operator-only READINESS_PROFILES; neither coordinator jobs nor model output can choose probe destinations. Each phase has one real elapsed deadline across metadata reads, HTTP/TCP/read-only JSON-RPC probes, and polling. A sample requires all containers running, any reported Docker health healthy (or absent/none unless explicitly required), and every configured probe passing. Identity, creation/start time, restart count, or container-set changes reset consecutive stability. Missing optional metadata is retained as missing, rather than inferred. Probe requests have five-second deadlines and bounded response bodies; redirects are disabled and diagnostics retain only safe names and generic errors. Raw MCP details preserve optional container metadata omitted by the typed client, while the coordinator's v1 container projection stays unchanged.
The deterministic log heuristic compares signatures within the same container, so an old failure in another service cannot hide a new regression. Non-fatal wording does not count as a fatal signature; a later genuine fatal occurrence still does.
Nexus receives redacted, bounded JSON evidence batches with no tools or conversation history. The planner accounts for JSON escaping and UTF-8 boundaries, includes every final captured log window, and keeps baseline/candidate container pairs together when they fit. Oversized pairs are partitioned into explicitly scoped service windows. It never truncates a serialized object or silently drops later containers. Collection facts report stabilization, expected container counts, and log-read errors. At most three requests run concurrently, with a 64-request ceiling and a single overall analysis deadline. Completed batch findings survive other batch failures or deadline expiry; incomplete coverage is recorded in analyzerErrors and the summary. Critical findings are prioritized during bounded aggregation. Nexus remains advisory, and no Nexus result can turn a deterministic failure into a pass.
The optional Jev System One classifier is an observation aid, separate from the explanation model. Before each side's log capture is finalized, it can request at most two additional observations if ongoing initialization is evident with confidence and selected probability at least 0.8. Follow-ups wait five seconds; classifier requests are limited to five seconds and follow-up log reads to ten seconds. Empty captures also receive bounded rechecks. Classifier failures and uncertain decisions preserve the existing logs and do not block comparative analysis, package cleanup, or result delivery. It cannot declare a package healthy or filter evidence. Captured logs retain their most recent 64 KiB after redaction; the analysis path no longer clips them again at 16 KiB or ignores services after the twentieth entry.
V1 deliberately does not contain leases, fencing tokens, automatic reassignment, distributed persistence, worker scheduling, object storage, or per-worker authentication. The coordinator and persistence ports keep those additions possible without changing the deterministic execution core.