Skip to content

Commit c903fe6

Browse files
committed
Simplify baseline recovery policy
1 parent 0fd58b8 commit c903fe6

11 files changed

Lines changed: 72 additions & 310 deletions

File tree

‎.env.example‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,10 @@ STABILIZATION_TIMEOUT_MS=180000
2626
STABILIZATION_POLL_MS=5000
2727
STABILIZATION_REQUIRED_SAMPLES=3
2828
LOG_TAIL=300
29-
# Applies to targets the harness installed itself. Targets that were already
30-
# installed are always restored to their recorded pre-test version.
29+
# Applies when the target was absent before the run. Preexisting targets are
30+
# always restored to the baseline resolved for the run.
3131
CLEANUP_ENABLED=true
3232
CLEANUP_TIMEOUT_MS=60000
33-
# Keep expensive baselines installed between runs. Comma-separated DNP names.
34-
RETAIN_BASELINE_PACKAGES=near.dnp.dappnode.eth
3533

3634
# Optional advisory analysis
3735
NEXUS_API_KEY=

‎README.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,7 @@ PACKAGE_HARNESS_HEARTBEAT_SECONDS=20
4141

4242
The shared bearer token is appropriate for this small trusted v1 deployment. The protocol deliberately leaves room for later per-worker credentials or mTLS, but neither is part of this version. The worker never logs the token.
4343

44-
Use [`.env.example`](.env.example) for all timeout, Dappmanager, cleanup, and optional Nexus settings. `MCP_TIMEOUT_MS` bounds read-only Dappmanager calls at 30 seconds by default, while `MCP_MUTATION_TIMEOUT_MS` gives each install, update, or removal attempt 30 minutes by default (up to one hour). Transient transport and package-download failures are retried three times with bounded exponential backoff; tune this with `MCP_MUTATION_ATTEMPTS` and `MCP_MUTATION_RETRY_DELAY_MS`. `NEXUS_TIMEOUT_MS` gives advisory Nexus analysis five minutes by default. `RETAIN_BASELINE_PACKAGES` is a comma-separated allowlist for packages with expensive downloads: after the first successful baseline install, the harness records its expected version and immutable release reference, restores it after candidate testing, and reuses it on later runs instead of removing and downloading it again. `TROPIBOT_TIMEOUT_MS` bounds each coordinator request. The Dappmanager MCP token and Nexus key remain local and are never sent to Tropibot or persisted in a job record.
44+
Use [`.env.example`](.env.example) for all timeout, Dappmanager, cleanup, and optional Nexus settings. `MCP_TIMEOUT_MS` bounds read-only Dappmanager calls at 30 seconds by default, while `MCP_MUTATION_TIMEOUT_MS` gives each install, update, or removal attempt 30 minutes by default (up to one hour). Transient transport and package-download failures are retried three times with bounded exponential backoff; tune this with `MCP_MUTATION_ATTEMPTS` and `MCP_MUTATION_RETRY_DELAY_MS`. `NEXUS_TIMEOUT_MS` gives advisory Nexus analysis five minutes by default. If the target package was installed before a run, cleanup restores the exact baseline resolved for that run; otherwise cleanup removes the target. The pre-run package version never replaces the requested baseline. `TROPIBOT_TIMEOUT_MS` bounds each coordinator request. The Dappmanager MCP token and Nexus key remain local and are never sent to Tropibot or persisted in a job record.
4545

4646
## Local fake mode
4747

‎docker-compose.yml‎

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -27,7 +27,6 @@ services:
2727
LOG_TAIL: "300"
2828
CLEANUP_ENABLED: "true"
2929
CLEANUP_TIMEOUT_MS: "60000"
30-
RETAIN_BASELINE_PACKAGES: "near.dnp.dappnode.eth"
3130
TROPIBOT_URL: ""
3231
PACKAGE_HARNESS_WORKER_ID: ""
3332
PACKAGE_HARNESS_WORKER_TOKEN: ""

‎docs/architecture.md‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -20,11 +20,11 @@ Planned restart control is an in-process `accepting → draining → drained` st
2020

2121
`RunRecord.worker` holds the opaque claim token, mutation/cleanup flag, final worker error when no normal result exists, exact pending completion JSON, acknowledgement status, and a manual-recovery reason. The atomic file store writes a temporary file in `/data` and persists it atomically. Only one unacknowledged claimed record is allowed to proceed. Pending completion bytes take precedence over legacy delivery-related recovery markers during restart. Compatibility rejections retry automatically with capped backoff, and an acknowledgement clears any stale marker.
2222

23-
On startup the worker finds an unresolved record before it polls. It retries a pending completion first. If an active job might have changed Dappmanager, it checks the target, refuses the harness and core packages, and executes the persisted recovery plan: restore the exact pre-test or intentionally retained baseline, or remove a target that was absent before the run. It verifies the final inventory, then sends an `interrupted` worker error. It never reruns the interrupted test. Successfully reconciled claim loss releases the obsolete local claim and lets Tropibot decide whether another claim is allowed. Cleanup and completion-conflict holds keep the worker task alive but not ready. `POST /operator/recovery/continue` remains the compatible cleanup-only endpoint; the dashboard uses `POST /operator/recovery/action` to record verified cleanup, retry exact persisted completion bytes, or explicitly accept the coordinator's existing result and release local delivery state. Every action is scoped to an exact run, and conflict release is rejected until cleanup is confirmed. The claim loop also remains paused, rather than exiting, when Tropibot reports an unresolved lost job. The UI separately proxies Tropibot's worker-authenticated lost-job lookup and exact-job ready command through `GET /api/coordinator/lost-job` and `POST /operator/coordinator/ready`; the browser never receives the coordinator token. The latter always injects `cleanupConfirmed: true`, lets the operator choose whether Tropibot should requeue the package, and wakes polling only after Tropibot accepts the recovery. Ordinary delivery compatibility failures self-heal when Tropibot becomes compatible.
23+
On startup the worker finds an unresolved record before it polls. It retries a pending completion first. If an active job might have changed Dappmanager, it checks the target, refuses the harness and core packages, and executes the persisted recovery plan: restore the exact test baseline when the target existed before the run, or remove a target that was absent before the run. It verifies the final inventory, then sends an `interrupted` worker error. It never reruns the interrupted test. Successfully reconciled claim loss releases the obsolete local claim and lets Tropibot decide whether another claim is allowed. Cleanup and completion-conflict holds keep the worker task alive but not ready. `POST /operator/recovery/continue` remains the compatible cleanup-only endpoint; the dashboard uses `POST /operator/recovery/action` to record verified cleanup, retry exact persisted completion bytes, or explicitly accept the coordinator's existing result and release local delivery state. Every action is scoped to an exact run, and conflict release is rejected until cleanup is confirmed. The claim loop also remains paused, rather than exiting, when Tropibot reports an unresolved lost job. The UI separately proxies Tropibot's worker-authenticated lost-job lookup and exact-job ready command through `GET /api/coordinator/lost-job` and `POST /operator/coordinator/ready`; the browser never receives the coordinator token. The latter always injects `cleanupConfirmed: true`, lets the operator choose whether Tropibot should requeue the package, and wakes polling only after Tropibot accepts the recovery. Ordinary delivery compatibility failures self-heal when Tropibot becomes compatible.
2424

25-
The controller’s baseline and candidate paths share stabilization and capture. It always resolves the requested baseline first; an omitted `baselineRef` means the latest published release and never means “whatever is installed.” An existing target is reused only when its observed version matches the resolved baseline. Outside the retained-baseline allowlist, an original installed version remains the restoration target. For retained packages, the resolved published baseline becomes the restoration target so an unpublished candidate left on the node cannot be mistaken for a baseline. Stabilization requires a non-empty, all-running container list with the same sorted names across consecutive samples. Failed detail calls, empty lists, non-running states, and container-set changes reset the streak. Evidence history, log collection, redaction, and analysis input are bounded.
25+
The controller’s baseline and candidate paths share stabilization and capture. It always resolves the requested baseline first; an omitted `baselineRef` means the latest published release and never means “whatever is installed.” An existing target is reused only when its observed version matches the resolved baseline, and cleanup restores that resolved baseline after candidate testing regardless of the pre-run version. If the target was absent before the run, cleanup removes it. Stabilization requires a non-empty, all-running container list with the same sorted names across consecutive samples. Failed detail calls, empty lists, non-running states, and container-set changes reset the streak. Evidence history, log collection, redaction, and analysis input are bounded.
2626

27-
The Dappmanager adapter uses separate read and mutation timeouts. Install, update, and removal calls retry only failures classified as transient transport or package-download errors, with bounded exponential backoff. Before reissuing a timed-out operation, it reconciles observable state so an installed package, reached version, or completed removal is accepted without a duplicate mutation. Authentication, validation, required-setup, and malformed-response failures are terminal. Packages in `RETAIN_BASELINE_PACKAGES` receive an exact restore plan for the resolved published baseline; that makes expensive baselines reusable across jobs and replaces any stale unpublished candidate left by an interrupted run. Cleanup verifies the final package inventory; failed verification or unexpected packages introduced by the run make cleanup fail rather than producing a clean result.
27+
The Dappmanager adapter uses separate read and mutation timeouts. Install, update, and removal calls retry only failures classified as transient transport or package-download errors, with bounded exponential backoff. Before reissuing a timed-out operation, it reconciles observable state so an installed package, reached version, or completed removal is accepted without a duplicate mutation. Authentication, validation, required-setup, and malformed-response failures are terminal. The baseline preview alone determines the version used for testing and recovery; the pre-run installed version is ignored. Cleanup verifies the final package inventory; failed verification or unexpected packages introduced by the run make cleanup fail rather than producing a clean result.
2828

2929
Nexus receives only a single redacted, bounded request with no tools or conversation history. Its typed result is advisory; a failure falls back to the heuristic analyzer and no Nexus result can turn a deterministic failure into a pass.
3030

‎src/api/routes.rs‎

Lines changed: 1 addition & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -802,9 +802,7 @@ fn not_ready(message: String) -> Response {
802802

803803
#[cfg(test)]
804804
mod tests {
805-
use std::{
806-
collections::BTreeSet, error::Error, net::SocketAddr, path::PathBuf, time::Duration,
807-
};
805+
use std::{error::Error, net::SocketAddr, path::PathBuf, time::Duration};
808806

809807
use axum::{
810808
body::{Body, to_bytes},
@@ -840,7 +838,6 @@ mod tests {
840838
log_tail: 1,
841839
cleanup_enabled: true,
842840
cleanup_timeout: Duration::from_secs(1),
843-
retain_baseline_packages: BTreeSet::new(),
844841
nexus_api_key: None,
845842
nexus_base_url: "https://nexus.example/v1".to_owned(),
846843
nexus_model: "nexus/auto".to_owned(),

‎src/config.rs‎

Lines changed: 1 addition & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,4 @@
1-
use std::{collections::BTreeSet, env, net::SocketAddr, path::PathBuf, time::Duration};
1+
use std::{env, net::SocketAddr, path::PathBuf, time::Duration};
22

33
use thiserror::Error;
44

@@ -41,9 +41,6 @@ pub struct Config {
4141
pub log_tail: usize,
4242
pub cleanup_enabled: bool,
4343
pub cleanup_timeout: Duration,
44-
/// Packages whose resolved published baseline should remain installed and
45-
/// be restored after candidate tests.
46-
pub retain_baseline_packages: BTreeSet<String>,
4744
pub nexus_api_key: Option<String>,
4845
pub nexus_base_url: String,
4946
pub nexus_model: String,
@@ -161,7 +158,6 @@ impl Config {
161158
log_tail,
162159
cleanup_enabled: bool_value("CLEANUP_ENABLED", true)?,
163160
cleanup_timeout: millis("CLEANUP_TIMEOUT_MS", 60_000)?,
164-
retain_baseline_packages: package_list("RETAIN_BASELINE_PACKAGES")?,
165161
nexus_api_key: optional("NEXUS_API_KEY"),
166162
nexus_base_url: env::var("NEXUS_BASE_URL")
167163
.unwrap_or_else(|_| "https://nexus-api.dappnode.com/v1".to_owned()),
@@ -194,25 +190,6 @@ impl Config {
194190
}
195191
}
196192

197-
fn package_list(name: &'static str) -> Result<BTreeSet<String>, ConfigError> {
198-
let Some(value) = optional(name) else {
199-
return Ok(BTreeSet::new());
200-
};
201-
value
202-
.split(',')
203-
.map(str::trim)
204-
.filter(|value| !value.is_empty())
205-
.map(|value| {
206-
crate::model::DnpName::parse(value)
207-
.map(|name| name.to_string())
208-
.map_err(|error| ConfigError::Invalid {
209-
name,
210-
message: format!("invalid package '{value}': {error}"),
211-
})
212-
})
213-
.collect()
214-
}
215-
216193
#[derive(Debug, Error)]
217194
pub enum ConfigError {
218195
#[error("invalid configuration {name}: {message}")]

‎src/main.rs‎

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,6 @@ async fn main() -> Result<(), Box<dyn Error>> {
6868
stabilization_required_samples = config.stabilization_required_samples,
6969
cleanup_enabled = config.cleanup_enabled,
7070
cleanup_timeout_ms = config.cleanup_timeout.as_millis() as u64,
71-
retained_baseline_packages = ?config.retain_baseline_packages,
7271
log_tail_lines = config.log_tail,
7372
"Execution policy loaded"
7473
);
@@ -113,7 +112,6 @@ async fn main() -> Result<(), Box<dyn Error>> {
113112
log_tail: config.log_tail,
114113
cleanup_enabled: config.cleanup_enabled,
115114
cleanup_timeout: config.cleanup_timeout,
116-
retain_baseline_packages: config.retain_baseline_packages.clone(),
117115
},
118116
));
119117
let coordinator = CoordinatorClient::new(

‎src/model/run.rs‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -60,17 +60,17 @@ pub enum ManualRecoveryKind {
6060
pub enum TargetRecoveryPlan {
6161
/// The target did not exist before the run and must be removed.
6262
Remove,
63-
/// Return the target to an exact baseline. `retained` records that the
64-
/// harness intentionally keeps a newly installed baseline as a local cache.
63+
/// Return a preexisting target to the baseline resolved for its run.
6564
Restore {
66-
/// Immutable reference used for restoration. New records use the
67-
/// release's IPFS origin so a semver downgrade cannot be suppressed.
65+
/// Exact release reference when available, otherwise its resolved
66+
/// semantic version.
6867
baseline_ref: String,
6968
/// Version expected in package inventory after restoration. Older
7069
/// records omit this and use `baseline_ref` for both purposes.
7170
#[serde(default)]
7271
expected_version: Option<String>,
7372
#[serde(default)]
73+
/// Backward-compatible marker used only by older persisted records.
7474
retained: bool,
7575
},
7676
}

‎src/runner/cleanup.rs‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -298,7 +298,7 @@ pub async fn restore_target(
298298
return CleanupResult {
299299
status: CleanupStatus::TimedOut,
300300
leftover_packages: Vec::new(),
301-
error: Some("target package did not return to its original version".to_owned()),
301+
error: Some("target package did not return to its baseline version".to_owned()),
302302
};
303303
}
304304
Ok(Err(error)) => {
@@ -336,7 +336,7 @@ pub async fn restore_target(
336336
CleanupResult {
337337
status: CleanupStatus::TimedOut,
338338
leftover_packages: Vec::new(),
339-
error: Some("target package did not return to its original version".to_owned()),
339+
error: Some("target package did not return to its baseline version".to_owned()),
340340
}
341341
}
342342

0 commit comments

Comments
 (0)