pmaxhogan
diff --git a/‎README.md‎
Lines changed: 9 additions & 9 deletions b/‎README.md‎
Lines changed: 9 additions & 9 deletions
diff --git a/‎crates/driven-cli/src/inspect.rs‎
Lines changed: 116 additions & 0 deletions b/‎crates/driven-cli/src/inspect.rs‎
Lines changed: 116 additions & 0 deletions
diff --git a/‎crates/driven-cli/src/main.rs‎
Lines changed: 5 additions & 0 deletions b/‎crates/driven-cli/src/main.rs‎
Lines changed: 5 additions & 0 deletions
diff --git a/‎crates/driven-cli/tests/cli.rs‎
Lines changed: 170 additions & 0 deletions b/‎crates/driven-cli/tests/cli.rs‎
Lines changed: 170 additions & 0 deletions
@@ -106,7 +106,7 @@ Notes:
 - ⁴¹ Driven's checkmark is scoped to Windows, where a VSS snapshot lets a locked file (Outlook PST, running DB, VM disk) back up while it is held open. macOS has an equivalent behind an opt-in setting (Settings > macOS): a small privileged helper mounts a read-only APFS local snapshot so a *busy* file can be read: it is off by default, and it does nothing for a Full Disk Access denial. On both macOS and Linux, a file Driven cannot open is in any case classified precisely as a transient lock (`local.file_locked`) versus a macOS Full Disk Access denial (`local.permission_denied`) and skipped with a clear reason in the activity log, rather than misreported as a disk error. Linux has no snapshot equivalent and none is planned. See `design/DESIGN.md` §5.3 and §5.3.2.
 - ⁴² Driven's checkmark is scoped to the Google Drive destination. Keeping previous versions relies on a changed file's re-upload landing on a NEW remote object, which is true of a Drive create but not of a destination whose object key is derived from the file name: on S3-compatible stores, SFTP servers, and local folders the re-upload overwrites the previous copy. Driven therefore does not offer per-source versioning or a restore-by-date on those destinations at all, rather than reporting a point-in-time restore it cannot perform. For recoverability on S3, enable the provider's own bucket versioning. See [issue #220](https://github.com/pmaxhogan/driven/issues/220).
 - ⁴³ Google Drive (the original destination), any S3-compatible object store (AWS S3, Cloudflare R2, MinIO, Backblaze B2 in S3-compatible mode, Wasabi), any SSH server with SFTP (a NAS, home server, or VPS - added in v2.9.0, with TOFU host-key pinning and password or private-key auth), and a local or removable folder (USB drive, external disk, NAS share) - all behind one `RemoteStore` trait. OneDrive and other non-S3-compatible cloud APIs are not implemented.
-- ⁴⁴ Driven runs three independent periodic checks, not one: a weekly local re-hash (deep-verify) that catches local bit-rot, a startup / deep-verify remote-existence audit that catches an object deleted at the destination outside Driven, and, added in v2.5.0, a weekly rolling integrity scrub that re-checks each already-backed-up object's size against the destination and, where the destination can supply one, its content checksum too - catching remote-side corruption or tampering that neither of the other two checks can see. On by default; a source's population is swept in bounded slices (500 objects per run by default) rather than all at once. Checksum coverage depends on the destination: full on Google Drive and on a local-folder destination (which always re-hashes the bytes it just wrote); size-only for an S3 object uploaded via multipart, because S3 stores no plain content digest for one, only an ETag that digests the individual parts - the scrub reports those as unverifiable rather than guessing, never as falsely clean.
+- ⁴⁴ Driven runs four independent periodic checks, not one: a weekly local re-hash (deep-verify) that catches local bit-rot, a startup / deep-verify remote-existence audit that catches an object deleted at the destination outside Driven, and, added in v2.5.0, a weekly rolling integrity scrub that re-checks each already-backed-up object's size against the destination and, where the destination can supply one, its content checksum too - catching remote-side corruption or tampering that neither of the other two checks can see. On by default; a source's population is swept in bounded slices (500 objects per run by default) rather than all at once. Checksum coverage depends on the destination: full on Google Drive and on a local-folder destination (which always re-hashes the bytes it just wrote); size-only for an S3 object uploaded via multipart, because S3 stores no plain content digest for one, only an ETag that digests the individual parts - the scrub reports those as unverifiable rather than guessing, never as falsely clean. The fourth check is the restore drill: every one of the three above verifies the backup from the WRITE side ("is what we wrote still there?"), and none of them ever runs the reverse pipeline, so a broken restore path or an unusable key stays invisible until the day you actually need your data. Once a month Driven therefore restores a few of your backed-up files for real - download, decrypt, extract, verify the plaintext hash - into a temporary directory, then deletes them. On by default, three files per drill; both the cadence and the sample size are configurable, and the whole thing has a kill switch. A drill that verifies nothing (nothing restorable yet, or a locked keychain) is reported as inconclusive, never as a pass.
 
 Competitor rows were verified in July 2026 against rclone 1.74.4, restic 0.19.1,
 Duplicati 2.3.0.4, Backblaze Personal Backup 10.0.2, and Drive for desktop 128.0.
@@ -174,20 +174,20 @@ These move: check each project's current docs before relying on a cell.
   catching remote-side corruption that neither of those two checks can see.
   On by default, weekly, in bounded slices so a huge source never triggers
   an unbounded sweep.
+- Restore drill: the only check that exercises the READ side. Every other
+  check asks "is what we wrote still there?"; a drill periodically restores a
+  small, deterministically-sampled set of your backed-up files through the
+  real restore path - download, decrypt, extract, verify the plaintext hash -
+  into a temp directory, then deletes them, proving your data actually comes
+  back rather than merely that it is still stored. On by default, monthly,
+  three files per drill. Reports are counts and error codes only, so a drill
+  report can never leak an encrypted source's filenames.
 - rclone config importer (`driven-cli rclone`): point it at an existing
   `rclone.conf` and it tells you exactly what to enter for an `s3` or
   `drive` remote, rather than making you re-type every endpoint and key.
   Read-only - it never creates an account itself. See "Migrating from
   rclone" below.
 
-<!--
-DRAFT - do not uncomment until the corresponding PR merges. Flip each bullet
-on individually as its PR lands, then delete this comment wrapper.
-- Restore drill: a one-click "prove the backup actually restores" check.
-  (core engine landed, #215 DRAFT / unmerged - surface is not wired up to
-  anything a user can click yet.)
--->
-
 ## Install
 
 Download the installer for your platform from the
 
@@ -49,6 +49,21 @@ pub struct ScrubArgs {
     pub fail_on_drift: bool,
 }
 
+/// Args for `driven-cli drill`.
+#[derive(Debug, Args)]
+pub struct DrillArgs {
+    /// Path to the Driven state database.
+    #[arg(long, default_value = "state.db")]
+    pub db: PathBuf,
+    /// Maximum number of drill runs to show (newest first).
+    #[arg(long, default_value_t = 10)]
+    pub limit: u32,
+    /// Exit non-zero when the most recent run of any source could not restore a
+    /// file it sampled, so the command is usable as a monitoring check.
+    #[arg(long)]
+    pub fail_on_failure: bool,
+}
+
 /// Args for `driven-cli history`.
 #[derive(Debug, Args)]
 pub struct HistoryArgs {
@@ -319,6 +334,107 @@ pub async fn run_scrub(args: ScrubArgs) -> Result<()> {
     Ok(())
 }
 
+/// Everything `drill` needs, gathered in one pass so the rendering is trivial
+/// and the query logic is unit-testable against a seeded temp DB.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct DrillOverview {
+    /// The configured drill policy, as the engine would actually apply it
+    /// (clamped + defaulted by `load_drill_config`).
+    pub config: driven_core::drill::DrillConfig,
+    /// Recent runs across every source, newest first.
+    pub runs: Vec<driven_core::state::DrillRunRow>,
+    /// Display name per source id, so the report can name sources without the
+    /// caller re-querying. Sources deleted since a run was recorded are absent.
+    pub source_names: std::collections::HashMap<String, String>,
+}
+
+pub async fn gather_drill(repo: &dyn StateRepo, limit: u32) -> Result<DrillOverview> {
+    let config = driven_core::drill::load_drill_config(repo).await;
+    let runs = repo.list_drill_runs(None, limit.max(1)).await?;
+    let source_names = repo
+        .list_sources()
+        .await?
+        .into_iter()
+        .map(|s| (s.id.to_string(), s.display_name))
+        .collect();
+    Ok(DrillOverview {
+        config,
+        runs,
+        source_names,
+    })
+}
+
+pub async fn run_drill(args: DrillArgs) -> Result<()> {
+    let repo = open_existing(&args.db).await?;
+    let overview = gather_drill(&repo, args.limit).await?;
+
+    let c = &overview.config;
+    println!(
+        "restore drill: {}  every {}s  sample {}",
+        if c.enabled { "enabled" } else { "DISABLED" },
+        c.interval_secs,
+        c.sample_size,
+    );
+
+    if overview.runs.is_empty() {
+        println!("No restore drill has run yet.");
+        return Ok(());
+    }
+
+    // Counts and stable SPEC s24 error codes only, never paths - the same rule
+    // the persisted report follows, so piping this into a log or a bug report
+    // cannot leak an encrypted source's filenames.
+    for r in &overview.runs {
+        let name = overview
+            .source_names
+            .get(&r.source_id.to_string())
+            .map_or("(deleted source)", String::as_str);
+        println!(
+            "{}  {:<12}  {}  sampled {} (verified {}, skipped {}, failed {})",
+            fmt_epoch_ms(Some(r.started_at)),
+            r.report.outcome.label(),
+            name,
+            r.report.sampled,
+            r.report.verified,
+            r.report.skipped,
+            r.report.failed,
+        );
+        if !r.report.failure_codes.is_empty() {
+            let codes = r
+                .report
+                .failure_codes
+                .iter()
+                .map(|(code, n)| format!("{code} x{n}"))
+                .collect::<Vec<_>>()
+                .join(", ");
+            println!("    failures: {codes}");
+        }
+    }
+
+    if args.fail_on_failure {
+        // "Most recent run PER SOURCE", same rule as `scrub --fail-on-drift`: a
+        // source that failed a month ago and has drilled clean since is not a
+        // live problem, and reporting it as one would make the check useless as
+        // a monitor.
+        //
+        // Deliberately keyed on `failed`, NOT on the outcome: an INCONCLUSIVE
+        // run (nothing restorable, or every candidate skipped because a key was
+        // unavailable) is not evidence of a broken backup, and failing a monitor
+        // on a locked keychain would train people to ignore it.
+        let mut seen: std::collections::HashSet<String> = std::collections::HashSet::new();
+        let mut failed = 0u64;
+        for r in &overview.runs {
+            if seen.insert(r.source_id.to_string()) {
+                failed = failed.saturating_add(r.report.failed);
+            }
+        }
+        if failed > 0 {
+            anyhow::bail!("{failed} sampled file(s) could not be restored from the backup");
+        }
+    }
+    Ok(())
+}
+
 pub async fn run_verify(args: InspectArgs) -> Result<()> {
     let repo = open_existing(&args.db).await?;
     let rows = gather_status(&repo).await?;
 
@@ -84,6 +84,10 @@ enum Command {
     /// the local state database. With `--fail-on-drift`, exits non-zero when
     /// the latest run of any source found drift it could not repair.
     Scrub(inspect::ScrubArgs),
+    /// Show the restore-drill configuration and recent drill reports from the
+    /// local state database. With `--fail-on-failure`, exits non-zero when the
+    /// latest run of any source could not restore a file it sampled.
+    Drill(inspect::DrillArgs),
     /// Restore one backup source to a local directory (no GUI), optionally
     /// byte-verifying the result against the original folder. Exits non-zero
     /// when any file fails to restore or fails verification.
@@ -179,6 +183,7 @@ async fn main() -> anyhow::Result<()> {
             rclone::RcloneCommand::Import(a) => rclone::run_import(a).await,
         },
         Command::Scrub(args) => inspect::run_scrub(args).await,
+        Command::Drill(args) => inspect::run_drill(args).await,
         Command::Restore(args) => restore::run_restore(args).await,
     }
 }
 
@@ -171,6 +171,7 @@ fn top_level_help_lists_every_subcommand() {
             .and(predicate::str::contains("verify"))
             .and(predicate::str::contains("rclone"))
             .and(predicate::str::contains("scrub"))
+            .and(predicate::str::contains("drill"))
             .and(predicate::str::contains("restore")),
     );
 }
@@ -305,6 +306,175 @@ fn drifted_report() -> driven_core::scrub::ScrubReport {
     r
 }
 
+/// Seed `path` with one source and `runs` recorded restore-drill reports.
+fn seed_drill_runs(path: &Path, runs: &[driven_core::drill::DrillReport]) {
+    block_on(async {
+        let repo = SqliteStateRepo::open(path).await.unwrap();
+        let acc = AccountId::new_v4();
+        let src = SourceId::new_v4();
+        repo.upsert_account(&account(acc)).await.unwrap();
+        repo.upsert_source(&source(acc, src, "Docs")).await.unwrap();
+        for (i, report) in runs.iter().enumerate() {
+            repo.insert_drill_run(&driven_core::state::NewDrillRun {
+                source_id: src,
+                started_at: i as i64 + 1,
+                finished_at: i as i64 + 2,
+                report: report.clone(),
+            })
+            .await
+            .unwrap();
+        }
+    });
+}
+
+/// Build a drill report out of `(verified, skipped, failures)`.
+fn drill_report(
+    verified: u64,
+    skipped: u64,
+    failures: &[driven_core::types::ErrorCode],
+) -> driven_core::drill::DrillReport {
+    let mut r = driven_core::drill::DrillReport::default();
+    for _ in 0..verified {
+        r.record(&driven_core::drill::DrillAttempt::Verified);
+    }
+    for _ in 0..skipped {
+        r.record(&driven_core::drill::DrillAttempt::Skipped {
+            code: driven_core::types::ErrorCode::CryptoKeyMissing,
+        });
+    }
+    for code in failures {
+        r.record(&driven_core::drill::DrillAttempt::Failed { code: *code });
+    }
+    r.finish();
+    r
+}
+
+#[test]
+fn drill_on_a_fresh_database_reports_the_shipped_policy_and_no_runs() {
+    let dir = tempfile::tempdir().unwrap();
+    let db = dir.path().join("state.db");
+    seed_empty(&db);
+
+    cli()
+        .args(["drill", "--db"])
+        .arg(&db)
+        .assert()
+        .success()
+        .stdout(
+            // The shipped posture: on, monthly, three files.
+            predicate::str::contains("restore drill: enabled")
+                .and(predicate::str::contains("every 2592000s"))
+                .and(predicate::str::contains("sample 3"))
+                .and(predicate::str::contains("No restore drill has run yet.")),
+        );
+}
+
+#[test]
+fn drill_renders_recorded_runs_with_counts_and_codes_only() {
+    let dir = tempfile::tempdir().unwrap();
+    let db = dir.path().join("state.db");
+    let passed = drill_report(3, 0, &[]);
+    let failed = drill_report(
+        1,
+        1,
+        &[
+            driven_core::types::ErrorCode::CryptoDecryptFailed,
+            driven_core::types::ErrorCode::CryptoDecryptFailed,
+        ],
+    );
+    seed_drill_runs(&db, &[passed, failed]);
+
+    cli()
+        .args(["drill", "--db"])
+        .arg(&db)
+        .assert()
+        .success()
+        .stdout(
+            predicate::str::contains("passed")
+                .and(predicate::str::contains("failed"))
+                .and(predicate::str::contains("verified 1"))
+                .and(predicate::str::contains("skipped 1"))
+                // The breakdown is aggregated with a count, and the code is a
+                // closed-vocabulary dotted string - never a path.
+                .and(predicate::str::contains("crypto.decrypt_failed x2"))
+                .and(predicate::str::contains("Docs")),
+        );
+}
+
+/// An INCONCLUSIVE run must not render as a pass. "We restored nothing" reading
+/// as "we restored everything successfully" is the exact lie the feature exists
+/// to prevent.
+#[test]
+fn drill_renders_an_inconclusive_run_distinctly_from_a_pass() {
+    let dir = tempfile::tempdir().unwrap();
+    let db = dir.path().join("state.db");
+    seed_drill_runs(&db, &[drill_report(0, 2, &[])]);
+
+    cli()
+        .args(["drill", "--db"])
+        .arg(&db)
+        .assert()
+        .success()
+        .stdout(
+            predicate::str::contains("inconclusive").and(predicate::str::contains("verified 0")),
+        );
+}
+
+/// `--fail-on-failure` makes the command usable as a monitoring check.
+#[test]
+fn drill_fail_on_failure_exits_non_zero_only_when_the_latest_run_still_failed() {
+    let dir = tempfile::tempdir().unwrap();
+    let db = dir.path().join("state.db");
+    seed_drill_runs(
+        &db,
+        &[drill_report(
+            2,
+            0,
+            &[driven_core::types::ErrorCode::DriveUnreachable],
+        )],
+    );
+
+    cli()
+        .args(["drill", "--fail-on-failure", "--db"])
+        .arg(&db)
+        .assert()
+        .failure()
+        .stderr(predicate::str::contains("could not be restored"));
+
+    // A LATER passing run clears it: a source that failed once and has drilled
+    // clean since is not a live problem.
+    let dir2 = tempfile::tempdir().unwrap();
+    let db2 = dir2.path().join("state.db");
+    seed_drill_runs(
+        &db2,
+        &[
+            drill_report(2, 0, &[driven_core::types::ErrorCode::DriveUnreachable]),
+            drill_report(3, 0, &[]),
+        ],
+    );
+    cli()
+        .args(["drill", "--fail-on-failure", "--db"])
+        .arg(&db2)
+        .assert()
+        .success();
+}
+
+/// An all-SKIPPED (inconclusive) run must NOT fail the monitor. A locked
+/// keychain is not evidence of a broken backup, and failing a monitoring check
+/// on it would train operators to ignore the check that matters.
+#[test]
+fn drill_fail_on_failure_ignores_an_inconclusive_run() {
+    let dir = tempfile::tempdir().unwrap();
+    let db = dir.path().join("state.db");
+    seed_drill_runs(&db, &[drill_report(0, 3, &[])]);
+
+    cli()
+        .args(["drill", "--fail-on-failure", "--db"])
+        .arg(&db)
+        .assert()
+        .success();
+}
+
 #[test]
 fn version_flag_prints_version() {
     cli()