Skip to content

Commit 85f6d6a

Browse files
pmaxhoganclaude
pmaxhogan
and
claude
authored
feat: real-world benchmark suite comparing driven with rclone (#178)
Adds a benchmark suite that measures Driven's **real backup engine** against `rclone` on a live Google Drive account, across the two workloads that dominate real backup sets and both a cold upload and an incremental re-run. The suite exists to answer one question honestly: is Driven constrained by hardware (CPU, disk, network, the Drive API) or by its own algorithms? A competitor doing roughly the same work is the cheapest way to tell those apart, and it catches regressions a synthetic microbenchmark never would. ## One deliberate deviation from the brief, up front The brief said to drive `driven-cli sync`. **I did not**, because reading it showed it is a debug driver, not the engine: it walks only the *top level* of the source folder (`crates/driven-cli/src/main.rs:394` - "V1 debug driver: top-level files only (no recursion)"), reads whole files into memory, keeps no state database, and uploads sequentially. Pointed at the `tiny-deep` fixture it would have uploaded **zero files**, and on multi-gigabyte files it would exhaust memory. Benchmarking it would have measured a debugging tool and published the number as Driven's. Instead the harness assembles the same stack `src-tauri/src/assembly.rs` does - `SqliteStateRepo` -> `DefaultExecutor` (adaptive upload pool, AIMD pacer) -> `SyncOrchestrator` - and calls `run_cycle` against a live `GoogleDriveStore`, so the real scan -> plan -> execute -> verify pipeline is what gets timed. This is new wiring: nothing in the repo previously ran the headless core against real Drive. ## What's here - **`crates/driven-bench`** - a single `driven-bench` binary with three subcommands: `run` (the matrix), `fixture` (build/clean trees without uploading), and a hidden `agent-sync` that runs **one** engine cycle and prints a JSON metrics line. - **`bench/README.md`** - prerequisites, scales, costs, safety rails, and an explicit "what is and is not apples-to-apples" section. - **`bench/run.ps1`** and `just bench` / `just bench-fixture` / `just bench-fixture-clean`. - **`.github/workflows/bench.yml`** - `workflow_dispatch` (scale, tools) plus `v*` tag pushes at the `smoke` scale. Never on `pull_request`, never on a plain push. Time-boxed at 180 minutes; clean skip when secrets are absent. ### Design choices worth reviewing **Both tools run as child processes.** Driven's engine could have run in-process, but then its CPU-time and peak-memory columns would silently include fixture generation and the harness's own Drive calls, and would not be comparable to rclone's. The harness re-invokes itself with `agent-sync`, so both tools are measured by identical OS accounting (`GetProcessTimes` / `GetProcessMemoryInfo` on Windows, `getrusage(RUSAGE_CHILDREN)` on Unix). **API-call counts come from a `RemoteStore` decorator**, the same seam the executor already uses for `BreakerReportingStore` - no core change. rclone exposes no request counter, so that cell renders as `-`, meaning "not measurable", never `0`. **rclone auth needs no token-minting request.** Its config carries a *non-empty* placeholder access token that is already expired, and rclone refreshes it from the same refresh token Driven uses. (An *empty* `access_token` makes rclone treat the whole token as unparseable and report "there's no refresh token" - verified empirically before building anything on it.) **The report separates scan time from upload time.** A total cannot answer the question the suite exists for - on the million-tiny-files shape, is a slow cold pass bound by the local walk and hashing, or by Drive round-trips? The agent consumes the orchestrator's event stream while the cycle runs (a timestamp cannot be recovered from a buffered event, and this also stops losing events to broadcast lag) and reports the boundary. rclone interleaves listing with transferring, so its cell stays blank rather than invented. **restic was considered and deliberately excluded.** It stores a chunked, deduplicating repository rather than a mirror, so its "upload" is a different operation, and on a re-run its deduplication would flatter it on exactly the workload this suite measures. Adding it would produce a bigger table, not a more honest one. The rationale is in the README, not just here. **Fixtures are seeded and incompressible** (SplitMix64), so both tools see byte-identical input and no tool scores on test-data entropy it would never see on photos and archives. The mutation step changes content only - no creates, no deletes - which keeps `rclone copy` a fair match rather than requiring the riskier `rclone sync`. ### Safety rails - The destination folder id must be explicit (`--dest` or `DRIVEN_E2E_DEST_FOLDER_ID`); no default, no discovery. Checked before a byte is generated. - All writes go under one `driven-bench-<uuid>` folder, with a subfolder per scenario. Cleanup trashes **that folder by the id it was created with** - the suite never lists the destination and never matches by name, so it cannot touch anything it did not create. A cleanup failure prints the id to trash by hand. - Uploads are capped at 2 GiB by default; exceeding it is an error telling you to pass `--full` or lower `--scale`. - Credentials are read from the environment only (never the keychain) and never printed. ## Smoke run (real Drive, dedicated automation account) Release build, Windows, 20 logical CPUs, rclone v1.74.4. Both tools, both shapes, both phases; run folder trashed afterwards. **huge** - 2 files, 16.0 MiB | Tool | Phase | Wall s | MiB/s | Files | API calls | CPU s | Peak RSS | Conc | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | driven | cold | 3.4 | 4.72 | 2 | 6 | 0.6 | 49.8 MiB | 16 | | driven | incremental | 2.4 | 3.37 | 1 | 4 | 0.6 | 37.3 MiB | 16 | | rclone | cold | 4.4 | 3.63 | 2 | - | 0.8 | 83.3 MiB | 4 | | rclone | incremental | 1.8 | 4.48 | 1 | - | 0.4 | 67.0 MiB | 4 | **tiny-deep** - 300 files, 591.7 KiB, nested 5 deep | Tool | Phase | Wall s | files/s | Files | API calls | CPU s | Peak RSS | Conc | | --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | | driven | cold | 78.6 | 3.8 | 300 | 489 | 2.0 | 28.4 MiB | 16 | | driven | incremental | 2.4 | 0.4 | 1 | 7 | 0.4 | 26.1 MiB | 16 | | rclone | cold | 152.5 | 2.0 | 300 | - | 1.3 | 59.9 MiB | 4 | | rclone | incremental | 10.6 | 0.1 | 1 | - | 0.6 | 56.6 MiB | 4 | Reading it: the incremental rows are the point. Driven re-detects a single changed file in **2.4 s and 7 API calls**; rclone takes **10.6 s** because it re-lists the remote every time. The state database is doing its job. The concurrency columns differ because each tool runs at its **stock** settings - `--rclone-transfers` equalises them if you want to isolate the algorithms. These numbers are a pipeline proof, not a verdict: 300 files is far too small to conclude anything from, and cross-host comparisons are meaningless. Run `just bench` (~610 MiB per tool) for numbers worth quoting. ## Gates - `cargo fmt --all -- --check` - clean - `cargo clippy --workspace --all-targets -- -D warnings` - clean (exit 0) - `cargo test --workspace` - clean (exit 0); every crate green, including driven-bench's 49 tests - `driven-bench` ships 49 unit tests: fixture determinism and depth, mutate / restore round-trips, crash-left-mutated recovery, rclone stats parsing, the metrics marker line, upload-cap arithmetic, dest-id refusal, dotenv precedence, and report rendering (including that an unmeasurable cell is a dash, not a zero). ## Notes for the reviewer - The coverage gate auto-includes any new crate via `--workspace --exclude`, so `driven-bench` is added to the exclusion list alongside `src-tauri` and `driven-chaos` in `coverage.yml`, `scripts/coverage.sh` and the `just coverage` recipe. A harness that mostly spawns processes and needs live credentials would otherwise drag the gate down for no signal. - `cargo test --workspace` does build and test the new crate. Its tests are fast and need no credentials or network; the credential-dependent paths simply are not exercised there. - Nothing here runs on PR CI or dev-branch builds, by design. 🤖 Generated with [Claude Code](https://claude.com/claude-code) https://claude.ai/code/session_01JLB3E2Jm7knNJd37fVpH8X --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
1 parent af8f048 commit 85f6d6a

19 files changed

Lines changed: 3769 additions & 5 deletions

File tree

‎.github/workflows/bench.yml‎

Lines changed: 131 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,131 @@
1+
name: Bench
2+
3+
# Real-world benchmark suite: Driven's engine vs rclone (bench/README.md).
4+
#
5+
# COST POLICY: this workflow uploads REAL bytes to a REAL Google account and
6+
# takes minutes to hours. It therefore NEVER runs on `pull_request` and NEVER on
7+
# a plain push - only:
8+
#
9+
# workflow_dispatch - on demand, with a chosen scale and tool list.
10+
# v* tag pushes - at the SMOKE scale only, as a release-time check that
11+
# the suite still works and nothing has fallen off a
12+
# cliff. Same gating shape as chaos.yml's real-drive job.
13+
#
14+
# Like `chaos-real-drive`, the job degrades to a clean SKIP (never red) when the
15+
# credentials are absent, so a fork or a rotated-away secret does not fail a
16+
# release. Every job is time-boxed so a hung upload cannot burn hours of runner
17+
# budget.
18+
19+
on:
20+
workflow_dispatch:
21+
inputs:
22+
scale:
23+
description: "Fixture scale"
24+
type: choice
25+
default: smoke
26+
options: [smoke, small, medium, full]
27+
tools:
28+
description: "Comma-separated tools to measure"
29+
type: string
30+
default: "driven,rclone"
31+
push:
32+
tags: ["v*"]
33+
34+
concurrency:
35+
# One benchmark at a time: two concurrent runs would contend for the same
36+
# uplink and produce numbers that mean nothing. Never cancel a running one -
37+
# a cancelled run leaves its uploaded folder behind.
38+
group: ${{ github.workflow }}
39+
cancel-in-progress: false
40+
41+
permissions:
42+
contents: read
43+
44+
env:
45+
CARGO_TERM_COLOR: always
46+
RUST_BACKTRACE: 1
47+
# The harness boots the headless core, whose state layer uses sqlx
48+
# compile-time-checked queries; CI has no live DB (same as ci.yml).
49+
SQLX_OFFLINE: "true"
50+
CARGO_PROFILE_DEV_DEBUG: "0"
51+
52+
jobs:
53+
bench:
54+
name: benchmark (${{ inputs.scale || 'smoke' }})
55+
runs-on: ubuntu-latest
56+
# A `full` run is meant to take hours; everything else finishes long before
57+
# this. The cap exists so a hung upload cannot run until the 6h default.
58+
timeout-minutes: 180
59+
env:
60+
DRIVEN_E2E_REFRESH_TOKEN: ${{ secrets.DRIVEN_E2E_REFRESH_TOKEN }}
61+
DRIVEN_E2E_DEST_FOLDER_ID: ${{ secrets.DRIVEN_E2E_DEST_FOLDER_ID }}
62+
DRIVEN_OAUTH_CLIENT_ID: ${{ secrets.DRIVEN_OAUTH_CLIENT_ID }}
63+
DRIVEN_OAUTH_CLIENT_SECRET: ${{ secrets.DRIVEN_OAUTH_CLIENT_SECRET }}
64+
# A tag push has no inputs; the release-time check is deliberately small.
65+
BENCH_SCALE: ${{ inputs.scale || 'smoke' }}
66+
BENCH_TOOLS: ${{ inputs.tools || 'driven,rclone' }}
67+
steps:
68+
- uses: actions/checkout@v7
69+
70+
- name: Check for credentials
71+
id: creds
72+
# Absent secrets are a clean skip, not a failure: forks and rotated
73+
# secrets must not turn a release tag red.
74+
run: |
75+
if [ -n "$DRIVEN_E2E_REFRESH_TOKEN" ] && [ -n "$DRIVEN_E2E_DEST_FOLDER_ID" ] \
76+
&& [ -n "$DRIVEN_OAUTH_CLIENT_SECRET" ]; then
77+
echo "present=true" >> "$GITHUB_OUTPUT"
78+
else
79+
echo "present=false" >> "$GITHUB_OUTPUT"
80+
echo "Bench credentials are not available; skipping the benchmark run."
81+
fi
82+
83+
- uses: dtolnay/rust-toolchain@stable
84+
if: steps.creds.outputs.present == 'true'
85+
86+
- name: Install build deps
87+
if: steps.creds.outputs.present == 'true'
88+
run: |
89+
sudo apt-get update
90+
sudo apt-get install -y libssl-dev
91+
92+
- name: Install rclone
93+
if: steps.creds.outputs.present == 'true'
94+
run: |
95+
sudo apt-get install -y rclone
96+
rclone version
97+
98+
- uses: Swatinem/rust-cache@v2
99+
if: steps.creds.outputs.present == 'true'
100+
with:
101+
# Builds a subset of the workspace, so it restores the same per-OS
102+
# cache ci.yml warms. This workflow never runs on main, so it is
103+
# purely restore-only.
104+
shared-key: "workspace"
105+
save-if: false
106+
107+
- name: Run the benchmark
108+
if: steps.creds.outputs.present == 'true'
109+
run: |
110+
# The upload cap is deliberate; the larger scales opt out of it
111+
# explicitly rather than the harness silently ignoring it.
112+
full=""
113+
case "$BENCH_SCALE" in
114+
medium|full) full="--full" ;;
115+
esac
116+
cargo run --release -p driven-bench -- run \
117+
--scale "$BENCH_SCALE" \
118+
--tools "$BENCH_TOOLS" \
119+
$full
120+
121+
- name: Publish the report to the run summary
122+
# `always()` so a failed benchmark still shows its table - the numbers
123+
# are the point, and a partial run is usually the interesting one.
124+
if: always() && steps.creds.outputs.present == 'true'
125+
run: |
126+
latest=$(ls -1t bench/results/*.md 2>/dev/null | head -n 1 || true)
127+
if [ -n "$latest" ]; then
128+
cat "$latest" >> "$GITHUB_STEP_SUMMARY"
129+
else
130+
echo "No benchmark report was produced." >> "$GITHUB_STEP_SUMMARY"
131+
fi

‎.github/workflows/coverage.yml‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -12,10 +12,13 @@ name: Coverage
1212
# there is no baseline, so the gate is informational (fail-open) for that one
1313
# run, then enforces from the next PR onward.
1414
#
15-
# Scope: the library crates (`--exclude src-tauri --exclude driven-chaos`).
16-
# src-tauri is a thin IPC layer over driven-core; driven-chaos is the stress
17-
# harness. Both are excluded from the measured/report set (their tests are not
18-
# run for coverage), but `--workspace --exclude` still auto-includes any NEW
15+
# Scope: the library crates (`--exclude src-tauri --exclude driven-chaos
16+
# --exclude driven-bench`). src-tauri is a thin IPC layer over driven-core;
17+
# driven-chaos is the stress harness; driven-bench is the benchmark harness,
18+
# which mostly spawns child processes and talks to a real Google account and so
19+
# is largely unreachable without credentials. All three are excluded from the
20+
# measured/report set (their tests are not run for coverage), but
21+
# `--workspace --exclude` still auto-includes any NEW
1922
# crate in the gate, which a hand-maintained `-p` list would silently miss. The
2023
# Vue/TS app (`ui/`) is measured in full. (telemetry-worker is its own toolchain
2124
# and is out of scope for this gate.)
@@ -88,6 +91,7 @@ jobs:
8891
- name: Rust coverage (library crates)
8992
run: |
9093
cargo llvm-cov --workspace --exclude src-tauri --exclude driven-chaos \
94+
--exclude driven-bench \
9195
--summary-only --json --output-path coverage-rust.json
9296
RUST_PCT=$(jq '.data[0].totals.lines.percent' coverage-rust.json)
9397
echo "HEAD_RUST=$RUST_PCT" >> "$GITHUB_ENV"

‎Cargo.lock‎

Lines changed: 25 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎Cargo.toml‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@ members = [
1313
"crates/driven-cli",
1414
"crates/driven-test-fixtures",
1515
"crates/driven-chaos",
16+
"crates/driven-bench",
1617
"src-tauri",
1718
]
1819
# M9b (SPEC s16): the telemetry Cloudflare Worker has its own (TypeScript/wrangler)

0 commit comments

Comments
 (0)