Skip to content

Latest commit

 

History

History
95 lines (67 loc) · 19.9 KB

proactive-bug-discovery.md

File metadata and controls

95 lines (67 loc) · 19.9 KB

Proactive bug discovery

This inventory distinguishes deterministic regression replay from exploratory campaigns. A green replay job means the checked-in corpus passed. It does not mean a scheduled fuzz, mutation, or native lifecycle campaign ran. Source revision, working-tree changes, exact commands, tool versions, elapsed budgets and individual results accompany each execution under target/discovery/.

Maintained boundary inventory

Boundary and owner Invariant and independent oracle Normal / boundary / negative-control cases Platform, source and evidence
SSE parser, C5 Chunk partitions cannot change a complete frame; parser byte limits remain hard. Oracle compares decoded item and payload across every split. Normal checked corpus; exact-limit CRLF split; interior CR and hard-buffer rejection. Linux fuzz at c42327c1, owner regression from fa1ca671; fuzz totals, defect replay. Synthetic bytes only.
NDJSON parser, C5 Newline partitioning cannot change admission or payload at the content limit. Oracle compares exact decoded JSON text. Normal checked corpus; exact-limit CRLF split; interior CR and hard-buffer rejection. Linux fuzz at c42327c1, owner regression from fa1ca671; framing evidence. Provider-specific JSON meaning stays in owner tests.
Provider response reducer, C5 Identity, ordering, fragment, terminal and EOF rules remain stable. Owner-contract assertions are the semantic oracle; twin reducers check determinism only. Complete message/tool lifecycles; gaps, conflicts, duplicates and incomplete UTF-8; post-terminal event rejection. Structured minimization reduces 9 records to a deletion-minimal 4 while retaining the exact diagnostic and reachability. Linux fuzz at c42327c1; checked corpus plus provider.rs. No live provider call.
Provider failover and pending effects, C5/G4 Retry/failover cannot replay an admitted external effect. Oracle combines exact request counts, receipt decisions and owner protocol terminal state. Successful provider sequence; disconnect after accepted tool completion; pending and completed receipt controls prevent redispatch. ProductRunner and protocol owner suites on Linux, including the disconnect/receipt interaction; receipt recovery evidence. Live-provider transport failure remains outside this synthetic campaign.
Working-context selection, C6 Required closure fits allocation and the hard available-token ceiling; a small graph oracle computes the expected closure. Normal required chain; exact capacity and one-token-short boundary; optional expansion cannot displace required roots; pressure plus real malformed SSE framing preserves completed work through deterministic fallback. Linux fuzz at c42327c1; context/framing interaction; automatic mutation is unreachable inside the Verus macro, so the curated canary is required. Token estimates do not claim provider-exact tokenization.
Product recovery lifecycle, G0 Explicit cancellation remains terminal; shutdown interruption remains retryable; completed work never reopens. Durable JSON phase and exact provider-request count are independent oracles. Graceful cancel/shutdown race; acknowledged cancel followed by child kill; late cancel after completion; generic retry of completed chat; repeated failed startup admission. Linux, commits 16d5fa87, 6dd8059b, b8997287; pilot, abrupt expansion, replay manifests. No power-loss claim.
Product-record persistence, G0 After a write/fsync/rename failure, reopen sees the complete old record or complete new record. Oracle decodes the canonical file independently. Normal persist; failures before write, file sync, rename, after rename and Unix directory sync; missed-schedule control. Linux owner tests; persistence evidence. Resume bytes are exercised by separate restore tests.
Effect receipt ledger, G4 One provider call ID authorizes at most one effect; completed receipts replay exactly; started commands remain ambiguous after crash. Oracle uses ledger bytes plus an independently synced effect counter. Normal begin/complete/replay; truncated tail and Started/Completed child-kill windows; malformed complete frame and reused-ID controls. Linux, commits 16d5fa87, 310db7e7; receipt manifests, reused-ID manifest, mutation results.
Command runtime, G4 Output bounds, stdin/signal control, cancellation and terminal exit have one durable terminal decision and no surviving owned child; concurrent runtimes allocate distinct durable ordinals through transient SQLite writer contention. Normal exit/input; output limit with sleeping descendant; ready-barrier cancellation; invalid/repeated control requests; fresh and initialized ordinal databases with a held writer lock. Native Linux owner suites; output-limit evidence, hosted contention evidence, defect replay. Windows/macOS execution is supplied by native hosted qualification.
AgentDriver durability, B1/G0 Duplicate envelopes do not advance semantics twice; gaps fail; lost tools become Indeterminate once; retry budgets begin after conservative finalization. SQLite journal rows and B1 ledger totals are independent oracles. Normal message completion; duplicate and gap; driver drop/journal reopen; lost tool classification twice; decreasing-usage rejection. Linux at base f75f5dee, campaign commit 0f0536bf; 67-test evidence. This fixture drops process-local ownership; it does not claim SIGKILL or power-loss qualification.
Process registry storage, process owner Reopen restores one complete manifest and removes staging/previous files without inventing identity. Oracle compares raw manifest bytes and reconciled process identity. Normal completed replacement; interrupted staging replacement; six named permission/storage-full failpoints; missed-occurrence control. Native Linux owner tests; manifest evidence. Private codec remains owner-local; kernel ENOSPC and power loss remain unsupported.
Process supervisor, process owner Output-limit or user-cancel termination reaps the owned tree, joins readers and persists the same terminal value observed live. Normal exit; output limit with descendant-held pipes; exact output barrier plus cancellation; separately owned sibling survives harness timeout. Native Linux owner and runner tests; process evidence. Corresponding Windows Job Object and macOS process-group results must come from hosted native jobs.
POSIX uninstall lifecycle, packaging owner Package files cannot be removed until owned supervisor absence is proven; controller failure remains visible and retryable. Oracle reads exit status, registration, binary and sibling bytes. Normal removal; Linux/macOS controller failure; loaded registration without file; explicit not-found; timed-out container; retry. Rootless Linux container, pinned Alpine digest, 12 cases at fab1aa1d; evidence, manifest. Fake supervisors do not qualify native systemd-user or launchd.
Native product qualification, platform owners Same-run native artifacts must pass process/lifecycle shards and preserve unrelated roots and processes. H2 reports and process census are the oracle. Normal install/run/uninstall; absent supervisor; access failure/retry; Windows locked files and sibling preservation. macos-15, macos-15-intel, windows-2025, windows-11-arm; exact commands and still-required observations are in the native runbook.
Mutation campaigns, slice owners Every discovered mutant has exactly one outcome; a clean baseline is mandatory; meaningful guards have a designated behavioral canary. Context curated canary; 32 receipt and 23 cancellation mutants; empty inventory, missed, timeout and unclassified outcomes cannot pass. Cancellation shards run the owning product_run:: namespace with a 120-second per-test bound inside the eight-minute campaign cap. Linux cargo-mutants 27.1.0; per-invariant mapping and counts, first hosted pilot and remediations. Hosted results across eight receipt and twelve cancellation shards remain revision-specific.
Campaign executor, A0 Missing inputs, timeouts, failed commands and incomplete shards cannot pass; owned process groups are reaped; each stream is retained within 8 MiB; the final bundle stays within 64 MiB; tracked and non-ignored workspace files remain byte-stable. Successful and failed child; timeout with descendant and sibling; escaped pipe holder; bounded prefix/tail capture; oversized bundle and symlink rejection; changed-source census; missing corpus and mutation outcomes. Local Linux tests plus Windows target strict Clippy; workflow uses ten-minute jobs, two mutation workers and 30-day artifacts. Native containment remains a corresponding-OS evidence obligation.

Every row names remaining platform or model limits explicitly. A limitation is not counted as an executed test. Source/configuration JSON under each target/discovery/ operation binds raw local or hosted output to its exact SHA, dirty patch, Rust version, OS, architecture, operation and compiler fan-out.

Replaying regressions

Populate the candidate's locked dependency cache once with cargo fetch --locked. Then run:

cargo xtask discovery-replay

This command validates that all four fuzz binaries and discovery-replay are registered and every corpus contains regular files of at most 8192 bytes. It runs each corpus through the stable binary with --locked --offline, at most 110 seconds per target. A missing target, empty corpus, failed assertion or timeout is an error. Individual ordinary replay is:

cargo run --locked --offline -p peritus-bug-discovery --bin discovery-replay -- sse crates/app/testing/peritus-bug-discovery/corpus/sse

Change both target and corpus suffix for the other registered targets. Preserve raw input bytes, target name and minimized regression together.

Bounded discovery

Production Rust remains pinned by the existing workspace toolchain. The explicit setup commands install harness tools; PR replay never invokes them:

cargo xtask discovery-setup-fuzz
cargo xtask discovery-setup-mutation

Fuzz setup installs nightly-2026-08-09 with rust-src and cargo-fuzz 0.13.2; mutation setup installs cargo-mutants 27.1.0. No setup command changes the default toolchain. A native C++ compiler is required by libFuzzer. The engine invocation uses --no-cfg-fuzzing, so production validation cannot be bypassed by cargo-fuzz's usual global cfg.

cargo xtask discovery-fuzz-sse
cargo xtask discovery-fuzz-ndjson
cargo xtask discovery-fuzz-working-state
cargo xtask discovery-fuzz-provider-sequence
cargo xtask discovery-posix-lifecycle
cargo xtask discovery-mutation-context
cargo xtask discovery-mutation-receipt
cargo xtask discovery-mutation-cancellation

Each fuzz campaign copies the corpus into its evidence directory, uses seed 881, an 8192-byte maximum input, 2048 MiB RSS limit, 30-second per-input timeout and 120-second engine budget. AddressSanitizer and the semantic oracles remain active; leak detection is disabled because traced child environments make LeakSanitizer abort after otherwise successful Rust campaigns. The complete build/run process has an eight-minute bound. A success exit also requires the engine's completion log to demonstrate nonzero executions and its full allotted budget. Build failure, absent completion evidence, timeout and oracle failure cannot become successful campaigns.

The POSIX lifecycle campaign accepts only docker or podman, pulls the exact reviewed Alpine 3.22 manifest digest, and runs every scenario with networking disabled and strict prerequisite enforcement. Set PERITUS_CONTAINER_ENGINE=podman for a local Podman run; scheduled CI uses Docker by default. A missing engine, failed pull, skipped scenario or zero executed scenarios fails the operation.

Cargo-fuzz 0.13.2 has no locked-input forwarding switch. The wrapper first resolves metadata with --locked --offline, disables Cargo network access during fuzzing and compares the complete lockfile before and after execution. A changed lockfile invalidates the campaign and remains visible for investigation. Checked-in corpora are never modified by the engine.

Mutation commands require committed tracked source, then retain --list --json with exact source ranges and diffs. Zero discovery is an error classified as unreachable, not test effectiveness. Nonempty inventories clone the committed tree into a short, private disposable directory, then run a clean unmodified baseline and the same tests for each in-place mutant inside that clone. Receipt campaigns run their full package suite; cancellation campaigns always retain the owning product_run:: namespace with a 120-second test ceiling. A direct AF_UNIX bind probe records host capability without removing the owning lifecycle tests. The wrapper retains denied lints, locked offline Cargo, one mutation worker, two compiler jobs, process-group containment, bounded compiler cache and evidence while excluding untracked coordination files.

Receipt commands append -0 through -7, for example cargo xtask discovery-mutation-receipt-0; cancellation commands append -0 through -11. The twelve cancellation shards retain all 23 discovered mutants with at most two mutants per current shard, leaving room for the measured baseline and test costs. The wrapper retains both full and selected inventories, an explicit outside-this-shard count, the per-slice shard count, and selected outcomes; an empty shard fails. Scheduled mutation jobs cover all eight receipt and twelve cancellation shards with at most two running concurrently. Eight minutes bounds the whole campaign; build and test phases have additional limits. All discovered mutants must have exactly one recognized outcome before completion can be reported. Partial inventories retain an untested count. caught, missed, unviable, timeout, unclassified outcomes and baseline failure are separate; equivalence requires independent review. No mutation coverage score is inferred.

Minimization and accepted failures

Preserve the original crash artifact and failing logical signature before minimizing:

cargo +nightly-2026-08-09 fuzz tmin sse target/discovery/CASE/crash-INPUT --features fuzzing --no-cfg-fuzzing --fuzz-dir crates/app/testing/peritus-bug-discovery -- -max_total_time=120 -rss_limit_mb=2048 -timeout=30

CASE and crash-INPUT are placeholders for the reported evidence path. Replay the minimized input through discovery-replay; reduce structured operations or fault points only while retaining the same oracle failure and required reachability. A minimized byte input that no longer reaches the original boundary is not an accepted reproducer. The owner must retain an ordinary regression that fails before the fix and passes afterward. Accepted chaos findings additionally require three independent fresh-fixture reproductions, persisted state observations, and a final process/filesystem census.

CI status and evidence

The checked workflow runs stable replay for pull requests. Weekly and manually dispatched campaigns use one Linux job per fuzz target and mutation slice/shard, with a ten-minute ceiling and failure-independent shards. The policy test audits the exact inventory, toolchain, pre-Cargo configuration check, commands, trigger conditions and unconditional artifact upload. Missing artifacts are errors. Host job timeouts can interrupt setup or builds before the discovery engine begins; that remains incomplete work, never a clean campaign.

Every operation writes completion.json as incomplete before launch. Each child also records its invocation before launch and a terminal command result afterward. Engine logs, mutated-source inventories, private corpus and isolated compiler-cache state remain under the operation's evidence directory. Volatile compiler and engine files use an owned OS temporary directory outside the Cargo workspace; successful completion requires removing it after process-group reaping. Each stdout and stderr capture drains concurrently, retains a bounded prefix and tail within 8 MiB, and records observed bytes and truncation. A timed-out capture is cancelled before its buffered bytes may touch finalized evidence. Finalization removes disposable compiler scratch and caps the retained operation bundle at 64 MiB. Oversized diagnostics keep a prefix and tail; bundle.json records original size and SHA-256, retained size, removal or truncation, and any omitted manifest rows. All JSON reports, source/canary patches, tool versions, mutation inventories, and libFuzzer crash-, timeout-, oom-, and leak- inputs are preserved; the campaign fails if this required evidence cannot fit. The source census binds both the initial/final HEAD identity and all tracked and non-ignored untracked bytes. The runner owns a fresh Unix process group or Windows Job Object, terminates the owned group on timeout and after root exit, and reaps its direct child. A pipe that remains open after the five-second cleanup budget leaves the command report incomplete instead of hanging the campaign. The Linux qualification checks that a reached descendant is no longer executing and a separately owned sibling survives; a detached-process fixture proves an escaped pipe holder cannot convert cleanup into an unbounded wait or mutate finalized evidence, but it does not claim arbitrary detached-process containment. Final accepted-failure evidence must include any additional native census required by the campaign.

Local infrastructure tests are execution evidence only for their actual host. Hosted job timing and non-Linux containment remain unverified until those jobs run against the delivered revision. A shard that times out remains incomplete even if another shard passes; aggregate all eight receipt or twelve cancellation inventories before describing that whole slice as completed. Do not reduce the inventory to make a pass appear complete.