optiflow

Observe first. Prove relationships. Plan safely.

optiflow is a local-first Rust CLI for inventorying messy media collections, proving byte-identical duplicate groups, calculating reclaimable storage, and producing immutable review plans.

Version 0.1.0 is deliberately read-only. It has no apply, delete, replace, move, quarantine, or optimization command.

Current capabilities

The capabilities below describe merged source at 5be461c, checked on 2026-09-17. They are not a promise about every installed 0.1.0 binary: this source includes work added after the v0.1.0 release tag.

• Scan one or more files and directories without modifying them.
• Exclude hidden trees, symbolic links, filesystem crossings, and optiflow's own state directory by default.
• Classify files by inspected content rather than filename extension.
• Collect optional container, stream, codec, dimensions, duration, sample-rate, channel, and bitrate metadata through ffprobe JSON.
• Derive deterministic read-only lossless-PNG review opportunities from current, provider-bound evidence without creating outputs or estimating savings.
• Persist observations and reusable analysis in a local SQLite database.
• Narrow exact-duplicate candidates by byte length before complete hashing.
• Calculate complete streaming BLAKE3-256 hashes only for size candidates.
• Produce exact-duplicate groups with transparent evidence.
• Calculate potential reclaimable storage without deleting anything.
• Publish schema-versioned run, report, and plan artifacts as coherent, marker-sealed sets.
• Generate deterministic review plans with apply-time safety preconditions.
• Emit human-readable or stable JSON output for flow and other pipelines.
• Load explicitly selected, operator-locked extension manifests for typed inspectors, analyzers, policy contributors, planners, validators, report/export providers, and read-only lifecycle observers.

Media capability matrix

OptiFlow does not yet produce optimized media or apply replacements. Inventory, optional probing, review evidence, and candidate validation are separate capabilities. Exact deduplication here means evidence and review planning, not deletion.

Legend: ✓ merged implementation; ~ conditional or limited subset; P planned only; — no implementation or selected format-specific profile. Each status links to its evidence or precise roadmap boundary.

Files	Inventory / exact dedup	Optional probe	Review evidence	Candidate validation	Candidate production	Apply / replace	Maturity / evidence
Any regular file (baseline)	✓	—	—	—	—	P	Format-independent inventory
PNG	✓	~	~	~	P	P	Review + library validator
JPEG	✓	~	—	P	P	P	Probe eligible; encoding planned
GIF	✓	~	—	—	—	P	No GIF producer selected
SVG	✓	—	—	—	—	P	Inventory only
WebP	✓	~	—	P	P	P	Delivery-format evaluation
AVIF / HEIF	✓	~	—	P	P	P	AVIF evaluation; HEIF inventory/probe
JPEG XL	✓	~	—	P	P	P	Delivery-format evaluation
TIFF, RAW, other images	✓	~	—	—	—	P	Recognition varies; preserve RAW
Audio	✓	~	—	P	P	P	Stream inspection; transforms planned
Video	✓	~	—	P	P	P	Stream inspection; transforms planned

The baseline row covers arbitrary readable regular files; media rows add the listed conditional capabilities. Classification inspects bytes, not suffixes. The locked infer recognizers only establish classification eligibility; optional ffprobe coverage depends on the exact installed provider and the input. SVG has no built-in image recognizer. RAW and container recognition is partial. P does not mean supported: the modern-format entries are evaluation candidates, and planned apply refers to the shared transaction engine, not a promise of optimization for every row.

PNG review opportunities do not prove that a candidate can be made smaller. The separate byte validator is library-only: static noninterlaced 8-bit RGB/RGBA with a documented metadata subset, exact sample/chunk preservation, and a strictly smaller encoded candidate. It does not run during scans or create files. See the capability evidence and optimizer strategy for source ownership, limits, OxiPNG sequencing, and the image_optim comparison.

Requirements

• Rust stable with edition 2024 support
• macOS or Linux
• Optional: ffprobe for media stream inventory
• Optional: jq for the repository smoke test

On macOS:

brew install ffmpeg jq rust

Quick start

cargo build --release

./target/release/optiflow doctor

./target/release/optiflow scan "/path/to/Media"

The scan prints its run identifier and writes immutable artifacts beneath the local state directory:

runs/<run-id>/
├── effective-policy.json
├── run.json
└── report.json

Generate a separate, review-only exact-duplicate plan:

./target/release/optiflow plan exact-duplicates \
  --run "<run-id>"

The generated plan-exact-duplicates.json explicitly declares "mutates_files": false. Its lexicographically first keep_path is only a stable review default; it is not presented as the objectively best copy.

Commands

optiflow doctor
optiflow scan <inputs...>
optiflow report <run-or-report-path>
optiflow plan exact-duplicates --run <run-or-report-path>
optiflow cache status
optiflow config validate
optiflow config show
optiflow config explain <setting>
optiflow extensions list --manifest <FILE> --lock <FILE>
optiflow extensions inspect <EXTENSION_ID> --manifest <FILE> --lock <FILE>
optiflow extensions doctor --manifest <FILE> --lock <FILE>

Global options:

--state-directory <DIRECTORY>  Override persistent local state
--config <FILE>                Select one explicit configuration file
--no-config                    Disable configuration-file loading
--output-format <FORMAT>       Select human or JSON command-result output
--json                         Compatible alias for --output-format json

Scan policy options:

--follow-symlinks      Follow symbolic links
--no-follow-symlinks   Explicitly keep link following disabled
--include-hidden       Include hidden files and directories
--exclude-hidden       Explicitly exclude hidden paths
--cross-filesystems    Cross filesystem boundaries
--stay-on-filesystem   Explicitly preserve filesystem boundaries
--probe                Explicitly enable optional ffprobe inspection
--no-probe             Skip optional ffprobe metadata extraction

The defaults are intentionally conservative for external drives and large archives.

Configuration follows compiled defaults < user file < nearest project optiflow.toml < recognized environment < explicit CLI. See Configuration and effective policy for the strict optiflow.config.v1 schema, exact locations and variables, supported settings, path rules, provenance, fingerprints, and locked invariants.

JSON pipeline contract

Every JSON invocation emits one optiflow.command-result.v1 envelope. The envelope carries the typed outcome, exact process exit code, coverage, diagnostics, committed artifact references, and domain result. Its schema is schemas/command-result.schema.json.

Machine results and committed domain artifacts have independent identifiers:

Artifact	Identifier	Schema
Command result	optiflow.command-result.v1	schemas/command-result.schema.json
Configuration	optiflow.config.v1	schemas/config-v1.schema.json
Effective policy	optiflow.effective-policy.v1	schemas/effective-policy-v1.schema.json
Artifact set	optiflow.artifact-set.v1	schemas/artifact-set-v1.schema.json
Run	optiflow.run.v5	schemas/run.schema.json
Report	optiflow.report.v6	schemas/report.schema.json
Plan	optiflow.plan.v5	schemas/plan.schema.json
Media profile evidence	optiflow.media-profile-evidence.v1	schemas/media-profile-evidence-v1.schema.json
Extension manifest	optiflow.extension-manifest.v1	schemas/extension-manifest-v1.schema.json
Extension lock	optiflow.extension-lock.v1	schemas/extension-lock-v1.schema.json
Extension invocation	optiflow.extension-invocation.v1	schemas/extension-invocation-v1.schema.json
Extension result	optiflow.extension-result.v1	schemas/extension-result-v1.schema.json

Use --output-format json (or --json) for subprocess integration. JSON owns stdout; human primary output uses stdout and human diagnostics use stderr. Complete success is 0, while a valid result with reduced coverage is 3. See CLI outcome contract for the complete stable exit matrix, stream rules, signal behavior, and shell examples.

State locations

• macOS: ~/Library/Application Support/optiflow
• Linux: $XDG_STATE_HOME/optiflow or ~/.local/state/optiflow
• Override: --state-directory or OPTIFLOW_STATE_DIRECTORY

The primary database stays on a local filesystem. optiflow uses SQLite's rollback journal rather than assuming WAL-safe behavior on removable or network volumes.

Safety boundary

An exact group in v0.1.0 means equal byte length plus an equal complete BLAKE3 content hash. Before a future destructive operation, every plan precondition requires optiflow to:

1. Re-read filesystem metadata.
2. Reject files changed since planning.
3. Recalculate the complete BLAKE3 hash.
4. Perform byte-for-byte confirmation.
5. Refuse mutation if any precondition fails.

See Safety Model for the complete invariant set. The safe extension SDK defines the separate provider declaration, operator trust, deterministic resolution, and core result- acceptance boundaries. Extensions receive no source mutation, destructive, signing, or publication authority. The artifact-set commit protocol defines coherent scan/plan publication, marker validation, and crash recovery. The handle-bound observation protocol documents how OptiFlow rejects replacement and in-read races without mixing evidence. The media-profile evidence contract defines the first lossless-PNG review profile, exact provider provenance, limitation semantics, and why an opportunity is neither an output nor a savings guarantee.

Ecosystem integration

optiflow is independently installable and useful as a standalone CLI. In the Ego Hygiene suite, flow is the unified orchestration facade: it invokes optiflow through the CLI and consumes the versioned JSON artifacts. Sibling tools do not embed optiflow's source or take an unversioned Rust dependency on its default branch.

Development

task validate
task performance:check

Equivalent commands:

cargo fmt --all -- --check
cargo clippy --locked --all-targets --all-features -- -D warnings
cargo test --locked --all-targets
task contracts
task extensions:examples
./scripts/smoke-test.sh
python3 scripts/run-performance-baseline.py \
  --binary "target/release/optiflow" \
  --budgets "performance/budgets-v1.json" \
  --output "target/performance-baseline.json" \
  --enforce

The end-to-end test uses synthetic files in a temporary directory, includes spaces and Unicode in paths, and verifies that both duplicate inputs remain unchanged after scanning and planning.

The performance baseline uses a separate synthetic fixture to enforce tolerant release-mode ceilings for cold discovery, complete candidate hashing, warm-cache reuse, peak resident memory, and committed artifact size. It never reads user media or invokes optional probes.

Dependency admission, security reporting, supported binary targets, signed release verification, and immutable rollback are defined in the release and dependency policy and security policy.

Documentation

The product documentation is built with Zensical from the checked-in Markdown under docs/. Its Python environment is pinned independently from the Rust crate with uv.lock.

task docs:serve
task docs:build
task site:serve

The strict documentation build validates internal links and anchors. The site build generates an architecture portal from the canonical root corpus, then composes it with the LaunchKit-derived landing source, Zensical output, and canonical JSON Schemas into a collision-checked dist/ artifact. See Site publication architecture for the boundary between the landing page, architecture, documentation, generated API reference, schemas, repository intelligence, and release guidance.

Project documents

• Complete architecture document inventory
• Architecture portal source and generation contract
• Purpose, vision, and principles
• Architecture
• MVP specification
• Safety model
• Artifact-set commit protocol
• Handle-bound observation protocol
• State model
• JSON contract
• Media-profile evidence
• Media capabilities and optimizer strategy
• CLI outcome contract
• Configuration and effective policy
• Safe extension SDK
• Generated extension capability reference
• Development model
• Performance budgets
• Release and dependency policy
• Security policy
• Site publication architecture
• Cloud-native placement
• Roadmap

License

MIT