feat: orchestrate and report incremental hashing (#12)

## Why

The first two PRs add the data needed for incremental hashing and a
conservative way to identify affected targets. This PR makes the feature
available in `hash-persister`.

Existing invocations remain full computations and continue to write the
existing comparison-file format. Incremental mode is explicitly opt-in
and is only an optimization: whenever `hash-persister` cannot prove
reuse is safe, it runs the existing full computation and still produces
a correct, complete output.

## User-facing workflow

The first seed is created by requesting the seedable artifact and
Bazel's `query` backend, which exposes the loading-phase dependency
graph without Bazel build configurations:

```sh
hash-persister \
  --query-backend=query \
  --seedable-output \
  --output "hashes-${BASE_SHA}.json" \
  "${BASE_SHA}"
```

The next revision can then reuse it:

```sh
hash-persister \
  --query-backend=query \
  --seed-file "hashes-${BASE_SHA}.json" \
  --seed-sha "${BASE_SHA}" \
  --output "hashes-${NEW_SHA}.json" \
  "${NEW_SHA}"
```

`--seed-file` supplies the earlier complete artifact; `--seed-sha`
states which revision it must represent. Supplying a seed automatically
makes the new output seedable, including when incremental execution
falls back to full hashing. This ensures a successful run can always
seed the next revision.

`--seedable-output` without a seed performs full hashing but writes the
larger v9 artifact containing dependency edges and compatibility
metadata. Without either flag, full mode keeps writing the legacy
v8-shaped artifact. Seedable output requires `--query-backend=query`;
ordinary `cquery` usage is unaffected.

## Incremental execution

Before reusing anything, `hash-persister` checks that the seed:

- uses the current seed-capable format and contains dependency edges;
- represents the revision passed through `--seed-sha`;
- has a compatibility fingerprint matching the current Bazel release,
target expression, query backend, Bazel options, filtering inputs,
rule-class fingerprints, and hashing version;
- contains valid SHA-256 target hashes.

It then computes the changed paths between the seed and destination
revisions. Rename detection is disabled so a move is conservatively
represented as a deletion plus an addition. The planner from PR #11
either requests a full fallback or returns dirty packages and their
affected reverse dependencies.

Only that reduced set is queried and rehashed. Hashes for unaffected
targets satisfy dependency lookups from the seeded in-memory cache. The
old graph is used only to choose the conservative initial dirty set; the
scoped query observes the dirty packages and their dependency closure at
the destination revision. Those fresh results then replace the dirty
portion of the seed: deleted targets disappear, newly added targets are
discovered, added or removed dependency edges replace their old
versions, and unaffected entries remain. The output is therefore
complete rather than merely a patch over the seed. If no seeded target
is affected, the artifact can advance to the new revision without a
Bazel query.

Generated query expressions may contain many explicit labels.
Expressions up to 64 KiB remain ordinary command-line arguments; larger
expressions are passed through Bazel's `--query_file` rather than
risking the operating system's command-line length limit. Logs show a
bounded preview instead of printing an arbitrarily large expression.

## Fallback versus failure

An incompatible seed, a repository-wide or package-boundary change, an
unsupported target expression, a Git-diff problem, or a scoped-query
problem prevents safe reuse but does not prevent a correct full result.
These conditions log a stable fallback code and run full hashing.

Errors that also prevent full hashing remain failures. This distinction
lets CI treat fallback as a performance outcome rather than a
correctness or availability failure. Codes such as
`seed_compatibility_mismatch`, `package_boundary_change`, and
`scoped_query_error` are bounded for metrics; path-specific details
remain diagnostic text.

## Rollout verification and metrics

`--verify-seed` runs the incremental and full computations for the same
destination and compares their target hashes. It succeeds only if
incremental mode was actually used and both results agree. On
disagreement, it reports a bounded sample of differences and preserves
the full result as the output. This is intended for sampled rollout
validation because it deliberately performs both computations.

The primary purpose of `--execution-report <path>` is to let the CI
wrapper that invokes `hash-persister` emit metrics. `hash-persister`
writes a small, versioned JSON handoff rather than depending on a
particular metrics client or backend; the wrapper reads it after the
process exits and maps its fields into the surrounding CI system's
metrics.

The report is written for successful incremental runs, full-mode
fallbacks, and execution failures. It records whether incremental mode
was requested and actually used, a stable fallback code, and bounded
workload counts such as changed files, dirty targets, recomputed
targets, reused targets, and total targets. These support metrics such
as incremental-use and fallback rates, fallback reasons, and the
proportion of hashes reused. Human-readable fallback and error details
remain available for diagnosis but are not intended to become metric
dimensions.

The README added here documents the mode, both CLI workflows, fallback
behavior, verification, and execution reporting.

## Review guidance

The most important boundaries are seed validation, the transition from
incremental work to a full fallback, and merging fresh results back into
a complete seed. In particular, a fallback must remain seedable, clean
entries must be retained, and every dirty entry must be replaced or
removed.

## Stack

1. [Seedable persistence and compatible cache
seeding](#10)
2. [Conservative dirty-set planning and scoped
queries](#11)
3. **This PR:** CLI orchestration, verification, fallback handling,
documentation, and execution reporting

Author: mattnworb

README.md

@@ -62,6 +62,60 @@ Target Determinator now includes hash persistence capabilities for optimized CI
 
 These tools enable faster target determination in CI by pre-computing hashes once per commit and reusing them across multiple comparisons.
 
+### Incremental hash persistence
+
+A normal `hash-persister` run asks Bazel for the complete target graph selected by `--targets`, computes every target hash, and writes the existing comparison artifact for the requested Git revision. It does not include the dependency edges needed by incremental hashing. Passing `--seedable-output` opts into a larger artifact that also contains those edges and compatibility metadata.
+
+Incremental mode starts with a seedable artifact from an earlier revision, uses the Git diff and the persisted dependency graph to find targets that may have changed, and queries and hashes only that smaller set. Hashes for unaffected targets are copied into the new artifact, which is again complete and seedable for a later run.
+
+Incremental mode is enabled by passing both `--seed-file` and `--seed-sha`. It currently requires the faster, configuration-independent `query` backend:
+
+```sh
+hash-persister \
+  --working-directory . \
+  --query-backend=query \
+  --output "hashes-${NEW_SHA}.json" \
+  --seed-file "hashes-${BASE_SHA}.json" \
+  --seed-sha "${BASE_SHA}" \
+  "${NEW_SHA}"
+```
+
+`--seed-file` is the JSON artifact produced for the revision named by `--seed-sha`. The seed must use the current seed-capable artifact format and must have been created with compatible hashing inputs, including the Bazel release, target expression, query backend, Bazel options, and rule-class fingerprints. These inputs are represented by a compatibility fingerprint embedded in the artifact. Supplying `--seed-file` automatically makes the new output seedable, including when incremental execution falls back to a full computation.
+
+Incremental hashing is an optimization rather than a weaker correctness mode. If `hash-persister` cannot prove that reuse is safe, it logs a bounded fallback code and performs a normal full computation. This includes incompatible or malformed seeds and changes that can affect Bazel loading or package boundaries without appearing in the persisted target graph, such as changes to Starlark, workspace or module metadata, Bazel configuration files, or BUILD-file boundaries. The resulting output is still a complete artifact for the requested revision and can seed a later incremental run.
+
+To create the first compatible seed, explicitly request seedable output while selecting the `query` backend:
+
+```sh
+hash-persister \
+  --working-directory . \
+  --query-backend=query \
+  --seedable-output \
+  --output "hashes-${BASE_SHA}.json" \
+  "${BASE_SHA}"
+```
+
+#### Verifying incremental results
+
+`--verify-seed` runs both incremental and full hashing for the destination revision and compares the resulting target hashes:
+
+```sh
+hash-persister \
+  --working-directory . \
+  --query-backend=query \
+  --output "hashes-${NEW_SHA}.json" \
+  --seed-file "hashes-${BASE_SHA}.json" \
+  --seed-sha "${BASE_SHA}" \
+  --verify-seed \
+  "${NEW_SHA}"
+```
+
+Verification exits nonzero if the two results differ or if incremental execution falls back to full hashing. When the two computations differ, the output path receives the full result so it remains safe for investigation and downstream use. A fallback stops verification rather than treating two full computations as evidence that incremental mode is correct. Verification is intended for rollout checks and sampling rather than the normal fast path because it deliberately performs both computations.
+
+#### Execution reports
+
+Pass `--execution-report <path>` to write a versioned, machine-readable JSON summary. The report distinguishes the requested mode from the mode actually used, records success or failure and any fallback code, and includes counts for changed files, dirty packages and targets, recomputed targets, reused targets, and total targets. This lets CI systems emit bounded metrics without parsing human-readable logs.
+
 ## driver binary
 
 `driver` is a binary which implements a simple CI pipeline; it runs the same logic as `target-determinator`, then tests all identified targets.

hash-persister/BUILD.bazel

@@ -1,4 +1,4 @@
-load("@io_bazel_rules_go//go:def.bzl", "go_library")
+load("@io_bazel_rules_go//go:def.bzl", "go_library", "go_test")
 load("//rules:multi_platform_go_binary.bzl", "multi_platform_go_binary")
 
 go_library(
@@ -12,6 +12,13 @@ go_library(
     ],
 )
 
+go_test(
+    name = "hash-persister_test",
+    srcs = ["hash-persister_test.go"],
+    embed = [":hash-persister_lib"],
+    deps = ["//pkg"],
+)
+
 multi_platform_go_binary(
     name = "hash-persister",
     embed = [":hash-persister_lib"],