Skip to content

Commit c9b6bcd

Browse files
authored
feat: plan conservative incremental hashing queries (#11)
## Why [PR #10](#10) adds a seedable hash file containing the previous revision's target hashes and direct dependency graph. Given that seed and a Git diff, the next problem is deciding which targets must be recomputed. This PR adds a conservative planner that turns changed file paths into a reduced Bazel query. “Dirty” here means **must be recomputed**, not necessarily **known to have a different final hash**. The planner narrows the query only when the seed contains enough information to prove the narrower query is safe; otherwise it requests a full computation. This remains library functionality in this PR. `hash-persister` starts using it in [PR #12](#12). ## From changed files to affected targets Suppose `lib/value.go` belongs to Bazel package `//lib` and `//app:binary` depends on `//lib:library`: 1. A change to `lib/value.go` marks `//lib` dirty. 2. Every target previously known in `//lib` is marked directly dirty. 3. The persisted graph is traversed in reverse, marking `//app:binary` and any further dependents dirty as well. A source file is assigned to the nearest enclosing package known to the seed. This follows Bazel package boundaries: a target can only refer to files in its own package or through labels owned by another package. A changed file with no enclosing known package cannot be an input of a seeded target and can be ignored unless it is one of the repository-wide fallback triggers described below. The result retains dirty package names as well as target labels. If a BUILD file changes, targets may have been added or removed since the seed was written. Querying the whole dirty package discovers additions and avoids referring to deleted labels; querying only labels remembered by the seed would do neither. ## When the planner refuses to narrow the query Some files can change Bazel loading, repository resolution, package ownership, or hashing behavior without appearing as ordinary target dependency edges. The planner requests full hashing for these cases, including: - Starlark files and workspace, module, or repository metadata; - Bazel configuration, version, ignore, lock, and repository-state files; - files used by rule-class fingerprints; - deleted or renamed BUILD files; - a BUILD file that creates a package absent from the seed. These cases receive a stable, bounded fallback code for automation and metrics, plus a human-readable detail naming the triggering path. Keeping paths out of the code prevents unbounded metric dimensions. ## Preserving the caller's target expression The reduced query contains two kinds of entries: - dirty packages use package wildcards such as `//lib:all`, so added and removed targets are handled; - affected targets in unchanged packages use explicit labels such as `//app:binary`, because their BUILD files have not changed. Those entries form a smaller replacement for the repository-wide `//...` universe in the caller's original Bazel query expression. For example, filters applied to `//...` remain applied to the reduced universe rather than being discarded. Only local `//...` occurrences are replaced; external-repository expressions such as `@repo//...` retain their original meaning. An expression without a local `//...` cannot be transformed safely by this mechanism, so orchestration will fall back to the full query. ## Review guidance The central correctness property is that every changed path must do one of two things: reach every target whose hash might be affected, or request a full computation. Package ownership, BUILD-file boundary changes, reverse-dependency propagation, and preservation of the original Bazel query semantics are therefore the most important review areas. ## Stack 1. [Seedable persistence and compatible cache seeding](#10) 2. **This PR:** dirty-set planning and scoped queries 3. [`hash-persister` orchestration, verification, fallback handling, and execution reporting](#12)
2 parents 6e03840 + 164b48e commit c9b6bcd

5 files changed

Lines changed: 824 additions & 0 deletions

File tree

pkg/BUILD.bazel

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7,9 +7,11 @@ go_library(
77
"bazel_info.go",
88
"cache.go",
99
"configurations.go",
10+
"dirty_set.go",
1011
"hash_cache.go",
1112
"hash_persistence.go",
1213
"normalizer.go",
14+
"scoped_query.go",
1315
"target_determinator.go",
1416
"targets_list.go",
1517
"walker.go",
@@ -36,9 +38,11 @@ go_test(
3638
name = "pkg_test",
3739
srcs = [
3840
"cache_test.go",
41+
"dirty_set_test.go",
3942
"hash_cache_test.go",
4043
"hash_persistence_test.go",
4144
"normalizer_test.go",
45+
"scoped_query_test.go",
4246
"seed_hashes_test.go",
4347
"target_determinator_test.go",
4448
"walker_test.go",

pkg/dirty_set.go

Lines changed: 270 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,270 @@
1+
package pkg
2+
3+
import (
4+
"path/filepath"
5+
"sort"
6+
"strings"
7+
)
8+
9+
// DirtySetResult contains the computed dirty set and metadata.
10+
type DirtySetResult struct {
11+
// DirtyLabels is the set of target labels directly affected by changes.
12+
DirtyLabels map[string]bool
13+
// DirtyStarLabels is DirtyLabels plus all transitive reverse deps.
14+
DirtyStarLabels map[string]bool
15+
// DirtyPackages is the sorted list of packages containing changed files
16+
// (source or BUILD). Targets in these packages must be re-listed with a
17+
// package wildcard (e.g. //pkg:all) rather than by explicit label, so
18+
// that targets added to or removed from the package are handled.
19+
DirtyPackages []string
20+
// NeedsFallback is true when a change requires a full rehash.
21+
NeedsFallback bool
22+
// FallbackReason describes why a fallback was triggered.
23+
FallbackReason string
24+
// FallbackCode is a stable, bounded identifier suitable for reporting and
25+
// metrics. FallbackReason remains the human-readable detail.
26+
FallbackCode string
27+
}
28+
29+
// ComputeDirtySet determines which targets need rehashing based on changed
30+
// files and the persisted edge map.
31+
//
32+
// changedFiles maps file paths (relative to workspace root) to their git
33+
// diff status code (M, A, D, R, etc.).
34+
// edges maps target labels to their direct dependency labels (from the seed
35+
// file's TargetEdges).
36+
// allLabels is the complete set of labels known to the seed file (union of
37+
// TargetEdges keys and values, plus TargetHashes keys). Used both to find
38+
// labels in dirty packages and to derive the set of known packages.
39+
// ruleClassFingerprintFiles is the set of workspace-relative paths used for
40+
// rule-class fingerprints; a change to any of these triggers a fallback.
41+
//
42+
// A changed source file is attributed to its owning package by walking up
43+
// the directory tree to the nearest package known to the seed. This mirrors
44+
// how Bazel assigns files to packages: globs cannot cross package
45+
// boundaries, and a label //pkg:path/file requires pkg to be a package, so
46+
// the nearest enclosing package is the only one whose targets can reference
47+
// the file. Files with no enclosing known package cannot be inputs to any
48+
// seeded target and are ignored.
49+
//
50+
// Fallback (full rehash) is triggered by files that can change loading,
51+
// repository resolution, or package boundaries without appearing in target
52+
// edges: Starlark, module/workspace/repository metadata, Bazel configuration,
53+
// rule-class fingerprint files, deleted or renamed BUILD files, and BUILD
54+
// files for packages unknown to the seed.
55+
func ComputeDirtySet(
56+
changedFiles map[string]string,
57+
edges map[string][]string,
58+
allLabels map[string]bool,
59+
ruleClassFingerprintFiles map[string]bool,
60+
) *DirtySetResult {
61+
result := &DirtySetResult{
62+
DirtyLabels: make(map[string]bool),
63+
DirtyStarLabels: make(map[string]bool),
64+
}
65+
66+
knownPackages := make(map[string]bool)
67+
labelsByPackage := make(map[string][]string)
68+
for label := range allLabels {
69+
pkg := labelToPackage(label)
70+
knownPackages[pkg] = true
71+
labelsByPackage[pkg] = append(labelsByPackage[pkg], label)
72+
}
73+
74+
// First pass: fallback triggers.
75+
for filePath, status := range changedFiles {
76+
base := filepath.Base(filePath)
77+
78+
if ruleClassFingerprintFiles[filePath] {
79+
result.NeedsFallback = true
80+
result.FallbackCode = "rule_fingerprint_change"
81+
result.FallbackReason = "rule-class fingerprint file changed: " + filePath
82+
return result
83+
}
84+
85+
if isFallbackTrigger(base) {
86+
result.NeedsFallback = true
87+
result.FallbackCode = "unsafe_file_change"
88+
result.FallbackReason = "fallback trigger file changed: " + filePath
89+
return result
90+
}
91+
92+
if base == "BUILD" || base == "BUILD.bazel" {
93+
if status == "D" || strings.HasPrefix(status, "R") {
94+
result.NeedsFallback = true
95+
result.FallbackCode = "package_boundary_change"
96+
result.FallbackReason = "package deleted or renamed: " + filePath
97+
return result
98+
}
99+
if !knownPackages[fileToPackage(filePath)] {
100+
result.NeedsFallback = true
101+
result.FallbackCode = "package_boundary_change"
102+
result.FallbackReason = "BUILD file for package unknown to seed: " + filePath
103+
return result
104+
}
105+
}
106+
}
107+
108+
// Second pass: map changed files to dirty packages and labels.
109+
dirtyPackages := make(map[string]bool)
110+
for filePath := range changedFiles {
111+
base := filepath.Base(filePath)
112+
113+
var pkg string
114+
if base == "BUILD" || base == "BUILD.bazel" {
115+
// A BUILD file change dirties its own package (verified known above).
116+
pkg = fileToPackage(filePath)
117+
} else {
118+
// A source file belongs to the nearest enclosing known package.
119+
owner, ok := owningPackage(filePath, knownPackages)
120+
if !ok {
121+
// No enclosing package: the file cannot be an input to any
122+
// seeded target (see function comment).
123+
continue
124+
}
125+
pkg = owner
126+
}
127+
128+
if !dirtyPackages[pkg] {
129+
dirtyPackages[pkg] = true
130+
for _, label := range labelsByPackage[pkg] {
131+
result.DirtyLabels[label] = true
132+
}
133+
}
134+
}
135+
136+
for pkg := range dirtyPackages {
137+
result.DirtyPackages = append(result.DirtyPackages, pkg)
138+
}
139+
sort.Strings(result.DirtyPackages)
140+
141+
// Propagate dirtiness through reverse deps.
142+
rdeps := BuildRdeps(edges)
143+
144+
queue := make([]string, 0, len(result.DirtyLabels))
145+
for label := range result.DirtyLabels {
146+
result.DirtyStarLabels[label] = true
147+
queue = append(queue, label)
148+
}
149+
150+
for len(queue) > 0 {
151+
current := queue[0]
152+
queue = queue[1:]
153+
154+
for _, rdep := range rdeps[current] {
155+
if !result.DirtyStarLabels[rdep] {
156+
result.DirtyStarLabels[rdep] = true
157+
queue = append(queue, rdep)
158+
}
159+
}
160+
}
161+
162+
return result
163+
}
164+
165+
func isFallbackTrigger(basename string) bool {
166+
if strings.HasSuffix(basename, ".bzl") {
167+
return true
168+
}
169+
lowerBasename := strings.ToLower(basename)
170+
if strings.HasSuffix(lowerBasename, ".lock") ||
171+
strings.Contains(lowerBasename, "-lock.") ||
172+
strings.Contains(lowerBasename, "_lock.") {
173+
return true
174+
}
175+
if basename == ".bazelrc" || strings.HasPrefix(basename, ".bazelrc.") ||
176+
strings.HasSuffix(basename, ".bazelrc") || strings.HasSuffix(basename, ".rc") {
177+
return true
178+
}
179+
switch basename {
180+
case "MODULE.bazel", "MODULE.bazel.lock",
181+
"WORKSPACE", "WORKSPACE.bazel", "WORKSPACE.bzlmod",
182+
"REPO.bazel", "VENDOR.bazel",
183+
"maven_install.json",
184+
".bazelversion", ".bazelignore", ".gitmodules":
185+
return true
186+
}
187+
if strings.HasSuffix(basename, ".MODULE.bazel") {
188+
return true
189+
}
190+
return false
191+
}
192+
193+
func fileToPackage(filePath string) string {
194+
dir := filepath.Dir(filePath)
195+
if dir == "." {
196+
return "//"
197+
}
198+
return "//" + filepath.ToSlash(dir)
199+
}
200+
201+
// owningPackage walks up the directory tree from filePath and returns the
202+
// nearest enclosing package present in knownPackages.
203+
func owningPackage(filePath string, knownPackages map[string]bool) (string, bool) {
204+
dir := filepath.ToSlash(filepath.Dir(filePath))
205+
for {
206+
var pkg string
207+
if dir == "." || dir == "/" || dir == "" {
208+
pkg = "//"
209+
} else {
210+
pkg = "//" + dir
211+
}
212+
if knownPackages[pkg] {
213+
return pkg, true
214+
}
215+
if pkg == "//" {
216+
return "", false
217+
}
218+
parent := filepath.ToSlash(filepath.Dir(dir))
219+
if parent == dir {
220+
return "", false
221+
}
222+
dir = parent
223+
}
224+
}
225+
226+
// LabelPackage returns the package portion of a label, e.g. "//foo/bar" for
227+
// "//foo/bar:baz". Repository prefixes are stripped.
228+
func LabelPackage(label string) string {
229+
return labelToPackage(label)
230+
}
231+
232+
func labelToPackage(label string) string {
233+
if idx := strings.Index(label, "//"); idx >= 0 {
234+
label = label[idx:]
235+
}
236+
if idx := strings.IndexByte(label, ':'); idx >= 0 {
237+
return label[:idx]
238+
}
239+
return label
240+
}
241+
242+
// BuildRdeps constructs a reverse-dependency map from an edge map.
243+
func BuildRdeps(edges map[string][]string) map[string][]string {
244+
rdeps := make(map[string][]string)
245+
for label, deps := range edges {
246+
for _, dep := range deps {
247+
rdeps[dep] = append(rdeps[dep], label)
248+
}
249+
}
250+
for dep := range rdeps {
251+
sort.Strings(rdeps[dep])
252+
}
253+
return rdeps
254+
}
255+
256+
// CollectAllLabels builds the complete set of labels from the edge map
257+
// (both keys and values) and the hash map keys.
258+
func CollectAllLabels(edges map[string][]string, hashLabels map[string]map[string]string) map[string]bool {
259+
all := make(map[string]bool)
260+
for lbl := range edges {
261+
all[lbl] = true
262+
for _, dep := range edges[lbl] {
263+
all[dep] = true
264+
}
265+
}
266+
for lbl := range hashLabels {
267+
all[lbl] = true
268+
}
269+
return all
270+
}

0 commit comments

Comments
 (0)