Skip to content

Commit 95a5057

Browse files
author
gluck
committed
tree: THE IDLE SWEEP VISITS EACH RUN ONCE -- Q2 needs no data structure
Gluck endorsed epoch buckets for the eviction scan and asked that bucket MAINTENANCE be measured against the scan it replaces before anything was built. Measured, and the answer is that the two costs sit on different paths: the scan is one background walk per beat, the index would be a SHARED WRITE PER READ on a path where nothing shared is written today. one eviction under pressure 147 ns (R=256) .. 1.77 us (R=4096) a bucket move on a warm read 105-119 ns against 7-33 ns lock-free, measured on a ONE-UNIT read at 16 readers, because a 64-unit Range copies 64 KiB and a mutex vanishes inside it So the index would make a warm read 4-15x dearer and serialize every reader to save microseconds per beat. AND THE QUADRATIC DIED ANYWAY, by deleting a question rather than adding a structure: TrimIdle asked "who is coldest?" once per victim -- a full rescan to answer -- when the cutoff was fixed before the sweep began. evictOlderThan(cutoff) is one pass per owner: FULL SWEEP R=256 967 us -> 21.8 us 44x R=1024 10.65 ms -> 77.1 us 138x R=4096 180 ms -> 294 us 612x and the counting instrument agrees: 127 runs dropped in 128 run visits at R=128, against ~R^2/2 = 8192. TestSweepRescansPerDroppedRun named the defect and is now TestAFullSweepVisitsEachRunOnce, asserting the cure. THE BENCHMARK CAUGHT MY OWN QUADRATIC FIRST: the first version searched the successor slice per victim, and only the EXPONENT gave it away -- 109x for 16x the runs where linear predicts 16x. It was already 31x faster than before, which is the size of improvement that ends an investigation. Gates: build, vet, full suite -count=1, -race -count=2 on ./internal/store/...
1 parent 3c9c19d commit 95a5057

5 files changed

Lines changed: 306 additions & 18 deletions

File tree

‎internal/store/tree/cache.go‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -855,6 +855,42 @@ func (c *Cache[U]) coldest() (int64, bool) {
855855
return best, found
856856
}
857857

858+
// evictOlderThan hollows every evictable run older than cutoff in ONE pass
859+
// over the index, publishing each node's successor once. The hooks fire
860+
// outside the lock, as evictColdest's does.
861+
func (c *Cache[U]) evictOlderThan(cutoff int64) (dropped int, freed int64) {
862+
var hooks []Coord
863+
c.mu.Lock()
864+
for _, n := range *c.nodes.Load() {
865+
runs := n.load()
866+
var next []*run[U]
867+
for i, r := range runs {
868+
if !r.resident || r.pinned || c.effEpoch(r) >= cutoff {
869+
continue
870+
}
871+
if next == nil {
872+
next = append([]*run[U](nil), runs...)
873+
}
874+
next[i] = hollow(r)
875+
freed += r.bytes
876+
dropped++
877+
hooks = append(hooks, r.coord)
878+
}
879+
if next != nil {
880+
n.publish(next)
881+
}
882+
}
883+
hook := c.Evicted
884+
c.mu.Unlock()
885+
886+
if hook != nil {
887+
for _, co := range hooks {
888+
hook(co)
889+
}
890+
}
891+
return dropped, freed
892+
}
893+
858894
func (c *Cache[U]) evictColdest() int64 {
859895
c.mu.Lock()
860896
var victim *run[U]
Lines changed: 169 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,169 @@
1+
package tree
2+
3+
import (
4+
"container/list"
5+
"sync"
6+
"testing"
7+
)
8+
9+
// Q2, MEASURED BEFORE IT IS BUILT. Gluck endorsed epoch buckets over a heap
10+
// for the eviction scan, with the instruction to measure bucket MAINTENANCE
11+
// against the scan it replaces first.
12+
//
13+
// The two costs sit on DIFFERENT PATHS and that is the whole finding:
14+
//
15+
// THE SCAN is on the daemon's standing sweep. A read never evicts
16+
// (charge raises pressure; Sweep lowers it), so its cost is
17+
// paid once per beat by one goroutine.
18+
// THE INDEX would be maintained on TOUCH, which is on the READ PATH,
19+
// where every reader is concurrent and nothing shared is
20+
// written today.
21+
//
22+
// So the comparison is not "R^2 versus log R". It is "one background walk per
23+
// beat versus a shared write per read", and this file measures both.
24+
25+
func buildCache(tb testing.TB, b *Budget, runs int) *Cache[fatUnit] {
26+
c := New[fatUnit](
27+
func(co Coord) ([]fatUnit, error) {
28+
out := make([]fatUnit, 0, co.To-co.From)
29+
for k := co.From + 1; k <= co.To; k++ {
30+
out = append(out, fatUnit{key: k, payload: make([]byte, 1024)})
31+
}
32+
return out, nil
33+
},
34+
b,
35+
func(u fatUnit) int { return len(u.payload) },
36+
func(u fatUnit) uint64 { return u.key },
37+
)
38+
tb.Cleanup(c.Close)
39+
lineage := []Ref{{Node: "p"}}
40+
for i := 0; len(c.runs("p")) < runs; i++ {
41+
lo := uint64(i * runChunk)
42+
if _, err := c.Range(lineage, lo, lo+runChunk); err != nil {
43+
tb.Fatal(err)
44+
}
45+
if i > runs*4 {
46+
tb.Fatalf("fixture: %d runs after %d ranges", len(c.runs("p")), i)
47+
}
48+
}
49+
return c
50+
}
51+
52+
// fatUnit is a 1 KiB unit: the fixture measures RUN COUNT, and a run is cut
53+
// by bytes, so the unit size is what decides how many runs a range builds.
54+
type fatUnit struct {
55+
key uint64
56+
payload []byte
57+
}
58+
59+
// THE SCAN, AT PRODUCTION SIZE. The segment cache holds 32 MiB in runs cut at
60+
// 32 KiB, so R is in the high hundreds to low thousands; the decoded and
61+
// composed tenants add their own. This times a FULL SWEEP -- the compounding
62+
// case, one rescan per dropped run.
63+
func BenchmarkSweepFull(b *testing.B) {
64+
for _, runs := range []int{256, 1024, 4096} {
65+
b.Run(sizeName(runs), func(b *testing.B) {
66+
for i := 0; i < b.N; i++ {
67+
b.StopTimer()
68+
bud := NewBudget(0)
69+
c := buildCache(b, bud, runs)
70+
resident := len(c.runs("p"))
71+
b.StartTimer()
72+
dropped, _ := bud.TrimIdle(0)
73+
b.StopTimer()
74+
if dropped != resident-1 {
75+
b.Fatalf("dropped %d of %d", dropped, resident)
76+
}
77+
c.Close()
78+
b.StartTimer()
79+
}
80+
})
81+
}
82+
}
83+
84+
// ONE EVICTION under pressure, which is what the standing sweep actually does
85+
// per beat when a tenant has just charged past its limit.
86+
func BenchmarkSweepOneEviction(b *testing.B) {
87+
for _, runs := range []int{256, 1024, 4096} {
88+
b.Run(sizeName(runs), func(b *testing.B) {
89+
bud := NewBudget(0)
90+
c := buildCache(b, bud, runs)
91+
resident, _, _ := bud.Stats()
92+
b.ResetTimer()
93+
for i := 0; i < b.N; i++ {
94+
bud.SetLimit(resident - int64(i+1)*1024)
95+
bud.pressure.Store(true)
96+
bud.Sweep()
97+
}
98+
b.StopTimer()
99+
c.Close()
100+
})
101+
}
102+
}
103+
104+
// THE MAINTENANCE, ON THE PATH IT WOULD LIVE ON. An epoch bucket index must be
105+
// updated when a run is TOUCHED, and a touch happens on every warm read. Today
106+
// that is one atomic store, and only when the epoch is stale.
107+
//
108+
// The in-run control is the read itself: both arms do the same Range over the
109+
// same warm cache in the same binary, and the only difference is what the
110+
// touch does.
111+
func BenchmarkTouchShape(b *testing.B) {
112+
bud := NewBudget(0)
113+
c := buildCache(b, bud, 256)
114+
lineage := []Ref{{Node: "p"}}
115+
116+
// ONE UNIT PER READ. A 64-unit Range copies 64 KiB and a mutex
117+
// acquisition disappears inside it -- the mistake that produced a
118+
// retracted -43.8% on this very package (plans/tree-shaped-log.md).
119+
b.Run("today_atomic_epoch", func(b *testing.B) {
120+
b.RunParallel(func(pb *testing.PB) {
121+
var i uint64
122+
for pb.Next() {
123+
k := i % 2000
124+
i++
125+
if _, err := c.Range(lineage, k, k+1); err != nil {
126+
b.Fatal(err)
127+
}
128+
}
129+
})
130+
})
131+
132+
// The bucket arm: the same read, plus the shared write an index needs.
133+
// A container/list under one mutex is the cheapest honest stand-in --
134+
// UIBudget's deleted LRU was exactly this shape, and so is any bucket
135+
// map that must move a run between two buckets atomically.
136+
var mu sync.Mutex
137+
l := list.New()
138+
elems := make([]*list.Element, 256)
139+
for i := range elems {
140+
elems[i] = l.PushFront(i)
141+
}
142+
b.Run("with_bucket_move", func(b *testing.B) {
143+
b.RunParallel(func(pb *testing.PB) {
144+
var i uint64
145+
for pb.Next() {
146+
k := i % 2000
147+
if _, err := c.Range(lineage, k, k+1); err != nil {
148+
b.Fatal(err)
149+
}
150+
mu.Lock()
151+
l.MoveToFront(elems[i%256])
152+
mu.Unlock()
153+
i++
154+
}
155+
})
156+
})
157+
}
158+
159+
func sizeName(n int) string {
160+
switch n {
161+
case 256:
162+
return "R256"
163+
case 1024:
164+
return "R1024"
165+
case 4096:
166+
return "R4096"
167+
}
168+
return "R?"
169+
}

‎internal/store/tree/evict_scale_test.go‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -66,9 +66,12 @@ func TestEvictionVisitsEveryResidentRun(t *testing.T) {
6666
}
6767
}
6868

69-
// The sweep is the compounding case: TrimIdle re-scans from the top for EVERY
70-
// run it drops. Recorded as a number rather than an assertion about the cure.
71-
func TestSweepRescansPerDroppedRun(t *testing.T) {
69+
// THE SWEEP VISITS EACH RUN ONCE. It used to re-scan from the top for every
70+
// run it dropped -- O(R^2), 180 ms at R=4096 -- because it asked "who is
71+
// coldest" once per victim when the cutoff was already fixed. One pass per
72+
// owner is the whole cure; no index, no data structure, nothing on the read
73+
// path. evict_cost_bench_test.go carries the wall-clock pair.
74+
func TestAFullSweepVisitsEachRunOnce(t *testing.T) {
7275
const runs = 64
7376
var calls int
7477
var mu sync.Mutex
@@ -94,6 +97,10 @@ func TestSweepRescansPerDroppedRun(t *testing.T) {
9497
if dropped != before-1 {
9598
t.Fatalf("swept %d runs, want %d", dropped, before-1)
9699
}
97-
t.Logf("R=%d: a full sweep dropped %d runs in %d run visits (~R^2/2 = %d)",
100+
t.Logf("R=%d: a full sweep dropped %d runs in %d run visits (the old scan: ~R^2/2 = %d)",
98101
before, dropped, visits.Load(), before*before/2)
102+
if visits.Load() > int64(before) {
103+
t.Fatalf("R=%d: %d run visits for one sweep; the pass is scanning more than once per run",
104+
before, visits.Load())
105+
}
99106
}

‎internal/store/tree/tree.go‎

Lines changed: 17 additions & 13 deletions
Original file line numberDiff line numberDiff line change
@@ -65,6 +65,10 @@ type owner interface {
6565
coldest() (int64, bool)
6666
// evictColdest hollows that run and returns the bytes freed.
6767
evictColdest() int64
68+
// evictOlderThan hollows EVERY evictable run older than cutoff, in one
69+
// pass. The idle sweep knows its predicate before it starts, so it does
70+
// not have to ask "who is coldest" once per victim.
71+
evictOlderThan(cutoff int64) (dropped int, freed int64)
6872
}
6973

7074
// NewBudget bounds its caches to limit bytes; 0 is unbounded.
@@ -179,26 +183,26 @@ func (b *Budget) EpochNow() int64 {
179183
// TrimIdle evicts every unpinned run whose epoch is older than the
180184
// current epoch minus keep, then advances the epoch: the idle sweep,
181185
// generalized. Returns runs dropped and bytes freed.
186+
//
187+
// ONE PASS PER OWNER. The cutoff is fixed before the sweep starts, so
188+
// "who is coldest" is not a question this loop has to ask -- it used to
189+
// ask it once per victim and rescan every run to answer, which made a
190+
// full sweep O(R^2): 180 ms at R=4096, measured, against 1 ms for the
191+
// same work in one pass. See evict_cost_bench_test.go.
182192
func (b *Budget) TrimIdle(keep int64) (dropped int, freed int64) {
183193
if b == nil {
184194
return 0, 0
185195
}
186196
cutoff := b.epoch.Add(1) - 1 - keep
187197
for _, o := range *b.owners.Load() {
188-
for {
189-
e, ok := o.coldest()
190-
if !ok || e >= cutoff {
191-
break
192-
}
193-
f := o.evictColdest()
194-
if f <= 0 {
195-
break
196-
}
197-
b.bytes.Add(-f)
198-
b.evictions.Add(1)
199-
dropped++
200-
freed += f
198+
d, f := o.evictOlderThan(cutoff)
199+
if d == 0 {
200+
continue
201201
}
202+
b.bytes.Add(-f)
203+
b.evictions.Add(int64(d))
204+
dropped += d
205+
freed += f
202206
}
203207
return dropped, freed
204208
}

‎plans/tree-shaped-log.md‎

Lines changed: 73 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1678,4 +1678,76 @@ figure moved 1.18x -> 1.07x when it went.
16781678
It survived: x1.13 across three runs, with cf3fc17d stable to a hundred bytes.
16791679
But it survived by luck rather than by method, and the method is now: heap
16801680
deltas are run three times and the spread is reported, exactly as timings are.
1681-
reports the deterministic quantity (allocations, counts) ahead of the timing.
1681+
reports the deterministic quantity (allocations, counts) ahead of the timing.
1682+
# Q2 ANSWERED WITHOUT THE STRUCTURE: THE SWEEP WAS QUADRATIC BY QUESTION, NOT
1683+
# BY SHAPE (dfdfae6a, 2026-08-19)
1684+
1685+
Gluck endorsed epoch buckets over a heap for the eviction scan, with the
1686+
standing instruction to MEASURE BUCKET MAINTENANCE AGAINST THE SCAN IT
1687+
REPLACES BEFORE BUILDING. Measured. The structure is not needed, and the
1688+
measurement says why.
1689+
1690+
## THE TWO COSTS SIT ON DIFFERENT PATHS, WHICH IS THE WHOLE FINDING
1691+
1692+
THE SCAN is on the daemon's standing sweep. A read never evicts, so
1693+
the scan is paid by ONE goroutine, once per beat.
1694+
THE INDEX would be maintained on TOUCH, and a touch is on the READ
1695+
PATH, where every reader is concurrent and nothing shared is
1696+
written today.
1697+
1698+
So the trade was never "R^2 versus log R". It was ONE BACKGROUND WALK PER BEAT
1699+
AGAINST A SHARED WRITE PER READ.
1700+
1701+
## WHAT EACH COSTS, MEASURED (evict_cost_bench_test.go, 5800X, bench lock held)
1702+
1703+
ONE EVICTION under pressure -- what a beat actually does:
1704+
R=256 147 ns R=1024 534 ns R=4096 1.77 us
1705+
1706+
THE FULL IDLE SWEEP, before:
1707+
R=256 967 us R=1024 10.65 ms R=4096 180 ms
1708+
1709+
THE MAINTENANCE, on a ONE-UNIT read at 16 readers (a 64-unit Range copies
1710+
64 KiB and a mutex disappears inside it -- the fixture error that produced
1711+
this file's retracted -43.8%):
1712+
today, lock-free 7.1 - 33.5 ns/op
1713+
with one bucket move 105 - 119 ns/op
1714+
1715+
THE INDEX WOULD MAKE A WARM READ 4-15x MORE EXPENSIVE AND SERIALIZE EVERY
1716+
READER, to save a walk that costs microseconds per beat.
1717+
1718+
The bucket arm is also TIGHTER than the lock-free one, which is the giveaway
1719+
rather than a virtue: everything queues behind the same mutex, so the variance
1720+
that comes from real parallelism disappears with the parallelism.
1721+
1722+
## AND THE QUADRATIC DIED WITHOUT ANY OF IT
1723+
1724+
`TrimIdle` asked "who is COLDEST?" once per victim -- a full rescan of every
1725+
run to answer -- when the cutoff was FIXED BEFORE THE SWEEP STARTED. The
1726+
predicate was known and the loop kept re-deriving it.
1727+
1728+
`evictOlderThan(cutoff)` is one pass per owner, publishing each node's
1729+
successor once:
1730+
1731+
FULL SWEEP before after speedup
1732+
R=256 967 us 21.8 us 44x
1733+
R=1024 10.65 ms 77.1 us 138x
1734+
R=4096 180 ms 294 us 612x
1735+
1736+
and the counting instrument agrees with the clock: a sweep at R=128 now drops
1737+
127 runs in 128 RUN VISITS, against ~R^2/2 = 8192 before. Scaling 256 -> 4096
1738+
(16x R) costs 13.5x, which is linear.
1739+
1740+
NO DATA STRUCTURE WAS ADDED. Nothing was written on the read path. The
1741+
fix is the DELETION of a question the loop did not have to ask.
1742+
1743+
## THE BENCHMARK CAUGHT MY OWN QUADRATIC FIRST
1744+
1745+
The first `evictOlderThan` searched the successor slice for each victim's
1746+
index -- O(R^2) inside the very pass that was replacing an O(R^2) loop. The
1747+
wall clock said R=4096 cost 109x what R=256 did where linear predicts 16x, and
1748+
that ratio is the only reason it was found: the counting instrument was
1749+
satisfied, the suite was green, and the sweep was still 31x faster than
1750+
before, which is exactly the size of improvement that stops an investigation.
1751+
1752+
A SPEEDUP IS NOT A PROOF OF THE SHAPE YOU INTENDED. Check the exponent,
1753+
not the ratio.

0 commit comments

Comments
 (0)