11---
2- status : in-progress
2+ status : done
33depends : []
44specs : []
55issues :
66 - 132
7- pr :
7+ pr : 134
88---
99
1010# Plan: fix boot OOM — load sheets sequentially, not concurrently
@@ -38,10 +38,12 @@ returned state shape/order.
3838
3939## Validation
4040
41- - [ ] Live pod boots and reaches ` /api/health/ready ` 200 on the standard node
42- size (heap measured ~ 0.48 GB retained via temporary boot instrumentation).
43- - [ ] ` npm run -w apps/api type-check && npm run lint ` clean; loader/store tests pass.
44- - [ ] Deployed ` :sandbox ` image boots without OOM.
41+ - [x] Live pod boots and reaches ` /api/health/ready ` 200 on the standard node
42+ size, 0 restarts, ~ 0.48 GB retained (per-phase boot instrumentation).
43+ - [x] ` npm run -w apps/api type-check && npm run lint ` clean; loader/store/memory
44+ tests pass (12/12).
45+ - [x] Deployed ` :sandbox ` image boots without OOM on the normal config
46+ (heap 2048 / limit 2560Mi).
4547
4648## Risks
4749
@@ -50,4 +52,24 @@ returned state shape/order.
5052
5153## Notes
5254
55+ - ** The real bug was concurrency, not memory size or data volume.** Per-phase
56+ heap logging in the live pod showed reconcile + push-daemon + private-store
57+ load all sat at ~ 60 MB; the entire balloon was the concurrent ` Promise.all `
58+ read in ` loadInMemoryState ` . Retained state is ~ 0.48 GB.
59+ - The diagnosis ruled out several earlier hypotheses: public people count (the
60+ prune didn't change the crash), the private store (11 MB), and reconcile/
61+ push-daemon git-object caching (flat at boot). The gitsheets read path * is*
62+ memory-heavy per read, but the OOM was specifically the eleven reads peaking
63+ together.
64+ - ** Sibling changes from the same investigation:** #131 (heap 1536→2048,
65+ limit 2Gi→2.5Gi) was a mitigation, now redundant but harmless headroom — left
66+ as-is. #133 (spam prune) shipped to production ` published ` (31,832→18,203
67+ people); independently worthwhile but not the OOM fix.
68+
5369## Follow-ups
70+
71+ - ** Tracked as #132 ** — broader heap-footprint work (the gitsheets per-read
72+ cost and ~ 2× retention of raw TOML + parsed records). Optional now that boot
73+ fits comfortably, but the lever if the dataset grows.
74+ - ** Deferred (ops):** the throwaway ` ghcr.io/codeforphilly/codeforphilly-ng:heapdiag `
75+ image tag was pushed during diagnosis; harmless but can be deleted from GHCR.
0 commit comments