Summary
Node 11.1.1 resolved the memory regression reported in #3669, but full mainnet sync
still uses noticeably more memory than the 11.0.1 baseline:
- Peak RSS +37% (15.43 -> 21.16 GiB)
- Peak GC live bytes +55% (10.28 -> 15.94 GiB)
This is not the same fault as #3669. That one was an oscillation - heap swinging between
19 and 36 GiB because LedgerDB garbage collection was gated on snapshotting, fixed by
ouroboros-consensus#2227.
Node 11.1.1 no longer oscillates; it sits on a higher plateau than 11.0.1 did, which
suggests a different cause rather than an incomplete fix.
Under investigation. It is not yet clear whether this is expected for the V2 LedgerDB
backend or a genuine second regression, and there is no consensus fix for it yet.
Full report, including per-era timings and graphs:
https://tests.cardano.intersectmbo.org/test_results/sync_reports/mainnet_11_1_1.html
Measurements
Full mainnet syncs from genesis on the same host (32 cores, 134 GB RAM, V2InMemory
LedgerDB backend, ghc-9.6). Memory is parsed from the node's own Resources: trace
lines, so GC live bytes is what the collector could not reclaim, as distinct from heap
the RTS happens to be holding.
|
11.0.1 |
11.1.1 |
change |
| Peak RSS |
15.43 GiB |
21.16 GiB |
+37% |
| Mean RSS |
12.95 GiB |
15.67 GiB |
+21% |
| Peak GC live bytes |
10.28 GiB |
15.94 GiB |
+55% |
| Mean GC live bytes |
6.03 GiB |
6.86 GiB |
+14% |
| Major GCs |
1,896 |
1,940 |
+2% |
| Total allocation |
336 TB |
286 TB |
-15% |
| Sync time |
133,009 s |
110,606 s |
-16.8% |
| Mean CPU load |
152% |
157% |
+3.3% |
Node revision under test: c2ebdc87dfe07706a83e52f219e712c60d1b0a56.
Speed is not affected and in fact improved by 6 h 13 min. Allocation went down while
retention went up, so this is not extra garbage being produced. Major GC count is normal,
so the collector is running as often as it did on 11.0.1 - it simply has more it cannot
free.
The heap profile is healthy, just higher
- 11.0.1 - monotone staircase to a 15.34 GiB plateau reached at slot 134 M. 21 heap
decreases across 130,674 samples, nothing above 20 GiB.
- 11.1.1 - also a staircase, plateauing near 19-20 GiB. 90 heap decreases, 3.5% of
samples above 20 GiB, none above 25 GiB.
For contrast, 11.1.0 had 204 heap decreases with 34.7% of samples above 20 GiB and 6.0%
above 30 GiB. The sawtooth is gone.
It is not chain growth
The 11.1.1 run covered 8.2 M more slots and 2.8 GB more chain than the 11.0.1 run, which
is far too little to explain the gap:
- The divergence starts around slot 40 M and widens monotonically from there.
- Truncating all runs to a common window of slots <= 188 M leaves the peaks unchanged:
15.43 GiB for 11.0.1 against 21.16 GiB for 11.1.1.
- Node 11.1.1 crosses the 15.34 GiB level that 11.0.1 uses as its ceiling at slot 71 M,
roughly halfway through Alonzo, with more than half the chain still to sync.
Peak RSS by slot band, GiB:
| Slot band |
11.0.1 |
11.1.1 |
| 0-10 M |
2.24 |
2.13 |
| 20-30 M |
4.09 |
4.84 |
| 40-50 M |
8.26 |
9.69 |
| 60-70 M |
11.49 |
15.33 |
| 80-90 M |
13.55 |
18.12 |
| 100-110 M |
14.15 |
19.01 |
| 120-130 M |
15.43 |
21.16 |
| 150-160 M |
15.43 |
20.36 |
| 180-190 M |
15.43 |
20.36 |
Early sync is at parity - 11.1.1 is actually marginally lower through the Byron era - and
the two curves separate from Alonzo onward.
Why it matters
A 21 GiB peak is comfortable on the 134 GB test host, and no OOM occurred. It is less
comfortable for operators on 16-32 GB machines, which is the constituency the 11.0.1
figure of 15.4 GiB was already close to.
Summary
Node 11.1.1 resolved the memory regression reported in #3669, but full mainnet sync
still uses noticeably more memory than the 11.0.1 baseline:
This is not the same fault as #3669. That one was an oscillation - heap swinging between
19 and 36 GiB because LedgerDB garbage collection was gated on snapshotting, fixed by
ouroboros-consensus#2227.
Node 11.1.1 no longer oscillates; it sits on a higher plateau than 11.0.1 did, which
suggests a different cause rather than an incomplete fix.
Under investigation. It is not yet clear whether this is expected for the V2 LedgerDB
backend or a genuine second regression, and there is no consensus fix for it yet.
Full report, including per-era timings and graphs:
https://tests.cardano.intersectmbo.org/test_results/sync_reports/mainnet_11_1_1.html
Measurements
Full mainnet syncs from genesis on the same host (32 cores, 134 GB RAM,
V2InMemoryLedgerDB backend, ghc-9.6). Memory is parsed from the node's own
Resources:tracelines, so
GC live bytesis what the collector could not reclaim, as distinct from heapthe RTS happens to be holding.
Node revision under test:
c2ebdc87dfe07706a83e52f219e712c60d1b0a56.Speed is not affected and in fact improved by 6 h 13 min. Allocation went down while
retention went up, so this is not extra garbage being produced. Major GC count is normal,
so the collector is running as often as it did on 11.0.1 - it simply has more it cannot
free.
The heap profile is healthy, just higher
decreases across 130,674 samples, nothing above 20 GiB.
samples above 20 GiB, none above 25 GiB.
For contrast, 11.1.0 had 204 heap decreases with 34.7% of samples above 20 GiB and 6.0%
above 30 GiB. The sawtooth is gone.
It is not chain growth
The 11.1.1 run covered 8.2 M more slots and 2.8 GB more chain than the 11.0.1 run, which
is far too little to explain the gap:
15.43 GiB for 11.0.1 against 21.16 GiB for 11.1.1.
roughly halfway through Alonzo, with more than half the chain still to sync.
Peak RSS by slot band, GiB:
Early sync is at parity - 11.1.1 is actually marginally lower through the Byron era - and
the two curves separate from Alonzo onward.
Why it matters
A 21 GiB peak is comfortable on the 134 GB test host, and no OOM occurred. It is less
comfortable for operators on 16-32 GB machines, which is the constituency the 11.0.1
figure of 15.4 GiB was already close to.