Skip to content

Commit 3331335

Browse files
perf(bench): add tail-latency percentiles and optional jemalloc/tcmalloc baselines (#116)
## Summary Implements **ROADMAP item 9.4** (issue #111), the last Milestone-9 item, decided in [ADR-0045](docs/adr/0045-benchmark-percentiles-and-external-baselines.md) (**extends** [ADR-0014](docs/adr/0014-microbenchmark-methodology-pool-vs-malloc.md), does not supersede it). It closes the two residual spec-review (#105 §6.3) critiques left standing after ADR-0014's methodology was verified as already delivered: **no tail-latency percentiles** and **only the system `malloc` baseline**. Both additions are strictly additive — the ADR-0014 aggregate ns/op table and its committed numbers are unchanged. ## Tail-latency percentiles (`--percentiles`) An opt-in mode that times the work **per operation** (one sample per op, vs the aggregate path's one per repeat) and emits a **separate** TSV table — `p50 / p90 / p99 / p999 + samples`. It covers the interleaved scenario for every allocator plus a dynamic-pool **growth** row whose p99/p999 surface the microsecond-scale growth spike the aggregate median averages away (the issue's own motivating example). Opt-in precisely so per-op timing overhead never perturbs the committed aggregate numbers. **Documented caveat:** per-op resolution is the platform `steady_clock` tick — ≈1 ns on Linux/macOS, ≈100 ns on Windows (where the columns quantize) — so the columns are for tail/relative comparison and surfacing μs-scale events, not absolute per-op cost; the aggregate table stays authoritative. ## Optional jemalloc / tcmalloc baselines CMake-**feature-detected** (`find_library` + `find_path`): when present, each appears as an extra comparison row in the aggregate and percentile tables, driven through `mallocx`/`dallocx` and `tc_malloc`/`tc_free` so they don't override the system `malloc` row. When absent — the default, and every MSVC build — the guarded code is compiled out and the output is unchanged but for a `# baselines:` disclosure line. Spec §3.3's zero-external-dependency posture is preserved. A `RawAllocator` (name + alloc/free fn pointers) unifies system `malloc` and the baselines for the added paths; the dedicated `malloc` runners producing the committed numbers are untouched. ## Validation - Built + ran the bench under MSVC (`bench` preset): default output schema intact (only a `# baselines: malloc` line added), `--percentiles` table works, and the dynamic-pool growth `p999` clearly shows the growth spike (≈7 µs) even through the 100 ns Windows clock quantization. - `clang-format` clean; `clang-tidy` clean on the changed file (incl. tidying two pre-existing `#if defined(__clang__)` → `#ifdef`). - `consistency_lint.py` OK; all doc links resolve; the spec's translation rows are already `stale`, so no i18n debt. - The jemalloc/tcmalloc paths (not installable on the maintainer's MSVC box) are exercised by a new Linux **`bench-baselines`** CI cell that installs both allocators, asserts they were feature-detected, and asserts the baseline rows + percentile table are present — gating on exit code 0, not numbers (ADR-0014 §8). ## Docs New ADR-0045 (+ index row) with a forward-reference note added to ADR-0014; spec §6.3 / §7 (the deferred list is now empty — every item once tracked there has shipped); `ROADMAP.md` (9.4 → done, closing Milestone 9); `CHANGELOG.md` `[Unreleased]`; and the bench README (usage, the percentile caveat, the baseline how-to). Per the established sequencing, the `README.md` performance-section refresh is deferred to the `v1.2.0` release PR to keep this PR off the translated docs surface. Closes #111. 🤖 Generated with [Claude Code](https://claude.com/claude-code) --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent 91883cb commit 3331335

10 files changed

Lines changed: 354 additions & 13 deletions

File tree

.github/workflows/ci.yml

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -459,6 +459,56 @@ jobs:
459459
"$bin" --iterations 10000 --repeats 3
460460
echo "OK — bench binary ran to completion."
461461
462+
# ---------------------------------------------------------------------------
463+
# ROADMAP §9.4 — external-allocator baselines + tail-latency percentiles
464+
# (ADR-0045). External baselines are measured the safe way: re-run the SAME
465+
# bench under LD_PRELOAD, which swaps the whole process allocator (jemalloc /
466+
# tcmalloc take over global malloc on load, so they cannot be linked or dlopen'd
467+
# beside the system allocator). Exercises the opt-in --percentiles table too.
468+
# Linux only; never touches the MSVC leg. Asserts exit code 0, not numbers.
469+
# ---------------------------------------------------------------------------
470+
bench-baselines:
471+
name: bench / external baselines + percentiles
472+
runs-on: ubuntu-24.04
473+
steps:
474+
- name: Check out
475+
uses: actions/checkout@v6
476+
477+
- name: Install CMake and Ninja
478+
uses: lukka/get-cmake@latest
479+
480+
- name: Install jemalloc and tcmalloc runtime libraries
481+
shell: bash
482+
run: |
483+
sudo apt-get update
484+
sudo apt-get install -y libjemalloc2 libtcmalloc-minimal4t64
485+
486+
- name: Configure and build (bench preset)
487+
shell: bash
488+
run: |
489+
export CC=gcc CXX=g++
490+
cmake --preset bench
491+
cmake --build --preset bench
492+
493+
- name: Baseline via LD_PRELOAD + the percentile table
494+
shell: bash
495+
run: |
496+
set -euo pipefail
497+
bin="build/bench/src/bench/cpp/it/d4np/memorypool/pool_vs_malloc_bench"
498+
# LD_PRELOAD resolves a soname through the loader cache (ldconfig ran on
499+
# install), so no filesystem search is needed.
500+
run() { # label, LD_PRELOAD value ("" = none), expected "# allocator:" text
501+
echo "=== allocator: $1 ==="
502+
out="$(LD_PRELOAD="$2" "$bin" --scenario all --iterations 20000 --repeats 3 --percentiles)"
503+
echo "$out"
504+
echo "$out" | grep -q "# allocator: $3" || { echo "FAIL: allocator disclosure ($1)"; exit 1; }
505+
echo "$out" | grep -q "p99_ns/op" || { echo "FAIL: percentile table missing ($1)"; exit 1; }
506+
}
507+
run "system malloc" "" "system malloc"
508+
run "jemalloc" "libjemalloc.so.2" "libjemalloc.so.2"
509+
run "tcmalloc" "libtcmalloc_minimal.so.4" "libtcmalloc_minimal.so.4"
510+
echo "OK — pool benchmarked against system malloc, jemalloc, and tcmalloc; percentile table present."
511+
462512
# ---------------------------------------------------------------------------
463513
# thread-safety — build and run the existing (single-threaded) test suite
464514
# under each non-default PBR_MEMORY_POOL_THREAD_SAFETY policy (ADR-0020 /

CHANGELOG.md

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -20,6 +20,19 @@ dated version block (`## [X.Y.Z] — YYYY-MM-DD`) when a release PR closes a mil
2020

2121
### Added
2222

23+
- **Benchmark extension — tail-latency percentiles and optional jemalloc/tcmalloc baselines.**
24+
The pool-vs-malloc microbenchmark gains an opt-in `--percentiles` mode that emits a separate
25+
per-operation **p50/p90/p99/p999** table — the dynamic-pool growth row surfaces the
26+
microsecond-scale growth spike the aggregate median hides — and **jemalloc** / **tcmalloc**
27+
baselines measured the safe way, by re-running the bench under **`LD_PRELOAD`** (those
28+
allocators take over global `malloc` on load, so they cannot be linked or `dlopen`'d beside
29+
the system allocator without crashing). A `# allocator:` header line discloses which allocator
30+
each run measured; the bench carries no allocator-specific code, so the default build stays
31+
zero-external-dependency (spec §3.3). Both are strictly additive: the
32+
[ADR-0014](docs/adr/0014-microbenchmark-methodology-pool-vs-malloc.md) aggregate ns/op table
33+
and its committed numbers are unchanged. A Linux `bench-baselines` CI cell installs both
34+
allocators and exercises the new paths (non-asserting on numbers, per ADR-0014 §8).
35+
[ADR-0045](docs/adr/0045-benchmark-percentiles-and-external-baselines.md). Closes #111.
2336
- **Coverage-guided fuzzing harness for the pool surface.** A new
2437
[`pool_fuzz.cpp`](src/test/cpp/it/d4np/memorypool/pool_fuzz.cpp) drives randomized
2538
`alloc` / `free(valid)` / `free(NULL)` / `free(foreign)` sequences — over both fixed and

ROADMAP.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -139,7 +139,7 @@ Goal: a coherent, post-`v1.0.0` wave of **additive, ABI-compatible** capabilitie
139139
- [x] 9.1 **`std::pmr::memory_resource` adapter** (`PoolMemoryResource`) — the "door left open" in [ADR-0018](docs/adr/0018-stl-allocator-adapter.md): a `std::pmr::memory_resource` subclass binding one `Pool` so any `std::pmr`-aware container can draw from it through `std::pmr::polymorphic_allocator`, without the `PoolAllocator<T>` per-type rebind. Deterministic `(bytes, alignment)` routing to the bound pool — over-sized / over-aligned requests delegate to a configurable upstream resource, and exhaustion of a pool-eligible request throws `std::bad_alloc` rather than falling back (preserving the deterministic deallocate routing) — with `is_equal` by `(pool, upstream)` identity, gated behind `PBR_MEMORY_POOL_HAS_PMR` where `<memory_resource>` is available. Header-only, additive, ABI-compatible. Decided in [ADR-0042](docs/adr/0042-pmr-memory-resource-adapter.md); implemented in [`pool_memory_resource.hpp`](src/main/cpp/it/d4np/memorypool/pool_memory_resource.hpp) with [`pool_memory_resource_test.cpp`](src/test/cpp/it/d4np/memorypool/pool_memory_resource_test.cpp) (issue #107).
140140
- [x] 9.2 **Opt-in debug hardening** — freed-block poisoning, canaries, and free-list safe-linking (which also yields double-free detection); zero cost when the gate is off (issue #109). Decided in [ADR-0043](docs/adr/0043-opt-in-debug-hardening.md); implemented in [`memory_pool.cpp`](src/main/cpp/it/d4np/memorypool/memory_pool.cpp) behind the compile-time `PBR_MEMORY_POOL_HARDENING` knob (a `harden` CMake preset), with the swappable violation-handler surface in [`pool_hardening.hpp`](src/main/cpp/it/d4np/memorypool/pool_hardening.hpp) and [`pool_hardening_test.cpp`](src/test/cpp/it/d4np/memorypool/pool_hardening_test.cpp) (CTest `pool_hardening`). The "canary" is realized as one trailing **guard word** living in *added* slot stride — so the user-visible `block_size` and the ADR-0009 alignment guarantee are unchanged and the default build is byte-for-byte unchanged (the mechanism is fully compiled out). Poisoning (`0xDE`) catches use-after-free on the next allocation, the guard word catches a write past `block_size` and a double-free, and glibc-style safe-linking (`ptr XOR (slot_addr >> 12)`) protects the in-band next-link. A `harden` CI matrix cell builds and tests the hardened configuration on each Tier-1 platform. Works with fixed and dynamic pools across all three thread-safety policies.
141141
- [x] 9.3 **Coverage-guided fuzzing harness** for the pool surface — a libFuzzer target, time-boxed in CI (issue #108). Decided in [ADR-0044](docs/adr/0044-coverage-guided-fuzzing-harness.md); implemented in [`pool_fuzz.cpp`](src/test/cpp/it/d4np/memorypool/pool_fuzz.cpp) as a stateful opcode interpreter with a shadow oracle that asserts the no-alias, canary-intact, foreign/NULL-no-op ([ADR-0012](docs/adr/0012-foreign-pointer-and-out-of-range-pointer-policy.md)) and `InstrumentedPool` accounting invariants across fixed and dynamic pools. One engine-agnostic source yields both the Clang-only libFuzzer target `pool_fuzz` (opt-in `PBR_MEMORY_POOL_BUILD_FUZZERS`, a `fuzz` preset under `-fsanitize=fuzzer,address,undefined`) and the always-built standalone [`pool_fuzz_replay`](src/test/cpp/it/d4np/memorypool/pool_fuzz_corpus/) target that replays the seed corpus as a portable regression gate (CTest `pool_fuzz_replay`) on every platform, including MSVC. A dedicated `fuzz` CI job replays the corpus and fuzzes for a bounded time on every PR; a crash is filed in the bug ledger ([ADR-0039](docs/adr/0039-bug-ledger-and-triage-protocol.md)). Test-only and additive — the release build and the benchmark numbers are untouched.
142-
- [ ] 9.4 **Benchmark extension** — external allocator baselines (jemalloc / tcmalloc) and p99 tail-latency reporting (issue #111).
142+
- [x] 9.4 **Benchmark extension** — external allocator baselines (jemalloc / tcmalloc) and p99 tail-latency reporting (issue #111). Decided in [ADR-0045](docs/adr/0045-benchmark-percentiles-and-external-baselines.md) (extends [ADR-0014](docs/adr/0014-microbenchmark-methodology-pool-vs-malloc.md)); implemented in [`pool_vs_malloc_bench.cpp`](src/bench/cpp/it/d4np/memorypool/pool_vs_malloc_bench.cpp). An opt-in `--percentiles` mode adds a separate per-operation p50/p90/p99/p999 table (the dynamic-pool growth row surfaces the amortized-growth spike the median hides); jemalloc / tcmalloc baselines are measured by re-running the bench under `LD_PRELOAD` (they take over global `malloc` on load, so they cannot be linked or `dlopen`'d beside the system allocator without crashing — an in-process attempt was tried and rejected); a `# allocator:` header discloses which allocator each run measured. The bench carries no allocator-specific code, so the default build keeps zero external dependencies (spec §3.3) and the [ADR-0014](docs/adr/0014-microbenchmark-methodology-pool-vs-malloc.md) aggregate table and its committed numbers are unchanged. A Linux `bench-baselines` CI cell re-runs the bench under each allocator's preload and asserts the disclosure + percentile table (non-asserting on numbers).
143143

144144
---
145145

docs/adr/0014-microbenchmark-methodology-pool-vs-malloc.md

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,11 @@
11
# ADR-0014: Microbenchmark methodology — pool vs. `malloc`
22

3-
- **Status:** Accepted
3+
- **Status:** Accepted (extended by [ADR-0045](0045-benchmark-percentiles-and-external-baselines.md))
44
- **Date:** 2026-06-11
55
- **Deciders:** Daniel Polo (maintainer), Claude (architect agent)
6-
- **Related:** spec §6.3, ROADMAP §2.9, [ADR-0005](0005-toolchain-matrix-and-supported-platforms.md) §3 (compiler matrix), [ADR-0006](0006-code-style-and-static-analysis-baseline.md) (style + lint baseline), [ADR-0009](0009-free-list-layout-block-size-constraints-and-alignment-guarantee.md) §2 (block-size constraints the benchmark must respect), [ADR-0013](0013-doxygen-for-api-markdown-for-narrative.md) (narrative goes in Markdown — bench reports too)
6+
- **Related:** spec §6.3, ROADMAP §2.9, [ADR-0005](0005-toolchain-matrix-and-supported-platforms.md) §3 (compiler matrix), [ADR-0006](0006-code-style-and-static-analysis-baseline.md) (style + lint baseline), [ADR-0009](0009-free-list-layout-block-size-constraints-and-alignment-guarantee.md) §2 (block-size constraints the benchmark must respect), [ADR-0013](0013-doxygen-for-api-markdown-for-narrative.md) (narrative goes in Markdown — bench reports too), [ADR-0045](0045-benchmark-percentiles-and-external-baselines.md) (adds tail-latency percentiles + optional jemalloc/tcmalloc baselines)
7+
8+
> **Extended by [ADR-0045](0045-benchmark-percentiles-and-external-baselines.md)** (2026-07-09): an opt-in `--percentiles` per-operation table (p50/p90/p99/p999) and optional feature-detected jemalloc/tcmalloc baselines. Both are strictly additive — the §4 aggregate statistics, the §6 output contract, and the committed numbers below are unchanged.
79
810
## Context
911

0 commit comments

Comments
 (0)