perf(kernel): persistent worker pool instead of threads per batch (6% faster)

[springfall2008]

Stacked on #4540 — base is perf/kernel-batch-abi, so this diff shows only the pool work. Retarget to main once #4540 merges.

What

pk_run_batch built and joined a fresh std::vector<std::thread> on every call. It now dispatches to a pool whose workers park on a condition variable for the life of the process.

Why

#4540 made planning 35% faster but enabling kernel threading on top bought only 1.6%. Instrumenting pk_run_batch over one benchmark plan showed why:

	serial	threaded (16)
calls / jobs	25,091 / 253,899	same
std::threads created	0	201,545
wall time in pk_run_batch	3131 ms	2409 ms
— of which thread creation	—	1507 ms

The parallel work compresses well — 3131ms of simulation becomes 902ms across ~8 lanes — but 1507ms of thread creation ate almost all of it. The median batch is 6 jobs, so each call was paying ~8 thread creations at 7.5µs to cover ~180µs of work.

Result

20-scenario benchmark, interleaved, best of 3, all bit-identical:

threads	normal	with each job 8× dearer
0 (serial)	26.33s	48.92s
4	24.89s	32.03s
6	24.71s	29.98s
8	24.91s	29.85s
16	25.04s	28.94s

6.1% on top of #4540, and more on hardware where the kernel dominates.

End to end, against current main

With #4540 underneath it, interleaved best of 3, each side on its own Python and kernel, all 20 scenarios identical:

20-scenario suite	main	both PRs
serial (threads: 0)	41.70s	26.26s	37.0% faster
default (threads: auto)	39.53s	24.82s	37.2% faster

This PR is the second 5.5% of that: 26.26s serial to 24.82s at auto on the same tree. For comparison main's process pool bought it 41.70s to 39.53s, 5.2% — so the two parallelism mechanisms are worth about the same, and the pool is the cheaper of them, with no fork, no pickling and no PRED_GLOBAL.

Two things measurement changed

Broadcasting was the wrong wake strategy. Built with a single notify_all, the pool was worth 1.3% — a six-job batch woke all fifteen workers and eleven went straight back to sleep, ~250,000 wasted wakeups per plan. Each worker now has its own flag and condvar so only the needed lanes wake: 1.3% → 3.7% on that change alone. The caller takes lane 0 itself, so a six-job batch wakes five.

A threads: auto cap was measured and rejected. The fast-machine curve peaks below the core count, which argues for capping. But with the kernel made 8× dearer the curve stops turning over entirely, and capping at 4 would cost 10.7% there against 1.3% for no cap at all. auto stays at the core count; resolve_batch_threads carries the numbers.

Also included: one scratch per thread rather than one per call. That is a no-op on its own (26.291s vs 26.344s, inside noise — the allocator never leaves its thread cache) and is kept only because the parked workers need it.

Risk

A live thread pool inside a ctypes-loaded library. Handled explicitly:

• fork — threads don't survive it, so a child would inherit a pool whose workers don't exist and hang. A pthread_atfork child handler drops the pool; the next batch builds a fresh one.
• concurrent callers — ctypes releases the GIL, so two Python threads could dispatch at once. A dispatch mutex serialises them.
• thread creation failure — caught; the batch clamps to the lanes that exist and falls back to inline.
• shutdown — the pool is deliberately never destroyed, so no static destructor runs while CPython tears down.

No ABI or parity bump — results are unchanged, so an older binary still loads and runs correctly, just without the speedup. All six architecture binaries rebuilt.

Testing

Full suite green (211 tests), 20 scenarios byte-identical, pre-commit clean. New tests cover worker reuse, lane allocation across 7 thread/job combinations, cross-context scratch reuse, concurrent dispatch, and a batch inside a forked child.

Every new test was mutation-checked, and two were not good enough first time:

• Removing the pthread_atfork handler makes the forked child deadlock; the test kills it on a deadline and reports the timeout rather than hanging the suite.
• The concurrency test initially passed with the dispatch mutex deleted — at 25 rounds over 4 lanes the GIL keeps the callers from overlapping often enough. At 400 rounds over 8 lanes it deadlocks within seconds. The round count is load-bearing and the test says so.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Introduces a persistent worker pool inside the C++ prediction kernel so pk_run_batch no longer pays std::thread creation/join overhead per batch call, improving threaded-batch performance while keeping results bit-identical.

Changes:

• Replace per-call thread creation in pk_run_batch with a process-lifetime PkThreadPool and per-thread scratch reuse.
• Add fork/concurrent-caller safety handling (atfork handler + dispatch mutex) and expose kernel counters for test verification.
• Add/extend tests to validate worker reuse, lane allocation, cross-context scratch reuse, concurrent dispatch, and fork behavior; factor thread-count resolution into resolve_batch_threads.

Reviewed changes

Copilot reviewed 6 out of 12 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/superpowers/specs/2026-08-16-kernel-thread-pool-design.md	Design/spec documenting pool + scratch reuse rationale, risks, and measurement results.
apps/predbat/prediction_kernel.cpp	Implements thread-local scratch, persistent worker pool, atfork handling, dispatch serialization, and test hooks.
apps/predbat/plan.py	Adds resolve_batch_threads() and routes threads setting through it.
apps/predbat/tests/test_kernel_parity.py	Adds kernel pool/scratch/concurrency/fork regression tests.
apps/predbat/tests/test_prediction_batch.py	Adds unit test for threads → kernel lane resolution behavior.
.cspell/custom-dictionary-workspace.txt	Adds new low-level terms used in code/tests (e.g., atfork/pthread/waitpid).

Suppressed comments (1)

apps/predbat/prediction_kernel.cpp:1486

• The pthread_atfork handler functions are internal callbacks, but they’re currently defined with external linkage, which can unnecessarily expand the library’s exported symbol surface. Mark them static (or move them into an anonymous namespace) so they remain private implementation details.

void pk_atfork_prepare()
{
    g_dispatch_mutex.lock();
}

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.