diff --git a/.cspell/custom-dictionary-workspace.txt b/.cspell/custom-dictionary-workspace.txt index 1f59c0f7f..d5ff0f1ef 100644 --- a/.cspell/custom-dictionary-workspace.txt +++ b/.cspell/custom-dictionary-workspace.txt @@ -25,6 +25,7 @@ armhf armv ASHP asyncio +atfork authoriser autodocstring autoflake @@ -51,6 +52,7 @@ byok cadata calib Cantarell +CDLL cdn Cerbo cexxxx @@ -77,6 +79,7 @@ corruptplans cprofile creds crosscharge +ctypes customauthorizer customisation Customise @@ -378,6 +381,7 @@ preseeded prevs protobuf psum +pthread pvbat pvenergytotal pvlib @@ -447,6 +451,7 @@ slugifies slugify slugifying snakeviz +snprintf socb socketloop socs @@ -472,6 +477,7 @@ stdlib stepline Stromligning strptime +strtod Strømligning subclassing submessage @@ -489,6 +495,7 @@ temperation Teslemetry testname thirdparty +threadpool timea timeb timec @@ -520,6 +527,8 @@ unconfigured undiscounted unnormalised unparseable +unpickled +unpushed unsmoothed unstaged urlsafe @@ -530,10 +539,14 @@ venv Victron visibilitychange waf +waitpid Wallbox weblink welink Werror +WEXITSTATUS +WIFEXITED +WNOHANG workmode writeonly wrongsha diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 501f8ac52..fd751f981 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -88,44 +88,37 @@ The core prediction/planning happens in `Plan` mixin (`apps/predbat/plan.py`) wh - `optimise_detailed_pass()` - Fine-tune charge/export windows - `run_prediction()` - Execute battery simulation via `Prediction` class -### Thread Pool System +### Batched Predictions -**Multiprocessing for predictions** (in `apps/predbat/plan.py`): +There is no process pool and no Python thread pool. Every optimiser fan-out in `plan.py` is +launch-all-then-collect-all, so the launches queue instead of running and the whole fan-out is +handed to the C++ kernel in one call, which spreads it across its own threads with the GIL +released. Python itself is strictly serial. -```python -from multiprocessing import Pool, cpu_count - -# Pool created on demand -if not self.pool: - threads = self.get_arg("threads", "auto") - if threads == "auto": - self.pool = Pool(processes=cpu_count()) - else: - self.pool = Pool(processes=int(threads)) -``` +**How it works** (`apps/predbat/prediction_batch.py`, mixed into `Prediction`): -**How it works**: - -- `self.pool` is a `multiprocessing.Pool` instance (NOT thread pool - uses processes) -- Pool runs prediction scenarios in parallel using `pool.apply_async()` -- Wrapped functions in `prediction.py`: `wrapped_run_prediction_single()`, `wrapped_run_prediction_charge()`, etc. -- Global state shared via `PRED_GLOBAL` dict (copied to child processes) -- Pool configured via `threads` config option: `"auto"` for CPU count or specific number -- Returns `DummyThread` objects when pool disabled (for single-threaded testing) +- `Plan.launch_run_prediction_*()` calls `Prediction.queue_run_prediction_*()`, which builds the + trial inputs, appends a `BatchJob` to `self.pending_batch` and returns a `BatchHandle` +- Nothing has run at that point. The first `BatchHandle.get()` calls `flush_batch()`, which resolves + prediction-cache hits and intra-batch duplicates in Python and sends the remainder through one + `run_prediction_kernel_batch()` / `pk_run_batch` call; every handle in the fan-out is then resolved +- A job the kernel will not take (no kernel, stale binary, debug run) falls back to + `Prediction.run_prediction()`, which is also the direct path `thread_run_prediction_*()` uses +- `threads` config option sets `Prediction.batch_threads`, the kernel's thread count. `"auto"` uses + the CPU count. Results are identical at any thread count - it only trades CPU for planning time **Key pattern**: ```python -# Async call to run prediction in pool -if self.pool and self.pool._state == "RUN": - han = self.pool.apply_async(wrapped_run_prediction_charge, (args...)) - return han -else: - # Fallback to synchronous execution - result = wrapped_run_prediction_charge(args...) - return DummyThread(result) +# Queue the whole fan-out, then read the handles - the first get() runs all of it +handles = [self.launch_run_prediction_charge(soc, window_n, ...) for soc in try_socs] +results = [handle.get() for handle in handles] ``` +**Invariant**: a caller must not mutate any list or window dict it has passed to +`launch_run_prediction_*` until it has read the matching handle, because those inputs are not read +until the batch is flushed. + ### Component System (Critical Pattern) All major features inherit from `ComponentBase` (see `apps/predbat/component_base.py`): @@ -281,8 +274,7 @@ self.log("Error: Failed with: {}".format(e)) Core prediction happens in `apps/predbat/prediction.py`: - `Prediction` class runs battery simulation in 5-min steps -- Multiprocessing via `wrapped_run_prediction_*` functions -- Global state in `PRED_GLOBAL` dict for process sharing +- Batched predictions via `launch_run_prediction_*()` queuing jobs and `BatchHandle.get()` flushing through the C++ kernel - Optimisation in `apps/predbat/plan.py` via `optimise_all_windows()` ### State Management diff --git a/apps/predbat/annual.py b/apps/predbat/annual.py index 1d9c1413c..3e133c482 100644 --- a/apps/predbat/annual.py +++ b/apps/predbat/annual.py @@ -997,14 +997,14 @@ def prepare_sample(predbat, config, weather, tariff, load_source, day, midnight_ """ reset_sample_state(predbat) - # calculate_plan() spins up a multiprocessing Pool sized from args["threads"] (plan.py) - # unless it is exactly 0. The annual tool plans hundreds of individual days per run, each a - # small, fast calculation, so per-day pool creation is both wasted overhead and, on a - # spawn-based multiprocessing start method, unsafe: pool workers read the module-level - # PRED_GLOBAL dict in prediction.py, which is only populated in the parent process. This - # matches create_headless_predbat()'s own choice for a fully offline run; it is forced here - # too so run_day() behaves the same way against a caller-supplied PredBat instance (such as - # the standard unit test fixture) that has not gone through that bootstrap. + # calculate_plan() sizes the C++ kernel's batch thread count from args["threads"] (plan.py); it + # no longer spins up a Python multiprocessing Pool. pk_run_batch creates and joins its own + # std::thread pool fresh on every call, so leaving threads at its 'auto' default would pay that + # creation cost on each of the hundreds of small, fast per-day calculations the annual tool runs + # - overhead a plan this size cannot recoup. This matches create_headless_predbat()'s own choice + # for a fully offline run; it is forced here too so run_day() behaves the same way against a + # caller-supplied PredBat instance (such as the standard unit test fixture) that has not gone + # through that bootstrap. predbat.args["threads"] = 0 predbat.midnight_utc = midnight_utc diff --git a/apps/predbat/hass.py b/apps/predbat/hass.py index cf4044058..58d48479f 100644 --- a/apps/predbat/hass.py +++ b/apps/predbat/hass.py @@ -164,9 +164,7 @@ def log(self, msg, quiet=True): log_size = self.logfile.tell() if log_size > 10000000 and threading.current_thread() is threading.main_thread(): # Only rotate from the main thread to avoid race conditions with - # component threads that also call log() — a Threading.Lock cannot - # be used here because the logfile object must survive pickle for - # the multiprocessing pool. + # component threads that also call log(). for num_logs in range(max_logs - 1, 0, -1): filename = "predbat." + format(num_logs) + ".log" if os.path.isfile(filename): diff --git a/apps/predbat/load_ml_component.py b/apps/predbat/load_ml_component.py index 8b5425b16..5d58c2546 100644 --- a/apps/predbat/load_ml_component.py +++ b/apps/predbat/load_ml_component.py @@ -736,8 +736,10 @@ async def run(self, seconds, first): self.log("ML Component: No training needed, model age is {} hours".format(dp2(retrain_age_seconds / 3600.0))) if should_train or should_fetch: - # Set load_ml_calculating across all NumPy-heavy work (training + predict + save) - # so that the plan's multiprocessing pool is never fork()ed while Numpy threads are active. + # Set load_ml_calculating across all NumPy-heavy work (training + predict + save). This used + # to gate the plan's multiprocessing pool off while NumPy threads were active (fork() is + # unsafe with live threads); the pool is gone, so this is now purely a status flag surfaced + # via is_calculating() and logged by calculate_plan(). if self.base.prediction_started: self.log("ML Component: Waiting for current prediction cycle to complete before running ML work") while self.base.prediction_started: diff --git a/apps/predbat/marginal.py b/apps/predbat/marginal.py index b4a49c8b8..e1f6635dc 100644 --- a/apps/predbat/marginal.py +++ b/apps/predbat/marginal.py @@ -86,8 +86,7 @@ def calculate_marginal_costs(self): if minute in modified_load: modified_load[minute] += extra_per_step - # Create a fresh Prediction with the modified load; this updates PRED_GLOBAL - # which is safe since we run synchronously (pool is idle at this point) + # Create a fresh Prediction with the modified load. # No need to include 10% extra load as we only run normal simulations. pred = Prediction(self, self.pv_forecast_minute_step, self.pv_forecast_minute_step, modified_load, modified_load) diff --git a/apps/predbat/plan.py b/apps/predbat/plan.py index 187e4ebab..2c8c40ab6 100644 --- a/apps/predbat/plan.py +++ b/apps/predbat/plan.py @@ -13,18 +13,18 @@ Implements the search algorithm that finds optimal charge and discharge windows by exploring combinations of price thresholds, window sizes, and SoC targets. -Uses multi-threaded prediction runs to evaluate thousands of scenarios and select -the plan that minimises the overall cost metric. +The search itself is serial Python: each fan-out queues its scenarios through +launch_run_prediction_* and the first handle read flushes them all through one +call to the C++ prediction kernel, which is where the threading now lives. """ import copy -import traceback from datetime import datetime, timedelta -from multiprocessing import Pool, cpu_count +from multiprocessing import cpu_count from const import PREDICT_STEP, PV_SCENARIO_NOMINAL, PV_SCENARIO_PV10, PV_SCENARIO_PV90, TIME_FORMAT, MINUTE_WATT from utils import calc_percent_limit, dp0, dp1, dp2, dp3, dp4, remove_intersecting_windows, in_car_slot -from prediction import Prediction, wrapped_run_prediction_single, wrapped_run_prediction_charge, wrapped_run_prediction_charge_min_max, wrapped_run_prediction_export +from prediction import Prediction from prediction_kernel import kernel_status_summary from predbat_metrics import metrics import time @@ -35,6 +35,26 @@ PLAN_PASS_WINDOW_BUDGET = 8 +def resolve_batch_threads(threads, cpu_count_value): + """Map the threads setting onto how many kernel lanes one batch may use. + + 'auto' takes the core count and is deliberately not capped. On a fast machine the curve is very + flat and peaks slightly below the core count - measured on the 20-scenario benchmark, best of 3: + serial 26.33s, 4 threads 24.89s, 6 threads 24.71s, 8 threads 24.91s, 16 threads 25.04s - so a cap + looks attractive. But re-running with each job made eight times dearer, which is how a machine + where the kernel dominates behaves, the curve stops turning over entirely: 48.92s serial, 32.03s + at 4, 29.98s at 6, 29.85s at 8, 28.94s at 16. + + That makes the risk asymmetric. Capping at 4 costs 0.7% on the fast machine but 10.7% on the + kernel-heavy one, while not capping costs 1.3% at worst. The worst case for a low cap is far + worse than the worst case for none, so 'auto' is left alone and anyone who wants fewer lanes sets + threads: explicitly. + """ + if threads == "auto": + return max(cpu_count_value, 1) + return max(int(threads), 1) + + def slots_around(target_slots, slot_lengths): """ Return a list of slot lengths around the target slots @@ -64,32 +84,14 @@ def scenario_hash_entry(kind, window_n, value): return acc ^ (acc >> 31) -""" -Used to mimic threads when they are disabled -""" - - -class DummyThread: - def __init__(self, result): - """ - Store the data into the class - """ - self.result = result - time.sleep(0) # Yield control - - def get(self): - """ - Return the result - """ - return self.result - - class Plan: """Plan optimisation mixin for finding optimal charge/discharge windows. Implements the search algorithm that explores price thresholds, - window combinations, and SoC targets using multi-threaded prediction - runs to minimise the overall cost metric. + window combinations, and SoC targets to minimise the overall cost + metric. Scenarios are evaluated in batches: launch_run_prediction_* + queues them and reading the first handle runs the whole batch through + one C++ kernel call, which spreads it across its own threads. """ def dynamic_load(self): @@ -669,49 +671,36 @@ def optimise_charge_limit_price_threads( ) def launch_run_prediction_single(self, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step=PREDICT_STEP): + """Queue a prediction and return a handle to its result. + + Nothing runs here: the inputs are read when the batch is flushed, so no list or window dict + passed in may be mutated until the returned handle's get() has been called. """ - Launch a thread to run a prediction - """ - charge_limit = list(charge_limit) - export_limits = list(export_limits) - if self.pool and self.pool._state == "RUN": - han = self.pool.apply_async(wrapped_run_prediction_single, (charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step)) - else: - han = DummyThread(self.prediction.thread_run_prediction_single(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step)) - return han + return self.prediction.queue_run_prediction_single(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step) def launch_run_prediction_charge(self, loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Queue a prediction and return a handle to its result. + + Nothing runs here: the inputs are read when the batch is flushed, so no list or window dict + passed in may be mutated until the returned handle's get() has been called. """ - Launch a thread to run a prediction - """ - if self.pool and self.pool._state == "RUN": - han = self.pool.apply_async(wrapped_run_prediction_charge, (loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record)) - else: - han = DummyThread(self.prediction.thread_run_prediction_charge(loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record)) - return han + return self.prediction.queue_run_prediction_charge(loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record) def launch_run_prediction_charge_min_max(self, loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Queue a prediction and return a handle to its result. + + Nothing runs here: the inputs are read when the batch is flushed, so no list or window dict + passed in may be mutated until the returned handle's get() has been called. """ - Launch a thread to run a prediction - """ - if self.pool and self.pool._state == "RUN": - han = self.pool.apply_async(wrapped_run_prediction_charge_min_max, (loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record)) - else: - han = DummyThread(self.prediction.thread_run_prediction_charge_min_max(loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record)) - return han + return self.prediction.queue_run_prediction_charge_min_max(loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record) def launch_run_prediction_export(self, this_export_limit, start, window_n, try_charge_limit, charge_window, try_export_window, try_export, pv_scenario, all_n, end_record): + """Queue a prediction and return a handle to its result. + + Nothing runs here: the inputs are read when the batch is flushed, so no list or window dict + passed in may be mutated until the returned handle's get() has been called. """ - Launch a thread to run a prediction - """ - if self.pool and self.pool._state == "RUN": - han = self.pool.apply_async( - wrapped_run_prediction_export, - (this_export_limit, start, window_n, try_charge_limit, charge_window, try_export_window, try_export, pv_scenario, all_n, end_record), - ) - else: - han = DummyThread(self.prediction.thread_run_prediction_export(this_export_limit, start, window_n, try_charge_limit, charge_window, try_export_window, try_export, pv_scenario, all_n, end_record)) - return han + return self.prediction.queue_run_prediction_export(this_export_limit, start, window_n, try_charge_limit, charge_window, try_export_window, try_export, pv_scenario, all_n, end_record) def scenario_summary_title(self, record_time): txt = "" @@ -1334,35 +1323,19 @@ def calculate_plan(self, recompute=True, debug_mode=False, publish=True): # Creation prediction object self.prediction = Prediction(self, pv_forecast_minute_step, pv_forecast_minute10_step, load_minutes_step, load_minutes_step10, pv_forecast_minute90_step, load_minutes_step90) + # The kernel spreads one batched fan-out across threads with the GIL released for the whole + # call, so these are real cores - unlike a Python ThreadPool, which peaked at 1.15x on two + # threads and then degraded below serial (perf/threadpool-prototype). + self.prediction.batch_threads = resolve_batch_threads(self.get_arg("threads", "auto"), cpu_count()) + self.log("Prediction batch using {} kernel thread(s)".format(self.prediction.batch_threads)) kernel_message, kernel_is_warning = kernel_status_summary(self.prediction) self.log("{}Prediction kernel: {}".format("Warn: " if kernel_is_warning else "", kernel_message)) - # Check if LoadML is active and disable thread pools as it causes lockup due to race conditions with NumPy + # Check if LoadML is active - it used to force the process pool off, which no longer exists; + # the kernel's threads are C++ threads with no fork and no NumPy involvement load_ml_comp = self.components.get_component("load_ml") if self.components else None - load_ml_calculating = False if load_ml_comp: - load_ml_calculating = load_ml_comp.is_calculating() - self.log("LoadML is_calculating {}".format(load_ml_calculating)) - if load_ml_calculating and self.pool: - self.log("Disabling thread pool as LoadML is calculating to avoid lockups") - self.pool.close() - self.pool.join() - self.pool = None - - # Create pool - if not self.pool: - if load_ml_calculating: - self.log("Not using thread pool as LoadML is calculating to avoid lockups") - else: - threads = self.get_arg("threads", "auto") - if threads == "auto": - self.log("Creating pool of {} processes to match your CPU count".format(cpu_count())) - self.pool = Pool(processes=cpu_count()) - elif threads: - self.log("Creating pool of {} processes as per apps.yaml".format(int(threads))) - self.pool = Pool(processes=int(threads)) - else: - self.log("Not using threading as threads is set to 0 in apps.yaml") + self.log("LoadML is_calculating {}".format(load_ml_comp.is_calculating())) # Simulate current settings to get initial data metric, import_kwh_battery, import_kwh_house, export_kwh, soc_min, soc, soc_min_minute, battery_cycle, metric_keep, final_iboost, final_carbon_g = self.run_prediction( @@ -1617,16 +1590,6 @@ def calculate_plan(self, recompute=True, debug_mode=False, publish=True): self.log(line) self.publish_html_plan(pv_forecast_minute_step, pv_forecast_minute10_step, load_minutes_step, load_minutes_step10, self.end_record) - # Destroy pool - if self.pool: - try: - self.pool.close() - self.pool.join() - except Exception as e: - self.log("Warn: failed to close thread pool: {}".format(e)) - self.log("Warn: " + traceback.format_exc()) - self.pool = None - # Record planning duration for SLO metrics self.plan_last_duration_seconds = time.time() - plan_start_time metrics().planning_duration_seconds.observe(self.plan_last_duration_seconds) diff --git a/apps/predbat/predbat.py b/apps/predbat/predbat.py index 59f3485f9..c097a7463 100644 --- a/apps/predbat/predbat.py +++ b/apps/predbat/predbat.py @@ -68,7 +68,6 @@ INVERTER_QUICK_UPDATE_SECONDS, ) from config import APPS_SCHEMA, CONFIG_ITEMS -from prediction import reset_prediction_globals from utils import minutes_since_yesterday, dp1, dp2, dp3 from predheat import PredHeat from octopus import Octopus @@ -271,27 +270,10 @@ def time_abs_str(self, minute): """ return (self.midnight + timedelta(minutes=minute)).strftime("%m-%d %H:%M:%S") - def cleanup_pool(self): - """ - Terminate and clean up the multiprocessing pool if it is active. - - Ensures worker processes are properly terminated to prevent orphaned - processes when the prediction loop exits unexpectedly. - """ - if getattr(self, "pool", None): - try: - self.pool.terminate() - self.pool.join() - except Exception as e: - self.log("Warn: Failed to terminate multiprocessing pool: {}".format(e)) - self.log(traceback.format_exc()) - self.pool = None - def reset(self): """ Init stub """ - reset_prediction_globals() self.text_plan = "Computing please wait..." self.prediction_cache_enable = True self.base_load = 0 @@ -316,8 +298,6 @@ def reset(self): # Forecast.solar API request metrics for monitoring self.currency_symbols = self.args.get("currency_symbols", "£p") - self.cleanup_pool() - self.pool = None self.watch_list = [] self.restart_active = False self.inverter_needs_reset = False @@ -1177,15 +1157,6 @@ def download_predbat_version(self, version): # Kill the current threads self.log("Kill current threads before update") self.stop_thread = True - if self.pool: - self.log("Warn: Killing current threads before update...") - try: - self.pool.close() - self.pool.join() - except Exception as e: - self.log("Warn: Failed to close thread pool: {}".format(e)) - self.log("Warn: " + traceback.format_exc()) - self.pool = None # Perform the update self.log("Perform the update.....") @@ -1609,7 +1580,6 @@ def initialize(self): # "Predbat defaulted it" or "another component already overwrote it" (issue #4494 follow-up). self.args_from_apps_yaml = copy.deepcopy(self.args) self.apps_yaml_override_warned = set() # {arg} already warned about via set_arg_auto() - self.pool = None self.log("Predbat: Startup {}".format(__name__)) self.update_time(print=False) self.started_time = self.now_utc_real @@ -1731,16 +1701,6 @@ async def terminate(self): self.log("Info: Components stopped") await asyncio.sleep(0) - if hasattr(self, "pool"): - if self.pool: - self.log("Info: Terminating thread pool...") - try: - self.pool.close() - self.pool.join() - except Exception as e: - self.log("Warn: Failed to close thread pool {}".format(e)) - self.log("Warn: " + traceback.format_exc()) - self.pool = None self.log("Predbat terminated") def update_time_loop(self, cb_args): @@ -1773,7 +1733,6 @@ def update_time_loop(self, cb_args): # Always clear the active flag, even on early return or exception, so the # web spinner and predbat.active switch don't get stuck on self.expose_config("active", False) - self.cleanup_pool() elif not self.prediction_started: time_now = datetime.now() inverter_data_last_fetch = self.inverter_data_last_fetch @@ -1855,7 +1814,6 @@ def run_time_loop(self, cb_args): # Always clear the active flag, even on early return or exception, so the # web spinner and predbat.active switch don't get stuck on self.expose_config("active", False) - self.cleanup_pool() def run_time_loop_balance(self, cb_args): """ diff --git a/apps/predbat/prediction.py b/apps/predbat/prediction.py index 2cc7053a2..5b8ea3acb 100644 --- a/apps/predbat/prediction.py +++ b/apps/predbat/prediction.py @@ -19,54 +19,12 @@ from datetime import timedelta from const import PREDICT_STEP, PV_SCENARIO_PV10, PV_SCENARIO_PV90, RUN_EVERY, TIME_FORMAT -from operator import itemgetter from utils import remove_intersecting_windows, get_charge_rate_curve_cached, get_discharge_rate_curve_cached, find_charge_rate, calc_percent_limit, in_iboost_slot, in_car_slot, charge_curve_to_tuple +from prediction_batch import PredictionBatch, prediction_cache_key from prediction_kernel import create_kernel_context, kernel_supported, run_prediction_kernel -# Only assign globals once to avoid re-creating them with processes are forked -if not "PRED_GLOBAL" in globals(): - PRED_GLOBAL = {} - - -def reset_prediction_globals(): - global PRED_GLOBAL - PRED_GLOBAL = {} - - -def wrapped_run_prediction_single(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step): - """Reconstruct a Prediction from the global shared state and run a single-scenario prediction in a worker process.""" - global PRED_GLOBAL - pred = Prediction() - pred.__dict__ = PRED_GLOBAL["dict"].copy() - return pred.thread_run_prediction_single(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step) - - -def wrapped_run_prediction_charge(try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): - """Reconstruct a Prediction from the global shared state and run a charge-window trial prediction in a worker process.""" - global PRED_GLOBAL - pred = Prediction() - pred.__dict__ = PRED_GLOBAL["dict"].copy() - return pred.thread_run_prediction_charge(try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record) - - -def wrapped_run_prediction_charge_min_max(try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): - """Reconstruct a Prediction from the global shared state and run a charge min/max trial prediction in a worker process.""" - global PRED_GLOBAL - pred = Prediction() - pred.__dict__ = PRED_GLOBAL["dict"].copy() - return pred.thread_run_prediction_charge_min_max(try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record) - - -def wrapped_run_prediction_export(this_export_limit, start, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): - """Reconstruct a Prediction from the global shared state and run an export-window trial prediction in a worker process.""" - global PRED_GLOBAL - pred = Prediction() - pred.__dict__ = PRED_GLOBAL["dict"].copy() - return pred.thread_run_prediction_export(this_export_limit, start, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record) - - def get_diff(battery_draw, pv_dc, pv_ac, load_yesterday, inverter_loss, inverter_loss_recp): """ Get AC output difference @@ -94,12 +52,7 @@ def get_total_inverted(battery_draw, pv_dc, pv_ac, inverter_loss, inverter_hybri return total_inverted -# Pulls (start, end) from a window dict; used to build the prediction cache key without a Python -# level loop over every window -window_bounds = itemgetter("start", "end") - - -class Prediction: +class Prediction(PredictionBatch): """ Class to hold prediction input and output data and the run function """ @@ -110,7 +63,6 @@ def __init__(self, base=None, pv_forecast_minute_step=None, pv_forecast_minute10 pv_forecast_minute90_step and load_minutes_step90 fall back to the nominal step arrays when None, so every existing call site that never requests the pv90 scenario keeps working unchanged. """ - global PRED_GLOBAL if base: self.minutes_now = base.minutes_now self.log = base.log @@ -221,13 +173,38 @@ def __init__(self, base=None, pv_forecast_minute_step=None, pv_forecast_minute10 if self.prediction_kernel_enable: self.kernel_handle = create_kernel_context(self) - # Store this dictionary in global so we can reconstruct it in the thread without passing the data - PRED_GLOBAL["dict"] = self.__dict__.copy() + # Outside the `if base:` block on purpose: a Prediction built without a base is still a valid + # object and its first enqueue_prediction would otherwise raise AttributeError + self.pending_batch = [] + self.batch_threads = 1 - def thread_run_prediction_single(self, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step): + def _prepare_single(self, charge_limit, export_limits): + """Copy the caller's limit lists for a single-scenario trial - shared by thread_run_prediction_single and the batch path. + + The copy used to live in Plan.launch_run_prediction_single. It is kept here because the batch + path does not read these lists until the batch is flushed, so a trial has to own the copy it + will eventually be simulated from rather than share the caller's list. """ - Run single prediction in a thread + return list(charge_limit), list(export_limits) + + def _prepare_charge(self, try_soc, window_n, charge_limit, all_n): + """Build the trial charge limits - shared by thread_run_prediction_charge/_charge_min_max and the batch path""" + try_charge_limit = charge_limit.copy() + if all_n: + for set_n in all_n: + try_charge_limit[set_n] = try_soc + else: + try_charge_limit[window_n] = try_soc + return try_charge_limit + + def thread_run_prediction_single(self, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step): + """Run one single-scenario prediction now and return its result. + + Nothing runs in a Python thread any more: this is the direct synchronous path, kept as the + reference the batch is checked against and as what a queued job falls back to when the kernel + will not take it. """ + charge_limit, export_limits = self._prepare_single(charge_limit, export_limits) ( cost, @@ -250,17 +227,24 @@ def thread_run_prediction_single(self, charge_limit, charge_window, export_windo ) = self.run_prediction(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record=end_record, step=step, cache=self.prediction_cache_enable) return (cost, import_kwh_battery, import_kwh_house, export_kwh, soc_min, soc, soc_min_minute, battery_cycle, metric_keep, final_iboost, final_carbon_g) - def thread_run_prediction_charge(self, try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + def queue_run_prediction_single(self, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step): + """Queue a single-scenario prediction, returning a handle - the batch runs on the first get(). + + The window lists and dicts are read at flush time, not now, so the caller must not mutate + anything it passed in before calling get() on the returned handle. """ - Run prediction in a thread + charge_limit, export_limits = self._prepare_single(charge_limit, export_limits) + return self.enqueue_prediction(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step, self.prediction_cache_enable) + + def thread_run_prediction_charge(self, try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Run one charge-window trial prediction now and return its result. + + Nothing runs in a Python thread any more: this is the direct synchronous path, kept as the + reference the batch is checked against and as what a queued job falls back to when the kernel + will not take it. """ - try_charge_limit = charge_limit.copy() - if all_n: - for set_n in all_n: - try_charge_limit[set_n] = try_soc - else: - try_charge_limit[window_n] = try_soc + try_charge_limit = self._prepare_charge(try_soc, window_n, charge_limit, all_n) ( cost, @@ -295,17 +279,43 @@ def thread_run_prediction_charge(self, try_soc, window_n, charge_limit, charge_w final_carbon_g, ) - def thread_run_prediction_charge_min_max(self, try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + def queue_run_prediction_charge(self, try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Queue a charge-window trial prediction, returning a handle. + + The window lists and dicts are read at flush time, not now, so the caller must not mutate + anything it passed in before calling get() on the returned handle. """ - Run prediction in a thread + try_charge_limit = self._prepare_charge(try_soc, window_n, charge_limit, all_n) + return self.enqueue_prediction(try_charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, PREDICT_STEP, self.prediction_cache_enable) + + def scan_soc_range(self, predict_soc, window): + """Return the (min, max) SoC across a charge window - shared by the direct and batch min/max paths. + + The kernel computes the same range inline (see PkBatchJob.soc_range_start_step), so this is + only reached when a job falls back to the Python engine; the two must agree exactly, including + the clamping that collapses an empty range to a single value rather than leaving min above max. """ + min_soc = self.soc_max + max_soc = 0 + predict_minute_start = max(int((window["start"] - self.minutes_now) / 5) * 5, 0) + predict_minute_end = int((window["end"] - self.minutes_now) / 5) * 5 + for minute in range(predict_minute_start, predict_minute_end + 5, 5): + if minute in predict_soc: + min_soc = min(predict_soc[minute], min_soc) + max_soc = max(predict_soc[minute], max_soc) + max_soc = max(max_soc, min_soc) + min_soc = min(min_soc, max_soc) + return min_soc, max_soc - try_charge_limit = charge_limit.copy() - if all_n: - for set_n in all_n: - try_charge_limit[set_n] = try_soc - else: - try_charge_limit[window_n] = try_soc + def thread_run_prediction_charge_min_max(self, try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Run one charge-window trial prediction now and return its result plus the SoC range. + + Nothing runs in a Python thread any more: this is the direct synchronous path, kept as the + reference the batch is checked against and as what a queued job falls back to when the kernel + will not take it. + """ + + try_charge_limit = self._prepare_charge(try_soc, window_n, charge_limit, all_n) ( cost, @@ -329,15 +339,7 @@ def thread_run_prediction_charge_min_max(self, try_soc, window_n, charge_limit, min_soc = self.soc_max max_soc = 0 if not all_n: - window = charge_window[window_n] - predict_minute_start = max(int((window["start"] - self.minutes_now) / 5) * 5, 0) - predict_minute_end = int((window["end"] - self.minutes_now) / 5) * 5 - for minute in range(predict_minute_start, predict_minute_end + 5, 5): - if minute in predict_soc: - min_soc = min(predict_soc[minute], min_soc) - max_soc = max(predict_soc[minute], max_soc) - max_soc = max(max_soc, min_soc) - min_soc = min(min_soc, max_soc) + min_soc, max_soc = self.scan_soc_range(predict_soc, charge_window[window_n]) return ( cost, @@ -355,11 +357,28 @@ def thread_run_prediction_charge_min_max(self, try_soc, window_n, charge_limit, max_soc, ) - def thread_run_prediction_export(self, this_export_limit, start, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + def queue_run_prediction_charge_min_max(self, try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Queue a charge-window trial prediction that also reports the SoC range across that window. + + Uncached, exactly as the direct path is: the SoC range is not part of the cached result, so a + hit would answer with the wrong shape. + + The window lists and dicts are read at flush time, not now, so the caller must not mutate + anything it passed in before calling get() on the returned handle. """ - Run prediction in a thread + try_charge_limit = self._prepare_charge(try_soc, window_n, charge_limit, all_n) + range_window = None if all_n else charge_window[window_n] + return self.enqueue_prediction(try_charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, PREDICT_STEP, False, want_range=True, range_window=range_window) + + def _prepare_export(self, this_export_limit, start, window_n, export_window, export_limits, all_n): + """Build the trial export limits and window list - shared by thread_run_prediction_export and the batch path. + + The trial start is applied to a private copy of the window rather than written into the + caller's list: with a process pool each worker mutated its own unpickled copy, but a batched + fan-out shares one list across every job in the batch, so an in-place write would corrupt the + other trials of the same window. Only ["end"] is ever read back by the caller + (optimise_export), so nothing depends on the write being visible. """ - # Store try value into the window export_limits = export_limits.copy() if all_n: @@ -370,7 +389,19 @@ def thread_run_prediction_export(self, this_export_limit, start, window_n, charg # Adjust start window = export_window[window_n] start = min(start, window["end"] - 5) - export_window[window_n]["start"] = start + export_window = list(export_window) + export_window[window_n] = dict(window, start=start) + + return export_window, export_limits + + def thread_run_prediction_export(self, this_export_limit, start, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Run one export-window trial prediction now and return its result. + + Nothing runs in a Python thread any more: this is the direct synchronous path, kept as the + reference the batch is checked against and as what a queued job falls back to when the kernel + will not take it. + """ + export_window, export_limits = self._prepare_export(this_export_limit, start, window_n, export_window, export_limits, all_n) ( metricmid, @@ -393,6 +424,15 @@ def thread_run_prediction_export(self, this_export_limit, start, window_n, charg ) = self.run_prediction(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record=end_record, cache=self.prediction_cache_enable) return metricmid, import_kwh_battery, import_kwh_house, export_kwh, soc_min, soc, soc_min_minute, battery_cycle, metric_keep, final_iboost, final_carbon_g + def queue_run_prediction_export(self, this_export_limit, start, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Queue an export-window trial prediction, returning a handle. + + The window lists and dicts are read at flush time, not now, so the caller must not mutate + anything it passed in before calling get() on the returned handle. + """ + export_window, export_limits = self._prepare_export(this_export_limit, start, window_n, export_window, export_limits, all_n) + return self.enqueue_prediction(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, PREDICT_STEP, self.prediction_cache_enable) + def find_charge_window_optimised(self, charge_windows, charge_limit, is_export=False): """ Takes in an array of charge windows @@ -416,23 +456,23 @@ def run_prediction(self, charge_limit, charge_window, export_window, export_limi KERNEL_PARITY_REVISION (prediction_kernel.py) and PK_PARITY_REVISION (prediction_kernel.cpp) must both be bumped, and the kernel_parity test must pass (cd coverage && ./run_all --test kernel_parity). """ + # A saving run publishes predict_soc_best and friends, and it is the only run that does. If a + # batch were left pending, the next handle read would flush it and reset_kernel_run_state + # would blank exactly those attributes, emptying the published plan and the debug HTML; a job + # flushed after its inputs were mutated would also be cached under a key hashed from the old + # ones, poisoning the shared cache. Draining first makes both impossible. Every fan-out site + # already drains its handles before saving, so this is the class defending its own invariant + # rather than a live fix - and it costs the ~250k non-save calls a single test of a local. + if save and self.pending_batch: + self.flush_batch() + # The cache key is only wanted when the cache is actually in play - a saving run always # simulates - so it is not computed otherwise. Building it as one tuple hash keeps the # per-window hashing in C rather than looping in Python, which matters because this runs on # every simulation with a few hundred windows. sim_hash = None if cache and not save: - sim_hash = hash( - ( - tuple(charge_limit), - tuple(map(window_bounds, charge_window)), - tuple(export_limits), - tuple(map(window_bounds, export_window)), - pv_scenario, - end_record, - step, - ) - ) + sim_hash = prediction_cache_key(charge_limit, charge_window, export_limits, export_window, pv_scenario, end_record, step) cached_result = self.prediction_cache.get(sim_hash) if cached_result is not None: # Return cached result diff --git a/apps/predbat/prediction_batch.py b/apps/predbat/prediction_batch.py new file mode 100644 index 000000000..7070ba641 --- /dev/null +++ b/apps/predbat/prediction_batch.py @@ -0,0 +1,183 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Lazy batching of the prediction fan-out. + +Every optimiser fan-out in plan.py is launch-all-then-collect-all, so the launches can queue instead +of running: each one prepares its trial inputs, appends a BatchJob and returns a BatchHandle, and the +first get() runs the whole batch through a single pk_run_batch call. That pays the Python/C boundary +and the window marshalling once per fan-out instead of once per scenario, and it is the only shape +that lets the kernel spread the work across cores - a Python thread pool cannot (the GIL serialises +it) and a process pool cannot (the parent's pickling is a serial bottleneck); both were measured and +rejected. + +INVARIANT: a caller must not mutate any list or window dict it has handed to a queue_* entry point +until it has read the matching handle. The four fan-outs in plan.py satisfy this; the 20-scenario +byte-identical plan comparison is what keeps it that way. +""" + +from operator import itemgetter + +from const import PREDICT_STEP +from prediction_kernel import BatchJob, kernel_supported, run_prediction_kernel_batch + +# Pulls (start, end) from a window dict; used to build the prediction cache key without a Python +# level loop over every window +window_bounds = itemgetter("start", "end") + + +def prediction_cache_key(charge_limit, charge_window, export_limits, export_window, pv_scenario, end_record, step): + """Hash a simulation's inputs into its prediction cache key. + + Shared by Prediction.run_prediction and the batch runner so a scenario cannot land under two + different keys depending on which path reached it. Built as one tuple hash to keep the per-window + hashing in C rather than looping in Python, which matters because it runs on every simulation + with a few hundred windows. + """ + return hash( + ( + tuple(charge_limit), + tuple(map(window_bounds, charge_window)), + tuple(export_limits), + tuple(map(window_bounds, export_window)), + pv_scenario, + end_record, + step, + ) + ) + + +def shape_batch_result(job, result, soc_range_min, soc_range_max): + """Shape a full run_prediction result into what this job's thread_run_prediction_* returned""" + shaped = tuple(result[:11]) + if job.want_range: + shaped = shaped + (soc_range_min, soc_range_max) + return shaped + + +class BatchHandle: + """Result handle for a queued prediction: get() runs the batch if it has not run yet. + + Deliberately shaped like a future so a fan-out reads as launch-all-then-get-all, but nothing is + running in the background - the work happens inside the first get(). + """ + + __slots__ = ("pred", "job") + + def __init__(self, pred, job): + """Bind the handle to the Prediction that owns the batch and to its own job""" + self.pred = pred + self.job = job + + def get(self): + """Return this job's result, flushing the whole pending batch on the first call""" + if self.job.result is None: + self.pred.flush_batch() + return self.job.result + + +class PredictionBatch: + """Pending-batch state and flushing for Prediction. + + A mixin rather than a separate object so the pending list lives directly on the Prediction that + owns the kernel context, with no reference cycle between the two. + """ + + def enqueue_prediction(self, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step, cache, want_range=False, range_window=None): + """Queue one prediction and return a handle to its eventual result""" + # A range job is never cached, and that is structural rather than the caller's choice: the + # SoC range is not part of a cached result, so a hit would answer with (0.0, 0.0) and silently + # collapse the caller's SoC pruning envelope. Callers pass cache=False today; ignoring cache + # outright for a range job means a future caller cannot reintroduce that bug. + sim_hash = prediction_cache_key(charge_limit, charge_window, export_limits, export_window, pv_scenario, end_record, step) if cache and not want_range else None + + # The kernel tracks the SoC range itself over a step range, so a min/max job does not have to + # ship an 8928-entry SoC buffer. A negative start step means "no range asked for", which the + # kernel answers with (soc_max, 0) - exactly what the Python all_n path returns unclamped. + soc_range_start_step = -1 + soc_range_end_step = -1 + if range_window is not None: + predict_minute_start = max(int((range_window["start"] - self.minutes_now) / 5) * 5, 0) + predict_minute_end = int((range_window["end"] - self.minutes_now) / 5) * 5 + soc_range_start_step = predict_minute_start // PREDICT_STEP + soc_range_end_step = predict_minute_end // PREDICT_STEP + + job = BatchJob( + charge_limit, + charge_window, + export_window, + export_limits, + pv_scenario, + end_record, + step, + soc_range_start_step=soc_range_start_step, + soc_range_end_step=soc_range_end_step, + sim_hash=sim_hash, + want_range=want_range, + range_window=range_window, + ) + self.pending_batch.append(job) + return BatchHandle(self, job) + + def run_batch_job_python(self, job): + """Run one job through the normal prediction path, for when the kernel cannot take it""" + result = self.run_prediction(job.charge_limit, job.charge_window, job.export_window, job.export_limits, job.pv_scenario, end_record=job.end_record, step=job.step, cache=job.sim_hash is not None) + if not job.want_range: + return tuple(result[:11]) + if job.range_window is None: + return tuple(result[:11]) + (self.soc_max, 0) + return tuple(result[:11]) + self.scan_soc_range(result[11], job.range_window) + + def flush_batch(self): + """Run every pending job: cache hits first, then one kernel call, then any fallbacks""" + pending = self.pending_batch + if not pending: + return + # Cleared before anything runs, so a fallback that re-enters run_prediction cannot see a + # half-flushed batch or flush it a second time + self.pending_batch = [] + + cache = self.prediction_cache + kernel_jobs = [] + duplicates = [] + first_by_hash = {} + for job in pending: + if job.sim_hash is not None: + cached = cache.get(job.sim_hash) + if cached is not None: + job.result = shape_batch_result(job, cached, 0.0, 0.0) + continue + # Two identical trials in one fan-out used to be a cache hit for the second one; the + # batch has to collapse them itself or that hit is lost (~44% of calls are hits) + first = first_by_hash.get(job.sim_hash) + if first is not None: + duplicates.append((job, first)) + continue + first_by_hash[job.sim_hash] = job + kernel_jobs.append(job) + + if kernel_jobs: + # kernel_supported() is a property of the Prediction, not of the individual job - it also + # refuses debug runs, which would otherwise reach the kernel here when the direct path + # would have used the Python engine. save is always None for a queued job: a saving run + # calls run_prediction directly and never fans out. + batched = run_prediction_kernel_batch(self, kernel_jobs, self.batch_threads) if kernel_supported(self, None, PREDICT_STEP) else None + for index, job in enumerate(kernel_jobs): + entry = batched[index] if batched is not None else None + if entry is None or entry[0] is None: + job.result = self.run_batch_job_python(job) + continue + result, soc_range_min, soc_range_max = entry + if job.sim_hash is not None: + # Stored without the SoC/car data to save memory, mirroring run_prediction + cache[job.sim_hash] = result[:11] + ([], []) + result[13:] + job.result = shape_batch_result(job, result, soc_range_min, soc_range_max) + + for job, first in duplicates: + job.result = first.result diff --git a/apps/predbat/prediction_kernel.cpp b/apps/predbat/prediction_kernel.cpp index 1746a6497..0be4dd4f8 100644 --- a/apps/predbat/prediction_kernel.cpp +++ b/apps/predbat/prediction_kernel.cpp @@ -16,7 +16,10 @@ // Build: bash apps/predbat/build_kernel.sh (g++/clang, C++17, no dependencies) #include +#include +#include #include +#include #include #include #include @@ -24,9 +27,22 @@ #include #include #include +// std::system_error, thrown when the system refuses a thread. libc++ pulls this in through , +// libstdc++ does not, so leaving it implicit builds on macOS and fails on GCC. +#include +#include #include -#define PK_ABI_VERSION 3 +#if defined(__unix__) || defined(__APPLE__) +#include +#endif + +// ABI 4: PkScenario::soc_out may be null, and pk_run_batch was added. The null is the reason this had +// to move - Python now passes no SoC buffer for every cached run, and an ABI 3 binary writes to it +// unconditionally, so loading one against this Python segfaults on the first prediction rather than +// falling back. Bumping makes the loader reject it and use the Python engine, which is the whole +// point of the check. +#define PK_ABI_VERSION 4 #define PK_PARITY_REVISION 5 #define PK_MAX_CARS 4 #define PK_RUN_EVERY 5 // const.py RUN_EVERY @@ -55,6 +71,71 @@ int32_t calc_percent_limit(double charge_limit, double soc_max) return std::min(static_cast((charge_limit / soc_max * 100.0) + 0.5), 100); } +// The SoC percent that Python's round(soc, 1) -> calc_percent_limit() pair produces. +// +// The 1dp rounding is not a modelling decision - it is there so utils.get_charge_rate_curve_cached's +// lru_cache hits - but rounding before the percent is taken occasionally moves the integer, so the +// kernel has to reproduce it exactly. +int32_t percent_via_round(double soc, double soc_max) +{ + return calc_percent_limit(round_py(soc, 1), soc_max); +} + +// Smallest SoC whose percent_via_round() reaches p, found by bisecting the double's bit pattern. +// +// Non-negative doubles compare in the same order as their bit patterns read as integers, so this is +// an exact bisection over every representable value in the bracket rather than an approximation: +// the boundary it returns is the true one, to the last bit. +double smallest_soc_for_percent(double soc_max, int32_t p, double hi_d) +{ + if (percent_via_round(0.0, soc_max) >= p) { + return 0.0; + } + if (percent_via_round(hi_d, soc_max) < p) { + return std::numeric_limits::infinity(); + } + uint64_t lo_bits, hi_bits; + double lo_d = 0.0; + std::memcpy(&lo_bits, &lo_d, sizeof(lo_bits)); + std::memcpy(&hi_bits, &hi_d, sizeof(hi_bits)); + while (hi_bits - lo_bits > 1) { + const uint64_t mid_bits = lo_bits + (hi_bits - lo_bits) / 2; + double mid_d; + std::memcpy(&mid_d, &mid_bits, sizeof(mid_d)); + if (percent_via_round(mid_d, soc_max) >= p) { + hi_bits = mid_bits; + } else { + lo_bits = mid_bits; + } + } + double result; + std::memcpy(&result, &hi_bits, sizeof(result)); + return result; +} + +// Boundaries of the 101 SoC percent buckets, so the hot loop can map SoC to percent with a binary +// search over doubles instead of a round_py (snprintf + strtod, ~96ns and the most expensive thing +// in the loop - and unusable under threading, since it serialises on the C library's global locale). +// out[p] is the smallest SoC that reads as p percent; out[0] is unused. +void build_soc_percent_thresholds(std::vector &out, double soc_max) +{ + out.assign(101, 0.0); + if (soc_max <= 0) { + return; + } + const double hi = soc_max * 2.0 + 1.0; + for (int32_t p = 1; p <= 100; p++) { + out[p] = smallest_soc_for_percent(soc_max, p, hi); + } + // upper_bound needs a non-decreasing sequence; the map is monotone so this only guards against + // an unreachable percent leaving an infinity in front of a later finite entry + for (int32_t p = 2; p <= 100; p++) { + if (out[p] < out[p - 1]) { + out[p] = out[p - 1]; + } + } +} + // Per-plan static context passed from Python, arrays are all n_steps long // (index k covers relative minute k*5) unless stated otherwise. // Field order MUST match the ctypes Structure in prediction_kernel.py exactly. @@ -184,6 +265,95 @@ struct PkResult { int32_t iboost_running_full; }; +// One scenario in a pk_run_batch() call; field order MUST match the ctypes Structure in +// prediction_kernel.py. +// +// This is deliberately a separate type from PkScenario rather than extra fields on it: pk_run keeps +// its existing layout, so a kernel binary predating the batch entry point still loads and runs, and +// Python falls back to looping pk_run when pk_run_batch is absent. +// +// soc_out may be null. A batched scenario is always a cached (non-saving) run, whose caller discards +// the per-minute SoC series, and materialising one buffer per job would cost ~84MB on a large batch. +// The one thing callers do want from the series is the SoC range across a charge window, so the +// kernel tracks that inline over [soc_range_start_step, soc_range_end_step] instead - set +// soc_range_start_step < 0 to skip it. +struct PkBatchJob { + const double *charge_limit; + const int32_t *charge_start; + const int32_t *charge_end; + const double *export_limits; + const int32_t *export_start; + const int32_t *export_end; + double *soc_out; // optional, null to skip + + int32_t n_charge; + int32_t n_export; + int32_t pv_scenario; + int32_t end_record; + int32_t step; + int32_t soc_range_start_step; // <0 to skip the SoC range scan + int32_t soc_range_end_step; // inclusive +}; + +// Result for one batched scenario; field order MUST match the ctypes Structure in +// prediction_kernel.py. soc_range_min/max mirror thread_run_prediction_charge_min_max's scan over +// predict_soc and are only meaningful when the job asked for a range. +struct PkBatchResult { + PkResult result; + double soc_range_min; + double soc_range_max; + int32_t status; // 0 = ok, matching pk_run's return codes otherwise + int32_t pad; +}; + +// Reusable per-simulation scratch buffers. +// +// build_window_membership assigns n_steps entries and the clip helpers clear-then-push, so all of +// these can be reused across scenarios without reallocating - which matters a lot under threading: +// each scenario would otherwise heap-allocate ~70KB, and 1200 of those per batch put every worker +// thread in contention on the allocator. Measured 4 threads at 0.6x of serial before this, because +// the simulation is cheap enough that malloc, not arithmetic, was the bottleneck. +// +// See thread_scratch() below: one of these now serves every batch a thread runs, not one per call. + +// Counts PkScratch constructions for the life of the library, for pk_scratch_construct_count(). The +// suite asserts this stops growing with the number of batches rather than trusting the reuse. +std::atomic g_scratch_constructions{0}; + +// Counts worker threads started for the life of the library, for pk_worker_thread_count(). Thread +// creation measured 7.5us each and a benchmark plan used to start 201,545 of them - 1.5s of a 26s +// plan, which is what the pool exists to stop paying. The suite asserts this stops growing. +std::atomic g_worker_threads_started{0}; + +struct PkScratch { + std::vector charge_member, export_member; + std::vector clipped_start, clipped_end; + std::vector clipped_limit; + + PkScratch() + { + g_scratch_constructions.fetch_add(1, std::memory_order_relaxed); + } +}; + +// One scratch per thread, reused for that thread's lifetime. +// +// Every field is fully rewritten before it is read on each run - build_window_membership assigns +// exactly n_steps entries and the clip helpers clear-then-push - which is the same invariant that +// already let one scratch serve a whole stride within a batch. Extending it across calls means a +// thread grows these buffers once instead of once per call: a benchmark plan issues ~25,000 batch +// calls, and at a median of 6 jobs each the ~70KB allocation was the same order as the simulation it +// wrapped. +// +// Thread-local rather than per-context because nothing here survives a simulation, so it never needs +// to belong to a particular context. A thread that later runs a context with a different n_steps +// simply has the vectors reassigned to the new length. +static PkScratch &thread_scratch() +{ + static thread_local PkScratch scratch; + return scratch; +} + // Deep-copied context storage so Python-side buffers can be freed after create struct ContextStore { std::vector rate_import, rate_export, alert_keep; @@ -193,6 +363,7 @@ struct ContextStore { std::vector charge_curve, discharge_curve; std::vector carbon, gas_rate, iboost_plan_load; std::vector car_load_flat, car_rate_flat; + std::vector soc_percent_threshold; // see build_soc_percent_thresholds PkContext ctx; }; @@ -222,6 +393,14 @@ inline double get_total_inverted(double battery_draw, double pv_dc, double pv_ac // Mirror of utils.py get_charge_rate_curve_cached()/get_discharge_rate_curve_cached(). // soc_key is passed exactly as Python does (usually round(soc, 1), but raw soc for the DC-rate lookups). // temp_cap_base is the pre-computed find_battery_temperature_cap() value before the min with max_rate. +inline double rate_curve_pct(int32_t soc_percent, double rate_setting, double rate_max, double temp_cap_base, const double *curve, double rate_min) +{ + double max_rate = rate_max * curve[soc_percent]; + double max_rate_cap = std::min(temp_cap_base, rate_max); + max_rate = std::min(max_rate, max_rate_cap); + return std::max(std::min(rate_setting, max_rate), rate_min); +} + inline double rate_curve(double soc_key, double rate_setting, double rate_max, double temp_cap_base, const double *curve, double soc_max, double rate_min) { int32_t soc_percent = calc_percent_limit(soc_key, soc_max); @@ -362,6 +541,7 @@ int64_t pk_context_create(const PkContext *in) return 0; } auto store = std::make_unique(); + build_soc_percent_thresholds(store->soc_percent_threshold, in->soc_max); size_t n = static_cast(in->n_steps); store->rate_import.assign(in->rate_import, in->rate_import + n); store->rate_export.assign(in->rate_export, in->rate_export + n); @@ -422,20 +602,39 @@ void pk_context_free(int64_t handle) // Run one prediction scenario. Returns 0 on success, non-zero on error. // Mirrors the hot loop of Prediction.run_prediction() (prediction.py:385-1200) for the // supported configuration: save=None, debug off, step=5 (cars, iBoost and carbon included). -int32_t pk_run(int64_t handle, const PkScenario *s, PkResult *out) +// Look up a context by handle, or null when the handle is unknown. The returned pointer stays valid +// as long as the caller does not free the context concurrently, which no caller does: contexts are +// created once per plan and released by a weakref finaliser once the Prediction is gone. +static const ContextStore *lookup_context(int64_t handle) { - const PkContext *c; - { - std::lock_guard lock(g_context_mutex); - auto it = g_contexts.find(handle); - if (it == g_contexts.end()) { - return 1; - } - c = &it->second->ctx; + std::lock_guard lock(g_context_mutex); + auto it = g_contexts.find(handle); + if (it == g_contexts.end()) { + return nullptr; } + return it->second.get(); +} + +// Simulate one scenario against an already-resolved context. +// +// Shared by pk_run and pk_run_batch, and safe to call concurrently on one context: c is const, every +// working value below is a local, and results go only to the caller's own out/soc_range slots. +// +// soc_out may be null (see PkBatchJob). soc_range_min/max are optional out-params: when non-null and +// soc_range_start_step >= 0, they receive the min/max of the rounded SoC over that inclusive step +// range, mirroring the predict_soc scan in Prediction.thread_run_prediction_charge_min_max. +static int32_t pk_run_one(const ContextStore *store, const PkScenario *s, PkResult *out, int32_t soc_range_start_step, int32_t soc_range_end_step, double *soc_range_min_out, double *soc_range_max_out, PkScratch &scratch) +{ + const PkContext *c = &store->ctx; + const std::vector &soc_pct_threshold = store->soc_percent_threshold; if (!s || !out || s->step != 5) { return 2; } + // Seeded to match Prediction.thread_run_prediction_charge_min_max, which starts min at soc_max + // and max at 0 so a range covering no recorded step leaves them in that state + double soc_range_min = c->soc_max; + double soc_range_max = 0.0; + const bool want_soc_range = soc_range_start_step >= 0; const int32_t pv_scenario = s->pv_scenario; const bool is_pv10 = pv_scenario == 1; @@ -445,12 +644,14 @@ int32_t pk_run(int64_t handle, const PkScenario *s, PkResult *out) const bool inverter_hybrid = c->inverter_hybrid != 0; // Window membership - prediction.py:494-495 / find_charge_window_optimised - std::vector charge_window_optimised, export_window_optimised; + std::vector &charge_window_optimised = scratch.charge_member; + std::vector &export_window_optimised = scratch.export_member; // The caller hands over the raw charge windows; clipping them against the export windows used to // be done in Python on every simulation, which cost more than the simulation itself - std::vector clipped_start, clipped_end; - std::vector clipped_limit; + std::vector &clipped_start = scratch.clipped_start; + std::vector &clipped_end = scratch.clipped_end; + std::vector &clipped_limit = scratch.clipped_limit; clip_intersecting_charge_windows(clipped_start, clipped_end, clipped_limit, s->n_charge, s->charge_start, s->charge_end, s->charge_limit, s->n_export, s->export_start, s->export_end, s->export_limits); const int32_t n_charge_clipped = static_cast(clipped_start.size()); @@ -570,8 +771,28 @@ int32_t pk_run(int64_t handle, const PkScenario *s, PkResult *out) record = false; } - // Save SoC prediction data - prediction.py:627-628 (kernel always fills the buffer) - s->soc_out[k] = round_py(soc, 3); + // Save SoC prediction data - prediction.py:627-628. Batched scenarios pass a null buffer + // (their caller discards the series); the rounded value is still what feeds the SoC range, + // so the range matches a Python scan over predict_soc exactly. + // round_py is snprintf+strtod, ~96ns and by far the most expensive thing in this loop, so it + // is only paid when something actually consumes the value. soc itself is never rounded - the + // result feeds soc_out and the SoC range and nothing else - so skipping it when a batched + // scenario wants neither cannot change the simulation. + const bool in_soc_range = want_soc_range && k >= soc_range_start_step && k <= soc_range_end_step; + if (s->soc_out || in_soc_range) { + const double soc_rounded = round_py(soc, 3); + if (s->soc_out) { + s->soc_out[k] = soc_rounded; + } + if (in_soc_range) { + if (soc_rounded < soc_range_min) { + soc_range_min = soc_rounded; + } + if (soc_rounded > soc_range_max) { + soc_range_max = soc_rounded; + } + } + } // Get load and pv forecast - prediction.py:657-659 double pv_now = pv_step[k]; @@ -700,11 +921,14 @@ int32_t pk_run(int64_t handle, const PkScenario *s, PkResult *out) } } - // Current real charge rate - prediction.py:777-786 - const double soc_round1 = round_py(soc, 1); - double charge_rate_now_curve = rate_curve(soc_round1, charge_rate_now, battery_rate_max_charge, c->temp_charge_cap[k], c->charge_curve, soc_max, battery_rate_min) * battery_rate_max_scaling; + // Current real charge rate - prediction.py:777-786. + // Python rounds SoC to 1dp here purely to quantise an lru_cache key, but the rounding moves + // the percent at bucket edges so it is observable. The precomputed bucket boundaries give + // the identical percent from a binary search, with no round_py in the loop at all. + const int32_t soc_percent_round1 = soc >= 0.0 ? static_cast(std::upper_bound(soc_pct_threshold.begin() + 1, soc_pct_threshold.end(), soc) - (soc_pct_threshold.begin() + 1)) : percent_via_round(soc, soc_max); + double charge_rate_now_curve = rate_curve_pct(soc_percent_round1, charge_rate_now, battery_rate_max_charge, c->temp_charge_cap[k], c->charge_curve, battery_rate_min) * battery_rate_max_scaling; double charge_rate_now_curve_step = charge_rate_now_curve * step; - double discharge_rate_now_curve = rate_curve(soc_round1, discharge_rate_now, battery_rate_max_discharge, c->temp_discharge_cap[k], c->discharge_curve, soc_max, battery_rate_min) * battery_rate_max_scaling_discharge; + double discharge_rate_now_curve = rate_curve_pct(soc_percent_round1, discharge_rate_now, battery_rate_max_discharge, c->temp_discharge_cap[k], c->discharge_curve, battery_rate_min) * battery_rate_max_scaling_discharge; double discharge_rate_now_curve_step = discharge_rate_now_curve * step; const double battery_to_min = std::max(soc - reserve_expected, 0.0) * battery_loss_discharge; @@ -727,7 +951,7 @@ int32_t pk_run(int64_t handle, const PkScenario *s, PkResult *out) export_rate_adjust = 1 - (export_limit_now - static_cast(static_cast(export_limit_now))); } discharge_rate_now = battery_rate_max_export * export_rate_adjust; - discharge_rate_now_curve = rate_curve(soc_round1, discharge_rate_now, battery_rate_max_export, c->temp_discharge_cap[k], c->discharge_curve, soc_max, battery_rate_min) * battery_rate_max_scaling_discharge; + discharge_rate_now_curve = rate_curve_pct(soc_percent_round1, discharge_rate_now, battery_rate_max_export, c->temp_discharge_cap[k], c->discharge_curve, battery_rate_min) * battery_rate_max_scaling_discharge; discharge_rate_now_curve_step = discharge_rate_now_curve * step; battery_draw = std::min(discharge_rate_now_curve_step, battery_to_min); @@ -813,7 +1037,7 @@ int32_t pk_run(int64_t handle, const PkScenario *s, PkResult *out) // find_charge_rate with set_charge_low_power off (always the case for scenario runs) // reduces to the max rate and its curve value - utils.py:1145,1237-1238 charge_rate_now = battery_rate_max_charge_combined; - charge_rate_now_curve = rate_curve(soc_round1, battery_rate_max_charge_combined, battery_rate_max_charge_combined, c->temp_charge_cap[k], c->charge_curve, soc_max, battery_rate_min) * battery_rate_max_scaling; + charge_rate_now_curve = rate_curve_pct(soc_percent_round1, battery_rate_max_charge_combined, battery_rate_max_charge_combined, c->temp_charge_cap[k], c->charge_curve, battery_rate_min) * battery_rate_max_scaling; charge_rate_now_curve_step = charge_rate_now_curve * step; battery_draw = -std::max({std::min(charge_rate_now_curve_step, std::max(charge_limit_n - soc, pv_now)), 0.0, -battery_to_max}); @@ -1074,7 +1298,335 @@ int32_t pk_run(int64_t handle, const PkScenario *s, PkResult *out) out->iboost_running = iboost_running; out->iboost_running_solar = iboost_running_solar; out->iboost_running_full = iboost_running_full; + if (soc_range_min_out) { + if (want_soc_range) { + // Mirrors the two clamping lines that follow the scan in + // thread_run_prediction_charge_min_max, so an empty range collapses to a single value + // rather than leaving min above max + const double clamped_max = soc_range_max > soc_range_min ? soc_range_max : soc_range_min; + const double clamped_min = soc_range_min < clamped_max ? soc_range_min : clamped_max; + *soc_range_min_out = clamped_min; + *soc_range_max_out = clamped_max; + } else { + // No range asked for. Python's all_n path skips the scan AND the clamping, returning + // (soc_max, 0) untouched, so clamping here would hand back (soc_max, soc_max) instead. + *soc_range_min_out = c->soc_max; + *soc_range_max_out = 0.0; + } + } return 0; } +// Run one scenario. Kept with its original signature and struct so a binary built before the batch +// entry point existed still satisfies the loader and every single-scenario caller. +int32_t pk_run(int64_t handle, const PkScenario *s, PkResult *out) +{ + const ContextStore *c = lookup_context(handle); + if (!c) { + return 1; + } + return pk_run_one(c, s, out, -1, -1, nullptr, nullptr, thread_scratch()); // c is the store +} + +// Run n_jobs scenarios against one context in a single call. +// +// The point is to pay the Python/C boundary and the context lookup once for a whole fan-out of +// scenarios instead of once each: the per-call Python around pk_run measured ~31us against ~55us of +// simulation, and neither Python threads nor a process pool can recover it (the GIL serialises the +// former, pickling the latter). +// +// Per-job failures are reported in that job's status field rather than aborting the batch, so one +// malformed scenario cannot discard the rest of the fan-out. +// Simulate one batched scenario into its result slot. Split out so the serial and threaded paths +// below share exactly one copy of the marshalling. +static void run_batch_job(const ContextStore *c, const PkBatchJob &job, PkBatchResult &out, PkScratch &scratch) +{ + PkScenario scenario; + scenario.charge_limit = job.charge_limit; + scenario.charge_start = job.charge_start; + scenario.charge_end = job.charge_end; + scenario.export_limits = job.export_limits; + scenario.export_start = job.export_start; + scenario.export_end = job.export_end; + scenario.soc_out = job.soc_out; + scenario.n_charge = job.n_charge; + scenario.n_export = job.n_export; + scenario.pv_scenario = job.pv_scenario; + scenario.end_record = job.end_record; + scenario.step = job.step; + out.soc_range_min = 0.0; + out.soc_range_max = 0.0; + out.pad = 0; + out.status = pk_run_one(c, &scenario, &out.result, job.soc_range_start_step, job.soc_range_end_step, &out.soc_range_min, &out.soc_range_max, scratch); +} + + +// A pool of worker threads that outlives every batch and every context. +// +// pk_run_batch used to build and join a fresh std::vector per call. Measured over one +// benchmark plan that was 201,545 thread creations costing 1.5s - against 3.1s of simulation, so the +// creation cost was cancelling most of what the parallelism won and threading measured 1.6% overall. +// Parking workers on a condition variable replaces each creation with a wake. +// +// Workers park until `generation` changes, then take lane `index` of a stride partition. The calling +// thread takes lane 0 itself, so a six-job batch wakes five workers rather than six and the caller +// does useful work instead of blocking. Nobody holds the mutex while simulating; run_batch_job only +// reads the shared const context and writes its own result slot, which is what made the previous +// per-call threading safe and is unchanged here. +class PkThreadPool { + public: + // Start workers until at least `want` exist. Grows, never shrinks. Returns how many there are. + int32_t reserve_workers(int32_t want) + { + std::lock_guard lock(mutex); + while (static_cast(workers.size()) < want) { + const int32_t lane = static_cast(workers.size()) + 1; // lane 0 is the caller + std::unique_ptr worker(new Worker()); + try { + worker->thread = std::thread(&PkThreadPool::worker_loop, this, lane, worker.get()); + } catch (const std::system_error &) { + // Out of thread resources - keep what we have and let the caller use fewer lanes. + break; + } + workers.push_back(std::move(worker)); + g_worker_threads_started.fetch_add(1, std::memory_order_relaxed); + } + return static_cast(workers.size()); + } + + // Run jobs[0..n_jobs) across `use` lanes, the calling thread taking lane 0. `use` must be at + // least 1 and no more than worker_count() + 1. + void run(const ContextStore *c, const PkBatchJob *jobs, int32_t n_jobs, PkBatchResult *results, int32_t use) + { + { + std::lock_guard lock(mutex); + cur_ctx = c; + cur_jobs = jobs; + cur_results = results; + cur_n_jobs = n_jobs; + cur_use = use; + outstanding = use - 1; + // Each worker has its own flag and its own condition variable, so exactly the lanes this + // batch needs are woken. Broadcasting instead would wake every worker in the pool - with + // a median batch of six jobs against sixteen workers that was ~250,000 wakeups per plan + // spent entirely on threads that immediately went back to sleep. + for (int32_t lane = 1; lane < use; lane++) { + workers[lane - 1]->has_work = true; + } + } + for (int32_t lane = 1; lane < use; lane++) { + workers[lane - 1]->ready.notify_one(); + } + + PkScratch &scratch = thread_scratch(); + for (int32_t i = 0; i < n_jobs; i += use) { + run_batch_job(c, jobs[i], results[i], scratch); + } + + std::unique_lock lock(mutex); + work_done.wait(lock, [this] { return outstanding == 0; }); + } + + private: + // One parked worker. The flag is written under the pool mutex before its condition variable is + // signalled, so a wake cannot be lost even if the worker has not re-parked yet. + struct Worker { + std::condition_variable ready; + bool has_work = false; + std::thread thread; + }; + + void worker_loop(int32_t lane, Worker *self) + { + std::unique_lock lock(mutex); + for (;;) { + self->ready.wait(lock, [self] { return self->has_work; }); + self->has_work = false; + const ContextStore *c = cur_ctx; + const PkBatchJob *jobs = cur_jobs; + PkBatchResult *results = cur_results; + const int32_t n_jobs = cur_n_jobs; + const int32_t use = cur_use; + lock.unlock(); + + PkScratch &scratch = thread_scratch(); + for (int32_t i = lane; i < n_jobs; i += use) { + run_batch_job(c, jobs[i], results[i], scratch); + } + + lock.lock(); + if (--outstanding == 0) { + work_done.notify_one(); + } + } + } + + std::mutex mutex; + std::condition_variable work_done; // the caller waits here + const ContextStore *cur_ctx = nullptr; + const PkBatchJob *cur_jobs = nullptr; + PkBatchResult *cur_results = nullptr; + int32_t cur_n_jobs = 0; + int32_t cur_use = 1; + int32_t outstanding = 0; + std::vector> workers; +}; + +// The pool is created on first threaded use and deliberately never destroyed: a static destructor +// running while CPython tears the process down risks a deadlock for no benefit, and the OS reclaims +// the threads at exit. +std::atomic g_pool{nullptr}; +std::mutex g_pool_create_mutex; +// Held for a whole batch. Python can enter pk_run_batch from two threads at once because ctypes +// releases the GIL, and one shared pool cannot serve two batches concurrently - its published job +// state would race. Nothing in predbat does this today; serialising is the safe answer if it ever does. +std::mutex g_dispatch_mutex; + +#if defined(__unix__) || defined(__APPLE__) +// Threads do not survive fork(), so a child would inherit a pool whose workers do not exist and hang +// waiting for them. The prepare/parent pair keep the dispatch mutex consistent across the fork, and +// the child drops the pool so its next batch builds a fresh one. Nothing here allocates or takes a +// new lock, which is what the child handler is allowed to do; the dead pool leaks in the child. +void pk_atfork_prepare() +{ + g_dispatch_mutex.lock(); +} + +void pk_atfork_parent() +{ + g_dispatch_mutex.unlock(); +} + +void pk_atfork_child() +{ + g_dispatch_mutex.unlock(); + g_pool.store(nullptr, std::memory_order_relaxed); +} +#endif + +// Fetch the pool, creating it on first use. +static PkThreadPool *pool_singleton() +{ + PkThreadPool *pool = g_pool.load(std::memory_order_acquire); + if (pool) { + return pool; + } + std::lock_guard lock(g_pool_create_mutex); + pool = g_pool.load(std::memory_order_relaxed); + if (!pool) { +#if defined(__unix__) || defined(__APPLE__) + static std::once_flag atfork_once; + std::call_once(atfork_once, [] { pthread_atfork(pk_atfork_prepare, pk_atfork_parent, pk_atfork_child); }); +#endif + pool = new PkThreadPool(); + g_pool.store(pool, std::memory_order_release); + } + return pool; +} + +// Run n_jobs scenarios against one context in a single call. +// +// The context lookup and the Python/C boundary are paid once for a whole fan-out rather than once +// per scenario, but the real point is n_threads: 19209 separate ctypes calls cannot be parallelised +// usefully from Python (the GIL serialises threads, and a process pool pays pickling), whereas one +// batch can split the work across cores with no GIL involved at all. +// +// n_threads <= 1 runs serially. Per-job failures land in that job's status field rather than +// aborting the batch, so one malformed scenario cannot discard the rest of the fan-out. +int32_t pk_run_batch(int64_t handle, const PkBatchJob *jobs, int32_t n_jobs, PkBatchResult *results, int32_t n_threads) +{ + const ContextStore *c = lookup_context(handle); + if (!c) { + return 1; + } + if (!jobs || !results || n_jobs < 0) { + return 2; + } + // Scenarios are independent - each reads the shared const context and writes only its own result + // slot - so they are split across threads by stride. + if (n_threads > 1 && n_jobs > 1) { + std::lock_guard dispatch(g_dispatch_mutex); + int32_t use = n_threads < n_jobs ? n_threads : n_jobs; + PkThreadPool *pool = pool_singleton(); + // Clamp to the lanes that actually exist: reserve_workers stops early if the system refuses a + // thread, and a batch must never wait on a worker that was never started. + const int32_t lanes = pool->reserve_workers(use - 1) + 1; + if (use > lanes) { + use = lanes; + } + if (use > 1) { + pool->run(c, jobs, n_jobs, results, use); + return 0; + } + // No workers available - fall through and run the batch inline. + } + for (int32_t i = 0; i < n_jobs; i++) { + run_batch_job(c, jobs[i], results[i], thread_scratch()); + } + return 0; +} + + +// Test hook: how many PkScratch instances have been constructed since the library was loaded. +// +// The scratch buffers are the only heap allocation a simulation makes, and the kernel's whole +// batching win rests on not paying them per call - so the suite asserts this counter stops growing +// with the number of batches rather than trusting the code to be reusing them. +int64_t pk_scratch_construct_count(void) +{ + return g_scratch_constructions.load(std::memory_order_relaxed); +} + + +// Test hook: how many worker threads have been started since the library was loaded. +// +// Batches used to build and join a fresh set of threads on every call. The suite asserts this count +// stops growing once the pool is warm, because a thread creation costs several microseconds against +// a median batch of six jobs - it was cancelling the parallelism it was there to provide. +int64_t pk_worker_thread_count(void) +{ + return g_worker_threads_started.load(std::memory_order_relaxed); +} + + +// Test hook: sweep SoC densely and confirm the precomputed bucket boundaries give exactly the same +// percent as the round_py path they replace. Returns the number of disagreements (0 = equivalent). +int32_t pk_verify_soc_percent_table(double soc_max, int32_t samples) +{ + std::vector table; + build_soc_percent_thresholds(table, soc_max); + int32_t bad = 0; + const double hi = soc_max * 1.2 + 0.5; + for (int32_t i = 0; i <= samples; i++) { + const double soc = hi * static_cast(i) / static_cast(samples); + const int32_t want = percent_via_round(soc, soc_max); + const int32_t got = static_cast(std::upper_bound(table.begin() + 1, table.end(), soc) - (table.begin() + 1)); + if (want != got) { + bad++; + } + } + // Also probe either side of every boundary, where a disagreement would actually hide + for (int32_t p = 1; p <= 100; p++) { + const double edge = table[p]; + if (!std::isfinite(edge)) { + continue; + } + for (int32_t d = -2; d <= 2; d++) { + double probe = edge; + for (int32_t n = 0; n < std::abs(d); n++) { + probe = d < 0 ? std::nextafter(probe, -1.0) : std::nextafter(probe, 1e18); + } + if (probe < 0) { + continue; + } + const int32_t want = percent_via_round(probe, soc_max); + const int32_t got = static_cast(std::upper_bound(table.begin() + 1, table.end(), probe) - (table.begin() + 1)); + if (want != got) { + bad++; + } + } + } + return bad; +} + } // extern "C" diff --git a/apps/predbat/prediction_kernel.py b/apps/predbat/prediction_kernel.py index 2275e9951..27f5f21b9 100644 --- a/apps/predbat/prediction_kernel.py +++ b/apps/predbat/prediction_kernel.py @@ -31,16 +31,19 @@ from utils import get_curve_value, find_battery_temperature_cap, in_car_slot, in_iboost_slot # Expected ABI/parity revisions of the shared library (see prediction_kernel.cpp) -KERNEL_ABI_VERSION = 3 +KERNEL_ABI_VERSION = 4 KERNEL_PARITY_REVISION = 5 # Maximum number of cars supported by the kernel (PK_MAX_CARS in prediction_kernel.cpp) KERNEL_MAX_CARS = 4 -# Module-level loader state, inherited by forked worker processes +# Module-level loader state - the library is loaded once per process and shared by every Prediction KERNEL_LIB = None KERNEL_LOAD_TRIED = False KERNEL_STATUS = "not loaded" +# True when the loaded binary exposes pk_run_batch. Probed rather than required so a kernel built +# before the batch entry point existed still runs every scenario through the single-call path. +KERNEL_HAS_BATCH = False class PkContext(ctypes.Structure): @@ -176,6 +179,69 @@ class PkResult(ctypes.Structure): ] +class PkBatchJob(ctypes.Structure): + """ctypes mirror of the PkBatchJob struct in prediction_kernel.cpp (field order must match exactly)""" + + _fields_ = [ + ("charge_limit", ctypes.POINTER(ctypes.c_double)), + ("charge_start", ctypes.POINTER(ctypes.c_int32)), + ("charge_end", ctypes.POINTER(ctypes.c_int32)), + ("export_limits", ctypes.POINTER(ctypes.c_double)), + ("export_start", ctypes.POINTER(ctypes.c_int32)), + ("export_end", ctypes.POINTER(ctypes.c_int32)), + ("soc_out", ctypes.POINTER(ctypes.c_double)), + ("n_charge", ctypes.c_int32), + ("n_export", ctypes.c_int32), + ("pv_scenario", ctypes.c_int32), + ("end_record", ctypes.c_int32), + ("step", ctypes.c_int32), + ("soc_range_start_step", ctypes.c_int32), + ("soc_range_end_step", ctypes.c_int32), + ] + + +class PkBatchResult(ctypes.Structure): + """ctypes mirror of the PkBatchResult struct in prediction_kernel.cpp (field order must match exactly)""" + + _fields_ = [ + ("result", PkResult), + ("soc_range_min", ctypes.c_double), + ("soc_range_max", ctypes.c_double), + ("status", ctypes.c_int32), + ("pad", ctypes.c_int32), + ] + + +class BatchJob: + """One queued prediction: its trial inputs, how the result should be shaped, and the result itself. + + run_prediction_kernel_batch reads only the input fields (charge_limit, charge_window, + export_window, export_limits, pv_scenario, end_record) and the SoC range steps. step, sim_hash, + want_range, range_window and result belong to the batch runner in prediction_batch.py, which is + the only thing that creates these; they live here so the ctypes layer can be tested on its own. + step is deliberately not one of the kernel's inputs - run_prediction_kernel_batch overrides it + with PREDICT_STEP, so it is only read when a job falls back to the Python engine. + """ + + __slots__ = ("charge_limit", "charge_window", "export_window", "export_limits", "pv_scenario", "end_record", "step", "soc_range_start_step", "soc_range_end_step", "sim_hash", "want_range", "range_window", "result") + + def __init__(self, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step, soc_range_start_step=-1, soc_range_end_step=-1, sim_hash=None, want_range=False, range_window=None): + """Record one job's trial inputs; result stays None until the batch is flushed""" + self.charge_limit = charge_limit + self.charge_window = charge_window + self.export_window = export_window + self.export_limits = export_limits + self.pv_scenario = pv_scenario + self.end_record = end_record + self.step = step + self.soc_range_start_step = soc_range_start_step + self.soc_range_end_step = soc_range_end_step + self.sim_hash = sim_hash + self.want_range = want_range + self.range_window = range_window + self.result = None + + def kernel_library_candidates(): """Return the ordered list of shared library paths to try loading. @@ -217,10 +283,11 @@ def kernel_status_summary(pred): def load_kernel(log=None): """Load and verify the kernel shared library, returns the ctypes library or None. - The result is cached module-wide (including across forked workers); failures - are logged once and result in a permanent fallback to the Python engine. + The result is cached module-wide, so the library is loaded and verified once + per process; failures are logged once and result in a permanent fallback to + the Python engine. """ - global KERNEL_LIB, KERNEL_LOAD_TRIED, KERNEL_STATUS + global KERNEL_LIB, KERNEL_LOAD_TRIED, KERNEL_STATUS, KERNEL_HAS_BATCH if KERNEL_LOAD_TRIED: return KERNEL_LIB @@ -248,6 +315,14 @@ def load_kernel(log=None): lib.pk_context_free.argtypes = [ctypes.c_int64] lib.pk_run.restype = ctypes.c_int32 lib.pk_run.argtypes = [ctypes.c_int64, ctypes.POINTER(PkScenario), ctypes.POINTER(PkResult)] + try: + lib.pk_run_batch.restype = ctypes.c_int32 + # The trailing c_int32 is n_threads - pk_run_batch takes five arguments, and ctypes + # will happily pass a fourth-and-a-bit if the binding says otherwise + lib.pk_run_batch.argtypes = [ctypes.c_int64, ctypes.POINTER(PkBatchJob), ctypes.c_int32, ctypes.POINTER(PkBatchResult), ctypes.c_int32] + KERNEL_HAS_BATCH = True + except AttributeError: + KERNEL_HAS_BATCH = False KERNEL_LIB = lib KERNEL_STATUS = "loaded from {}".format(path) if log: @@ -288,8 +363,11 @@ def double_array(values): positional arguments and is several times slower, which matters because these are rebuilt on every simulation. from_buffer returns a view over the array.array, and ctypes keeps the backing object alive through the view's _objects, so the buffer cannot be collected while the kernel is - using it. Each pool worker is a separate process (multiprocessing with fork), so no buffer is - ever shared between workers. + using it. The view is never written to once built: it is filled on the calling thread before + pk_run/pk_run_batch is entered and the kernel only ever reads it, so even a buffer deliberately + shared between jobs (the window memo in run_prediction_kernel_batch) is only read concurrently - + the kernel's worker threads never see a buffer change under them, because the one thread that + could change it is blocked inside the call. An iterable is taken rather than a list so callers can hand over a map/generator and skip materialising an intermediate list - array.array consumes it in C and coerces each item to a double @@ -493,6 +571,138 @@ def kernel_supported(pred, save, step): return not save and not pred.debug_enable and getattr(pred, "kernel_handle", 0) != 0 +def kernel_result_tuple(pred, result, predict_soc): + """Assemble the 17-field Prediction.run_prediction() result tuple from a PkResult. + + Shared by the single-scenario and batch paths - mirrors prediction.py:626-628, 1266-1284. + """ + car_charging_soc_next = pred.car_charging_soc_next[:] + if result.car_soc_next_valid: + for car_n in range(pred.num_cars): + car_charging_soc_next[car_n] = result.car_soc_next[car_n] + + iboost_next = result.iboost_next if pred.iboost_enable else pred.iboost_next + + return ( + round(result.final_metric, 4), + round(result.import_kwh_battery, 4), + round(result.import_kwh_house, 4), + round(result.export_kwh, 4), + round(result.soc_min, 4), + round(result.final_soc, 4), + result.soc_min_minute, + round(result.battery_cycle, 4), + round(result.metric_keep, 4), + round(result.final_iboost, 4), + round(result.final_carbon_g, 4), + predict_soc, + car_charging_soc_next, + iboost_next, + bool(result.iboost_running), + bool(result.iboost_running_solar), + bool(result.iboost_running_full), + ) + + +def reset_kernel_run_state(pred): + """Clear the per-run state attributes exactly as the Python engine does - prediction.py:414-422 + + Non-save runs never populate these, so they simply stay empty/False; a kernel run has to leave + the Prediction in the same state a Python run would. + """ + pred.predict_soc_best = {} + pred.predict_metric_best = {} + pred.predict_iboost_best = {} + pred.predict_carbon_best = {} + pred.predict_clipped_best = {} + pred.iboost_running = False + pred.iboost_running_solar = False + pred.iboost_running_full = False + + +def run_prediction_kernel_batch(pred, jobs, n_threads=1): + """Run a whole fan-out of prediction scenarios through the kernel in one pk_run_batch call. + + Returns a list of (result_tuple, soc_range_min, soc_range_max) in job order, where result_tuple + is the same 17-field tuple Prediction.run_prediction() returns, or None for a job the kernel + refused - the caller re-runs those through the Python engine. Returns None when the batch could + not be run at all (no kernel, no batch entry point, or a non-zero return code). + + Whether the kernel is allowed to answer at all is the caller's decision, not this function's: + kernel_supported() also refuses debug and saving runs, and flush_batch checks it before getting + here. + + Batched jobs never materialise the per-minute SoC series: their callers all discard it, and one + buffer per job would be ~84MB on a large batch. The one thing a caller does want from it - the + SoC range across a charge window - the kernel tracks inline over + [soc_range_start_step, soc_range_end_step] instead. + """ + lib = KERNEL_LIB + if not lib or not KERNEL_HAS_BATCH or not getattr(pred, "kernel_handle", 0): + return None + + n_jobs = len(jobs) + job_array = (PkBatchJob * n_jobs)() + result_array = (PkBatchResult * n_jobs)() + # Every ctypes buffer behind a pointer field has to outlive the call. ctypes' own keepalive already + # covers this (b_base walks up to job_array._objects), so this list is belt-and-braces: the limit + # arrays are held explicitly here rather than relying on ctypes' container keepalive. + buffers = [] + # The charge fan-outs reuse the same window lists across every job, so their start/end arrays are + # marshalled once per distinct list instead of once per job - which is most of what those batches + # save. Export windows share this memo only when all_n is truthy; otherwise _prepare_export builds + # a fresh one per job. Keyed on identity, with the list retained so an id() cannot be recycled + # mid-batch. This relies on no caller mutating a window list between enqueue and flush (see + # prediction_batch.py). + window_cache = {} + + def window_arrays(windows): + """Marshal a window list's start/end arrays, reusing an earlier job's arrays where possible""" + entry = window_cache.get(id(windows)) + if entry is None: + entry = (int32_array([window["start"] for window in windows]), int32_array([window["end"] for window in windows]), windows) + window_cache[id(windows)] = entry + return entry + + for index, job in enumerate(jobs): + charge_start, charge_end, _ = window_arrays(job.charge_window) + export_start, export_end, _ = window_arrays(job.export_window) + charge_limit = double_array(job.charge_limit) + export_limits = double_array(job.export_limits) + buffers.append((charge_limit, export_limits)) + + pk_job = job_array[index] + pk_job.charge_limit = charge_limit + pk_job.charge_start = charge_start + pk_job.charge_end = charge_end + pk_job.export_limits = export_limits + pk_job.export_start = export_start + pk_job.export_end = export_end + pk_job.soc_out = None + pk_job.n_charge = len(job.charge_window) + pk_job.n_export = len(job.export_window) + pk_job.pv_scenario = int(job.pv_scenario) + pk_job.end_record = job.end_record + pk_job.step = PREDICT_STEP # the caller's step is ignored - see run_prediction_kernel + pk_job.soc_range_start_step = job.soc_range_start_step + pk_job.soc_range_end_step = job.soc_range_end_step + + return_code = lib.pk_run_batch(pred.kernel_handle, job_array, n_jobs, result_array, max(int(n_threads), 1)) + if return_code != 0: + return None + + reset_kernel_run_state(pred) + + results = [] + for index in range(n_jobs): + batch_result = result_array[index] + if batch_result.status != 0: + results.append((None, 0.0, 0.0)) + continue + results.append((kernel_result_tuple(pred, batch_result.result, {}), batch_result.soc_range_min, batch_result.soc_range_max)) + return results + + def run_prediction_kernel(pred, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step, cache): """Run one prediction scenario through the C++ kernel. @@ -523,7 +733,11 @@ def run_prediction_kernel(pred, charge_limit, charge_window, export_window, expo scenario.export_limits = double_array(export_limits) scenario.export_start = int32_array([window["start"] for window in export_window]) scenario.export_end = int32_array([window["end"] for window in export_window]) - soc_out = (ctypes.c_double * n_steps)() + # A cached run discards the per-minute SoC series (see the `if not cache` block below), so the + # buffer is not allocated and the kernel is told to skip filling it. That skips a round_py per + # step, which is snprintf+strtod and the single most expensive thing in the kernel's hot loop - + # measured 52.8us -> 28.5us per scenario, for a value that was being thrown away. + soc_out = (ctypes.c_double * n_steps)() if not cache else None scenario.soc_out = soc_out scenario.n_charge = len(charge_window) scenario.n_export = len(export_window) @@ -536,18 +750,9 @@ def run_prediction_kernel(pred, charge_limit, charge_window, export_window, expo if return_code != 0: return None - # Reset the per-run state attributes exactly as the Python engine does - prediction.py:414-422 - # (non-save runs never populate these, so they remain empty/False) - pred.predict_soc_best = {} - pred.predict_metric_best = {} - pred.predict_iboost_best = {} - pred.predict_carbon_best = {} - pred.predict_clipped_best = {} - pred.iboost_running = False - pred.iboost_running_solar = False - pred.iboost_running_full = False + reset_kernel_run_state(pred) - # Assemble the same return value as the Python engine - prediction.py:626-628, 1266-1284 + # Assemble the same return value as the Python engine predict_soc = {} if not cache: # Indexed loop, not dict(zip(range(...), soc_out)): iterating a ctypes array boxes each double @@ -555,29 +760,4 @@ def run_prediction_kernel(pred, charge_limit, charge_window, export_window, expo for k in range(n_steps): predict_soc[k * PREDICT_STEP] = soc_out[k] - car_charging_soc_next = pred.car_charging_soc_next[:] - if result.car_soc_next_valid: - for car_n in range(pred.num_cars): - car_charging_soc_next[car_n] = result.car_soc_next[car_n] - - iboost_next = result.iboost_next if pred.iboost_enable else pred.iboost_next - - return ( - round(result.final_metric, 4), - round(result.import_kwh_battery, 4), - round(result.import_kwh_house, 4), - round(result.export_kwh, 4), - round(result.soc_min, 4), - round(result.final_soc, 4), - result.soc_min_minute, - round(result.battery_cycle, 4), - round(result.metric_keep, 4), - round(result.final_iboost, 4), - round(result.final_carbon_g, 4), - predict_soc, - car_charging_soc_next, - iboost_next, - bool(result.iboost_running), - bool(result.iboost_running_solar), - bool(result.iboost_running_full), - ) + return kernel_result_tuple(pred, result, predict_soc) diff --git a/apps/predbat/prediction_kernel_lib_aarch64.so b/apps/predbat/prediction_kernel_lib_aarch64.so index 2f989b602..eda553983 100755 Binary files a/apps/predbat/prediction_kernel_lib_aarch64.so and b/apps/predbat/prediction_kernel_lib_aarch64.so differ diff --git a/apps/predbat/prediction_kernel_lib_armv7l.so b/apps/predbat/prediction_kernel_lib_armv7l.so index eb1b953dc..b26eda510 100755 Binary files a/apps/predbat/prediction_kernel_lib_armv7l.so and b/apps/predbat/prediction_kernel_lib_armv7l.so differ diff --git a/apps/predbat/prediction_kernel_lib_darwin_arm64.so b/apps/predbat/prediction_kernel_lib_darwin_arm64.so index 83a19542c..0431deb3d 100755 Binary files a/apps/predbat/prediction_kernel_lib_darwin_arm64.so and b/apps/predbat/prediction_kernel_lib_darwin_arm64.so differ diff --git a/apps/predbat/prediction_kernel_lib_darwin_x86_64.so b/apps/predbat/prediction_kernel_lib_darwin_x86_64.so index feb706535..7300ca9bd 100755 Binary files a/apps/predbat/prediction_kernel_lib_darwin_x86_64.so and b/apps/predbat/prediction_kernel_lib_darwin_x86_64.so differ diff --git a/apps/predbat/prediction_kernel_lib_i686.so b/apps/predbat/prediction_kernel_lib_i686.so index f42ccbdef..05939aabc 100755 Binary files a/apps/predbat/prediction_kernel_lib_i686.so and b/apps/predbat/prediction_kernel_lib_i686.so differ diff --git a/apps/predbat/prediction_kernel_lib_x86_64.so b/apps/predbat/prediction_kernel_lib_x86_64.so index 9eca7a369..9cb9287e0 100755 Binary files a/apps/predbat/prediction_kernel_lib_x86_64.so and b/apps/predbat/prediction_kernel_lib_x86_64.so differ diff --git a/apps/predbat/tests/test_download.py b/apps/predbat/tests/test_download.py index 5ba19ef1b..b2372e7e2 100644 --- a/apps/predbat/tests/test_download.py +++ b/apps/predbat/tests/test_download.py @@ -1693,7 +1693,6 @@ def _test_predbat_download_main_uses_configured_repository(my_predbat): class DummyPredBat: def __init__(self): self.stop_thread = False - self.pool = None def log(self, *_args, **_kwargs): return None @@ -1725,7 +1724,6 @@ def _test_predbat_download_tag_uses_default_repository(my_predbat): class DummyPredBat: def __init__(self): self.stop_thread = False - self.pool = None def log(self, *_args, **_kwargs): return None diff --git a/apps/predbat/tests/test_export_commitment.py b/apps/predbat/tests/test_export_commitment.py index fdd046ac7..c7ca43d0e 100644 --- a/apps/predbat/tests/test_export_commitment.py +++ b/apps/predbat/tests/test_export_commitment.py @@ -23,7 +23,6 @@ def setup_single_export_window(my_predbat, rate_import=10.0, rate_export=30.0, b my_predbat.load_user_config() my_predbat.fetch_config_options() reset_inverter(my_predbat) - my_predbat.pool = None my_predbat.forecast_minutes = 24 * 60 end_record = my_predbat.forecast_minutes diff --git a/apps/predbat/tests/test_infra.py b/apps/predbat/tests/test_infra.py index 0c8b3220a..0a8160017 100644 --- a/apps/predbat/tests/test_infra.py +++ b/apps/predbat/tests/test_infra.py @@ -9,7 +9,7 @@ # pylint: disable=attribute-defined-outside-init from datetime import datetime, timedelta -from prediction import wrapped_run_prediction_single, Prediction +from prediction import Prediction from matplotlib import pyplot as plt import asyncio import numpy as np @@ -805,7 +805,7 @@ def simple_scenario( metric_keep, final_iboost, final_carbon_g, - ) = wrapped_run_prediction_single(charge_limit_best, charge_window_best, export_window_best, export_limit_best, pv10, end_record=(my_predbat.end_record), step=5) + ) = prediction.thread_run_prediction_single(charge_limit_best, charge_window_best, export_window_best, export_limit_best, pv10, end_record=(my_predbat.end_record), step=5) else: ( metric, diff --git a/apps/predbat/tests/test_kernel_parity.py b/apps/predbat/tests/test_kernel_parity.py index 816c4c1c7..bdcd49025 100644 --- a/apps/predbat/tests/test_kernel_parity.py +++ b/apps/predbat/tests/test_kernel_parity.py @@ -26,7 +26,11 @@ import gc import os import random +import signal import subprocess +import sys +import threading +import time import prediction_kernel from const import PV_SCENARIO_NOMINAL, PV_SCENARIO_PV10, PV_SCENARIO_PV90 @@ -423,6 +427,19 @@ def run_marshalling_tests(): print("ERROR: {}([]) should be empty, got length {}".format(name, len(empty))) failed = True + # pk_run_batch's binding used to be missing its trailing n_threads entry in argtypes. That bug is invisible + # to the batch parity test: ctypes silently accepts more call arguments than argtypes declares, and + # a garbage thread count only changes how pk_run_batch splits work across threads - the results are + # thread-count-independent by design. Only asserting the binding itself catches a regression here. + if prediction_kernel.KERNEL_HAS_BATCH: + expected_argtypes = (ctypes.c_int64, ctypes.POINTER(prediction_kernel.PkBatchJob), ctypes.c_int32, ctypes.POINTER(prediction_kernel.PkBatchResult), ctypes.c_int32) + actual_argtypes = tuple(prediction_kernel.KERNEL_LIB.pk_run_batch.argtypes) + if actual_argtypes != expected_argtypes: + print("ERROR: pk_run_batch.argtypes is {} but the C++ signature (handle, jobs, n_jobs, results, n_threads) needs {}".format(actual_argtypes, expected_argtypes)) + failed = True + else: + print("SKIP: pk_run_batch argtypes check - kernel does not expose pk_run_batch") + if not failed: print("PASS") return failed @@ -849,6 +866,207 @@ def run_clipping_parity_tests(my_predbat, count=250): return failed +def build_batch_jobs(my_predbat, rng, count, window_variants=1): + """Build a list of BatchJob scenarios plus the Prediction they run against. + + window_variants controls how many distinct charge/export window lists the batch draws from, cycled + by job index (job N uses variant N % window_variants). The default of 1 shares a single list across + every job, as the real min/max fan-out does when it scans one window at a time. A caller wanting + per-job window lists - to exercise run_prediction_kernel_batch's identity-keyed window memo, whose + cache key is id(windows), not its content - passes a larger value. + """ + apply_random_scenario(my_predbat, rng) + pv_step, pv10_step, load_step, load10_step = make_step_data(my_predbat, rng) + my_predbat.prediction_kernel_enable = True + prediction = Prediction(my_predbat, pv_step, pv10_step, load_step, load10_step) + prediction.kernel_handle = create_kernel_context(prediction) + + charge_windows = [make_windows(rng, my_predbat.minutes_now, my_predbat.forecast_minutes, rng.randint(2, 5)) for _ in range(window_variants)] + export_windows = [make_windows(rng, my_predbat.minutes_now, my_predbat.forecast_minutes, rng.randint(2, 5)) for _ in range(window_variants)] + end_record = my_predbat.forecast_minutes + + jobs = [] + for index in range(count): + charge_window = charge_windows[index % window_variants] + export_window = export_windows[index % window_variants] + charge_limit = [round(rng.uniform(0, my_predbat.soc_max), 2) for _ in charge_window] + export_limits = [rng.choice([100.0, 99.0, 0.0, round(rng.uniform(1, 99), 1)]) for _ in export_window] + pv_scenario = rng.choice([PV_SCENARIO_NOMINAL, PV_SCENARIO_PV10, PV_SCENARIO_PV90]) + # Every third job asks for the SoC range over a charge window, as the min/max fan-out does + range_window = charge_window[index % len(charge_window)] if (index % 3) == 0 else None + start_step = -1 + end_step = -1 + if range_window is not None: + start_step = max(int((range_window["start"] - my_predbat.minutes_now) / 5) * 5, 0) // 5 + end_step = int((range_window["end"] - my_predbat.minutes_now) / 5) * 5 // 5 + jobs.append( + prediction_kernel.BatchJob( + charge_limit, + charge_window, + export_window, + export_limits, + pv_scenario, + end_record, + 5, + soc_range_start_step=start_step, + soc_range_end_step=end_step, + want_range=range_window is not None, + range_window=range_window, + ) + ) + return prediction, jobs + + +def build_batch_reference(prediction, jobs): + """Run one pk_run per job to build the reference results run_prediction_kernel_batch is checked against. + + The SoC series is materialised (unlike the batch path) so the range can be scanned in Python + exactly as thread_run_prediction_charge_min_max does. Returns None if a reference run itself fails. + """ + reference = [] + for job in jobs: + single = run_prediction_kernel(prediction, job.charge_limit, job.charge_window, job.export_window, job.export_limits, job.pv_scenario, job.end_record, 5, False) + if single is None: + print("ERROR: reference pk_run failed") + return None + if job.range_window is None: + soc_range = (prediction.soc_max, 0) + else: + soc_range = prediction.scan_soc_range(single[11], job.range_window) + reference.append((single, soc_range)) + return reference + + +def check_batch_results(jobs, reference, batched, n_threads): + """Compare one pk_run_batch call's results against the build_batch_reference() reference, returns True on failure""" + if len(batched) != len(jobs): + print("ERROR: pk_run_batch returned {} results for {} jobs".format(len(batched), len(jobs))) + return True + + failed = False + for index, (result_tuple, soc_range_min, soc_range_max) in enumerate(batched): + single, (expect_min, expect_max) = reference[index] + if result_tuple is None: + print("ERROR: job {} reported a non-zero status at {} threads".format(index, n_threads)) + failed = True + continue + for field, name in enumerate(RESULT_NAMES): + if result_tuple[field] != single[field]: + print("ERROR: job {} at {} threads differs on {}: batch {} single {}".format(index, n_threads, name, result_tuple[field], single[field])) + failed = True + # The batch never materialises the SoC series - that is what makes it affordable. Checked + # against the single path rather than against an empty dict on its own, so this fails both + # if the batch starts filling the series and if the reference stopped filling it (which + # would make the batch's emptiness prove nothing). + if result_tuple[11] or not single[11]: + print("ERROR: job {} SoC series: batch has {} entries, single reference has {}".format(index, len(result_tuple[11]), len(single[11]))) + failed = True + # car_charging_soc_next is recorded at minute 0, before any trial limit can move it, so it is + # the same value for every job in the fan-out - comparing it job by job pins that the batch + # assembles it like the single path does, but cannot catch a mis-routed result. That the + # kernel fills it at all is asserted once per run in run_batch_parity_tests. + for field, name in [(12, "car_charging_soc_next"), (13, "iboost_next"), (14, "iboost_running"), (15, "iboost_running_solar"), (16, "iboost_running_full")]: + if result_tuple[field] != single[field]: + print("ERROR: job {} at {} threads differs on {}: batch {} single {}".format(index, n_threads, name, result_tuple[field], single[field])) + failed = True + if soc_range_min != expect_min or soc_range_max != expect_max: + print("ERROR: job {} at {} threads SoC range {} {} expected {} {}".format(index, n_threads, soc_range_min, soc_range_max, expect_min, expect_max)) + failed = True + return failed + + +def run_batch_parity_tests(my_predbat, count=60): + """Check pk_run_batch matches a loop of pk_run exactly, at every thread count, returns True on failure. + + The batch path is the only one Python will use after this refactor, and it is the only one that + can run scenarios concurrently, so it is checked against the single-scenario path it replaces + rather than against the Python engine (which run_random_sweep_tests already pins pk_run against). + Results must be bit-identical, not merely close: the whole point of the fan-out is that plans do + not change. + """ + print("**** Running kernel batch parity tests ****") + if not prediction_kernel.KERNEL_HAS_BATCH: + print("SKIP: kernel does not expose pk_run_batch") + return False + + prediction, jobs = build_batch_jobs(my_predbat, random.Random(4321), count) + if not prediction.kernel_handle: + print("ERROR: batch parity kernel context creation failed") + return True + + reference = build_batch_reference(prediction, jobs) + if reference is None: + return True + + failed = False + # The per-job car_charging_soc_next comparison in check_batch_results is only worth anything if + # the kernel actually fills that field - otherwise both sides are the Prediction's own baseline + # copied through, and the comparison is two copies of the same list. Seed 4321 gives this + # scenario cars; assert it rather than assume it, so changing the seed cannot silently hollow the + # comparison out. + if not prediction.num_cars: + print("ERROR: batch parity scenario has no cars - the car_charging_soc_next comparison is vacuous") + failed = True + elif reference[0][0][12][: prediction.num_cars] == prediction.car_charging_soc_next[: prediction.num_cars]: + print("ERROR: car_charging_soc_next was not filled by the kernel, it is the Prediction's baseline {}".format(prediction.car_charging_soc_next)) + failed = True + + for n_threads in (1, 2, 4, 8): + batched = prediction_kernel.run_prediction_kernel_batch(prediction, jobs, n_threads) + if batched is None: + print("ERROR: pk_run_batch failed at {} threads".format(n_threads)) + return True + failed |= check_batch_results(jobs, reference, batched, n_threads) + + if not failed: + print("Batch parity: {} scenarios bit-identical to pk_run at 1, 2, 4 and 8 threads".format(count)) + return failed + + +def run_batch_window_variant_tests(my_predbat, count=30, window_variants=6): + """Check the batch's per-job window marshalling picks the right list, not a cached neighbour's. + + build_batch_jobs's default (window_variants=1) shares one charge/export window list across the + whole batch, which exercises run_prediction_kernel_batch's window memo only on its cache-hit path: + every job asks for the same id(windows), so a memo keyed on anything coarser than list identity + would still pass. Task 4's real fan-out mixes windows freely between jobs, so here several distinct + window lists are cycled across the jobs (some jobs sharing a list, most not) and checked against the + same per-scenario pk_run reference run_batch_parity_tests uses - a job that picked up another list's + start/end arrays would simulate the wrong charge/export times and show up as a wrong result. + """ + print("**** Running kernel batch window variant tests ****") + if not prediction_kernel.KERNEL_HAS_BATCH: + print("SKIP: kernel does not expose pk_run_batch") + return False + + prediction, jobs = build_batch_jobs(my_predbat, random.Random(9876), count, window_variants=window_variants) + if not prediction.kernel_handle: + print("ERROR: batch window variant kernel context creation failed") + return True + + distinct_charge = len({id(job.charge_window) for job in jobs}) + distinct_export = len({id(job.export_window) for job in jobs}) + if distinct_charge < 2 or distinct_export < 2: + print("ERROR: batch window variant test only produced {} charge and {} export window list(s) - not exercising the memo".format(distinct_charge, distinct_export)) + return True + + reference = build_batch_reference(prediction, jobs) + if reference is None: + return True + + failed = False + for n_threads in (1, 4): + batched = prediction_kernel.run_prediction_kernel_batch(prediction, jobs, n_threads) + if batched is None: + print("ERROR: pk_run_batch failed at {} threads".format(n_threads)) + return True + failed |= check_batch_results(jobs, reference, batched, n_threads) + + if not failed: + print("Batch window variant parity: {} scenarios across {} charge and {} export window lists bit-identical to pk_run".format(count, distinct_charge, distinct_export)) + return failed + + def kernel_available(): """Ensure the kernel library is built and loaded, returns (available, required_failure)""" if not ensure_kernel_built(): @@ -878,6 +1096,351 @@ def run_model_kernel_tests(my_predbat): my_predbat.prediction_kernel_enable = False +def run_soc_percent_table_tests(): + """Check the kernel's SoC percent bucket table is exactly the round_py path it replaced. + + The hot loop maps SoC to a curve percent through precomputed bucket boundaries instead of + round(soc, 1), which is only legitimate if the two agree for every SoC that can occur. The kernel + exposes pk_verify_soc_percent_table for this: it sweeps the SoC range and, more importantly, + probes either side of all 101 boundaries, which is the only place a disagreement could hide. + + A failure here means plans would silently diverge from the Python engine at a bucket edge, so it + is checked across the range of battery sizes rather than one convenient value. + """ + print("**** Running SoC percent table tests ****") + lib = prediction_kernel.KERNEL_LIB + if not lib or not hasattr(lib, "pk_verify_soc_percent_table"): + print("SKIP: kernel does not expose pk_verify_soc_percent_table") + return False + + lib.pk_verify_soc_percent_table.restype = ctypes.c_int32 + lib.pk_verify_soc_percent_table.argtypes = [ctypes.c_double, ctypes.c_int32] + failed = False + # Spread of real battery capacities, including awkward non-round ones where the percent + # boundaries do not line up with tidy SoC values + for soc_max in (0.5, 2.0, 4.8, 5.0, 9.5, 10.0, 13.5, 20.0, 25.6, 30.0, 100.0): + bad = lib.pk_verify_soc_percent_table(ctypes.c_double(soc_max), 200000) + if bad: + print("ERROR: SoC percent table disagrees with round_py in {} places for soc_max {}".format(bad, soc_max)) + failed = True + if not failed: + print("SoC percent table matches round_py for every sampled SoC and every bucket boundary") + return failed + + +def run_scratch_reuse_tests(my_predbat, batches=10): + """Check a thread builds its scratch buffers once, not once per batch, returns True on failure. + + The scratch vectors are the only heap allocation a simulation makes - roughly 70KB once they have + grown to n_steps - and a benchmark plan issues around 25,000 batch calls, so paying that per call + is the difference between the allocator and the arithmetic being the bottleneck. The kernel + exposes pk_scratch_construct_count for this: the count must stop growing with the number of + batches rather than tracking it. + """ + print("**** Running kernel scratch reuse tests ****") + lib = prediction_kernel.KERNEL_LIB + if not lib or not hasattr(lib, "pk_scratch_construct_count"): + print("SKIP: kernel does not expose pk_scratch_construct_count") + return False + + lib.pk_scratch_construct_count.restype = ctypes.c_int64 + lib.pk_scratch_construct_count.argtypes = [] + + prediction, jobs = build_batch_jobs(my_predbat, random.Random(99), 4) + if not prediction.kernel_handle: + print("ERROR: scratch reuse kernel context creation failed") + return True + + # One batch first, so any first-touch allocation is already paid before the count is taken + prediction_kernel.run_prediction_kernel_batch(prediction, jobs, 1) + before = lib.pk_scratch_construct_count() + for _ in range(batches): + if prediction_kernel.run_prediction_kernel_batch(prediction, jobs, 1) is None: + print("ERROR: scratch reuse batch call failed") + return True + grew = lib.pk_scratch_construct_count() - before + + if grew: + print("ERROR: {} batches built {} scratch buffers - they should be reused, not rebuilt per call".format(batches, grew)) + return True + print("Scratch reuse: {} batches on one thread built 0 further scratch buffers".format(batches)) + return False + + +def run_scratch_cross_context_tests(my_predbat, rounds=3): + """Check a reused scratch cannot leak between contexts of different sizes, returns True on failure. + + One scratch now serves every batch a thread ever runs, and Predbat builds a fresh context each + plan cycle - with a different n_steps whenever forecast_minutes changes. The buffers are only safe + to reuse because every field is fully rewritten before it is read: build_window_membership assigns + exactly n_steps entries and the clip helpers clear before pushing. If that ever stopped holding, a + large context followed by a small one would read the large one's tail. + + So the contexts are deliberately run largest-first and then interleaved, and each must keep + returning what it returned on its own. + """ + print("**** Running kernel scratch cross-context tests ****") + if not prediction_kernel.KERNEL_HAS_BATCH: + print("SKIP: kernel does not expose pk_run_batch") + return False + + saved_forecast = my_predbat.forecast_minutes + try: + cases = [] + # Largest first: a later, smaller context is the one that would read a stale tail + for forecast_minutes in (48 * 60, 12 * 60): + my_predbat.forecast_minutes = forecast_minutes + prediction, jobs = build_batch_jobs(my_predbat, random.Random(forecast_minutes), 6) + if not prediction.kernel_handle: + print("ERROR: cross-context kernel context creation failed for {} minutes".format(forecast_minutes)) + return True + cases.append((forecast_minutes, prediction, jobs, build_batch_reference(prediction, jobs))) + + failed = False + for round_n in range(rounds): + for forecast_minutes, prediction, jobs, reference in cases: + batched = prediction_kernel.run_prediction_kernel_batch(prediction, jobs, 1) + if batched is None: + print("ERROR: cross-context batch call failed for {} minutes".format(forecast_minutes)) + return True + for index, (result_tuple, soc_range_min, soc_range_max) in enumerate(batched): + single, (expect_min, expect_max) = reference[index] + if result_tuple is None: + print("ERROR: round {} {} minutes job {} reported a non-zero status".format(round_n, forecast_minutes, index)) + failed = True + continue + for field, name in enumerate(RESULT_NAMES): + if result_tuple[field] != single[field]: + print("ERROR: round {} {} minutes job {} differs on {}: got {} want {}".format(round_n, forecast_minutes, index, name, result_tuple[field], single[field])) + failed = True + if soc_range_min != expect_min or soc_range_max != expect_max: + print("ERROR: round {} {} minutes job {} SoC range {} {} want {} {}".format(round_n, forecast_minutes, index, soc_range_min, soc_range_max, expect_min, expect_max)) + failed = True + if failed: + break + + if not failed: + print("Scratch cross-context: {} rounds interleaving {} and {} minute contexts, all identical".format(rounds, cases[0][0], cases[1][0])) + return failed + finally: + my_predbat.forecast_minutes = saved_forecast + + +def run_worker_reuse_tests(my_predbat, batches=10, n_threads=4): + """Check threaded batches reuse parked workers instead of starting new ones, returns True on failure. + + A benchmark plan issues ~25,000 threaded batch calls with a median of six jobs each. Starting a + thread per lane per call measured 201,545 threads and 1.5s of a 26s plan - more than the + parallelism was returning, which is why threading was worth 1.6% before the pool. The kernel + exposes pk_worker_thread_count for this: once warm, the count must stop growing with the number of + batches. + """ + print("**** Running kernel worker reuse tests ****") + lib = prediction_kernel.KERNEL_LIB + if not lib or not hasattr(lib, "pk_worker_thread_count"): + print("SKIP: kernel does not expose pk_worker_thread_count") + return False + + lib.pk_worker_thread_count.restype = ctypes.c_int64 + lib.pk_worker_thread_count.argtypes = [] + + prediction, jobs = build_batch_jobs(my_predbat, random.Random(1234), 8) + if not prediction.kernel_handle: + print("ERROR: worker reuse kernel context creation failed") + return True + + # Warm first, so the pool has grown to n_threads before the count is taken + prediction_kernel.run_prediction_kernel_batch(prediction, jobs, n_threads) + before = lib.pk_worker_thread_count() + for _ in range(batches): + if prediction_kernel.run_prediction_kernel_batch(prediction, jobs, n_threads) is None: + print("ERROR: worker reuse batch call failed") + return True + grew = lib.pk_worker_thread_count() - before + + if grew: + print("ERROR: {} batches at {} threads started {} more workers - they should be reused, not restarted per call".format(batches, n_threads, grew)) + return True + print("Worker reuse: {} batches at {} threads started 0 further workers".format(batches, n_threads)) + return False + + +def compare_batch_to_reference(label, batched, reference): + """Compare a batch's results against a per-job pk_run reference, returns True on failure""" + failed = False + for index, (result_tuple, soc_range_min, soc_range_max) in enumerate(batched): + single, (expect_min, expect_max) = reference[index] + if result_tuple is None: + print("ERROR: {} job {} reported a non-zero status".format(label, index)) + failed = True + continue + for field, name in enumerate(RESULT_NAMES): + if result_tuple[field] != single[field]: + print("ERROR: {} job {} differs on {}: got {} want {}".format(label, index, name, result_tuple[field], single[field])) + failed = True + if soc_range_min != expect_min or soc_range_max != expect_max: + print("ERROR: {} job {} SoC range {} {} want {} {}".format(label, index, soc_range_min, soc_range_max, expect_min, expect_max)) + failed = True + return failed + + +def run_pool_lane_tests(my_predbat): + """Check lane allocation across batch and pool sizes, returns True on failure. + + The pool is sized by the largest thread count it has been asked for and never shrinks, so most + batches use fewer lanes than there are workers - the median batch is six jobs against a pool built + for more. Idle workers must sit out without touching the completion count, and a batch must never + wait on a lane that was never dispatched. These combinations walk a batch smaller than the pool, + the pool growing, the pool being asked for less than it has, and the single-job and single-thread + bypasses. + """ + print("**** Running kernel pool lane tests ****") + if not prediction_kernel.KERNEL_HAS_BATCH: + print("SKIP: kernel does not expose pk_run_batch") + return False + + prediction, jobs = build_batch_jobs(my_predbat, random.Random(2468), 8) + if not prediction.kernel_handle: + print("ERROR: pool lane kernel context creation failed") + return True + reference = build_batch_reference(prediction, jobs) + + failed = False + for n_threads, n_jobs in ((16, 3), (8, 8), (2, 8), (16, 8), (4, 1), (1, 8), (3, 8)): + subset = jobs[:n_jobs] + batched = prediction_kernel.run_prediction_kernel_batch(prediction, subset, n_threads) + if batched is None: + print("ERROR: pool lane batch failed at {} threads {} jobs".format(n_threads, n_jobs)) + return True + failed |= compare_batch_to_reference("threads={} jobs={}".format(n_threads, n_jobs), batched, reference[:n_jobs]) + + if not failed: + print("Pool lanes: 7 thread/job combinations all bit-identical to pk_run") + return failed + + +def run_pool_concurrency_tests(my_predbat, rounds=400, n_threads=8): + """Check two Python threads can dispatch batches at once, returns True on failure. + + ctypes releases the GIL, so predbat could enter pk_run_batch from two threads simultaneously. One + shared pool cannot serve two batches at once - the published job pointers would race - so dispatch + is serialised by a mutex. Nothing does this today, which is why it needs a test rather than an + assumption: each thread checks its own job set against its own reference, so a crossed batch shows + up as wrong values, not just a crash. + + The round count is load-bearing and was chosen by mutation, not taste. With the dispatch mutex + deleted, 25 rounds over 4 lanes passes happily - the GIL keeps the two callers from overlapping + often enough for the race to land - while 400 rounds over 8 lanes deadlocks within seconds and is + caught by the join timeout below. The healthy case costs no measurable time either way, so do not + trim these numbers to make the suite look faster. + """ + print("**** Running kernel pool concurrency tests ****") + if not prediction_kernel.KERNEL_HAS_BATCH: + print("SKIP: kernel does not expose pk_run_batch") + return False + + sets = [] + for seed in (11, 22): + prediction, jobs = build_batch_jobs(my_predbat, random.Random(seed), 6) + if not prediction.kernel_handle: + print("ERROR: pool concurrency kernel context creation failed") + return True + sets.append((prediction, jobs, build_batch_reference(prediction, jobs))) + + errors = [] + + def hammer(label, prediction, jobs, reference): + """Run many batches on one context and record any mismatch against its own reference""" + for _ in range(rounds): + batched = prediction_kernel.run_prediction_kernel_batch(prediction, jobs, n_threads) + if batched is None: + errors.append("{}: batch call failed".format(label)) + return + if compare_batch_to_reference(label, batched, reference): + errors.append("{}: result mismatch".format(label)) + return + + workers = [threading.Thread(target=hammer, args=("concurrent-{}".format(n), prediction, jobs, reference)) for n, (prediction, jobs, reference) in enumerate(sets)] + for worker in workers: + worker.start() + for worker in workers: + worker.join(timeout=120) + for worker in workers: + if worker.is_alive(): + print("ERROR: concurrent dispatch did not finish within 120s - suspect a deadlock") + return True + + if errors: + for error in errors: + print("ERROR: {}".format(error)) + return True + print("Pool concurrency: 2 threads x {} batches, all bit-identical to pk_run".format(rounds)) + return False + + +def run_pool_fork_tests(my_predbat, n_threads=4, timeout_s=60): + """Check a batch still runs in a forked child, returns True on failure. + + Threads do not survive fork(), so a child inherits a pool whose workers do not exist. Without the + pthread_atfork handler that drops the pool, the child's first batch waits forever on lanes that + will never report - a hang, not a crash, which is why this test kills the child on a deadline and + treats the timeout as the failure rather than waiting on it. + """ + print("**** Running kernel pool fork tests ****") + if not prediction_kernel.KERNEL_HAS_BATCH: + print("SKIP: kernel does not expose pk_run_batch") + return False + + if not hasattr(os, "fork"): + print("SKIP: platform has no fork()") + return False + + prediction, jobs = build_batch_jobs(my_predbat, random.Random(3690), 6) + if not prediction.kernel_handle: + print("ERROR: pool fork kernel context creation failed") + return True + reference = build_batch_reference(prediction, jobs) + + # Warm the pool in the parent, so the child inherits one whose workers are gone + if prediction_kernel.run_prediction_kernel_batch(prediction, jobs, n_threads) is None: + print("ERROR: pool fork parent warm-up batch failed") + return True + + sys.stdout.flush() + sys.stderr.flush() + pid = os.fork() + if pid == 0: + # Child: os._exit throughout, so the parent's buffers are never flushed twice + try: + batched = prediction_kernel.run_prediction_kernel_batch(prediction, jobs, n_threads) + if batched is None: + os._exit(3) + os._exit(1 if compare_batch_to_reference("fork-child", batched, reference) else 0) + except BaseException: + os._exit(2) + + deadline = time.time() + timeout_s + status = None + while time.time() < deadline: + waited, raw_status = os.waitpid(pid, os.WNOHANG) + if waited == pid: + status = raw_status + break + time.sleep(0.05) + + if status is None: + os.kill(pid, signal.SIGKILL) + os.waitpid(pid, 0) + print("ERROR: forked child did not finish within {}s - the inherited pool deadlocked".format(timeout_s)) + return True + if not os.WIFEXITED(status) or os.WEXITSTATUS(status) != 0: + print("ERROR: forked child exited with status {} (1 = mismatch, 2 = exception, 3 = batch failed)".format(status)) + return True + + print("Pool fork: a batch in a forked child matched pk_run") + return False + + def run_kernel_parity_tests(my_predbat): """Compare the C++ prediction kernel against the Python engine, returns True on failure""" print("**** Running kernel parity tests ****") @@ -889,11 +1452,28 @@ def run_kernel_parity_tests(my_predbat): state = snapshot_scenario_state(my_predbat) try: failed = run_marshalling_tests() + failed |= run_soc_percent_table_tests() failed |= run_edge_case_tests(my_predbat) if not failed: failed |= run_random_sweep_tests(my_predbat) if not failed: failed |= run_clipping_parity_tests(my_predbat) + if not failed: + failed |= run_batch_parity_tests(my_predbat) + if not failed: + failed |= run_batch_window_variant_tests(my_predbat) + if not failed: + failed |= run_scratch_reuse_tests(my_predbat) + if not failed: + failed |= run_scratch_cross_context_tests(my_predbat) + if not failed: + failed |= run_worker_reuse_tests(my_predbat) + if not failed: + failed |= run_pool_lane_tests(my_predbat) + if not failed: + failed |= run_pool_concurrency_tests(my_predbat) + if not failed: + failed |= run_pool_fork_tests(my_predbat) finally: restore_scenario_state(my_predbat, state) diff --git a/apps/predbat/tests/test_prediction_batch.py b/apps/predbat/tests/test_prediction_batch.py new file mode 100644 index 000000000..3bfd2046b --- /dev/null +++ b/apps/predbat/tests/test_prediction_batch.py @@ -0,0 +1,575 @@ +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the batched prediction fan-out. + +launch_run_prediction_* no longer runs anything: it queues a job on the Prediction and returns a +handle whose first get() flushes the whole batch through one pk_run_batch call. These tests pin what +that deferral relies on: that a trial owns its inputs rather than sharing the caller's, that a queued +job returns exactly what the direct thread_run_prediction_* path returns for every job shape and step, +that nothing runs before the first get() and one get() drains the batch, that cache hits and +intra-batch duplicates never reach the kernel while min/max jobs are never cached at all, that every +job the kernel will not take still runs on the engine the direct path would have used, and that every +handle in a multi-job batch comes back with its own job's result rather than a neighbour's. +""" + + +import random + +import prediction_batch +from const import PREDICT_STEP, PV_SCENARIO_NOMINAL, PV_SCENARIO_PV10 +from plan import resolve_batch_threads +from prediction import Prediction +from prediction_kernel import create_kernel_context +from tests.test_kernel_parity import apply_random_scenario, kernel_available, make_step_data, make_windows, restore_scenario_state, snapshot_scenario_state + + +def make_export_windows(minutes_now): + """Build a small deterministic export window layout for the trial-input tests""" + return [ + {"start": minutes_now + 60, "end": minutes_now + 120, "average": 15.0}, + {"start": minutes_now + 180, "end": minutes_now + 240, "average": 20.0}, + ] + + +def test_export_trial_does_not_mutate_caller_window(my_predbat): + """The export trial must build its own window list, returns True on failure. + + It used to write the trial start straight into the caller's window dict, which was only safe + because each pool worker mutated its own unpickled copy. A batched fan-out holds one shared list + across every job in the batch, so an in-place write would corrupt every other trial of the same + window. + """ + print("**** Running export trial input tests ****") + failed = False + minutes_now = my_predbat.minutes_now + export_window = make_export_windows(minutes_now) + export_limits = [100.0, 100.0] + original = [dict(window) for window in export_window] + + prediction = Prediction(my_predbat, {}, {}, {}, {}) + trial_window, trial_limits = prediction._prepare_export(5.0, minutes_now + 90, 0, export_window, export_limits, None) + + if export_window != original: + print("ERROR: _prepare_export mutated the caller's export window: {} vs {}".format(export_window, original)) + failed = True + if export_limits != [100.0, 100.0]: + print("ERROR: _prepare_export mutated the caller's export limits: {}".format(export_limits)) + failed = True + if trial_window[0]["start"] != minutes_now + 90: + print("ERROR: trial window start not applied, got {}".format(trial_window[0]["start"])) + failed = True + if trial_window[1] is not export_window[1]: + print("ERROR: untouched windows should be shared with the caller's list, not copied") + failed = True + if trial_limits[0] != 5.0: + print("ERROR: trial export limit not applied, got {}".format(trial_limits[0])) + failed = True + + # The trial start is clamped to at least 5 minutes before the window end + trial_window, _ = prediction._prepare_export(5.0, minutes_now + 200, 0, export_window, export_limits, None) + if trial_window[0]["start"] != minutes_now + 115: + print("ERROR: trial window start not clamped to end-5, got {}".format(trial_window[0]["start"])) + failed = True + + if not failed: + print("Export trial input tests passed") + return failed + + +def test_batch_state_exists_without_a_base(my_predbat): + """A Prediction built without a base must still be able to queue, returns True on failure. + + pending_batch and batch_threads used to be initialised only inside __init__'s `if base:` branch, + so a baseless Prediction raised AttributeError on its first enqueue. Nothing constructs one today + - the wrapped_* trampolines that did went with the process pool - which is exactly why the trap + is easy to walk back into. + """ + print("**** Running baseless Prediction tests ****") + failed = False + prediction = Prediction() + + if prediction.pending_batch != []: + print("ERROR: a baseless Prediction has pending_batch {}".format(prediction.pending_batch)) + failed = True + if prediction.batch_threads != 1: + print("ERROR: a baseless Prediction has batch_threads {}".format(prediction.batch_threads)) + failed = True + + handle = prediction.enqueue_prediction([], [], [], [], PV_SCENARIO_NOMINAL, 0, PREDICT_STEP, False) + if len(prediction.pending_batch) != 1: + print("ERROR: enqueue on a baseless Prediction left {} pending job(s)".format(len(prediction.pending_batch))) + failed = True + if handle.job.result is not None: + print("ERROR: enqueue on a baseless Prediction ran the job") + failed = True + + if not failed: + print("Baseless Prediction tests passed") + return failed + + +def make_batch_prediction(my_predbat, seed=7): + """Build a kernel-enabled Prediction plus a charge/export window layout to fan out over""" + rng = random.Random(seed) + apply_random_scenario(my_predbat, rng) + pv_step, pv10_step, load_step, load10_step = make_step_data(my_predbat, rng) + my_predbat.prediction_kernel_enable = True + prediction = Prediction(my_predbat, pv_step, pv10_step, load_step, load10_step) + prediction.kernel_handle = create_kernel_context(prediction) + charge_window = make_windows(rng, my_predbat.minutes_now, my_predbat.forecast_minutes, 4) + export_window = make_windows(rng, my_predbat.minutes_now, my_predbat.forecast_minutes, 4) + charge_limit = [round(my_predbat.soc_max / 2, 2)] * len(charge_window) + export_limits = [100.0] * len(export_window) + return prediction, charge_window, export_window, charge_limit, export_limits + + +def test_queued_matches_direct(my_predbat): + """Every queued prediction must return exactly what the direct thread path returns, returns True on failure""" + print("**** Running queued vs direct equivalence tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + try_soc = round(my_predbat.soc_max * 0.8, 2) + + cases = [ + ( + "single", + lambda: prediction.thread_run_prediction_single(charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, end_record, 5), + lambda: prediction.queue_run_prediction_single(charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, end_record, 5), + ), + ( + "charge", + lambda: prediction.thread_run_prediction_charge(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_PV10, None, end_record), + lambda: prediction.queue_run_prediction_charge(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_PV10, None, end_record), + ), + ( + "charge_min_max", + lambda: prediction.thread_run_prediction_charge_min_max(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record), + lambda: prediction.queue_run_prediction_charge_min_max(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record), + ), + ( + "charge_min_max_all_n", + lambda: prediction.thread_run_prediction_charge_min_max(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, [0, 2], end_record), + lambda: prediction.queue_run_prediction_charge_min_max(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, [0, 2], end_record), + ), + ( + "export", + lambda: prediction.thread_run_prediction_export(5.0, export_window[1]["start"] + 15, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record), + lambda: prediction.queue_run_prediction_export(5.0, export_window[1]["start"] + 15, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record), + ), + # The levels optimiser runs single predictions at a coarse "fast mode" step (plan_interval_minutes, + # 30 by default), which is the only caller that passes a step other than PREDICT_STEP. It reaches + # both the cache key and the job, so it needs its own case + ( + "single_coarse_step", + lambda: prediction.thread_run_prediction_single(charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, end_record, 30), + lambda: prediction.queue_run_prediction_single(charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, end_record, 30), + ), + ] + + for name, direct_call, queue_call in cases: + # The queued run goes first, so the direct run cannot be answered out of the cache the queued + # one just populated - and vice versa, hence the clear between them + prediction.prediction_cache = {} + queued = queue_call().get() + prediction.prediction_cache = {} + direct = direct_call() + if queued != direct: + print("ERROR: {} queued {} != direct {}".format(name, queued, direct)) + failed = True + + if not failed: + print("Queued predictions match the direct path for all {} shapes".format(len(cases))) + return failed + + +def test_batch_is_lazy(my_predbat): + """Nothing runs until the first get(), and one get() flushes the whole batch, returns True on failure""" + print("**** Running batch laziness tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + + handles = [prediction.queue_run_prediction_charge(round(soc, 2), 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) for soc in (1.0, 2.0, 3.0)] + if len(prediction.pending_batch) != 3: + print("ERROR: expected 3 pending jobs, got {}".format(len(prediction.pending_batch))) + failed = True + if any(handle.job.result is not None for handle in handles): + print("ERROR: a job ran before its handle was read") + failed = True + + handles[0].get() + if prediction.pending_batch: + print("ERROR: pending batch not drained by the first get(), {} left".format(len(prediction.pending_batch))) + failed = True + if any(handle.job.result is None for handle in handles): + print("ERROR: one get() did not flush the whole batch") + failed = True + + # A second read must not re-run anything + before = handles[2].job.result + if handles[2].get() is not before: + print("ERROR: reading a handle twice recomputed the result") + failed = True + + if not failed: + print("Batch laziness tests passed") + return failed + + +def test_batch_cache_and_dedup(my_predbat): + """Cache hits and intra-batch duplicates must never reach the kernel, returns True on failure""" + print("**** Running batch cache/dedup tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + prediction.prediction_cache_enable = True + + calls = [] + real_batch = prediction_batch.run_prediction_kernel_batch + + def spy(pred, jobs, n_threads): + """Record how many jobs each batch call was handed, then run it for real""" + calls.append(len(jobs)) + return real_batch(pred, jobs, n_threads) + + prediction_batch.run_prediction_kernel_batch = spy + try: + # Three identical jobs in one batch must collapse to one kernel job + prediction.prediction_cache = {} + handles = [prediction.queue_run_prediction_charge(4.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) for _ in range(3)] + results = [handle.get() for handle in handles] + if calls != [1]: + print("ERROR: expected one kernel job for three identical trials, got {}".format(calls)) + failed = True + if results[0] != results[1] or results[1] != results[2]: + print("ERROR: deduplicated jobs returned different results") + failed = True + + # And a repeat in a later batch must come from the cache without a kernel call at all + del calls[:] + again = prediction.queue_run_prediction_charge(4.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record).get() + if calls: + print("ERROR: a cached scenario still reached the kernel: {}".format(calls)) + failed = True + if again != results[0]: + print("ERROR: cached result {} differs from the computed one {}".format(again, results[0])) + failed = True + + # A min/max job must never be cached. The SoC range is not part of a cached result, so a hit + # would answer (0.0, 0.0), collapse the caller's SoC pruning envelope and silently drop + # candidate SoCs from the search. Checked twice: through the public entry point, and by + # asking enqueue_prediction for a cached range job directly - the second is what pins the + # guard as structural rather than a property of the one call site that passes cache=False. + prediction.prediction_cache = {} + for _ in range(2): + prediction.queue_run_prediction_charge_min_max(4.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record).get() + if prediction.prediction_cache: + print("ERROR: a queued min/max job populated the prediction cache: {}".format(len(prediction.prediction_cache))) + failed = True + + range_handle = prediction.enqueue_prediction(list(charge_limit), charge_window, export_window, list(export_limits), PV_SCENARIO_NOMINAL, end_record, PREDICT_STEP, True, want_range=True, range_window=charge_window[1]) + if range_handle.job.sim_hash is not None: + print("ERROR: a range job was given a cache key despite never being cacheable") + failed = True + range_handle.get() + if prediction.prediction_cache: + print("ERROR: a range job enqueued with cache=True populated the prediction cache: {}".format(len(prediction.prediction_cache))) + failed = True + finally: + prediction_batch.run_prediction_kernel_batch = real_batch + + if not failed: + print("Batch cache/dedup tests passed") + return failed + + +def test_queued_range_window_in_the_past(my_predbat): + """A range window that started before minutes_now must give the direct path's answer, returns True on failure. + + make_windows only ever emits windows starting at or after minutes_now, so the max(..., 0) clamp on + the range start step and the negative end step of a wholly-past window are otherwise never + executed. Both matter: the kernel derives the range from those two step numbers while a fallback + derives it from scan_soc_range, and the two have to agree exactly. + """ + print("**** Running past range window tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + minutes_now = my_predbat.minutes_now + window = charge_window[1] + + cases = [ + # Straddles minutes_now: the start step clamps to 0, the end step is positive + ("straddling", {"start": minutes_now - 120, "end": window["end"], "average": window["average"]}), + # Wholly behind minutes_now: both steps are negative, so no step is ever in range + ("wholly past", {"start": minutes_now - 240, "end": minutes_now - 60, "average": window["average"]}), + # Ends exactly at minutes_now, the boundary between the two cases above + ("ends at now", {"start": minutes_now - 180, "end": minutes_now, "average": window["average"]}), + ] + + for name, past_window in cases: + trial_window = list(charge_window) + trial_window[1] = past_window + prediction.prediction_cache = {} + queued = prediction.queue_run_prediction_charge_min_max(3.0, 1, charge_limit, trial_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record).get() + prediction.prediction_cache = {} + direct = prediction.thread_run_prediction_charge_min_max(3.0, 1, charge_limit, trial_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) + if queued != direct: + print("ERROR: {} range window queued {} != direct {}".format(name, queued, direct)) + failed = True + + if not failed: + print("Past range window tests passed for all three shapes") + return failed + + +def test_batch_fallbacks(my_predbat): + """A job the kernel cannot take still runs, and gives the Python engine's answer, returns True on failure""" + print("**** Running batch fallback tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + + # Two references, because the fallbacks land on two different engines. A batch the kernel refused + # still re-runs through run_prediction, which dispatches to the kernel; a Prediction the kernel + # will not serve at all re-runs on the Python engine. Those two agree to 1e-6 (kernel_parity), not + # bit-exactly, so each fallback is compared against the engine it actually used. + prediction.prediction_cache = {} + expected = prediction.thread_run_prediction_charge_min_max(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) + prediction.debug_enable = True + prediction.prediction_cache = {} + expected_python = prediction.thread_run_prediction_charge_min_max(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) + prediction.debug_enable = False + + # 1. Whole batch refused (no kernel, stale binary, or a non-zero return code) + real_batch = prediction_batch.run_prediction_kernel_batch + + def refuse_all(pred, jobs, n_threads): + """Stand in for a kernel that cannot run the batch at all""" + return None + + def refuse_each(pred, jobs, n_threads): + """Stand in for a kernel that ran the batch but rejected every job""" + return [(None, 0.0, 0.0)] * len(jobs) + + for name, stub in (("whole batch", refuse_all), ("per job", refuse_each)): + prediction_batch.run_prediction_kernel_batch = stub + try: + prediction.prediction_cache = {} + got = prediction.queue_run_prediction_charge_min_max(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record).get() + if got != expected: + print("ERROR: {} fallback returned {} expected {}".format(name, got, expected)) + failed = True + finally: + prediction_batch.run_prediction_kernel_batch = real_batch + + # 2. Debug runs must not reach the kernel from the batch when the direct path would not take it. + # This has to assert the mechanism, not the value: the two engines agree to ~1e-6 and every field + # is rounded to 4dp, so the tuples match whether or not the kernel_supported() gate exists. The + # gate is load-bearing because a debug run needs predict_soc_best and friends populated for the + # debug HTML, and reset_kernel_run_state clears exactly those. + calls = [] + + def spy_batch(pred, jobs, n_threads): + """Record that the kernel batch was reached at all, then run it for real""" + calls.append(len(jobs)) + return real_batch(pred, jobs, n_threads) + + prediction.debug_enable = True + prediction_batch.run_prediction_kernel_batch = spy_batch + try: + prediction.prediction_cache = {} + got = prediction.queue_run_prediction_charge_min_max(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record).get() + if calls: + print("ERROR: a debug run reached the kernel batch: {}".format(calls)) + failed = True + if got != expected_python: + print("ERROR: debug_enable fallback returned {} expected {}".format(got, expected_python)) + failed = True + finally: + prediction_batch.run_prediction_kernel_batch = real_batch + prediction.debug_enable = False + + # 3. Kernel genuinely unavailable for this Prediction + prediction.kernel_handle = 0 + prediction.prediction_cache = {} + got = prediction.queue_run_prediction_charge_min_max(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record).get() + if got != expected_python: + print("ERROR: kernel-less fallback returned {} expected {}".format(got, expected_python)) + failed = True + + if not failed: + print("Batch fallback tests passed") + return failed + + +def test_save_run_drains_pending_batch(my_predbat): + """A saving run must flush any pending batch before it runs, returns True on failure. + + A save run is the only one that populates predict_soc_best and friends, and it goes down the + Python engine because kernel_supported() refuses save. If a batch were left pending across it, + the next handle read would flush it and reset_kernel_run_state would blank exactly those + attributes - the published plan and the debug HTML would come back empty. All four fan-outs drain + their handles before any save run today, so this pins the guard, not a live bug: it is what stops + a fifth fan-out from introducing one. + """ + print("**** Running save-run batch drain tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + + handle = prediction.queue_run_prediction_charge(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) + if handle.job.result is not None: + print("ERROR: the queued job ran before the save run") + failed = True + + prediction.run_prediction(charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, end_record=end_record, save="best") + + if prediction.pending_batch: + print("ERROR: the save run left {} job(s) pending".format(len(prediction.pending_batch))) + failed = True + if handle.job.result is None: + print("ERROR: the save run did not flush the pending job") + failed = True + if not prediction.predict_soc_best: + print("ERROR: the save run published no predict_soc_best, so this test cannot see it being blanked") + failed = True + + # The point of the guard: reading the handle afterwards must not wipe what the save run published + published = dict(prediction.predict_soc_best) + handle.get() + if prediction.predict_soc_best != published: + print("ERROR: reading a handle after a save run changed predict_soc_best ({} entries, was {})".format(len(prediction.predict_soc_best), len(published))) + failed = True + + if not failed: + print("Save-run batch drain tests passed") + return failed + + +def test_batch_results_match_their_own_job(my_predbat): + """Every handle in a multi-job batch must get its own job's result, at any thread count, returns True on failure. + + The oracle is external, and it has to be: flush_batch resolves cache hits and intra-batch + duplicates before the kernel call and then walks the surviving jobs alongside the kernel's + results, so an off-by-one or reversed job->result mapping is easy to introduce and returns a + perfectly plausible cost for every trial. Comparing a batch against another batch cannot see + that - both would be wrong the same way - and neither can sweeping n_threads, which is forwarded + verbatim to C with nothing on the Python side varying with it. + + So each handle is compared against thread_run_prediction_charge for the same try_soc, with the + prediction cache cleared between the two runs so neither can be answered out of what the other + stored. The SoCs are checked to give genuinely distinct costs first, otherwise a misrouted result + could match by coincidence. + + The thread sweep rides along: pk_run_batch is pinned bit-identical across thread counts at the C + level (kernel_parity), so this pins that the Python side routes results the same way however the + kernel split the work. + """ + print("**** Running batch result routing tests ****") + failed = False + # Seed 5 rather than the shared default: this test needs ten trials with ten different costs, and + # the default scenario's battery saturates so that nine of the ten come out identical + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat, seed=5) + end_record = my_predbat.forecast_minutes + socs = [round(my_predbat.soc_max * fraction, 2) for fraction in (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0)] + # Every charge window takes the trial SoC, not just one: moving a single window's limit saturates + # once that window can no longer absorb more, which collapses the costs back together + all_n = list(range(len(charge_window))) + + # The oracle: one direct run per SoC, each from a cold cache so it is genuinely simulated + expected = [] + for soc in socs: + prediction.prediction_cache = {} + expected.append(prediction.thread_run_prediction_charge(soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, all_n, end_record)) + + costs = [result[0] for result in expected] + if len(set(costs)) != len(costs): + print("ERROR: the trial SoCs do not all give distinct costs ({}) - a misrouted result could match by luck".format(costs)) + failed = True + + for n_threads in (1, 2, 8): + prediction.batch_threads = n_threads + prediction.prediction_cache = {} + handles = [prediction.queue_run_prediction_charge(soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, all_n, end_record) for soc in socs] + results = [handle.get() for handle in handles] + for index, (got, want) in enumerate(zip(results, expected)): + if got != want: + print("ERROR: job {} (try_soc {}) at {} threads returned {} expected {}".format(index, socs[index], n_threads, got, want)) + failed = True + prediction.batch_threads = 1 + + if not failed: + print("Batch result routing tests passed - {} jobs matched their own direct run at 1, 2 and 8 threads".format(len(socs))) + return failed + + +def test_batch_thread_count_resolution(): + """Check how the threads setting maps onto kernel lanes, returns True on failure. + + 'auto' takes the core count uncapped. A cap was measured and rejected: on a fast machine the + curve is flat and peaks slightly below the core count, but with each job made eight times dearer + - how a machine where the kernel dominates behaves - it stops turning over, and a cap of 4 then + costs 10.7% against 1.3% for no cap at all. The zero and one cases both mean serial, which is what + annual.py and the benchmark harness rely on. + """ + print("**** Running batch thread count resolution tests ****") + failed = False + cases = [ + ("auto", 2, 2, "auto follows the core count"), + ("auto", 16, 16, "auto is not capped"), + ("auto", 64, 64, "auto is not capped on a large machine either"), + ("auto", 0, 1, "a nonsense core count still leaves one usable lane"), + (0, 32, 1, "0 means serial"), + (1, 32, 1, "1 means serial"), + (2, 32, 2, "an explicit value is honoured"), + (12, 32, 12, "an explicit value above the core count is honoured"), + ] + for threads, cores, expected, why in cases: + got = resolve_batch_threads(threads, cores) + if got != expected: + print("ERROR: threads={} cores={} gave {} lanes, expected {} ({})".format(threads, cores, got, expected, why)) + failed = True + + if not failed: + print("Batch thread count resolution tests passed") + return failed + + +def run_prediction_batch_tests(my_predbat): + """Run every batched prediction test, returns True on failure""" + failed = test_export_trial_does_not_mutate_caller_window(my_predbat) + failed |= test_batch_thread_count_resolution() + failed |= test_batch_state_exists_without_a_base(my_predbat) + + available, required_failure = kernel_available() + if not available: + print("WARNING: kernel not available - batch tests that need it are SKIPPED") + return failed or required_failure + + state = snapshot_scenario_state(my_predbat) + try: + failed |= test_queued_matches_direct(my_predbat) + failed |= test_queued_range_window_in_the_past(my_predbat) + failed |= test_batch_is_lazy(my_predbat) + failed |= test_batch_cache_and_dedup(my_predbat) + failed |= test_batch_fallbacks(my_predbat) + failed |= test_save_run_drains_pending_batch(my_predbat) + failed |= test_batch_results_match_their_own_job(my_predbat) + finally: + restore_scenario_state(my_predbat, state) + my_predbat.prediction_kernel_enable = False + + if failed: + print("**** Prediction batch tests FAILED ****") + else: + print("**** Prediction batch tests passed ****") + return failed diff --git a/apps/predbat/tests/test_pv90.py b/apps/predbat/tests/test_pv90.py index bf61b75e3..3844e2dc9 100644 --- a/apps/predbat/tests/test_pv90.py +++ b/apps/predbat/tests/test_pv90.py @@ -10,7 +10,6 @@ """Tests for the pv90 upside forecast scenario.""" from const import PV_SCENARIO_NOMINAL, PV_SCENARIO_PV10, PV_SCENARIO_PV90 -from plan import DummyThread from tests.test_infra import reset_inverter, reset_rates @@ -785,8 +784,8 @@ def _setup_calculate_plan_with_real_windows(my_predbat): window in each direction to search, so the counting assertions below are actually exercised. threads is forced to 0 - the same idiom already used by test_execute.py, test_random_scenarios.py and - test_single_debug.py - so calculate_plan() takes the synchronous DummyThread fallback instead of - spinning up a real multiprocessing Pool, keeping the test fast and deterministic. + test_single_debug.py - so calculate_plan() sizes the kernel's batch thread pool to a single thread + rather than the 'auto' cpu_count() default, keeping the test fast and deterministic. Returns a snapshot dict for _restore_calculate_plan_with_real_windows. """ @@ -1103,6 +1102,22 @@ def counting(loop_soc, window_n, charge_limit, charge_window, export_window, exp return failed +class FakeHandle: + """Minimal result-handle double with a get(), standing in for a real BatchHandle. + + Used to substitute an encoded result into a launch_run_prediction_* return value without routing + it back through the kernel/batch machinery. + """ + + def __init__(self, result): + """Store the pre-computed result tuple.""" + self.result = result + + def get(self): + """Return the stored result.""" + return self.result + + def test_pv90_charge_limit_results_paired_with_try_soc(my_predbat): """optimise_charge_limit's pv90 results must stay paired with the try_soc that produced them. @@ -1149,7 +1164,7 @@ def encoding_launch(loop_soc, window_n, charge_limit, charge_window, export_wind if pv_scenario in encoded_scenarios: encoded_result = list(handle.get()) encoded_result[0] = loop_soc - handle = DummyThread(tuple(encoded_result)) + handle = FakeHandle(tuple(encoded_result)) return handle original_launch_min_max = my_predbat.launch_run_prediction_charge_min_max @@ -1160,7 +1175,7 @@ def encoding_launch_min_max(loop_soc, window_n, charge_limit, charge_window, exp if pv_scenario in encoded_scenarios: encoded_result = list(handle.get()) encoded_result[0] = loop_soc - handle = DummyThread(tuple(encoded_result)) + handle = FakeHandle(tuple(encoded_result)) return handle original_optimise_charge_limit = my_predbat.optimise_charge_limit diff --git a/apps/predbat/unit_test.py b/apps/predbat/unit_test.py index 16fac4fd8..03973c7f5 100644 --- a/apps/predbat/unit_test.py +++ b/apps/predbat/unit_test.py @@ -23,6 +23,7 @@ from tests.test_model import run_model_tests from tests.test_predict_pv_power import run_predict_pv_power_tests from tests.test_kernel_parity import run_kernel_parity_tests, run_model_kernel_tests +from tests.test_prediction_batch import run_prediction_batch_tests from tests.test_execute import run_execute_tests from tests.test_load_car_energy import test_load_car_energy_warns_when_configured_entity_has_no_data from tests.test_octopus_slots import run_load_octopus_slots_tests @@ -297,6 +298,7 @@ def main(): ("predict_pv_power", run_predict_pv_power_tests, "predict_pv_power plan-interval scaling tests", False), ("model_kernel", run_model_kernel_tests, "Model tests run with the C++ prediction kernel enabled", False), ("kernel_parity", run_kernel_parity_tests, "C++ prediction kernel vs Python engine parity tests", False), + ("prediction_batch", run_prediction_batch_tests, "Batched prediction fan-out tests", False), ("inverter", run_inverter_tests, "Inverter tests", False), ("execute", run_execute_tests, "Execute tests", False), ("load_car_energy", test_load_car_energy_warns_when_configured_entity_has_no_data, "car_charging_energy configured-but-empty warning tests (#4458 follow-up)", False), diff --git a/apps/predbat/userinterface.py b/apps/predbat/userinterface.py index 236de5dce..1cbacc694 100644 --- a/apps/predbat/userinterface.py +++ b/apps/predbat/userinterface.py @@ -33,7 +33,6 @@ from predbat import THIS_VERSION DEBUG_EXCLUDE_LIST = [ - "pool", "ha_interface", "components", "prediction", diff --git a/docs/apps-yaml.md b/docs/apps-yaml.md index 40dc6e55b..770842dd3 100644 --- a/docs/apps-yaml.md +++ b/docs/apps-yaml.md @@ -286,10 +286,14 @@ This will remove the need for a DNS lookup of the IP address every time Predbat If defined sets the number of threads to use during plan calculation, the default is 'auto' which will use the same number of threads as you have CPUs in your system. +Predbat batches each group of simulations into a single call to the C++ prediction kernel, which then +spreads them across this many threads. Results do not depend on the thread count - the same plan is +produced at any setting - so this only trades CPU for planning time. + Valid values are: - 'auto' - Use the same number of threads as your CPU count -- '0' - Don't use threads - disabled +- '0' - Clamped up to a single kernel thread, i.e. each batch is simulated serially - 'N' - Use N threads, recommended values are between 2 and 8 ```yaml diff --git a/docs/superpowers/plans/2026-08-15-phase-b-batch-prediction.md b/docs/superpowers/plans/2026-08-15-phase-b-batch-prediction.md new file mode 100644 index 000000000..216198f37 --- /dev/null +++ b/docs/superpowers/plans/2026-08-15-phase-b-batch-prediction.md @@ -0,0 +1,1648 @@ +# Phase B: Batch Prediction Fan-Out Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `launch_run_prediction_*` queue work into a lazily-flushed batch that reaches the C++ kernel through one `pk_run_batch` call per fan-out, so the per-call Python overhead is paid once per batch and the kernel's threading can finally be used. + +**Architecture:** The four `launch_run_prediction_*` call sites in `plan.py` are already launch-all-then-collect-all, so they do not change shape. `launch_*` stops running anything: it prepares the trial inputs, appends a `BatchJob` to a pending list on the `Prediction`, and returns a `BatchHandle`. The first `BatchHandle.get()` flushes the whole pending batch — prediction-cache hits are resolved in Python, the misses are marshalled into one `PkBatchJob` array and run by a single `pk_run_batch` call (optionally multi-threaded inside C++), and each job's result is then shaped into exactly what its `thread_run_prediction_*` used to return. Anything the kernel cannot take falls back to the existing Python path, per job. + +**Tech Stack:** Python 3 (ctypes), C++ prediction kernel (`prediction_kernel.cpp`, ABI 3, parity revision 5, already built and committed), predbat's own `unit_test.py` harness. + +**Spec:** [PHASE_B_BATCH_REFACTOR.md](../../../PHASE_B_BATCH_REFACTOR.md) — the working brief this plan implements. Read it first; it carries the measurements, the traps and the dead ends that justify the design below. + +## Global Constraints + +- **Branch:** `perf/kernel-batch-abi` (4 commits, unpushed, based on `main`). All work continues on this branch. +- **No C++ changes.** The kernel side is done, verified and committed. If you find yourself editing `prediction_kernel.cpp`, stop — that means the design has drifted. (The one exception would be a genuine kernel bug, which then needs `PK_PARITY_REVISION` **and** `KERNEL_PARITY_REVISION` bumped and all 6 shipped binaries rebuilt.) +- **Byte-identical plans are the gate.** Every task that can change results must end with the 20-scenario comparison showing `20 unchanged, +0.0000` on metric, cost and all three PV futures. +- **Line length:** 256 chars (Black), 250 (Flake8). +- **Docstrings:** 100% coverage required (`interrogate`) — every function, method and class, including `__init__`. +- **Spelling:** British English (`en-gb`) via CSpell. New words go in `.cspell/custom-dictionary-workspace.txt`, which is auto-sorted on commit — re-stage after running pre-commit. +- **Naming:** `lower_case_with_underscores`. +- **Unit tests are mandatory for all new code** (CLAUDE.md), registered in `TEST_REGISTRY` in `apps/predbat/unit_test.py`. +- **Test output goes to a file, then you grep the file.** Never pipe a test run straight into grep — a wrong pattern means running it all again. +- **`cd` does not persist between Bash tool calls.** Every command below is written to be pasted whole. +- **Do not `git checkout` between branches with uncommitted work** — it carries the changes across. Commit first. + +## Invariant this design depends on + +**No caller may mutate a list or window dict it has passed to `launch_run_prediction_*` between that call and the matching `.get()`.** + +Today every launch either runs immediately (serial path) or pickles its arguments immediately (pool path), so this invariant did not exist. Batching defers the read to flush time, so it does now. All four call sites were read and already satisfy it, *after* Task 1 removes the one violation: + +| call site | fan-out | why it is safe | +|---|---|---| +| `plan.py:528-530` (`optimise_levels`) | `launch_run_prediction_single` ×3 | `try_charge_limit`/`try_export` are freshly materialised per scenario and never written again | +| `plan.py:1781-1799` (`optimise_charge_limit`) | `launch_run_prediction_charge_min_max` ×4-6 | `charge_limit`/`export_limits` are read-only across the block | +| `plan.py:1950-1955` (`optimise_charge_limit`) | `launch_run_prediction_charge` ×2-3 per SoC | `try_charge_limit` is a separate list, only written in the results loop *after* every `.get()` | +| `plan.py:2183-2186` (`optimise_export`) | `launch_run_prediction_export` ×2-3 per option | **violates it today** via `export_window[window_n]["start"] = start` — Task 1 fixes this | + +The 20-scenario byte-identical gate is what enforces the invariant in practice: any violation shows up as a changed plan immediately. + +## File Structure + +| File | Responsibility | Change | +|---|---|---| +| `apps/predbat/prediction_kernel.py` | ctypes/ABI layer only. Gains `BatchJob` (the queued-job record), `run_prediction_kernel_batch()` and `kernel_result_tuple()`; `pk_run_batch`'s argtypes get their missing `n_threads`. | Modify | +| `apps/predbat/prediction_batch.py` | **New.** `PredictionBatch` mixin (pending list, enqueue, flush, per-job Python fallback), `BatchHandle`, `prediction_cache_key()`. Imports `prediction_kernel` and `const` only — never `prediction`, so there is no import cycle. | Create | +| `apps/predbat/prediction.py` | `Prediction` gains the `PredictionBatch` mixin, the `_prepare_*` input builders shared by the direct and batch paths, `scan_soc_range()`, and the four `queue_run_prediction_*` entry points. | Modify | +| `apps/predbat/plan.py` | The four `launch_run_prediction_*` become one-line delegations to `queue_run_prediction_*`; the process pool goes away. | Modify | +| `apps/predbat/tests/test_kernel_parity.py` | Gains `run_batch_parity_tests()` — `pk_run_batch` against a loop of `pk_run`, at several thread counts. | Modify | +| `apps/predbat/tests/test_prediction_batch.py` | **New.** The batch runner's behaviour: equivalence with the direct path for all four kinds, laziness, dedup, cache hits, and both fallback routes. | Create | +| `apps/predbat/unit_test.py` | Register `prediction_batch` in `TEST_REGISTRY`. | Modify | +| `docs/apps-yaml.md` | `threads` changes meaning: kernel threads, not worker processes. | Modify | + +--- + +### Task 1: Stop the export trial mutating the caller's window + +**Files:** +- Modify: `apps/predbat/prediction.py:358-373` (`thread_run_prediction_export`) +- Test: `apps/predbat/tests/test_prediction_batch.py` (created here, one test; the rest of the file lands in Task 4) +- Modify: `apps/predbat/unit_test.py` (register the new test) + +**Interfaces:** +- Consumes: nothing from earlier tasks. +- Produces: `thread_run_prediction_export` no longer writes to `export_window[window_n]["start"]`. Task 3 splits this same function, and Task 4 relies on the window list being safe to hold across a deferred flush. + +This is the fix from `perf/threadpool-prototype` (`263f8178`, marked do-not-merge), ported across. It was only ever safe because each pool worker mutated its own unpickled copy. + +- [ ] **Step 1: Write the failing test** + +Create `apps/predbat/tests/test_prediction_batch.py`: + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Tests for the batched prediction fan-out. + +launch_run_prediction_* no longer runs anything: it queues a job on the Prediction and returns a +handle whose first get() flushes the whole batch through one pk_run_batch call. These tests pin the +two things that makes conditional - that a queued job returns exactly what the direct +thread_run_prediction_* path returns, and that a job the kernel cannot take still runs. +""" + +import random + +from prediction import Prediction + + +def make_export_windows(minutes_now): + """Build a small deterministic export window layout for the trial-input tests""" + return [ + {"start": minutes_now + 60, "end": minutes_now + 120, "average": 15.0}, + {"start": minutes_now + 180, "end": minutes_now + 240, "average": 20.0}, + ] + + +def test_export_trial_does_not_mutate_caller_window(my_predbat): + """The export trial must build its own window list, returns True on failure. + + It used to write the trial start straight into the caller's window dict, which was only safe + because each pool worker mutated its own unpickled copy. A batched fan-out holds one shared list + across every job in the batch, so an in-place write would corrupt every other trial of the same + window. + """ + print("**** Running export trial input tests ****") + failed = False + minutes_now = my_predbat.minutes_now + export_window = make_export_windows(minutes_now) + export_limits = [100.0, 100.0] + original = [dict(window) for window in export_window] + + prediction = Prediction(my_predbat, {}, {}, {}, {}) + trial_window, trial_limits = prediction._prepare_export(5.0, minutes_now + 90, 0, export_window, export_limits, None) + + if export_window != original: + print("ERROR: _prepare_export mutated the caller's export window: {} vs {}".format(export_window, original)) + failed = True + if export_limits != [100.0, 100.0]: + print("ERROR: _prepare_export mutated the caller's export limits: {}".format(export_limits)) + failed = True + if trial_window[0]["start"] != minutes_now + 90: + print("ERROR: trial window start not applied, got {}".format(trial_window[0]["start"])) + failed = True + if trial_window[1] is not export_window[1]: + print("ERROR: untouched windows should be shared with the caller's list, not copied") + failed = True + if trial_limits[0] != 5.0: + print("ERROR: trial export limit not applied, got {}".format(trial_limits[0])) + failed = True + + # The trial start is clamped to at least 5 minutes before the window end + trial_window, _ = prediction._prepare_export(5.0, minutes_now + 200, 0, export_window, export_limits, None) + if trial_window[0]["start"] != minutes_now + 115: + print("ERROR: trial window start not clamped to end-5, got {}".format(trial_window[0]["start"])) + failed = True + + if not failed: + print("Export trial input tests passed") + return failed + + +def run_prediction_batch_tests(my_predbat): + """Run every batched prediction test, returns True on failure""" + failed = test_export_trial_does_not_mutate_caller_window(my_predbat) + if failed: + print("**** Prediction batch tests FAILED ****") + else: + print("**** Prediction batch tests passed ****") + return failed +``` + +The test is written against `_prepare_export`, which does not exist yet — Step 4 of this task creates it, because extracting the helper *is* the fix: it is the only way to build the trial window without touching the caller's. Task 3 later joins the other three `thread_run_prediction_*` to the same pattern. + +- [ ] **Step 2: Register the test** + +In `apps/predbat/unit_test.py`, add the import next to the other test imports (they are alphabetical-ish, grouped by module): + +```python +from tests.test_prediction_batch import run_prediction_batch_tests +``` + +and add the registry entry to `TEST_REGISTRY`, next to the other prediction entries: + +```python + ("prediction_batch", run_prediction_batch_tests, "Batched prediction fan-out tests", False), +``` + +- [ ] **Step 3: Run the test to verify it fails** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all --test prediction_batch > /tmp/predbat-task1.log 2>&1; grep -nE "ERROR|FAIL|PASS|Traceback|AttributeError" /tmp/predbat-task1.log | head -20 +``` + +Expected: `AttributeError: 'Prediction' object has no attribute '_prepare_export'`. + +- [ ] **Step 4: Add `_prepare_export` and route the direct path through it** + +In `apps/predbat/prediction.py`, add the helper immediately above `thread_run_prediction_export`: + +```python + def _prepare_export(self, this_export_limit, start, window_n, export_window, export_limits, all_n): + """Build the trial export limits and window list - shared by thread_run_prediction_export and the batch path. + + The trial start is applied to a private copy of the window rather than written into the + caller's list: with a process pool each worker mutated its own unpickled copy, but a batched + fan-out shares one list across every job in the batch, so an in-place write would corrupt the + other trials of the same window. Only ["end"] is ever read back by the caller + (optimise_export), so nothing depends on the write being visible. + """ + export_limits = export_limits.copy() + + if all_n: + for window_id in all_n: + export_limits[window_id] = this_export_limit + else: + export_limits[window_n] = this_export_limit + # Adjust start + window = export_window[window_n] + start = min(start, window["end"] - 5) + export_window = list(export_window) + export_window[window_n] = dict(window, start=start) + + return export_window, export_limits +``` + +and replace the head of `thread_run_prediction_export` (everything from `# Store try value into the window` down to the line before the `(` of the result unpack) with: + +```python + export_window, export_limits = self._prepare_export(this_export_limit, start, window_n, export_window, export_limits, all_n) +``` + +- [ ] **Step 5: Run the test to verify it passes** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all --test prediction_batch > /tmp/predbat-task1.log 2>&1; grep -nE "ERROR|FAIL|passed|Traceback" /tmp/predbat-task1.log | head -20 +``` + +Expected: `Export trial input tests passed`, no ERROR lines. + +- [ ] **Step 6: Verify plans are byte-identical** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && python3 ../apps/predbat/unit_test.py --random-run --random-template cases/predbat_debug_agile1.yaml --random-scenarios cases/random_scenarios.yaml > /tmp/predbat-task1-run.log 2>&1; python3 ../apps/predbat/unit_test.py --random-compare cases/random_results.json random_results.json > /tmp/predbat-task1-cmp.log 2>&1; tail -30 /tmp/predbat-task1-cmp.log +``` + +Expected: 20 scenarios unchanged, `+0.0000` on metric, cost, and all three PV futures. Anything else means the mutation *was* being depended on — stop and investigate before continuing. + +- [ ] **Step 7: Run the full suite** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all > /tmp/predbat-task1-all.log 2>&1; grep -nE "FAILED|ERROR|Traceback|All tests passed|tests passed" /tmp/predbat-task1-all.log | tail -20 +``` + +- [ ] **Step 8: Commit** + +```bash +cd /Users/treforsouthwell/predbat/batpred && ./run_pre_commit && git add apps/predbat/prediction.py apps/predbat/tests/test_prediction_batch.py apps/predbat/unit_test.py && git commit -m "fix(prediction): build the export trial window privately instead of mutating the caller's + +The trial start was written straight into the caller's window dict, which was only safe because each +pool worker mutated its own unpickled copy. A batched fan-out shares one window list across every job +in the batch, so this has to be local. Ported from perf/threadpool-prototype. + +Co-Authored-By: Claude Opus 5 (1M context) " +``` + +(If pre-commit rewrites the cspell dictionary, `git add` it and re-run the commit.) + +--- + +### Task 2: Batch entry point in the ctypes layer + +**Files:** +- Modify: `apps/predbat/prediction_kernel.py:287-292` (argtypes), `:596-629` (result assembly) +- Test: `apps/predbat/tests/test_kernel_parity.py` + +**Interfaces:** +- Consumes: `PkBatchJob`, `PkBatchResult`, `KERNEL_HAS_BATCH` (already present on this branch). +- Produces: + - `class BatchJob` — the queued-job record, with `__slots__` `("charge_limit", "charge_window", "export_window", "export_limits", "pv_scenario", "end_record", "step", "soc_range_start_step", "soc_range_end_step", "sim_hash", "want_range", "range_window", "result")`. + - `kernel_result_tuple(pred, result, predict_soc) -> tuple` — the 17-field `run_prediction()` result tuple built from a `PkResult`. + - `run_prediction_kernel_batch(pred, jobs, n_threads=1) -> list | None` — returns one `(result_tuple_or_None, soc_range_min, soc_range_max)` per job in job order, or `None` if the whole batch could not run. + +**The bug this task fixes first:** `pk_run_batch` is declared in C++ as +`int32_t pk_run_batch(int64_t handle, const PkBatchJob *jobs, int32_t n_jobs, PkBatchResult *results, int32_t n_threads)` +but `prediction_kernel.py:289` gives it only four `argtypes`. Nothing calls it yet so nothing has broken, but a call built against that binding hands the C++ side an undefined `n_threads`. + +- [ ] **Step 1: Write the failing test** + +Add to `apps/predbat/tests/test_kernel_parity.py`: + +```python +def build_batch_jobs(my_predbat, rng, count): + """Build a list of BatchJob scenarios plus the Prediction they run against""" + apply_random_scenario(my_predbat, rng) + pv_step, pv10_step, load_step, load10_step = make_step_data(my_predbat, rng) + my_predbat.prediction_kernel_enable = True + prediction = Prediction(my_predbat, pv_step, pv10_step, load_step, load10_step) + prediction.kernel_handle = create_kernel_context(prediction) + + charge_window = make_windows(rng, my_predbat.minutes_now, my_predbat.forecast_minutes, rng.randint(2, 5)) + export_window = make_windows(rng, my_predbat.minutes_now, my_predbat.forecast_minutes, rng.randint(2, 5)) + end_record = my_predbat.forecast_minutes + + jobs = [] + for index in range(count): + charge_limit = [round(rng.uniform(0, my_predbat.soc_max), 2) for _ in charge_window] + export_limits = [rng.choice([100.0, 99.0, 0.0, round(rng.uniform(1, 99), 1)]) for _ in export_window] + pv_scenario = rng.choice([PV_SCENARIO_NOMINAL, PV_SCENARIO_PV10, PV_SCENARIO_PV90]) + # Every third job asks for the SoC range over a charge window, as the min/max fan-out does + range_window = charge_window[index % len(charge_window)] if (index % 3) == 0 else None + start_step = -1 + end_step = -1 + if range_window is not None: + start_step = max(int((range_window["start"] - my_predbat.minutes_now) / 5) * 5, 0) // 5 + end_step = int((range_window["end"] - my_predbat.minutes_now) / 5) * 5 // 5 + jobs.append( + prediction_kernel.BatchJob( + charge_limit, + charge_window, + export_window, + export_limits, + pv_scenario, + end_record, + 5, + soc_range_start_step=start_step, + soc_range_end_step=end_step, + want_range=range_window is not None, + range_window=range_window, + ) + ) + return prediction, jobs + + +def run_batch_parity_tests(my_predbat, count=60): + """Check pk_run_batch matches a loop of pk_run exactly, at every thread count, returns True on failure. + + The batch path is the only one Python will use after this refactor, and it is the only one that + can run scenarios concurrently, so it is checked against the single-scenario path it replaces + rather than against the Python engine (which run_random_sweep_tests already pins pk_run against). + Results must be bit-identical, not merely close: the whole point of the fan-out is that plans do + not change. + """ + print("**** Running kernel batch parity tests ****") + if not prediction_kernel.KERNEL_HAS_BATCH: + print("SKIP: kernel does not expose pk_run_batch") + return False + + failed = False + prediction, jobs = build_batch_jobs(my_predbat, random.Random(4321), count) + if not prediction.kernel_handle: + print("ERROR: batch parity kernel context creation failed") + return True + + # Reference: one pk_run per scenario, with the SoC series materialised so the range can be + # scanned in Python exactly as thread_run_prediction_charge_min_max does + reference = [] + for job in jobs: + single = run_prediction_kernel(prediction, job.charge_limit, job.charge_window, job.export_window, job.export_limits, job.pv_scenario, job.end_record, 5, False) + if single is None: + print("ERROR: reference pk_run failed") + return True + if job.range_window is None: + soc_range = (prediction.soc_max, 0) + else: + soc_range = prediction.scan_soc_range(single[11], job.range_window) + reference.append((single, soc_range)) + + for n_threads in (1, 2, 4, 8): + batched = prediction_kernel.run_prediction_kernel_batch(prediction, jobs, n_threads) + if batched is None: + print("ERROR: pk_run_batch failed at {} threads".format(n_threads)) + return True + if len(batched) != len(jobs): + print("ERROR: pk_run_batch returned {} results for {} jobs".format(len(batched), len(jobs))) + return True + for index, (result_tuple, soc_range_min, soc_range_max) in enumerate(batched): + single, (expect_min, expect_max) = reference[index] + if result_tuple is None: + print("ERROR: job {} reported a non-zero status at {} threads".format(index, n_threads)) + failed = True + continue + for field, name in enumerate(RESULT_NAMES): + if result_tuple[field] != single[field]: + print("ERROR: job {} at {} threads differs on {}: batch {} single {}".format(index, n_threads, name, result_tuple[field], single[field])) + failed = True + # The batch never materialises the SoC series - that is what makes it affordable + if result_tuple[11] != {}: + print("ERROR: job {} returned a SoC series from the batch path".format(index)) + failed = True + for field, name in [(12, "car_charging_soc_next"), (13, "iboost_next"), (14, "iboost_running"), (15, "iboost_running_solar"), (16, "iboost_running_full")]: + if result_tuple[field] != single[field]: + print("ERROR: job {} at {} threads differs on {}: batch {} single {}".format(index, n_threads, name, result_tuple[field], single[field])) + failed = True + if soc_range_min != expect_min or soc_range_max != expect_max: + print("ERROR: job {} at {} threads SoC range {} {} expected {} {}".format(index, n_threads, soc_range_min, soc_range_max, expect_min, expect_max)) + failed = True + + if not failed: + print("Batch parity: {} scenarios bit-identical to pk_run at 1, 2, 4 and 8 threads".format(count)) + return failed +``` + +and call it from `run_kernel_parity_tests`, after the clipping sweep: + +```python + if not failed: + failed |= run_batch_parity_tests(my_predbat) +``` + +The test uses `prediction.scan_soc_range`, which Task 3 creates. Add it in this task as part of Step 4 (Task 3 then routes `thread_run_prediction_charge_min_max` through it). + +- [ ] **Step 2: Run the test to verify it fails** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all --test kernel_parity > /tmp/predbat-task2.log 2>&1; grep -nE "ERROR|FAIL|Traceback|AttributeError|passed" /tmp/predbat-task2.log | head -20 +``` + +Expected: `AttributeError: module 'prediction_kernel' has no attribute 'BatchJob'`. + +- [ ] **Step 3: Fix the `pk_run_batch` binding** + +In `apps/predbat/prediction_kernel.py`, replace the argtypes line inside the `try:` at line 288-289: + +```python + lib.pk_run_batch.restype = ctypes.c_int32 + # The trailing c_int32 is n_threads - pk_run_batch takes five arguments, and ctypes + # will happily pass a fourth-and-a-bit if the binding says otherwise + lib.pk_run_batch.argtypes = [ctypes.c_int64, ctypes.POINTER(PkBatchJob), ctypes.c_int32, ctypes.POINTER(PkBatchResult), ctypes.c_int32] +``` + +- [ ] **Step 4: Add `BatchJob`, `kernel_result_tuple`, `run_prediction_kernel_batch` and `scan_soc_range`** + +In `apps/predbat/prediction_kernel.py`, add `BatchJob` after the `PkBatchResult` class: + +```python +class BatchJob: + """One queued prediction: its trial inputs, how the result should be shaped, and the result itself. + + run_prediction_kernel_batch reads only the input fields and the SoC range steps. sim_hash, + want_range, range_window and result belong to the batch runner in prediction_batch.py, which is + the only thing that creates these; they live here so the ctypes layer can be tested on its own. + """ + + __slots__ = ("charge_limit", "charge_window", "export_window", "export_limits", "pv_scenario", "end_record", "step", "soc_range_start_step", "soc_range_end_step", "sim_hash", "want_range", "range_window", "result") + + def __init__(self, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step, soc_range_start_step=-1, soc_range_end_step=-1, sim_hash=None, want_range=False, range_window=None): + """Record one job's trial inputs; result stays None until the batch is flushed""" + self.charge_limit = charge_limit + self.charge_window = charge_window + self.export_window = export_window + self.export_limits = export_limits + self.pv_scenario = pv_scenario + self.end_record = end_record + self.step = step + self.soc_range_start_step = soc_range_start_step + self.soc_range_end_step = soc_range_end_step + self.sim_hash = sim_hash + self.want_range = want_range + self.range_window = range_window + self.result = None +``` + +Then factor the result assembly out of `run_prediction_kernel` so the single and batch paths cannot drift. Add: + +```python +def kernel_result_tuple(pred, result, predict_soc): + """Assemble the 17-field Prediction.run_prediction() result tuple from a PkResult. + + Shared by the single-scenario and batch paths - mirrors prediction.py:626-628, 1266-1284. + """ + car_charging_soc_next = pred.car_charging_soc_next[:] + if result.car_soc_next_valid: + for car_n in range(pred.num_cars): + car_charging_soc_next[car_n] = result.car_soc_next[car_n] + + iboost_next = result.iboost_next if pred.iboost_enable else pred.iboost_next + + return ( + round(result.final_metric, 4), + round(result.import_kwh_battery, 4), + round(result.import_kwh_house, 4), + round(result.export_kwh, 4), + round(result.soc_min, 4), + round(result.final_soc, 4), + result.soc_min_minute, + round(result.battery_cycle, 4), + round(result.metric_keep, 4), + round(result.final_iboost, 4), + round(result.final_carbon_g, 4), + predict_soc, + car_charging_soc_next, + iboost_next, + bool(result.iboost_running), + bool(result.iboost_running_solar), + bool(result.iboost_running_full), + ) + + +def reset_kernel_run_state(pred): + """Clear the per-run state attributes exactly as the Python engine does - prediction.py:414-422 + + Non-save runs never populate these, so they simply stay empty/False; a kernel run has to leave + the Prediction in the same state a Python run would. + """ + pred.predict_soc_best = {} + pred.predict_metric_best = {} + pred.predict_iboost_best = {} + pred.predict_carbon_best = {} + pred.predict_clipped_best = {} + pred.iboost_running = False + pred.iboost_running_solar = False + pred.iboost_running_full = False + + +def run_prediction_kernel_batch(pred, jobs, n_threads=1): + """Run a whole fan-out of prediction scenarios through the kernel in one pk_run_batch call. + + Returns a list of (result_tuple, soc_range_min, soc_range_max) in job order, where result_tuple + is the same 17-field tuple Prediction.run_prediction() returns, or None for a job the kernel + refused - the caller re-runs those through the Python engine. Returns None when the batch could + not be run at all (no kernel, no batch entry point, or a non-zero return code). + + Whether the kernel is allowed to answer at all is the caller's decision, not this function's: + kernel_supported() also refuses debug and saving runs, and flush_batch checks it before getting + here. + + Batched jobs never materialise the per-minute SoC series: their callers all discard it, and one + buffer per job would be ~84MB on a large batch. The one thing a caller does want from it - the + SoC range across a charge window - the kernel tracks inline over + [soc_range_start_step, soc_range_end_step] instead. + """ + lib = KERNEL_LIB + if not lib or not KERNEL_HAS_BATCH or not getattr(pred, "kernel_handle", 0): + return None + + n_jobs = len(jobs) + job_array = (PkBatchJob * n_jobs)() + result_array = (PkBatchResult * n_jobs)() + # Every ctypes buffer behind a pointer field has to outlive the call. Structures reached through + # an array index do not keep their own _objects, so the buffers are held here rather than relying + # on ctypes' keepalive doing the right thing through the array. + buffers = [] + # A fan-out reuses the same window lists across most of its jobs, so their start/end arrays are + # marshalled once per distinct list instead of once per job - that is the bulk of the batching + # win. Keyed on identity, with the list retained so an id() cannot be recycled mid-batch. This + # relies on no caller mutating a window list between enqueue and flush (see prediction_batch.py). + window_cache = {} + + def window_arrays(windows): + """Marshal a window list's start/end arrays, reusing an earlier job's arrays where possible""" + entry = window_cache.get(id(windows)) + if entry is None: + entry = (int32_array([window["start"] for window in windows]), int32_array([window["end"] for window in windows]), windows) + window_cache[id(windows)] = entry + return entry + + for index, job in enumerate(jobs): + charge_start, charge_end, _ = window_arrays(job.charge_window) + export_start, export_end, _ = window_arrays(job.export_window) + charge_limit = double_array(job.charge_limit) + export_limits = double_array(job.export_limits) + buffers.append((charge_limit, export_limits)) + + pk_job = job_array[index] + pk_job.charge_limit = charge_limit + pk_job.charge_start = charge_start + pk_job.charge_end = charge_end + pk_job.export_limits = export_limits + pk_job.export_start = export_start + pk_job.export_end = export_end + pk_job.soc_out = None + pk_job.n_charge = len(job.charge_window) + pk_job.n_export = len(job.export_window) + pk_job.pv_scenario = int(job.pv_scenario) + pk_job.end_record = job.end_record + pk_job.step = PREDICT_STEP # the caller's step is ignored - see run_prediction_kernel + pk_job.soc_range_start_step = job.soc_range_start_step + pk_job.soc_range_end_step = job.soc_range_end_step + + return_code = lib.pk_run_batch(pred.kernel_handle, job_array, n_jobs, result_array, max(int(n_threads), 1)) + if return_code != 0: + return None + + reset_kernel_run_state(pred) + + results = [] + for index in range(n_jobs): + batch_result = result_array[index] + if batch_result.status != 0: + results.append((None, 0.0, 0.0)) + continue + results.append((kernel_result_tuple(pred, batch_result.result, {}), batch_result.soc_range_min, batch_result.soc_range_max)) + return results +``` + +Then rewrite the tail of `run_prediction_kernel` (from the `# Reset the per-run state attributes` comment to the end) to use the shared helpers: + +```python + reset_kernel_run_state(pred) + + # Assemble the same return value as the Python engine + predict_soc = {} + if not cache: + # Indexed loop, not dict(zip(range(...), soc_out)): iterating a ctypes array boxes each double + # through the sequence protocol and measured 34% slower than subscripting it. + for k in range(n_steps): + predict_soc[k * PREDICT_STEP] = soc_out[k] + + return kernel_result_tuple(pred, result, predict_soc) +``` + +Finally, add `scan_soc_range` to `Prediction` in `apps/predbat/prediction.py`, immediately above `thread_run_prediction_charge_min_max`: + +```python + def scan_soc_range(self, predict_soc, window): + """Return the (min, max) SoC across a charge window - shared by the direct and batch min/max paths. + + The kernel computes the same range inline (see PkBatchJob.soc_range_start_step), so this is + only reached when a job falls back to the Python engine; the two must agree exactly, including + the clamping that collapses an empty range to a single value rather than leaving min above max. + """ + min_soc = self.soc_max + max_soc = 0 + predict_minute_start = max(int((window["start"] - self.minutes_now) / 5) * 5, 0) + predict_minute_end = int((window["end"] - self.minutes_now) / 5) * 5 + for minute in range(predict_minute_start, predict_minute_end + 5, 5): + if minute in predict_soc: + min_soc = min(predict_soc[minute], min_soc) + max_soc = max(predict_soc[minute], max_soc) + max_soc = max(max_soc, min_soc) + min_soc = min(min_soc, max_soc) + return min_soc, max_soc +``` + +- [ ] **Step 5: Run the test to verify it passes** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all --test kernel_parity > /tmp/predbat-task2.log 2>&1; grep -nE "ERROR|FAIL|bit-identical|passed|Traceback" /tmp/predbat-task2.log | head -20 +``` + +Expected: `Batch parity: 60 scenarios bit-identical to pk_run at 1, 2, 4 and 8 threads`, and the parity suite passing. + +- [ ] **Step 6: Verify the single-scenario path still matches (the refactor above touched it)** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all --test model_kernel --test kernel_parity > /tmp/predbat-task2-model.log 2>&1; grep -nE "ERROR|FAILED|passed" /tmp/predbat-task2-model.log | tail -20 +``` + +- [ ] **Step 7: Commit** + +```bash +cd /Users/treforsouthwell/predbat/batpred && ./run_pre_commit && git add apps/predbat/prediction_kernel.py apps/predbat/prediction.py apps/predbat/tests/test_kernel_parity.py && git commit -m "feat(kernel): marshal a whole fan-out into one pk_run_batch call + +Adds BatchJob, run_prediction_kernel_batch and the shared result assembly, and gives pk_run_batch's +ctypes binding the n_threads argument it was missing. Window start/end arrays are marshalled once per +distinct window list rather than once per job, which is where the batching win comes from. + +Bit-identical to a loop of pk_run across 60 scenarios at 1, 2, 4 and 8 threads. + +Co-Authored-By: Claude Opus 5 (1M context) " +``` + +--- + +### Task 3: Split the trial-input building out of `thread_run_prediction_*` + +**Files:** +- Modify: `apps/predbat/prediction.py:227-394` (all four `thread_run_prediction_*`) + +**Interfaces:** +- Consumes: `_prepare_export` (Task 1), `scan_soc_range` (Task 2). +- Produces: + - `_prepare_single(charge_limit, export_limits) -> (list, list)` + - `_prepare_charge(try_soc, window_n, charge_limit, all_n) -> list` + - `_prepare_export(...)` — already exists from Task 1. + Task 4's `queue_run_prediction_*` call exactly these, so the direct and batch paths cannot drift. + +This step must be a **no-op**. It exists so the batch path never gets a second copy of the input building. + +- [ ] **Step 1: Add the two remaining prepare helpers** + +In `apps/predbat/prediction.py`, above `thread_run_prediction_single`: + +```python + def _prepare_single(self, charge_limit, export_limits): + """Copy the caller's limit lists for a single-scenario trial - shared by thread_run_prediction_single and the batch path. + + The copy used to live in Plan.launch_run_prediction_single, where it protected the caller's + lists from the pool pickling them; the batch path defers the read to flush time, which needs + the same protection, so it belongs with the other trial-input building. + """ + return list(charge_limit), list(export_limits) + + def _prepare_charge(self, try_soc, window_n, charge_limit, all_n): + """Build the trial charge limits - shared by thread_run_prediction_charge/_charge_min_max and the batch path""" + try_charge_limit = charge_limit.copy() + if all_n: + for set_n in all_n: + try_charge_limit[set_n] = try_soc + else: + try_charge_limit[window_n] = try_soc + return try_charge_limit +``` + +- [ ] **Step 2: Route the four thread functions through them** + +`thread_run_prediction_single` — insert as the first statement of the body: + +```python + charge_limit, export_limits = self._prepare_single(charge_limit, export_limits) +``` + +`thread_run_prediction_charge` and `thread_run_prediction_charge_min_max` — replace the five-line `try_charge_limit = charge_limit.copy()` block in each with: + +```python + try_charge_limit = self._prepare_charge(try_soc, window_n, charge_limit, all_n) +``` + +`thread_run_prediction_charge_min_max` — replace the scan block (`min_soc = self.soc_max` through `min_soc = min(min_soc, max_soc)`) with: + +```python + min_soc = self.soc_max + max_soc = 0 + if not all_n: + min_soc, max_soc = self.scan_soc_range(predict_soc, charge_window[window_n]) +``` + +`thread_run_prediction_export` — already routed through `_prepare_export` in Task 1. + +- [ ] **Step 3: Drop the now-duplicated copy in plan.py** + +`Plan.launch_run_prediction_single` in `apps/predbat/plan.py:675-676` still does its own `list()` copies. Delete those two lines — `_prepare_single` now owns that. + +- [ ] **Step 4: Verify the split changed nothing** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && python3 ../apps/predbat/unit_test.py --random-run --random-template cases/predbat_debug_agile1.yaml --random-scenarios cases/random_scenarios.yaml > /tmp/predbat-task3-run.log 2>&1; python3 ../apps/predbat/unit_test.py --random-compare cases/random_results.json random_results.json > /tmp/predbat-task3-cmp.log 2>&1; tail -30 /tmp/predbat-task3-cmp.log +``` + +Expected: 20 unchanged, `+0.0000` everywhere. This step alone should be a no-op — if it is not, the split is wrong. + +- [ ] **Step 5: Run the full suite** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all > /tmp/predbat-task3-all.log 2>&1; grep -nE "FAILED|ERROR|Traceback|tests passed" /tmp/predbat-task3-all.log | tail -20 +``` + +- [ ] **Step 6: Commit** + +```bash +cd /Users/treforsouthwell/predbat/batpred && ./run_pre_commit && git add apps/predbat/prediction.py apps/predbat/plan.py && git commit -m "refactor(prediction): split trial-input building out of thread_run_prediction_* + +Each thread_run_prediction_* built the trial inputs, ran the prediction and shaped the result. The +batch path needs the first and third but not the second, so they are split rather than copied - one +definition of what a trial input is, for both paths. + +No-op: 20 random scenarios byte-identical. + +Co-Authored-By: Claude Opus 5 (1M context) " +``` + +--- + +### Task 4: The lazy batch runner, serial + +**Files:** +- Create: `apps/predbat/prediction_batch.py` +- Modify: `apps/predbat/prediction.py` (mixin, `queue_run_prediction_*`, `prediction_cache_key` reuse, `__init__`) +- Modify: `apps/predbat/plan.py:671-714` (the four `launch_*`), `:27` (imports), `:1340-1365` (pool creation) +- Test: `apps/predbat/tests/test_prediction_batch.py` + +**Interfaces:** +- Consumes: `BatchJob`, `run_prediction_kernel_batch` (Task 2); `_prepare_single`, `_prepare_charge`, `_prepare_export`, `scan_soc_range` (Tasks 1 and 3). +- Produces: + - `prediction_batch.prediction_cache_key(charge_limit, charge_window, export_limits, export_window, pv_scenario, end_record, step) -> int` + - `prediction_batch.BatchHandle` with `.get()` — the drop-in for `AsyncResult`/`DummyThread`. + - `prediction_batch.PredictionBatch` mixin: `enqueue_prediction(...) -> BatchHandle`, `flush_batch()`, `run_batch_job_python(job) -> tuple`, attributes `pending_batch` (list) and `batch_threads` (int). + - `Prediction.queue_run_prediction_single/_charge/_charge_min_max/_export` — same signatures as the matching `thread_run_prediction_*`, returning a `BatchHandle`. + +`batch_threads` stays at 1 for this whole task. That isolates the marshalling win with no concurrency in play, so any divergence here is a Python bug and nothing else. + +- [ ] **Step 1: Write the failing tests** + +Extend `apps/predbat/tests/test_prediction_batch.py` (keep the Task 1 test, add these). Import at the top: + +```python +import prediction_batch +from const import PV_SCENARIO_NOMINAL, PV_SCENARIO_PV10 +from prediction_kernel import create_kernel_context +from tests.test_kernel_parity import apply_random_scenario, kernel_available, make_step_data, make_windows, restore_scenario_state, snapshot_scenario_state +``` + +```python +def make_batch_prediction(my_predbat, seed=7): + """Build a kernel-enabled Prediction plus a charge/export window layout to fan out over""" + rng = random.Random(seed) + apply_random_scenario(my_predbat, rng) + pv_step, pv10_step, load_step, load10_step = make_step_data(my_predbat, rng) + my_predbat.prediction_kernel_enable = True + prediction = Prediction(my_predbat, pv_step, pv10_step, load_step, load10_step) + prediction.kernel_handle = create_kernel_context(prediction) + charge_window = make_windows(rng, my_predbat.minutes_now, my_predbat.forecast_minutes, 4) + export_window = make_windows(rng, my_predbat.minutes_now, my_predbat.forecast_minutes, 4) + charge_limit = [round(my_predbat.soc_max / 2, 2)] * len(charge_window) + export_limits = [100.0] * len(export_window) + return prediction, charge_window, export_window, charge_limit, export_limits + + +def test_queued_matches_direct(my_predbat): + """Every queued prediction must return exactly what the direct thread path returns, returns True on failure""" + print("**** Running queued vs direct equivalence tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + try_soc = round(my_predbat.soc_max * 0.8, 2) + + cases = [ + ("single", lambda: prediction.thread_run_prediction_single(charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, end_record, 5), lambda: prediction.queue_run_prediction_single(charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, end_record, 5)), + ("charge", lambda: prediction.thread_run_prediction_charge(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_PV10, None, end_record), lambda: prediction.queue_run_prediction_charge(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_PV10, None, end_record)), + ("charge_min_max", lambda: prediction.thread_run_prediction_charge_min_max(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record), lambda: prediction.queue_run_prediction_charge_min_max(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record)), + ("charge_min_max_all_n", lambda: prediction.thread_run_prediction_charge_min_max(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, [0, 2], end_record), lambda: prediction.queue_run_prediction_charge_min_max(try_soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, [0, 2], end_record)), + ("export", lambda: prediction.thread_run_prediction_export(5.0, export_window[1]["start"] + 15, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record), lambda: prediction.queue_run_prediction_export(5.0, export_window[1]["start"] + 15, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record)), + ] + + for name, direct_call, queue_call in cases: + # The queued run goes first, so the direct run cannot be answered out of the cache the queued + # one just populated - and vice versa, hence the clear between them + prediction.prediction_cache = {} + queued = queue_call().get() + prediction.prediction_cache = {} + direct = direct_call() + if queued != direct: + print("ERROR: {} queued {} != direct {}".format(name, queued, direct)) + failed = True + + if not failed: + print("Queued predictions match the direct path for all five shapes") + return failed + + +def test_batch_is_lazy(my_predbat): + """Nothing runs until the first get(), and one get() flushes the whole batch, returns True on failure""" + print("**** Running batch laziness tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + + handles = [prediction.queue_run_prediction_charge(round(soc, 2), 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) for soc in (1.0, 2.0, 3.0)] + if len(prediction.pending_batch) != 3: + print("ERROR: expected 3 pending jobs, got {}".format(len(prediction.pending_batch))) + failed = True + if any(handle.job.result is not None for handle in handles): + print("ERROR: a job ran before its handle was read") + failed = True + + handles[0].get() + if prediction.pending_batch: + print("ERROR: pending batch not drained by the first get(), {} left".format(len(prediction.pending_batch))) + failed = True + if any(handle.job.result is None for handle in handles): + print("ERROR: one get() did not flush the whole batch") + failed = True + + # A second read must not re-run anything + before = handles[2].job.result + if handles[2].get() is not before: + print("ERROR: reading a handle twice recomputed the result") + failed = True + + if not failed: + print("Batch laziness tests passed") + return failed + + +def test_batch_cache_and_dedup(my_predbat): + """Cache hits and intra-batch duplicates must never reach the kernel, returns True on failure""" + print("**** Running batch cache/dedup tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + prediction.prediction_cache_enable = True + + calls = [] + real_batch = prediction_batch.run_prediction_kernel_batch + + def spy(pred, jobs, n_threads): + """Record how many jobs each batch call was handed, then run it for real""" + calls.append(len(jobs)) + return real_batch(pred, jobs, n_threads) + + prediction_batch.run_prediction_kernel_batch = spy + try: + # Three identical jobs in one batch must collapse to one kernel job + prediction.prediction_cache = {} + handles = [prediction.queue_run_prediction_charge(4.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) for _ in range(3)] + results = [handle.get() for handle in handles] + if calls != [1]: + print("ERROR: expected one kernel job for three identical trials, got {}".format(calls)) + failed = True + if results[0] != results[1] or results[1] != results[2]: + print("ERROR: deduplicated jobs returned different results") + failed = True + + # And a repeat in a later batch must come from the cache without a kernel call at all + del calls[:] + again = prediction.queue_run_prediction_charge(4.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record).get() + if calls: + print("ERROR: a cached scenario still reached the kernel: {}".format(calls)) + failed = True + if again != results[0]: + print("ERROR: cached result {} differs from the computed one {}".format(again, results[0])) + failed = True + finally: + prediction_batch.run_prediction_kernel_batch = real_batch + + if not failed: + print("Batch cache/dedup tests passed") + return failed + + +def test_batch_fallbacks(my_predbat): + """A job the kernel cannot take still runs, and gives the Python engine's answer, returns True on failure""" + print("**** Running batch fallback tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + + # Two references, because the fallbacks land on two different engines. A batch the kernel refused + # still re-runs through run_prediction, which dispatches to the kernel; a Prediction the kernel + # will not serve at all re-runs on the Python engine. Those two agree to 1e-6 (kernel_parity), not + # bit-exactly, so each fallback is compared against the engine it actually used. + prediction.prediction_cache = {} + expected = prediction.thread_run_prediction_charge_min_max(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) + prediction.debug_enable = True + prediction.prediction_cache = {} + expected_python = prediction.thread_run_prediction_charge_min_max(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) + prediction.debug_enable = False + + # 1. Whole batch refused (no kernel, stale binary, or a non-zero return code) + real_batch = prediction_batch.run_prediction_kernel_batch + + def refuse_all(pred, jobs, n_threads): + """Stand in for a kernel that cannot run the batch at all""" + return None + + def refuse_each(pred, jobs, n_threads): + """Stand in for a kernel that ran the batch but rejected every job""" + return [(None, 0.0, 0.0)] * len(jobs) + + for name, stub in (("whole batch", refuse_all), ("per job", refuse_each)): + prediction_batch.run_prediction_kernel_batch = stub + try: + prediction.prediction_cache = {} + got = prediction.queue_run_prediction_charge_min_max(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record).get() + if got != expected: + print("ERROR: {} fallback returned {} expected {}".format(name, got, expected)) + failed = True + finally: + prediction_batch.run_prediction_kernel_batch = real_batch + + # 2. Debug runs must not reach the kernel from the batch when the direct path would not take it + prediction.debug_enable = True + try: + prediction.prediction_cache = {} + got = prediction.queue_run_prediction_charge_min_max(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record).get() + if got != expected_python: + print("ERROR: debug_enable fallback returned {} expected {}".format(got, expected_python)) + failed = True + finally: + prediction.debug_enable = False + + # 3. Kernel genuinely unavailable for this Prediction + prediction.kernel_handle = 0 + prediction.prediction_cache = {} + got = prediction.queue_run_prediction_charge_min_max(3.0, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record).get() + if got != expected_python: + print("ERROR: kernel-less fallback returned {} expected {}".format(got, expected_python)) + failed = True + + if not failed: + print("Batch fallback tests passed") + return failed +``` + +and rewrite the runner at the bottom of the file: + +```python +def run_prediction_batch_tests(my_predbat): + """Run every batched prediction test, returns True on failure""" + failed = test_export_trial_does_not_mutate_caller_window(my_predbat) + + available, required_failure = kernel_available() + if not available: + print("WARNING: kernel not available - batch tests that need it are SKIPPED") + return failed or required_failure + + state = snapshot_scenario_state(my_predbat) + try: + failed |= test_queued_matches_direct(my_predbat) + failed |= test_batch_is_lazy(my_predbat) + failed |= test_batch_cache_and_dedup(my_predbat) + failed |= test_batch_fallbacks(my_predbat) + finally: + restore_scenario_state(my_predbat, state) + my_predbat.prediction_kernel_enable = False + + if failed: + print("**** Prediction batch tests FAILED ****") + else: + print("**** Prediction batch tests passed ****") + return failed +``` + +- [ ] **Step 2: Run the tests to verify they fail** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all --test prediction_batch > /tmp/predbat-task4.log 2>&1; grep -nE "ERROR|FAIL|Traceback|ModuleNotFound|AttributeError" /tmp/predbat-task4.log | head -20 +``` + +Expected: `ModuleNotFoundError: No module named 'prediction_batch'`. + +- [ ] **Step 3: Write `prediction_batch.py`** + +```python +# ----------------------------------------------------------------------------- +# Predbat Home Battery System +# Copyright Trefor Southwell 2026 - All Rights Reserved +# This application maybe used for personal use only and not for commercial use +# ----------------------------------------------------------------------------- +# fmt off +# pylint: disable=consider-using-f-string +# pylint: disable=line-too-long + +"""Lazy batching of the prediction fan-out. + +Every optimiser fan-out in plan.py is launch-all-then-collect-all, so the launches can queue instead +of running: each one prepares its trial inputs, appends a BatchJob and returns a BatchHandle, and the +first get() runs the whole batch through a single pk_run_batch call. That pays the Python/C boundary +and the window marshalling once per fan-out instead of once per scenario, and it is the only shape +that lets the kernel spread the work across cores - a Python thread pool cannot (the GIL serialises +it) and a process pool cannot (the parent's pickling is a serial bottleneck); both were measured and +rejected. + +INVARIANT: a caller must not mutate any list or window dict it has handed to a queue_* entry point +until it has read the matching handle. The four fan-outs in plan.py satisfy this; the 20-scenario +byte-identical plan comparison is what keeps it that way. +""" + +from operator import itemgetter + +from const import PREDICT_STEP +from prediction_kernel import BatchJob, kernel_supported, run_prediction_kernel_batch + +# Pulls (start, end) from a window dict; used to build the prediction cache key without a Python +# level loop over every window +window_bounds = itemgetter("start", "end") + + +def prediction_cache_key(charge_limit, charge_window, export_limits, export_window, pv_scenario, end_record, step): + """Hash a simulation's inputs into its prediction cache key. + + Shared by Prediction.run_prediction and the batch runner so a scenario cannot land under two + different keys depending on which path reached it. Built as one tuple hash to keep the per-window + hashing in C rather than looping in Python, which matters because it runs on every simulation + with a few hundred windows. + """ + return hash( + ( + tuple(charge_limit), + tuple(map(window_bounds, charge_window)), + tuple(export_limits), + tuple(map(window_bounds, export_window)), + pv_scenario, + end_record, + step, + ) + ) + + +def shape_batch_result(job, result, soc_range_min, soc_range_max): + """Shape a full run_prediction result into what this job's thread_run_prediction_* returned""" + shaped = tuple(result[:11]) + if job.want_range: + shaped = shaped + (soc_range_min, soc_range_max) + return shaped + + +class BatchHandle: + """Result handle for a queued prediction - the drop-in for the pool's AsyncResult""" + + __slots__ = ("pred", "job") + + def __init__(self, pred, job): + """Bind the handle to the Prediction that owns the batch and to its own job""" + self.pred = pred + self.job = job + + def get(self): + """Return this job's result, flushing the whole pending batch on the first call""" + if self.job.result is None: + self.pred.flush_batch() + return self.job.result + + +class PredictionBatch: + """Pending-batch state and flushing for Prediction. + + A mixin rather than a separate object so the pending list lives directly on the Prediction that + owns the kernel context, with no reference cycle between the two. + """ + + def enqueue_prediction(self, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step, cache, want_range=False, range_window=None): + """Queue one prediction and return a handle to its eventual result""" + sim_hash = prediction_cache_key(charge_limit, charge_window, export_limits, export_window, pv_scenario, end_record, step) if cache else None + + # The kernel tracks the SoC range itself over a step range, so a min/max job does not have to + # ship an 8928-entry SoC buffer. A negative start step means "no range asked for", which the + # kernel answers with (soc_max, 0) - exactly what the Python all_n path returns unclamped. + soc_range_start_step = -1 + soc_range_end_step = -1 + if range_window is not None: + predict_minute_start = max(int((range_window["start"] - self.minutes_now) / 5) * 5, 0) + predict_minute_end = int((range_window["end"] - self.minutes_now) / 5) * 5 + soc_range_start_step = predict_minute_start // PREDICT_STEP + soc_range_end_step = predict_minute_end // PREDICT_STEP + + job = BatchJob( + charge_limit, + charge_window, + export_window, + export_limits, + pv_scenario, + end_record, + step, + soc_range_start_step=soc_range_start_step, + soc_range_end_step=soc_range_end_step, + sim_hash=sim_hash, + want_range=want_range, + range_window=range_window, + ) + self.pending_batch.append(job) + return BatchHandle(self, job) + + def run_batch_job_python(self, job): + """Run one job through the normal prediction path, for when the kernel cannot take it""" + result = self.run_prediction(job.charge_limit, job.charge_window, job.export_window, job.export_limits, job.pv_scenario, end_record=job.end_record, step=job.step, cache=job.sim_hash is not None) + if not job.want_range: + return tuple(result[:11]) + if job.range_window is None: + return tuple(result[:11]) + (self.soc_max, 0) + return tuple(result[:11]) + self.scan_soc_range(result[11], job.range_window) + + def flush_batch(self): + """Run every pending job: cache hits first, then one kernel call, then any fallbacks""" + pending = self.pending_batch + if not pending: + return + # Cleared before anything runs, so a fallback that re-enters run_prediction cannot see a + # half-flushed batch or flush it a second time + self.pending_batch = [] + + cache = self.prediction_cache + kernel_jobs = [] + duplicates = [] + first_by_hash = {} + for job in pending: + if job.sim_hash is not None: + cached = cache.get(job.sim_hash) + if cached is not None: + job.result = shape_batch_result(job, cached, 0.0, 0.0) + continue + # Two identical trials in one fan-out used to be a cache hit for the second one; the + # batch has to collapse them itself or that hit is lost (~44% of calls are hits) + first = first_by_hash.get(job.sim_hash) + if first is not None: + duplicates.append((job, first)) + continue + first_by_hash[job.sim_hash] = job + kernel_jobs.append(job) + + if kernel_jobs: + # kernel_supported() is a property of the Prediction, not of the individual job - it also + # refuses debug runs, which would otherwise reach the kernel here when the direct path + # would have used the Python engine. save is always None for a queued job: a saving run + # calls run_prediction directly and never fans out. + batched = run_prediction_kernel_batch(self, kernel_jobs, self.batch_threads) if kernel_supported(self, None, PREDICT_STEP) else None + for index, job in enumerate(kernel_jobs): + entry = batched[index] if batched is not None else None + if entry is None or entry[0] is None: + job.result = self.run_batch_job_python(job) + continue + result, soc_range_min, soc_range_max = entry + if job.sim_hash is not None: + # Stored without the SoC/car data to save memory, mirroring run_prediction + cache[job.sim_hash] = result[:11] + ([], []) + result[13:] + job.result = shape_batch_result(job, result, soc_range_min, soc_range_max) + + for job, first in duplicates: + job.result = first.result +``` + +- [ ] **Step 4: Wire the mixin and the queue entry points into `Prediction`** + +In `apps/predbat/prediction.py`, replace the local `window_bounds` definition and the inline hash in `run_prediction` with the shared ones, and add the mixin. Imports: + +```python +from prediction_batch import PredictionBatch, prediction_cache_key +``` + +Delete the module-level `window_bounds = itemgetter("start", "end")` and its comment (and the now-unused `from operator import itemgetter` if nothing else uses it — check first). + +Class declaration: + +```python +class Prediction(PredictionBatch): +``` + +In `__init__`, after the `PRED_GLOBAL["dict"] = self.__dict__.copy()` line: + +```python + # Deliberately after the PRED_GLOBAL snapshot: a pool worker reconstructs its Prediction + # from that dict and never queues anything, so it has no business carrying a batch. + self.pending_batch = [] + self.batch_threads = 1 +``` + +In `run_prediction`, replace the inline `sim_hash = hash((...))` block with: + +```python + sim_hash = None + if cache and not save: + sim_hash = prediction_cache_key(charge_limit, charge_window, export_limits, export_window, pv_scenario, end_record, step) + cached_result = self.prediction_cache.get(sim_hash) + if cached_result is not None: + # Return cached result + return cached_result +``` + +Then add the four queue entry points, next to their `thread_run_prediction_*` twins: + +```python + def queue_run_prediction_single(self, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step): + """Queue a single-scenario prediction, returning a handle - the batch runs on the first get()""" + charge_limit, export_limits = self._prepare_single(charge_limit, export_limits) + return self.enqueue_prediction(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step, self.prediction_cache_enable) + + def queue_run_prediction_charge(self, try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Queue a charge-window trial prediction, returning a handle""" + try_charge_limit = self._prepare_charge(try_soc, window_n, charge_limit, all_n) + return self.enqueue_prediction(try_charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, PREDICT_STEP, self.prediction_cache_enable) + + def queue_run_prediction_charge_min_max(self, try_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Queue a charge-window trial prediction that also reports the SoC range across that window. + + Uncached, exactly as the direct path is: the SoC range is not part of the cached result, so a + hit would answer with the wrong shape. + """ + try_charge_limit = self._prepare_charge(try_soc, window_n, charge_limit, all_n) + range_window = None if all_n else charge_window[window_n] + return self.enqueue_prediction(try_charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, PREDICT_STEP, False, want_range=True, range_window=range_window) + + def queue_run_prediction_export(self, this_export_limit, start, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Queue an export-window trial prediction, returning a handle""" + export_window, export_limits = self._prepare_export(this_export_limit, start, window_n, export_window, export_limits, all_n) + return self.enqueue_prediction(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, PREDICT_STEP, self.prediction_cache_enable) +``` + +- [ ] **Step 5: Switch the launch sites and stop creating the pool** + +In `apps/predbat/plan.py`, replace all four `launch_run_prediction_*` bodies: + +```python + def launch_run_prediction_single(self, charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step=PREDICT_STEP): + """Queue a prediction and return a handle to its result""" + return self.prediction.queue_run_prediction_single(charge_limit, charge_window, export_window, export_limits, pv_scenario, end_record, step) + + def launch_run_prediction_charge(self, loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Queue a prediction and return a handle to its result""" + return self.prediction.queue_run_prediction_charge(loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record) + + def launch_run_prediction_charge_min_max(self, loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record): + """Queue a prediction and return a handle to its result""" + return self.prediction.queue_run_prediction_charge_min_max(loop_soc, window_n, charge_limit, charge_window, export_window, export_limits, pv_scenario, all_n, end_record) + + def launch_run_prediction_export(self, this_export_limit, start, window_n, try_charge_limit, charge_window, try_export_window, try_export, pv_scenario, all_n, end_record): + """Queue a prediction and return a handle to its result""" + return self.prediction.queue_run_prediction_export(this_export_limit, start, window_n, try_charge_limit, charge_window, try_export_window, try_export, pv_scenario, all_n, end_record) +``` + +Delete the `wrapped_run_prediction_*` import (line 27 becomes `from prediction import Prediction`) and the `from multiprocessing import Pool, cpu_count` import — both are unused now and Flake8 will fail otherwise. + +Delete the pool creation block (`# Create pool` through the `self.log("Not using threading...")` line) and the LoadML pool-disable block above it, keeping the LoadML status log: + +```python + # Check if LoadML is active - it used to force the process pool off, which no longer exists; + # the kernel's threads are C++ threads with no fork and no NumPy involvement + load_ml_comp = self.components.get_component("load_ml") if self.components else None + if load_ml_comp: + self.log("LoadML is_calculating {}".format(load_ml_comp.is_calculating())) + + # Serial for now - Task 5 wires this to the threads config + self.prediction.batch_threads = 1 +``` + +Place that `batch_threads` assignment immediately after `self.prediction = Prediction(...)` at line 1336. + +- [ ] **Step 6: Run the new tests to verify they pass** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all --test prediction_batch --test kernel_parity > /tmp/predbat-task4.log 2>&1; grep -nE "ERROR|FAIL|passed|Traceback" /tmp/predbat-task4.log | head -30 +``` + +- [ ] **Step 7: Verify plans are byte-identical — this is the real gate** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && python3 ../apps/predbat/unit_test.py --random-run --random-template cases/predbat_debug_agile1.yaml --random-scenarios cases/random_scenarios.yaml > /tmp/predbat-task4-run.log 2>&1; python3 ../apps/predbat/unit_test.py --random-compare cases/random_results.json random_results.json > /tmp/predbat-task4-cmp.log 2>&1; tail -30 /tmp/predbat-task4-cmp.log +``` + +Expected: 20 unchanged, `+0.0000` on metric, cost and all three PV futures. If a scenario moves, the most likely causes in order are: a caller mutating an input between launch and get (the invariant above), the min/max range steps disagreeing with the Python scan, or a cache key built differently on the two paths. + +- [ ] **Step 8: Run the full suite including the slow tests** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all > /tmp/predbat-task4-all.log 2>&1; grep -nE "FAILED|ERROR|Traceback|tests passed|seconds" /tmp/predbat-task4-all.log | tail -25 +``` + +- [ ] **Step 9: Record the serial benchmark** + +Sum `runtime_s` across the 20 scenarios, best of 3 runs. Run-to-run spread is 2-6%, so a single run proves nothing — an earlier single-run measurement in this work produced a bogus 18% "win" that was pure noise. + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && for i in 1 2 3; do python3 ../apps/predbat/unit_test.py --random-run --random-template cases/predbat_debug_agile1.yaml --random-scenarios cases/random_scenarios.yaml > /tmp/predbat-bench-serial-$i.log 2>&1; python3 -c "import json;print(round(sum(s['runtime_s'] for s in json.load(open('random_results.json'))['results']), 2))"; done +``` + +Baseline to beat: **31.6s**. + +- [ ] **Step 10: Commit** + +```bash +cd /Users/treforsouthwell/predbat/batpred && ./run_pre_commit && git add apps/predbat/prediction_batch.py apps/predbat/prediction.py apps/predbat/plan.py apps/predbat/tests/test_prediction_batch.py && git commit -m "perf(plan): batch the prediction fan-out into one kernel call + +launch_run_prediction_* no longer runs anything: it prepares the trial inputs, queues a job and +returns a handle whose first get() flushes the whole batch through a single pk_run_batch call. Window +start/end arrays are marshalled once per fan-out rather than once per scenario, cache hits and +intra-batch duplicates are resolved in Python first, and anything the kernel refuses falls back to the +Python engine per job. Serial for now. + +20 random scenarios byte-identical. + +Co-Authored-By: Claude Opus 5 (1M context) " +``` + +--- + +### Task 5: Turn on kernel threading + +**Files:** +- Modify: `apps/predbat/plan.py` (threads config → `batch_threads`) +- Modify: `docs/apps-yaml.md:285-297` (`threads` now means kernel threads) +- Test: `apps/predbat/tests/test_prediction_batch.py` + +**Interfaces:** +- Consumes: `Prediction.batch_threads` (Task 4), `run_prediction_kernel_batch`'s `n_threads` (Task 2). +- Produces: nothing new — `batch_threads` is now derived from `args["threads"]`. + +`pk_run_batch` was measured bit-identical at every thread count in the C-level test (Task 2 pins that), so any divergence here is a Python bug, not a race in the kernel. + +- [ ] **Step 1: Write the failing test** + +Add to `apps/predbat/tests/test_prediction_batch.py`: + +```python +def test_batch_threads_do_not_change_results(my_predbat): + """A threaded flush must return exactly what a serial one does, returns True on failure. + + pk_run_batch is pinned bit-identical across thread counts at the C level (kernel_parity), so this + is really checking the Python side: that results are matched back to their own job regardless of + how the kernel split the work. + """ + print("**** Running batch threading tests ****") + failed = False + prediction, charge_window, export_window, charge_limit, export_limits = make_batch_prediction(my_predbat) + end_record = my_predbat.forecast_minutes + socs = [round(my_predbat.soc_max * fraction, 2) for fraction in (0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1.0)] + + reference = None + for n_threads in (1, 2, 8): + prediction.batch_threads = n_threads + prediction.prediction_cache = {} + handles = [prediction.queue_run_prediction_charge(soc, 1, charge_limit, charge_window, export_window, export_limits, PV_SCENARIO_NOMINAL, None, end_record) for soc in socs] + results = [handle.get() for handle in handles] + if reference is None: + reference = results + elif results != reference: + for index, (got, want) in enumerate(zip(results, reference)): + if got != want: + print("ERROR: job {} at {} threads returned {} expected {}".format(index, n_threads, got, want)) + failed = True + prediction.batch_threads = 1 + + if not failed: + print("Batch threading tests passed - identical results at 1, 2 and 8 threads") + return failed +``` + +and call it from `run_prediction_batch_tests` alongside the others. + +- [ ] **Step 2: Run it and watch it pass already** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all --test prediction_batch > /tmp/predbat-task5.log 2>&1; grep -nE "ERROR|FAIL|passed|Traceback" /tmp/predbat-task5.log | head -20 +``` + +This one is expected to pass immediately — the plumbing landed in Task 4; the test exists to pin it before the config starts driving it. If it fails, stop: that is a real threading bug and the config must not be wired until it is understood. + +- [ ] **Step 3: Wire the config** + +In `apps/predbat/plan.py`, re-add the import: + +```python +from multiprocessing import cpu_count +``` + +and replace the placeholder from Task 4 with: + +```python + # The kernel spreads one batched fan-out across threads with the GIL released for the whole + # call, so these are real cores - unlike a Python ThreadPool, which peaked at 1.15x on two + # threads and then degraded below serial (perf/threadpool-prototype). + threads = self.get_arg("threads", "auto") + if threads == "auto": + self.prediction.batch_threads = cpu_count() + else: + self.prediction.batch_threads = max(int(threads), 1) + self.log("Prediction batch using {} kernel thread(s)".format(self.prediction.batch_threads)) +``` + +- [ ] **Step 4: Verify plans are byte-identical with threading on** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && python3 ../apps/predbat/unit_test.py --random-run --random-template cases/predbat_debug_agile1.yaml --random-scenarios cases/random_scenarios.yaml > /tmp/predbat-task5-run.log 2>&1; python3 ../apps/predbat/unit_test.py --random-compare cases/random_results.json random_results.json > /tmp/predbat-task5-cmp.log 2>&1; tail -30 /tmp/predbat-task5-cmp.log +``` + +Note the benchmark harness may set `threads: 0` (as `annual.py` does deliberately). Check which the run used — grep the run log for `Prediction batch using` — and if it is 1, re-run the benchmark with the template's `threads` raised to `auto` to actually exercise threading before claiming a number. + +- [ ] **Step 5: Benchmark, best of 3** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && for i in 1 2 3; do python3 ../apps/predbat/unit_test.py --random-run --random-template cases/predbat_debug_agile1.yaml --random-scenarios cases/random_scenarios.yaml > /tmp/predbat-bench-threaded-$i.log 2>&1; python3 -c "import json;print(round(sum(s['runtime_s'] for s in json.load(open('random_results.json'))['results']), 2))"; done +``` + +Record serial vs threaded against the 31.6s baseline. Remember what the brief predicts: the batch call itself is not faster (0.99x vs a loop of `pk_run`) — the win is the Python-side marshalling plus, now, the threads. Kernel time was already down to ~21.6% of plan time, so Amdahl caps what threading alone can return. + +- [ ] **Step 6: Update the docs** + +`docs/apps-yaml.md`, the `threads` section: + +```markdown +### threads + +If defined sets the number of threads to use during plan calculation, the default is 'auto' which will use the same number of threads as you have CPUs in your system. + +Predbat batches each group of simulations into a single call to the C++ prediction kernel, which then +spreads them across this many threads. Results do not depend on the thread count - the same plan is +produced at any setting - so this only trades CPU for planning time. + +Valid values are: + +- 'auto' - Use the same number of threads as your CPU count +``` + +Keep the rest of the section as it stands, and check whether the list below it still describes reality (e.g. any wording about "0 - no threading" is still correct: 0 means serial). + +```bash +cd /Users/treforsouthwell/predbat/batpred && grep -rn "threads" docs/*.md apps/predbat/*.yaml 2>/dev/null | grep -vi "threshold" | head -20 +``` + +Update anything else that describes threads as worker processes. + +- [ ] **Step 7: Full suite and commit** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all > /tmp/predbat-task5-all.log 2>&1; grep -nE "FAILED|ERROR|Traceback|tests passed" /tmp/predbat-task5-all.log | tail -20 +``` + +```bash +cd /Users/treforsouthwell/predbat/batpred && ./run_pre_commit && git add apps/predbat/plan.py apps/predbat/tests/test_prediction_batch.py docs/apps-yaml.md && git commit -m "perf(plan): run each batched fan-out across kernel threads + +args[threads] now sizes the kernel's thread pool rather than a pool of worker processes. The GIL is +released for the whole pk_run_batch call, so unlike a Python ThreadPool these are real cores. + +Identical results at 1, 2 and 8 threads; 20 random scenarios byte-identical. + +Co-Authored-By: Claude Opus 5 (1M context) " +``` + +--- + +### Task 6: Delete the process pool + +**Files:** +- Modify: `apps/predbat/plan.py` (`DummyThread`, pool close/join, unused imports) +- Modify: `apps/predbat/prediction.py` (`wrapped_run_prediction_*`, `PRED_GLOBAL`, `reset_prediction_globals`) +- Modify: `apps/predbat/predbat.py` (`self.pool` lifecycle) +- Modify: `apps/predbat/annual.py:1004`, `apps/predbat/marginal.py:89` (comments referencing `PRED_GLOBAL`) +- Delete: `PHASE_B_BATCH_REFACTOR.md` + +**Interfaces:** +- Consumes: everything from Tasks 1-5. +- Produces: no `self.pool`, no `PRED_GLOBAL`, no `DummyThread`, no `wrapped_run_prediction_*`. + +**Stated consequence:** with the pool gone, a Predbat install where the kernel is unavailable (no binary for the architecture, a stale binary, or `debug_enable` on) plans serially in Python where it used to fan out across processes. The pool was measured at 1.11x on 16 cores with the kernel active; it will be worth more than that without one. This is the brief's call, made explicitly — if that regression is not acceptable, the alternative is to keep the pool solely for the `not kernel_supported()` case, which means keeping `wrapped_run_prediction_*` and `PRED_GLOBAL` alive with it. + +- [ ] **Step 1: Find every reference before deleting anything** + +```bash +cd /Users/treforsouthwell/predbat/batpred && grep -rn "PRED_GLOBAL\|wrapped_run_prediction\|DummyThread\|reset_prediction_globals\|\.pool\b\|apply_async" apps/predbat/ --include="*.py" > /tmp/predbat-pool-refs.txt; cat /tmp/predbat-pool-refs.txt +``` + +Work the list top to bottom. Note `apps/predbat/tests/` may reference these too. + +- [ ] **Step 2: Delete the dead code** + +- `apps/predbat/prediction.py`: remove `PRED_GLOBAL`, `reset_prediction_globals`, all four `wrapped_run_prediction_*`, the `global PRED_GLOBAL` lines in `__init__`, and the `PRED_GLOBAL["dict"] = self.__dict__.copy()` line. `self.pending_batch = []` / `self.batch_threads = 1` stay where they are (the comment above them about the PRED_GLOBAL snapshot no longer applies — delete it). +- `apps/predbat/plan.py`: remove `DummyThread` and its preamble comment, and the pool `close()`/`join()` blocks at ~1621-1628. Check whether `import time` is still used elsewhere in the file before removing it. +- `apps/predbat/predbat.py`: remove the `self.pool` attribute and its terminate/close/join blocks (lines ~283-288, 320, 1180-1188, 1612, 1735-1743). +- `apps/predbat/annual.py`: the `predbat.args["threads"] = 0` line stays (it now means serial batching, which is still what a per-day annual run wants), but rewrite the comment — the PRED_GLOBAL/spawn-safety reasoning it gives is gone. Something like: "The annual tool plans hundreds of small, fast days per run, where splitting each fan-out across kernel threads costs more in thread setup than it returns." +- `apps/predbat/marginal.py:89`: drop the "this updates PRED_GLOBAL which is safe since we run synchronously" comment; the remaining sentence about not needing 10% extra load stays. + +- [ ] **Step 3: Verify nothing references the removed names** + +```bash +cd /Users/treforsouthwell/predbat/batpred && grep -rn "PRED_GLOBAL\|wrapped_run_prediction\|DummyThread\|reset_prediction_globals\|apply_async\|self\.pool" apps/predbat/ --include="*.py" +``` + +Expected: no output. + +- [ ] **Step 4: Full suite plus the parity gate** + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && python3 ../apps/predbat/unit_test.py --random-run --random-template cases/predbat_debug_agile1.yaml --random-scenarios cases/random_scenarios.yaml > /tmp/predbat-task6-run.log 2>&1; python3 ../apps/predbat/unit_test.py --random-compare cases/random_results.json random_results.json > /tmp/predbat-task6-cmp.log 2>&1; tail -30 /tmp/predbat-task6-cmp.log +``` + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all > /tmp/predbat-task6-all.log 2>&1; grep -nE "FAILED|ERROR|Traceback|tests passed|seconds" /tmp/predbat-task6-all.log | tail -25 +``` + +`annual_integration` is the one most likely to notice this task — it explicitly sets `threads: 0` and its comment about the pool is what changed. Check its timing in the log against the ~34s the brief records. + +- [ ] **Step 5: Delete the brief** + +```bash +cd /Users/treforsouthwell/predbat/batpred && git rm PHASE_B_BATCH_REFACTOR.md +``` + +- [ ] **Step 6: Commit** + +```bash +cd /Users/treforsouthwell/predbat/batpred && ./run_pre_commit && git add -A apps/predbat docs && git commit -m "refactor(plan): remove the prediction process pool + +Nothing has launched work into it since the fan-out started batching, so PRED_GLOBAL, the four +wrapped_run_prediction_* trampolines, DummyThread and the pool lifecycle all go with it. The pool was +measured at 1.11x on 16 cores, with the parent's per-call pickling as the serial bottleneck. + +20 random scenarios byte-identical. + +Co-Authored-By: Claude Opus 5 (1M context) " +``` + +--- + +## Final verification before the branch is offered for merge + +Run all four gates from the brief, in this order, and paste the actual output rather than summarising it: + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && python3 ../apps/predbat/unit_test.py --random-run --random-template cases/predbat_debug_agile1.yaml --random-scenarios cases/random_scenarios.yaml > /tmp/predbat-final-run.log 2>&1; python3 ../apps/predbat/unit_test.py --random-compare cases/random_results.json random_results.json > /tmp/predbat-final-cmp.log 2>&1; tail -30 /tmp/predbat-final-cmp.log +``` + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all --test kernel_parity --test prediction_batch > /tmp/predbat-final-parity.log 2>&1; grep -nE "ERROR|FAILED|passed" /tmp/predbat-final-parity.log | tail -20 +``` + +```bash +cd /Users/treforsouthwell/predbat/batpred/coverage && source setup.csh && ./run_all > /tmp/predbat-final-all.log 2>&1; grep -nE "FAILED|ERROR|Traceback|tests passed|seconds" /tmp/predbat-final-all.log | tail -25 +``` + +```bash +cd /Users/treforsouthwell/predbat/batpred && ./run_pre_commit +``` + +Required: 20 scenarios unchanged with `+0.0000` on metric, cost and all three PV futures; kernel parity green; full suite green including the slow tests; pre-commit clean (re-run and re-stage after the cspell dictionary auto-sorts). + +Then report, best of 3 each: serial batch vs threaded batch vs the 31.6s pre-Phase-B baseline. + +## What it actually measured, once built + +The plan was executed and these are the results, kept here because the hand-off notes this plan was +written from are deleted by the last task. + +**The win is the batching itself, and it is large.** Measured against `a131f12c`, the fork point, +with the kernel built from each commit's own source and confirmed active on both sides, runs +interleaved, best of 3, on a 16-CPU machine. All 20 scenarios byte-identical in every configuration. + +| 20-scenario benchmark | fork point | this branch | delta | +|---|---|---|---| +| parallelism off (`threads: 0`) | 41.74s | 26.22s | **37.2% faster** | +| each side's default (`threads: auto`) | 40.02s | 25.81s | **35.5% faster** | + +The 31.6s figure in the hand-off notes was measured in an earlier session under unknown conditions +and did not survive re-measurement — do not carry it forward. + +**Neither side's parallelism is worth much.** The fork point's 16-process pool bought 41.74s → 40.02s +(4%); this branch's 16 kernel threads buy 26.22s → 25.81s (1.6%), inside the 2-6% noise band. The +entire improvement is the batching and the marshalling it removes, not concurrency. + +**The fork point's process pool was also broken outside the `__main__` entry point.** Its workers +rebuild their `Prediction` from a module global populated only in the parent, which requires `fork` +start semantics; `hass.py` sets those only under `if __name__ == "__main__"`. On macOS (spawn by +default since 3.8) every worker therefore started empty and every scenario died with +`KeyError: 'dict'` — the benchmark above needed `fork` forced to get comparable numbers out of the +baseline at all. This branch removes that failure mode by removing the pool. + +**Why**, from a `n_jobs` census over one benchmark run: + +```text +25,091 pk_run_batch calls carrying 253,899 jobs +mean 10.12 jobs/call, median 6, p95 24, max 363 +97.4% of calls carry fewer than 32 jobs +``` + +This plan's own profile predicted fan-outs of ~1176, ~900 and ~628 jobs. The gap is mostly not a +batching failure: the large fan-out is scoped to one `(loop_price, coarse)` iteration and its ~1176 +launches become ~363 kernel jobs once the ~44% prediction-cache hit rate and the intra-batch dedup +have taken their share. The median of 6 is `optimise_charge_limit`'s min/max pre-pass, which is +genuinely that small. So the batch is collecting whole fan-outs — the fan-outs are just small. + +At ~6 jobs of ~30us each, `pk_run_batch` builds and joins six `std::thread`s to cover ~180us of work. +Thread setup dominates by construction. + +**Where the next change goes**, in order: + +1. **`pk_run_batch` in `prediction_kernel.cpp`** — replace the per-call `std::vector` + with a pool created once at `pk_context_create` and woken per batch. This is the smallest change + with the largest effect, and it makes threading pay at today's batch sizes without touching Python. +2. **`plan.py`, not `prediction_batch.py`**, if larger batches are wanted. The ceiling is + `optimise_charge_limit`'s internal serialisation: the min/max pre-pass computes the SoC-pruning + envelope that decides which `try_socs` are launched at all. Breaking that read-then-decide + dependency means either launching the unpruned SoC set speculatively or running the pre-pass for + every window before any trial — both change optimiser semantics and risk moving plans. + `enqueue_prediction`/`flush_batch` already accept arbitrarily large batches; they need no change. +3. **Not** a batch-size threshold before threading in `run_prediction_kernel_batch` — the C++ already + clamps threads to job count, and a Python-side knob would only hide (1). + +## Notes for whoever picks this up + +- **`PR #4536` (`perf/window-bounds-cache`) is in flight and independent**, worth -21% on its own. Batching partly subsumes it — a batch ships window bounds once per fan-out rather than per call — but the `sim_hash` half still matters, and `prediction_cache_key()` in `prediction_batch.py` is exactly where that half would land. If both branches merge, re-profile before assuming what is left. +- **`round_py` is poison in any hot loop** — `snprintf`+`strtod`, ~96ns, serialising on libc's global locale, measured 17x degradation at 8 threads. Two calls remain in the kernel (`iboost_next`, `car_soc_next`), both once per run. Do not add a third. +- **Any new test that calls `calculate_plan` must not use the shared `my_predbat` fixture** — it arrives carrying whatever earlier tests left on it and `calculate_plan` raises. Follow `run_annual_integration_isolated` in `unit_test.py`: take a fresh instance from `create_predbat()` and load a debug YAML. +- **Do not re-litigate the dead ends**: Python `ThreadPool` (1.15x peak, then below serial), the process pool itself (1.11x on 16 cores), and expecting `pk_run_batch` to beat a loop of `pk_run` on its own (0.99x). All three were measured on this work. diff --git a/docs/superpowers/specs/2026-08-16-kernel-thread-pool-design.md b/docs/superpowers/specs/2026-08-16-kernel-thread-pool-design.md new file mode 100644 index 000000000..7c4d3986b --- /dev/null +++ b/docs/superpowers/specs/2026-08-16-kernel-thread-pool-design.md @@ -0,0 +1,267 @@ +# Persistent scratch and thread pool in the prediction kernel + +**Status:** implemented. Three things below were changed by measurement during implementation and are +recorded in place: stage 1 is a no-op on its own, broadcasting to wake workers was the wrong choice, +and the `threads: auto` cap this design would have added was measured and rejected. See "What +measurement changed" at the end. + +**Branch:** `perf/kernel-thread-pool`, stacked on `perf/kernel-batch-abi` (PR #4540). Neither +`pk_run_batch` nor `PkScratch` exists on `main` — both arrive with that PR — so this work cannot be +based on `main`. The follow-up PR takes `perf/kernel-batch-abi` as its base so its diff shows only +this change, and is retargeted to `main` once #4540 merges. + +## Why + +PR #4540 replaced the per-scenario prediction fan-out with a batch that reaches the C++ kernel in one +`pk_run_batch` call. It made planning 35-37% faster, but enabling kernel threading on top of it +bought only 1.6% — inside the run-to-run noise band. + +A census over one 20-scenario benchmark run explains why: + +```text +25,091 pk_run_batch calls carrying 253,899 jobs +mean 10.12 jobs/call, median 6, p95 24, max 363 +97.4% of calls carry fewer than 32 jobs +``` + +The median batch is six jobs of roughly 30us each — about 180us of simulation. Against that, +`pk_run_batch` builds and joins a fresh `std::vector` on every call, and the serial path +constructs a fresh `PkScratch` whose first use heap-allocates roughly 70KB. Both costs are paid per +call, and at these batch sizes both are the same order as the work they wrap. + +The kernel's own comment already records that this class of cost dominates: scratch reuse was +introduced because "each scenario would otherwise heap-allocate ~70KB, and 1200 of those per batch +put every worker thread in contention on the allocator. Measured 4 threads at 0.6x of serial before +this, because the simulation is cheap enough that malloc, not arithmetic, was the bottleneck." + +This design removes both per-call costs. It is deliberately staged, because the two halves carry very +different risk and the first may be enough on its own. + +## Scope + +**In scope:** `apps/predbat/prediction_kernel.cpp`, its tests, and the six shipped binaries. + +**Out of scope:** any Python change. `n_threads` stays a per-call argument, the struct layouts do not +move, and the function signatures are unchanged, so `prediction_kernel.py` is untouched. + +**Not an ABI or parity change.** Results are bit-identical before and after, so neither +`PK_ABI_VERSION` nor `PK_PARITY_REVISION` is bumped. A binary built before this change still loads +and runs correctly, just without the speedup. The six architecture binaries are still rebuilt and +shipped. + +## Stage 1 — one scratch per thread + +`PkScratch` is constructed at three sites: `pk_run` (prediction_kernel.cpp:1279), each worker lambda +in `pk_run_batch` (:1341), and the serial batch loop (:1352). All three become one accessor: + +```cpp +// One scratch per thread, reused for that thread's lifetime. Every field is fully rewritten before +// it is read on each run - build_window_membership assigns exactly n_steps entries and the clip +// helpers clear-then-push - which is the same invariant that already lets one scratch serve a whole +// stride. Extending it across calls means a thread allocates these buffers once, not once per call. +// Thread-local rather than per-context because nothing here survives a simulation, so it never needs +// to belong to a particular context. +static PkScratch &thread_scratch() +{ + static thread_local PkScratch scratch; + return scratch; +} +``` + +Each of the three sites drops its local and passes `thread_scratch()` instead. + +### The risk this introduces + +Reuse within a call spans one context, so `n_steps` is constant. Reuse across calls spans contexts, +and Predbat builds a new context every plan cycle — with a different `n_steps` whenever +`forecast_minutes` changes. `build_window_membership` assigns exactly `n_steps` entries and the clip +vectors clear before pushing, so a shrink cannot expose stale tail data through any read path. + +That is an argument, not a proof, so it is pinned by a test (see Testing). + +### Expected effect + +Removes roughly 25,000 allocations of ~70KB per plan from the serial path — which is the default for +`threads: 0`, for `annual.py`, and for the whole test harness. Independently measurable and +independently shippable. + +## Stage 2 — one pool for the process + +A single pool, allocated on first threaded use and never destroyed, holding workers parked on a +condition variable. + +```cpp +class PkThreadPool { +public: + void reserve_workers(int32_t want); // grows, never shrinks + void run(const ContextStore *c, const PkBatchJob *jobs, int32_t n_jobs, PkBatchResult *results, int32_t use); +private: + std::mutex mutex; + std::condition_variable work_ready; // workers park here + std::condition_variable work_done; // the caller waits here + const ContextStore *cur_ctx = nullptr; + const PkBatchJob *cur_jobs = nullptr; + PkBatchResult *cur_results = nullptr; + int32_t cur_n_jobs = 0; + int32_t cur_use = 1; + uint64_t generation = 0; // bumped per batch; workers wake when it changes + int32_t outstanding = 0; // lanes still running + std::vector workers; +}; +``` + +### Dispatch + +The lane count is `use = min(n_threads, n_jobs, workers.size() + 1)` — bounded by what the caller +asked for, by how much work there is, and by how many workers actually exist (see Thread creation +failure). The `+ 1` is the calling thread, which takes a lane itself. + +The caller, holding `mutex`: + +1. publishes the batch — context, jobs, results, `n_jobs` and `use` +2. sets `outstanding = use - 1` +3. bumps `generation` +4. broadcasts on `work_ready` + +then releases the mutex and **runs lane 0 itself**, striding `i = 0; i < n_jobs; i += use`. This +matters at these batch sizes: six jobs wakes five workers rather than six, and the caller does useful +work instead of blocking. Finally it waits on `work_done` until `outstanding` reaches zero. + +Each worker parks on `work_ready` until `generation` changes, records the new generation +unconditionally so it cannot spin, and participates only if its lane index is `< cur_use`. Workers +outside that range park again without touching `outstanding`. A participating worker copies the batch +pointers locally, releases the mutex, strides `i = lane; i < n_jobs; i += use`, then re-acquires the +mutex and notifies `work_done` if it was the last one out. + +Workers simulate through `thread_scratch()` from stage 1, so a persistent worker allocates its +buffers on its first batch and reuses them for the life of the process. Stage 1 replaced the scratch +local in the per-call worker lambda; stage 2 replaces that lambda with a parked worker, and the +accessor carries over unchanged. + +Nobody holds the mutex while simulating. The partition is the same stride the current code uses, and +`run_batch_job` already only reads the const context and writes its own result slot — the property +that makes today's threading safe is unchanged. + +### Bypass + +`n_threads <= 1 || n_jobs <= 1` runs inline exactly as today and never touches the pool, so +`threads: 0`, `annual.py` and the test harness never allocate a worker. + +### Sizing + +`reserve_workers` grows to whatever `n_threads` asks for and never shrinks. A later, smaller request +simply wakes fewer lanes. + +## Error handling + +**Fork.** Threads do not survive `fork()`, so a child would inherit a pool whose workers do not exist +and hang waiting on them. Nothing in Predbat forks today — PR #4540 deleted the last thing that did — +but `hass.py` still calls `set_start_method("fork")`, so a future feature that spawns a process would +trip it. A `pthread_atfork` child handler sets the pool pointer to null: no allocation, no locking, +async-signal-safe. The child lazily builds a fresh pool on its next batch, and the dead object leaks +in the child's address space, which is standard and costs a few hundred bytes. Registered once via +`std::call_once` before the pool is first created, guarded by `#if defined(__unix__) || defined(__APPLE__)`. + +**Concurrent callers.** Two Python threads can enter `pk_run_batch` at once, because ctypes releases +the GIL. With one shared pool their published state would race, so a dispatch mutex is held for the +whole batch and concurrent callers serialise. Today they would each spawn private threads and run in +parallel, so this is a regression in a case nothing exercises; a data race is the worse option. + +**Thread creation failure.** `std::thread` throws `std::system_error` under resource exhaustion. +`reserve_workers` catches it, keeps whatever workers were created, and `run` clamps `use` to +`workers.size() + 1` — falling all the way back to inline if none exist. A home-automation daemon +should degrade, not crash. + +**Shutdown.** The pool is never destroyed, deliberately. A static destructor running while CPython +tears down risks a deadlock for no benefit; the OS reclaims the threads at exit. + +**Per-job failures** are unchanged: each job's `status` field still carries its own result, so one +malformed scenario cannot discard the fan-out. + +## Testing + +Every new test is mutation-checked — break the thing deliberately, confirm the test fails, restore. +That method found three real defects on PR #4540, two of them tests that could not fail. + +| Pins | Test | +|---|---| +| Stage 1's risk | Alternate two contexts with different `forecast_minutes` on one thread; each must match its single-context result. Stale tail data or a mis-sized buffer appears here and nowhere else. | +| Worker reuse | Run the 1/2/4/8-thread parity sweep twice. The first batch creates workers, the second wakes parked ones — different paths, and "works once, breaks on reuse" is the classic pool bug. | +| Lane arithmetic | `n_jobs < n_threads`: a 3-job batch at 16 threads, so `use = 3` and thirteen workers must park without touching `outstanding`. This is the median-6 production shape. | +| Grow then wake fewer | Call at 8 threads, then 2, then 16. | +| The dispatch mutex | Two Python threads calling `pk_run_batch` at once; results must match serial. | +| The `pthread_atfork` handler | `os.fork()`, run a batch in the child, compare against the parent's result, check the child's exit status. Skipped where fork is unavailable. Without this the handler is unverified code whose whole job is preventing a hang. | + +Existing gates that must continue to hold: the `kernel_parity` sweeps, the 20-scenario byte-identical +run (`+0.0000` on metric, cost and all three PV futures), and the full suite. + +## Measurement + +Interleaved, best of 3 per side, on one machine in one sitting — the method used for PR #4540's +numbers. + +- **Stage 1** on the serial benchmark (`threads: 0`), which is where the 25,000 allocations are. +- **Stage 2** on the threaded benchmark, with serial re-checked to confirm it has not regressed. + +Each stage is measured against the tree it modifies, so the two effects are attributable separately. +If stage 1 recovers most of the available time, stage 2 can be dropped rather than accepting its +concurrency risk for a small remainder. + +**Honest ceiling.** This turns a thread creation (tens of microseconds) into a wake (a few +microseconds), so threading can pay where it currently cannot. But the upside is still bounded by the +kernel's share of plan time, so single-digit to low-teens percent is the realistic expectation, not +anything resembling a 16x. + +## Deliverables + +- `apps/predbat/prediction_kernel.cpp`: `thread_scratch()`, `PkThreadPool`, the atfork handler, and + `pk_run_batch` rewired onto them. +- `apps/predbat/tests/test_kernel_parity.py`: the six tests above. +- All six shipped binaries rebuilt with `build_kernel_cross.sh` and confirmed to load. +- No change to any Python module. + +## What measurement changed + +Three parts of the design above did not survive contact with a benchmark. They are left in place so +the reasoning can be compared against what actually happened. + +**Stage 1 is a no-op on its own.** Removing ~25,000 allocations of ~70KB per plan changed nothing: +26.291s before, 26.344s after, interleaved best of 3 against ~1.8% run-to-run spread. The premise was +a misreading of the kernel's own comment, which recorded the allocator being the bottleneck under +four threads contending — a different problem from one thread repeatedly allocating and freeing the +same-sized block, which never leaves the allocator's thread cache. Stage 1 is kept only because the +pool's parked workers need per-thread scratch, and because its two tests are worth having. + +**Broadcasting to wake workers cost most of the win.** The design says "broadcasts on `work_ready`". +Built that way the pool was worth 1.3% (25.833s → 25.489s), because a six-job batch woke all fifteen +workers and eleven of them went straight back to sleep — roughly 250,000 wasted wakeups per plan. +Giving each worker its own flag and condition variable, so exactly the needed lanes wake, took it to +3.7% (26.008s → 25.056s). The shipped code does that; the single `work_ready` variable in the sketch +above does not exist. + +**The `threads: auto` cap was measured and rejected.** With the pool in place the fast-machine curve +peaks below the core count — serial 26.33s, 4 threads 24.89s, 6 threads 24.71s, 8 threads 24.91s, +16 threads 25.04s — which argues for capping `auto`. Re-running with each job made eight times dearer, +which is how a machine where the kernel dominates behaves, the curve stops turning over: 48.92s +serial, 32.03s at 4, 29.98s at 6, 29.85s at 8, 28.94s at 16. Capping at 4 therefore costs 0.7% on a +fast machine but 10.7% on a kernel-heavy one, while not capping costs 1.3% at worst. `auto` is left +as the core count; `resolve_batch_threads` in `plan.py` carries the numbers. + +**Final result:** 26.33s serial → 24.71s at 6 threads on the 20-scenario benchmark, **6.1%**, on top +of the 35% the batching work delivered. All 20 scenarios byte-identical at every thread count, in +both the normal and the kernel-heavy regime. + +## What the tests found + +Every new test was mutation-checked, and two were not good enough on the first attempt: + +- The **fork** test earns its place. With the `pthread_atfork` handler removed the forked child + deadlocks on workers that no longer exist; the test kills it on a deadline and reports the timeout + rather than hanging the suite. +- The **concurrency** test initially did not bite. With the dispatch mutex deleted, 25 rounds over 4 + lanes passed happily — the GIL keeps two callers from overlapping often enough for the race to + land. At 400 rounds over 8 lanes it deadlocks within seconds. The round count is therefore + load-bearing and the test says so. +- The **scratch cross-context** test cannot fail before its change, since without reuse there is + nothing to leak; it was verified by relaxing `build_window_membership`'s `assign` to skip shrinking, + which makes it fail immediately.