Skip to content

Commit fc0255a

Browse files
dandwhelanclaude
andcommitted
Snapshot the uncommitted Predbat edits running on the Pi
These nine files were edited in /opt/predbat/config on the Pi and never pushed. They are all listed in manifest.yaml, so the next update from the Predbat UI would mv -f over them and the work would be gone. Captured as-is from the running v712.18 install, with one exception: predbat.py takes main's version (keeping the now_utc timezone fix, which the Pi predates) plus only the Pi's own four-line addition, so neither change is lost. The inverter.py and predbat.py edits are one change - GivTCP REST export target writeback, where the discharge_target_soc register never reads back on some hardware. The "unwritable" set moves off the Inverter, which is rebuilt every cycle, onto the base, and rest_write_acknowledged() falls back to the POST result when the readback cannot confirm a write. The rest is annual/web_annual work: costs, weather and tariff. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1 parent f5784d3 commit fc0255a

9 files changed

Lines changed: 1033 additions & 33 deletions

File tree

‎apps/predbat/annual.py‎

Lines changed: 340 additions & 21 deletions
Large diffs are not rendered by default.

‎apps/predbat/annual_costs.py‎

Lines changed: 59 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,16 @@
4343
# figure, without doing any arithmetic themselves.
4444
"quoted_pv_gbp": 0.0,
4545
"quoted_total_gbp": 0.0,
46+
# Air source heat pump, installed. A typical 2025/26 UK retrofit including the
47+
# cylinder and the radiator work an MCS installer signs off.
48+
"heat_pump_install_gbp": 13000.0,
49+
# The Boiler Upgrade Scheme grant, paid to the installer and deducted from the
50+
# price the household actually pays, so it reduces CAPITAL rather than being an
51+
# income stream. Zero it for a household that does not qualify.
52+
"heat_pump_grant_gbp": 7500.0,
53+
# A real quote for the heat pump installation, before the grant. Zero means "no
54+
# quote" rather than "free", matching quoted_pv_gbp/quoted_total_gbp above.
55+
"quoted_heat_pump_gbp": 0.0,
4656
}
4757

4858
# Accepted in a stored config but ignored: an earlier revision asked for a battery-only
@@ -164,6 +174,55 @@ def build_costs(total_kwp, battery_kwh, settings):
164174
}
165175

166176

177+
def build_heat_costs(settings):
178+
"""Return the heat pump capital breakdown: list price, grant, and what is actually paid.
179+
180+
The grant reduces capital rather than appearing as a saving, because that is how the
181+
Boiler Upgrade Scheme works in practice - it is paid to the installer and the
182+
household is invoiced the difference. Treating it as an annual income stream would
183+
make the payback period depend on how long it was spread over, which is meaningless.
184+
185+
The net figure is floored at zero: a grant larger than the price would otherwise
186+
produce a negative capital cost and a nonsensical payback.
187+
"""
188+
quoted = float(settings.get("quoted_heat_pump_gbp", 0) or 0)
189+
install = quoted if quoted > 0 else float(settings.get("heat_pump_install_gbp", 0) or 0)
190+
grant = min(float(settings.get("heat_pump_grant_gbp", 0) or 0), install)
191+
return {
192+
"install_gbp": round(install, 2),
193+
"grant_gbp": round(grant, 2),
194+
"net_gbp": round(max(0.0, install - grant), 2),
195+
"quoted": quoted > 0,
196+
}
197+
198+
199+
def build_heat_payback(heat_scenarios, heat_costs, months_included):
200+
"""Return the heat pump payback rows, or a reason they could not be built.
201+
202+
One row per hardware scenario, because the answer genuinely differs between them:
203+
a heat pump's extra electricity is far cheaper for a household that can charge a
204+
battery overnight and run the heat pump off it than for one importing every kWh at
205+
the daytime rate. Collapsing that to a single number would hide the interaction this
206+
tool exists to measure.
207+
208+
Like ``build_payback``, this refuses a partial year rather than extrapolating one:
209+
heating is the most seasonal load in the house, so a year missing January is not
210+
eleven twelfths of an answer - it is missing the part that matters most.
211+
"""
212+
if not heat_scenarios:
213+
return {"available": False, "reason": "No month produced a usable heat pump result, so there is nothing to pay back."}
214+
if months_included != 12:
215+
return {
216+
"available": False,
217+
"reason": "Heat pump payback needs a full year, but only {} of 12 months could be modelled. Heating is strongly seasonal, so a partial year cannot be scaled up. The missing months are named in the caveats.".format(months_included),
218+
}
219+
220+
rows = {}
221+
for key, entry in heat_scenarios.items():
222+
rows[key] = payback_row(heat_costs["net_gbp"], entry["annual_saving_p"] / 100.0)
223+
return {"available": True, "scenarios": rows}
224+
225+
167226
def payback_row(capital_gbp, annual_saving_gbp, recurring_gbp=0.0):
168227
"""Return one payback row: capital divided by the net annual saving.
169228

‎apps/predbat/annual_tariff.py‎

Lines changed: 12 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -108,10 +108,20 @@ def __init__(self, config, log, predbat, storage=None, fetch_json=None, timezone
108108
self.unpaid_export_months = set()
109109

110110
def _resolve_url(self, url, name, dno_region=None):
111-
"""Substitute templated arguments such as {dno_region} into a tariff URL, without mutating predbat.args."""
111+
"""Substitute templated arguments such as {dno_region} into a tariff URL, without mutating predbat.args.
112+
113+
Prefilled URLs can come from two different sources with two different
114+
placeholder names for the same value: the tariff catalogue's own entries
115+
use ``{dno_region}``, while URLs copied from the user's live apps.yaml
116+
(``rates_import_octopus_url`` / ``rates_export_octopus_url`` - see
117+
web_annual.py's prefill_config) use Predbat's own long-standing
118+
``{octopus_region}`` convention. Both must resolve to the same region
119+
letter, so both keys are offered here regardless of which one the URL
120+
actually contains.
121+
"""
112122
if not url:
113123
return None
114-
extra_args = {"dno_region": dno_region} if dno_region else None
124+
extra_args = {"dno_region": dno_region, "octopus_region": dno_region} if dno_region else None
115125
return self.predbat.resolve_arg(name, url, indirect=False, extra_args=extra_args)
116126

117127
@staticmethod

‎apps/predbat/annual_weather.py‎

Lines changed: 122 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,7 +13,9 @@
1313
"""
1414

1515
import math
16-
from datetime import timedelta
16+
from datetime import datetime, timedelta
17+
18+
import pytz
1719

1820
from annual_http import fetch_json
1921
from solar_model import convert_azimuth, gti_hourly_to_period_kwh
@@ -23,6 +25,16 @@
2325

2426
HOURLY_VARIABLES = "global_tilted_irradiance,temperature_2m,wind_speed_10m"
2527

28+
# The heat model needs outdoor temperature and nothing else. Requested on its own rather
29+
# than lifted out of the PV download because a heat pump can be modelled with no solar
30+
# arrays configured at all, in which case there is no PV request to lift it from.
31+
TEMPERATURE_VARIABLE = "temperature_2m"
32+
33+
# A local day must have at least this many of its 24 hours present to be usable. Short
34+
# days are a real occurrence at the edges of the downloaded window and on the spring
35+
# clock change; a day missing more than a few hours would understate its heat demand.
36+
MIN_HOURS_PER_LOCAL_DAY = 20
37+
2638
# A month needs at least this many usable forecast/actual day pairs before its
2739
# measured P10 ratio is trusted over the flat fallback.
2840
MIN_DAYS_FOR_P10 = 7
@@ -230,6 +242,65 @@ def _derive_p10_ratios(self, actual_periods, forecast_periods, forecast_availabl
230242

231243
return ratios, fallback_months
232244

245+
def _build_temperature_url(self, year):
246+
"""Build the Open-Meteo request for a calendar year of hourly outdoor temperature.
247+
248+
No tilt or azimuth: temperature is a property of the site, not of any array, so
249+
this is a single request regardless of how many arrays are configured - or
250+
whether any are.
251+
"""
252+
return "{}?latitude={}&longitude={}&start_date={}-01-01&end_date={}-01-01&hourly={}&timezone=UTC".format(ARCHIVE_URL, self.latitude, self.longitude, year, year + 1, TEMPERATURE_VARIABLE)
253+
254+
@staticmethod
255+
def _temperature_payload_problem(data):
256+
"""Return why a temperature payload cannot be trusted, or None when it is usable."""
257+
if not data:
258+
return "no data"
259+
hourly = data.get("hourly", {})
260+
times = hourly.get("time", [])
261+
values = hourly.get(TEMPERATURE_VARIABLE, [])
262+
if not times or not values:
263+
return "no hourly values"
264+
if len(values) < len(times):
265+
return "truncated ({} stamps, {} values)".format(len(times), len(values))
266+
return None
267+
268+
async def fetch_temperatures(self, year):
269+
"""Download a calendar year of hourly outdoor temperature, keyed by tz-aware UTC hour.
270+
271+
Returns an empty dict when the download is unusable, leaving the caller to decide
272+
whether that is fatal - it is for a heat pump run, and irrelevant for one without.
273+
Cached like the PV series, so a repeated run costs no request.
274+
"""
275+
cache_key = "temperature_{}_{}_{}".format(year, self.latitude, self.longitude)
276+
data = None
277+
if self.storage:
278+
cached = await self.storage.load("annual", cache_key)
279+
if self._temperature_payload_problem(cached) is None:
280+
data = cached
281+
if not data:
282+
fetched = await self.fetch_json(self._build_temperature_url(year))
283+
problem = self._temperature_payload_problem(fetched)
284+
if problem is not None:
285+
self.log("Warn: Annual: outdoor temperature data for {} is unusable ({})".format(year, problem))
286+
return {}
287+
data = fetched
288+
if self.storage:
289+
await self.storage.save("annual", cache_key, data, format="json")
290+
291+
hourly = data["hourly"]
292+
series = {}
293+
for index, stamp_text in enumerate(hourly["time"]):
294+
value = hourly[TEMPERATURE_VARIABLE][index]
295+
if value is None:
296+
continue
297+
try:
298+
stamp = datetime.strptime(stamp_text, "%Y-%m-%dT%H:%M").replace(tzinfo=pytz.utc)
299+
except (ValueError, TypeError):
300+
continue
301+
series[stamp] = float(value)
302+
return series
303+
233304
async def fetch(self, year):
234305
"""Download and convert a calendar year, returning a populated WeatherYear."""
235306
actual_periods = await self._fetch_series(ARCHIVE_URL, year, "actual")
@@ -243,6 +314,56 @@ async def fetch(self, year):
243314
return WeatherYear(actual_periods, forecast_periods, ratios, forecast_available, fallback_months)
244315

245316

317+
def temperature_by_local_day(series, timezone):
318+
"""Regroup a UTC-keyed hourly temperature series into 24 values per LOCAL date.
319+
320+
The heat model works in local days because the load profile does: minute 0 of a
321+
``load_forecast`` is local midnight, so hour 0 of a day's heat demand has to be
322+
local midnight too. Feeding it a UTC-keyed series would shift the whole heating
323+
profile by the UTC offset - an hour in British Summer Time, which lands the evening
324+
peak in the wrong half-hourly rate slot.
325+
326+
Both clock changes are handled rather than discarded. In autumn two UTC hours map
327+
onto the same local hour, so they are averaged; in spring one local hour never
328+
happens, so it is filled from its neighbours along with any other isolated gap.
329+
A local day still short of ``MIN_HOURS_PER_LOCAL_DAY`` real readings is dropped
330+
entirely - that is the first and last day of the downloaded window, which only ever
331+
hold a partial day, and a day filled mostly from guesses would understate its heat.
332+
"""
333+
zone = pytz.timezone(timezone)
334+
collected = {}
335+
for stamp, value in series.items():
336+
local = stamp.astimezone(zone)
337+
collected.setdefault(local.date(), {}).setdefault(local.hour, []).append(value)
338+
339+
by_day = {}
340+
for day, hours in collected.items():
341+
if len(hours) < MIN_HOURS_PER_LOCAL_DAY:
342+
continue
343+
averaged = {hour: sum(values) / len(values) for hour, values in hours.items()}
344+
by_day[day] = _fill_missing_hours(averaged)
345+
return by_day
346+
347+
348+
def _fill_missing_hours(averaged):
349+
"""Return 24 hourly values, filling any gap from the nearest hour that has one.
350+
351+
Only ever called for a day already known to hold at least
352+
``MIN_HOURS_PER_LOCAL_DAY`` real readings, so a filled hour is always surrounded by
353+
real ones - this is interpolation across a clock change or an isolated dropout, not
354+
invention of a day that was never downloaded.
355+
"""
356+
present = sorted(averaged)
357+
filled = []
358+
for hour in range(24):
359+
if hour in averaged:
360+
filled.append(averaged[hour])
361+
continue
362+
nearest = min(present, key=lambda candidate: abs(candidate - hour))
363+
filled.append(averaged[nearest])
364+
return filled
365+
366+
246367
POSTCODE_URL = "https://api.postcodes.io/postcodes/{}"
247368

248369

‎apps/predbat/db_manager.py‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,7 @@
1717

1818
from datetime import timedelta, datetime, timezone
1919
import asyncio
20+
import sqlite3
2021
import time
2122
import traceback
2223
import threading
@@ -26,6 +27,7 @@
2627
IPC_TIMEOUT = 60.0 # Seconds to wait for IPC response
2728
MAINTENANCE_HOUR = 3 # Local hour at which nightly database maintenance runs
2829
IDLE_WAIT = 30.0 # Seconds to wait for new work before re-checking housekeeping
30+
DB_ERROR_RECOVER_AFTER = 5 # Consecutive corruption errors before rebuilding the connection
2931

3032

3133
class DatabaseManager(ComponentBase):
@@ -50,6 +52,7 @@ def initialize(self, db_enable, db_days, db_days_daily=365):
5052
self.api_started = False
5153
self.last_success_timestamp = None
5254
self.last_commit_time = datetime.now(timezone.utc)
55+
self.db_error_count = 0
5356

5457
def bridge_event(self, loop):
5558
"""
@@ -141,17 +144,50 @@ async def start(self):
141144
self.db_engine._commit_db()
142145
self.last_commit_time = now
143146
self.last_success_timestamp = datetime.now(timezone.utc)
147+
self.db_error_count = 0
144148

145149
except Exception as e:
146150
self.log(f"Error in database thread: {e}")
147151
self.log("Error: " + traceback.format_exc())
152+
self.recover_from_error(e)
148153

149154
if hasattr(self.db_engine, "_commit_db"):
150155
self.db_engine._commit_db()
151156
self.db_engine._close()
152157
self.log("db_manager: Stopped cleanly")
153158
self.api_started = False
154159

160+
def recover_from_error(self, error):
161+
"""
162+
Rebuild the connection after repeated corruption errors.
163+
164+
The queue loop otherwise has no recovery of its own - DatabaseEngine only retries
165+
and quarantines at startup - so a single bad page leaves Predbat running with
166+
db_primary set and every history write failing silently for the life of the process.
167+
That is exactly what happened on 2026-08-02, and it only cleared by luck.
168+
169+
A lock or a busy file is not corruption: another reader will let go on its own and
170+
reconnecting mid-contention would just make it worse, so those are left alone.
171+
"""
172+
if not isinstance(error, sqlite3.DatabaseError) or isinstance(error, sqlite3.OperationalError):
173+
return
174+
175+
self.db_error_count += 1
176+
if self.db_error_count < DB_ERROR_RECOVER_AFTER:
177+
return
178+
179+
self.log("Warn: db_manager: {} database errors in a row, rebuilding the connection".format(self.db_error_count))
180+
self.db_error_count = 0
181+
try:
182+
# _cleanup_db_retry reconnects, re-checks, and quarantines the file if it really
183+
# is damaged, so this recovers or starts a fresh database rather than looping
184+
self.db_engine._connect()
185+
self.db_engine._cleanup_db_retry()
186+
self.log("Info: db_manager: Database connection rebuilt")
187+
except Exception as e:
188+
self.log("Error: db_manager: Unable to recover the database: {}".format(e))
189+
self.log("Error: " + traceback.format_exc())
190+
155191
def run_maintenance(self):
156192
"""
157193
Run the nightly prune and vacuum once per day, in the small hours.

‎apps/predbat/fetch.py‎

Lines changed: 4 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -313,7 +313,10 @@ def fill_load_from_power(self, load_minutes, load_power_data):
313313

314314
# Fill zero periods with integrated power data
315315
if zero_periods:
316-
self.log("Warn: Found {} periods of zero load with power data, filling using power integration".format(len(zero_periods)))
316+
# Not a warning - the gap is found and repaired here, so this is routine. A real
317+
# historic outage would otherwise re-report itself every cycle until it aged out
318+
# of the history window.
319+
self.log("Info: Found {} periods of zero load with power data, filling using power integration".format(len(zero_periods)))
317320
# Print the first 5 periods for debugging
318321
for i, (period_start, period_end, base_value) in enumerate(zero_periods[:5]):
319322
start_timestamp = self.now_utc - timedelta(minutes=period_start)

0 commit comments

Comments
 (0)