Skip to content

Commit 36eaf6c

Browse files
authored
Fix crash and blind restart when Solcast drops past periods mid-day (#57) (#57)
Two related bugs triggered on 2026-06-01 at 10:38 UTC when the intra-day forecast refresh ran and Solcast had stopped returning the completed Morning period, leaving only Aftn and Eve. Bug 1 (crash): the intra-day refresh replaced today_period_forecast wholesale with the 2-period result while ordered_period_windows still held Morn, Aftn, Eve. _process_period_windows then crashed with KeyError: 'Morn' trying to look up today_period_forecast['Morn']. Bug 2 (blind restart): after the crash, the service restarted with a fresh SchedulerState (today_period_windows = {}). Because current_date already equalled today's date, _handle_forecast_refresh fell into the intra-day branch instead of the day-reset branch, computed daytime_periods = [], and fired no period handlers for the rest of the day. Fixes: - scheduler_operations: merge (not replace) today_period_forecast on intra-day refresh so Morn is preserved after Solcast drops it. - scheduler_operations: day-reset always anchors windows to the canonical ["Morn","Aftn","Eve"] triplet, so a mid-day restart doesn't halve the window widths (same root cause as the period-windows shift fixed in #54). - scheduler_coordinator: trigger day-reset when ordered_period_windows is empty (fresh startup) instead of falling into the intra-day branch. - scheduler_coordinator: guard _process_period_windows against any period absent from today_period_forecast as defence-in-depth. - Two new regression tests cover both scenarios.
1 parent 148828c commit 36eaf6c

4 files changed

Lines changed: 102 additions & 9 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1524,6 +1524,9 @@ python scripts/battery_throughput.py
15241524

15251525
## Recent Updates
15261526

1527+
**2026-06-01**
1528+
- **Fix: crash and blind restart when Solcast drops past periods mid-day.** Two related bugs surfaced when the monitor restarted at 10:38 UTC. (1) The intra-day forecast refresh replaced `today_period_forecast` wholesale with Solcast's 2-period result (Aftn, Eve) while `ordered_period_windows` still held all 3 periods — the coordinator then crashed with `KeyError: 'Morn'`. (2) On restart, `today_period_windows` is empty so the startup flow fell into the intra-day branch, produced `daytime_periods = []`, and fired no handlers for the rest of the day. Fixed by: merging (not replacing) `today_period_forecast` on intra-day refresh; always anchoring window boundaries to the canonical `["Morn","Aftn","Eve"]` triplet on day-reset so windows remain thirds even after a mid-day restart; triggering a day-reset when `ordered_period_windows` is empty at startup; and guarding `_process_period_windows` against any period absent from the forecast.
1529+
15271530
**2026-05-31**
15281531
- **Fix: export restart blocked after SOC floor hit until SOC recovers.** When a timed export was stopped early by the SOC floor, the 15-minute cooldown previously allowed a restart as soon as it expired — even if SOC had only recovered marginally above the floor. This caused brief oscillating re-exports (e.g. a 3-minute export) that immediately re-hit the floor. A per-period `soc_floor_hit` flag is now set in `day_state` when the SOC floor fires. The period-start export path then requires SOC to recover to `DAYTIME_TIMED_EXPORT_MIN_SOC_PERCENT + TIMED_EXPORT_FLOOR_HIT_RECOVERY_PERCENT` (default 40% + 7.5% = 47.5%) before restarting. The flag is per-period and resets at midnight. Configurable via `TIMED_EXPORT_FLOOR_HIT_RECOVERY_PERCENT` in `settings.py`.
15291532

‎logic/scheduler_coordinator.py‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -160,7 +160,7 @@ async def _handle_auth_refresh(self, now: datetime, today: date) -> None:
160160

161161
async def _handle_forecast_refresh(self, now: datetime, today: date) -> bool:
162162
"""Refresh forecast data if needed. Returns True if the tick should be skipped."""
163-
if today != self.state.current_date:
163+
if today != self.state.current_date or not self.state.ordered_period_windows:
164164
self.state.current_date = today
165165
try:
166166
await refresh_daily_data(self.state, self.logger, reset_day_state=True)
@@ -359,6 +359,8 @@ async def _process_period_windows(self, now: datetime, today: date) -> None:
359359

360360
if not night_tick_consumed:
361361
for period_index, (period, period_start) in enumerate(self.state.ordered_period_windows):
362+
if period not in self.state.today_period_forecast:
363+
continue
362364
solar_value, status = self.state.today_period_forecast[period]
363365
period_end_utc = (
364366
self.state.ordered_period_windows[period_index + 1][1]

‎logic/scheduler_operations.py‎

Lines changed: 16 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -60,8 +60,15 @@ async def refresh_daily_data(
6060
logger.info("[SCHEDULER] Refreshing daily forecast and sunrise/sunset data.")
6161
state.forecast_calibration = build_and_save_forecast_calibration()
6262
forecast_obj: SolarForecastProvider = create_solar_forecast_provider(logger)
63-
state.today_period_forecast = forecast_obj.get_todays_period_forecast()
63+
new_today = forecast_obj.get_todays_period_forecast()
6464
state.tomorrow_period_forecast = forecast_obj.get_tomorrows_period_forecast()
65+
if reset_day_state:
66+
state.today_period_forecast = new_today
67+
else:
68+
# Intra-day: merge rather than replace. Solcast stops returning past periods
69+
# (e.g. Morn after ~10:00), but today_period_windows still references them;
70+
# a full overwrite would cause a KeyError in _process_period_windows.
71+
state.today_period_forecast.update(new_today)
6572
logger.info(f"[SCHEDULER] Today's forecast: {state.today_period_forecast}")
6673
logger.info(f"[SCHEDULER] Tomorrow's forecast: {state.tomorrow_period_forecast}")
6774

@@ -95,17 +102,18 @@ async def refresh_daily_data(
95102
)
96103

97104
if reset_day_state:
98-
# Derive today's period windows from the full 3-period forecast at day start.
99-
# Intra-day refreshes must NOT recalculate these: once Morn completes Solcast
100-
# stops returning it, so the list shrinks to 2 periods and the windows would
101-
# be recalculated as halves instead of thirds, shifting Eve ~3 h too early.
102-
daytime_periods = order_daytime_periods(state.today_period_forecast)
103-
state.today_period_windows = derive_period_windows(sunrise_utc, sunset_utc, daytime_periods)
105+
# Always anchor windows to the full canonical period set. Solcast drops
106+
# past periods mid-day, so a mid-day restart would otherwise produce
107+
# 2-period windows (halves of the solar day) instead of the correct thirds.
108+
# _process_period_windows skips any period absent from today_period_forecast.
109+
canonical_periods = ["Morn", "Aftn", "Eve"]
110+
state.today_period_windows = derive_period_windows(sunrise_utc, sunset_utc, canonical_periods)
104111
state.ordered_period_windows = sorted(state.today_period_windows.items(), key=lambda item: item[1])
105112
state.day_state = {
106113
p: {"pre_set": False, "start_set": False, "clipping_export_set": False, "high_soc_export_set": False, "soc_floor_hit": False}
107-
for p in daytime_periods
114+
for p in canonical_periods
108115
}
116+
daytime_periods = canonical_periods
109117
else:
110118
daytime_periods = list(state.today_period_windows.keys())
111119
for period in daytime_periods:

‎tests/test_scheduler_helpers.py‎

Lines changed: 80 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -381,3 +381,83 @@ def test_is_pre_sunrise_discharge_window_false_outside_months() -> None:
381381
months_csv="4,5,6,7,8,9",
382382
lead_minutes=60,
383383
) is False
384+
385+
386+
# ---------------------------------------------------------------------------
387+
# Tests for mid-day restart / Solcast period-drop resilience (2026-06-01 crash)
388+
# ---------------------------------------------------------------------------
389+
390+
def _make_day_state(periods: list[str]) -> "dict[str, Any]":
391+
return {p: {"pre_set": False, "start_set": False, "clipping_export_set": False,
392+
"high_soc_export_set": False, "soc_floor_hit": False} for p in periods}
393+
394+
395+
def test_refresh_daily_data_intraday_preserves_dropped_periods(tmp_path, monkeypatch) -> None:
396+
"""Intra-day forecast refresh must merge, not replace, so past periods
397+
that Solcast has stopped returning are kept in today_period_forecast.
398+
Regression: 2026-06-01 crash KeyError: 'Morn'."""
399+
import asyncio
400+
from unittest.mock import patch as _patch, MagicMock
401+
from datetime import date
402+
from logic.scheduler_operations import refresh_daily_data
403+
from logic.scheduler_state import SchedulerState
404+
from logic.schedule_utils import derive_period_windows
405+
406+
sunrise = datetime(2026, 6, 1, 4, 5, tzinfo=timezone.utc)
407+
sunset = datetime(2026, 6, 1, 20, 51, tzinfo=timezone.utc)
408+
409+
state = SchedulerState(current_date=date(2026, 6, 1))
410+
state.today_period_forecast = {"Morn": (1059, "Red"), "Aftn": (2871, "Amber"), "Eve": (2466, "Amber")}
411+
state.today_period_windows = derive_period_windows(sunrise, sunset, ["Morn", "Aftn", "Eve"])
412+
state.ordered_period_windows = sorted(state.today_period_windows.items(), key=lambda x: x[1])
413+
state.day_state = _make_day_state(["Morn", "Aftn", "Eve"])
414+
415+
mock_provider = MagicMock()
416+
mock_provider.get_todays_period_forecast.return_value = {"Aftn": (2757, "Amber"), "Eve": (2330, "Amber")}
417+
mock_provider.get_tomorrows_period_forecast.return_value = {"Morn": (2142, "Amber")}
418+
419+
with _patch("logic.scheduler_operations.build_and_save_forecast_calibration", return_value={}), \
420+
_patch("logic.scheduler_operations.create_solar_forecast_provider", return_value=mock_provider), \
421+
_patch("logic.scheduler_operations.get_sunrise_sunset", side_effect=[
422+
("2026-06-01T04:05:14+00:00", "2026-06-01T20:51:26+00:00"),
423+
("2026-06-02T04:04:22+00:00", "2026-06-02T20:52:37+00:00"),
424+
]):
425+
asyncio.run(refresh_daily_data(state, _test_logger, reset_day_state=False))
426+
427+
assert "Morn" in state.today_period_forecast, "Morn must be preserved after intra-day merge"
428+
assert state.today_period_forecast["Aftn"] == (2757, "Amber")
429+
assert state.today_period_forecast["Eve"] == (2330, "Amber")
430+
431+
432+
def test_refresh_daily_data_reset_uses_canonical_period_windows(tmp_path, monkeypatch) -> None:
433+
"""Day-reset must always compute windows against canonical ['Morn','Aftn','Eve']
434+
even when Solcast only returns 2 periods (mid-day restart scenario).
435+
Regression: after 2026-06-01 crash, restart left ordered_period_windows=[]."""
436+
import asyncio
437+
from unittest.mock import patch as _patch, MagicMock
438+
from datetime import date
439+
from logic.scheduler_operations import refresh_daily_data
440+
from logic.scheduler_state import SchedulerState
441+
442+
state = SchedulerState(current_date=date(2026, 6, 1))
443+
444+
mock_provider = MagicMock()
445+
mock_provider.get_todays_period_forecast.return_value = {"Aftn": (2757, "Amber"), "Eve": (2330, "Amber")}
446+
mock_provider.get_tomorrows_period_forecast.return_value = {"Morn": (2142, "Amber"), "Aftn": (4831, "Green"), "Eve": (2455, "Amber")}
447+
448+
with _patch("logic.scheduler_operations.build_and_save_forecast_calibration", return_value={}), \
449+
_patch("logic.scheduler_operations.create_solar_forecast_provider", return_value=mock_provider), \
450+
_patch("logic.scheduler_operations.get_sunrise_sunset", side_effect=[
451+
("2026-06-01T04:05:14+00:00", "2026-06-01T20:51:26+00:00"),
452+
("2026-06-02T04:04:22+00:00", "2026-06-02T20:52:37+00:00"),
453+
]):
454+
asyncio.run(refresh_daily_data(state, _test_logger, reset_day_state=True))
455+
456+
assert set(state.today_period_windows.keys()) == {"Morn", "Aftn", "Eve"}
457+
assert len(state.ordered_period_windows) == 3
458+
assert set(state.day_state.keys()) == {"Morn", "Aftn", "Eve"}
459+
# Windows must be thirds of the solar day — Eve start must be > halfway
460+
morn_start = state.today_period_windows["Morn"]
461+
aftn_start = state.today_period_windows["Aftn"]
462+
eve_start = state.today_period_windows["Eve"]
463+
assert morn_start < aftn_start < eve_start

0 commit comments

Comments
 (0)