fix(octopus): don't rely on future IOG dispatch slots the car no longer needs

[chalfontchubby]

Fixes #4482.

Summary

• Octopus grants daytime Intelligent Go dispatch slots based on its own assumption of what the car needs, since it can't see the car's real SoC - only Predbat can, via car_charging_soc/car_charging_limit. load_octopus_slots() already caps car_charging_slots[car_n] at the car's real remaining requirement when octopus_intelligent_consider_full is on, zeroing the kwh of any slot beyond that - but rate_add_io_slots() (which decides which dispatch slots the house battery can rely on as a cheap rate) never looked at that, so it kept treating every granted slot as low rate regardless of whether the car's own plan still needed it.
• If Predbat had already deferred house battery charging into one of those "surplus" slots, the slot could disappear once Octopus notices the car has stopped drawing - leaving the battery undercharged with no cheap window left to make it up.
• Adds octopus_intelligent_limit_future_slots (expert mode, default Off, requires octopus_intelligent_consider_full to have any effect - warns at startup otherwise): a future out-of-window dispatch slot only counts as low rate for the house battery while it falls within a 30-min block car_charging_slots still lists a positive kwh for. A slot already underway or completed is trusted regardless (only future need is gated), and the fixed 23:30-05:30 window is never affected, since it's guaranteed cheap by the tariff itself.

Test plan

• 6 new rate_add_io_slots scenarios: partial-need-vs-full settlement period, car already full, current/completed dispatch exempted, switch off restores old behaviour, fixed window unaffected
• fetch_config_options warning test for the consider_full dependency
• ./run_all --quick
• ./run_pre_commit

[springfall2008]

Can you review this against: #4237

[chalfontchubby]

Checked against #4237 - no overlap or conflict, they're on different pathways:

• feat: strengthen Octopus Intelligent earlier-charge skew #4237 operates on self.io_adjusted, which is populated by fetch_octopus_rates() from Octopus's own rate-API is_intelligent_adjusted attribute across the whole import/export rate curve (octopus.py:2825, unrelated to any specific car). It's consumed downstream in plan.py's sort_window_by_price_combined() as a soft pence-level pricing gradient across a contiguous run of already-built charge/export windows.
• This PR (fix(octopus): don't rely on future IOG dispatch slots the car no longer needs #4483) operates on rate_add_io_slots(), which works from the explicit per-car dispatch slot list (octopus_slots/car_charging_slots) and decides, upstream, whether a future slot's low assumed_price gets written into rates[] at all. Nothing in it touches self.io_adjusted.

No file overlap either - #4237 only touches plan.py + its own new test module; this PR touches config.py/fetch.py/octopus.py/predbat.py + existing tests. Safe to merge in either order.

Functionally they're complementary rather than duplicative: this PR answers "should a future out-of-window slot be relied on for the house battery at all" (hard exclusion, gated on the car's real remaining SoC need), #4237 answers "given the slots that are being relied on, which within a run should be preferred first" (soft risk-decay ranking). A slot this PR excludes (needed=False) just carries its normal, non-discounted rate into #4237's sort stage, so #4237's few-pence gradient has negligible effect on it either way - it isn't priced as cheap in the first place.

One thing I can't verify from code alone: whether Octopus's is_intelligent_adjusted rate-curve flag (#4237's input) always lines up 1:1 with the specific per-car dispatch slots this PR reads (octopus_slots), or whether they can diverge in practice. Not something either PR depends on for correctness, but worth knowing if there's a real-world case where they disagree.

[Speshman]

One possible interaction with #4237/the Octopus rate input path may need testing. When metric_octopus_import is used, fetch_octopus_rates() can receive a dispatch period whose value_inc_vat is already the Intelligent low rate and whose is_intelligent_adjusted flag is true. By the time rate_add_io_slots() runs, rates[minute] may therefore already be 3.99p.

In that case, needed=False only prevents another low-rate assignment; it does not restore the already-adjusted rate to the normal out-of-window tariff. The block may consequently remain available to the optimiser.

I observed this shape in the live Octopus integration:

value_inc_vat: 0.03993
is_intelligent_adjusted: true

compared with the normal peak rate of 0.293222.

Could a test be added where the input rate is already low and io_adjusted is true, asserting that an unneeded future out-of-window block returns to the firm tariff? The implementation may also need to clear or ignore the corresponding io_adjusted marker so #4237 does not continue treating the rejected block as an Intelligent window.

The existing tests initialise all rates at 10p and rely on rate_add_io_slots() to reduce accepted blocks to 4p, so they do not currently cover this input path.

[chalfontchubby]

Excellent catch, thank you - traced it through and you're right.

fetch.py:978 runs rate_replicate(import_rates, self.io_adjusted, ...) before rate_add_io_slots() ever sees the data (fetch.py:981), and rate_replicate() only gap-fills minutes that are missing - it doesn't touch a minute that already has a real fetched value. So for a genuine Octopus Intelligent tariff, value_inc_vat can arrive already at the dispatch-discounted rate (your 3.99p example) for a period flagged is_intelligent_adjusted, before rate_add_io_slots() runs at all.

needed=False in this PR only skips the write (rates[minute] = assumed_price) - it was never built to undo an already-low rate, because on main today rate_add_io_slots() has no "reject" concept at all: it only ever adds a low rate for a slot it decides is valid, it never had to make one not low. This PR is the first thing that introduces that need, and I only tested it against fixtures that start every rate flat at 10p (so rate_add_io_slots() itself is always the one making a block cheap) - exactly the gap you spotted, not something the existing tests could have caught.

Proposed fix: when needed=False and the incoming rate at that slot is already below the normal tariff, restore it to self.rate_max_base (fetch.py:975 - the day's true peak rate, captured before any IO/saving-session/override distortion) rather than leaving it untouched. Also clear self.io_adjusted[slot_start] for the rejected block at the same time, so a future consumer of that flag (e.g. #4237's pricing gradient, once merged) doesn't keep treating a now-firm-tariff block as an Intelligent window.

rate_max_base is the closest existing signal I can find, but it's an approximation - the day's overall max, not necessarily that specific half-hour's true undiscounted rate. Since you've clearly got real API responses to hand: is there a better signal I should be restoring to instead (e.g. a standard/non-dispatch rate reported alongside the adjusted one), or is rate_max_base close enough in practice for how Octopus structures IOG tariffs?

Will add a test with rates pre-seeded low (mirroring your example) + io_adjusted=True, asserting an unneeded future block returns to the firm tariff, once the restore target is confirmed.

[Speshman]

I now have a live example confirming this input path and answering the restoration-target question.

For tonight, Octopus currently advertises these dispatches:

• 23:00-23:30: 1.753 kWh
• 00:30-04:30: 14.024 kWh
• 05:30-06:00: 0.5965 kWh

The car's actual SoC is 2%. With octopus_intelligent_consider_full enabled, Predbat forecasts it reaching 100% at approximately 03:15. Accordingly, Predbat assigns zero EV demand to the future 05:30-06:00 dispatch.

However, the Octopus rate event already supplies that surplus block as:

start: 2026-08-13T05:30:00+01:00
end: 2026-08-13T06:00:00+01:00
value_inc_vat: 0.03993
is_intelligent_adjusted: true
Predbat therefore still publishes 05:30-06:00 at 3.99p despite forecasting no EV demand in it. This demonstrates that needed=False must actively undo the adjusted input rate rather than merely skip writing another low rate.
Regarding the restoration value: the Octopus event does not provide an undiscounted value alongside an adjusted row. On this fixed IOG tariff, every ordinary out-of-window row is 0.293222 GBP/kWh and the event's max_rate is also 0.293222. Therefore, rate_max_base appears to be the appropriate conservative restoration target here.
One implementation detail: minute_data() populates self.io_adjusted for every minute covered by the adjusted period, not just slot_start (utils.py:598-603). The rejection path should therefore restore and clear every minute in the rejected block, for example while iterating it:
rates[minute] = self.rate_max_base
self.io_adjusted.pop(minute, None)
Clearing only self.io_adjusted[slot_start] would leave the other 29 minutes marked as Intelligent-adjusted.
This installation is currently running unpatched code, so this is evidence of the real input shape and current failure mode rather than a test of the revised PR implementation. I can also report whether Octopus later withdraws the surplus 05:30-06:00 dispatch after the car finishes charging.

[chalfontchubby]

Pushed the restoration fix discussed above.

A rejected slot (not needed, or the daily cap already reached) now restores rates[minute] to rate_max_base for every minute of the block, rather than just skipping the addition of a new low rate - previously it left whatever fetch_octopus_rates() had already put there untouched, which for a genuine Intelligent tariff can already be the dispatch-discounted rate. Also clears self.io_adjusted across the same range, not just slot_start - minute_data() sets it for every minute in an adjusted block, so a partial clear would leave the plan believing an already-restored minute was still IOG-adjusted.

New regression test simulates the exact scenario (rejected slot with rates[] pre-seeded as fetch_octopus_rates() would leave it) - confirmed it fails without this commit and passes with it. All 23 existing scenarios pass unchanged.

Full suite + pre-commit clean.

[Speshman]

Thanks, this now addresses the live failure mode I reported: restoring every minute to rate_max_base and clearing every corresponding io_adjusted marker is exactly what the pre-adjusted 3.99p input requires.

I noticed one possible side effect in the revised rejection path. The same destructive restoration also runs when the slot is rejected only because octopus_slot_max has been reached:

if needed and slots_per_day[day_offset] < octopus_slot_max:
...
else:
rates[minute] = self.rate_max_base
self.io_adjusted.pop(minute, None)

That changes the previous cap behavior even when octopus_intelligent_limit_future_slots is Off. It could also overwrite a legitimate tariff rate, including a guaranteed fixed-window rate, despite the PR intending 23:30–05:30 to remain unaffected.

Would it be safer to perform the active restore and io_adjusted clear only when needed is false? A slot rejected solely by the existing daily cap could retain the previous behavior unless its undiscounted source rate is known.

A regression test with the feature switch Off and octopus_slot_max already reached would confirm whether this behavior is intentional. A capped slot within 23:30–05:30 may also be worth covering.

I very much appreciate your work on this.

[chalfontchubby]

Pushed 3b1fedac addressing both points.

The restore scope: agreed, and confirmed the exact mechanism you flagged - needed stays True even with octopus_intelligent_limit_future_slots Off, since the daily octopus_slot_max cap is a pre-existing, unrelated mechanism. The destructive restore now only fires when needed is False; a slot rejected purely because the cap is already spent is left exactly as fetch_octopus_rates() set it, same as before this PR. Added the two tests you asked for - cap-only rejection with the feature Off, and one inside the fixed 23:30-05:30 window - both fail on the pre-fix code, pass now.

Second thing, flagged separately by Rik while reviewing this: a zero-kWh dispatch entry (a plug-independent SMART grid-flex event Octopus schedules that delivers no energy to the car) was still consuming a slot of the 12-slot daily budget under the old counting, which could crowd out real charging dispatches later in the day even on a day the car never got close to 12 real slots. Added octopus_slot_count_zero_kwh (default Off) so these are now exempt from both the cap and the needed check entirely - a real tariff discount, but not something the car-dispatch cap should be modelling in the first place. On (legacy) restores the old counting.

Thanks for reading the actual diff rather than just eyeballing the PR description - the cap-rejection path especially wasn't something we'd have caught without it.

[Speshman]

If you need anything else from me please just ask. I'm still quite new to all this so on a vertical learning curve 😁

[chalfontchubby]

Thanks @Speshman - I'm pretty new here myself. Subject to review from @springfall2008 now I think

[Speshman]

The main fix for #4482 still looks useful, but the branch is now well behind current main and newer changes affect the same Octopus slot code. Could this be rebased and retested?

I think the zero-kWh exemption should be removed from this PR or handled separately. At present, any slot decoded as zero kWh can bypass both the EV-need check and the daily slot cap. That can include malformed or incomplete input, not only genuine SMART grid-flex events.

The safer version would keep zero-kWh slots under the existing cap behaviour unless there is confirmed evidence that this exact event type always guarantees the cheap tariff.

Once rebased, it would also be useful to test against the newer IOG cap-splitting code and the io_adjusted planning changes.

[chalfontchubby]

Rebased onto current main (force-pushed) - 3 clean commits, no more merge noise. Verified rate_add_io_slots() itself was untouched on main since this branch diverged, so the rebase was purely mechanical (renamed a test-numbering collision with #4545's own new "Test 14" in test_fetch_config_options.py to avoid duplicate numbering, nothing functional).

Addressed the zero-kWh exemption concern: scoped zero_kwh_exempt to also require source == "SMART", not just kwh <= 0. decode_octopus_slot() silently coerces malformed/unparseable charge_in_kwh to 0.0 - indistinguishable by value alone from a genuine zero-kWh SMART event - so a garbage entry with any other source can no longer slip past both the cap and the #4482 need-check. Added test30_zero_kwh_non_smart_source_not_exempt, confirmed it fails without the scoping and passes with it.

Also checked against the newer IOG cap-splitting work (#4624/#4644, load_octopus_slots()) and the io_adjusted planning changes (#4237) - neither touches rate_add_io_slots(), confirmed by a clean auto-merge with no overlap, and test_octopus_slots.py's cap-split tests still pass unchanged.

Not stacking on #4528 for now - going to chase that one to land separately, and nothing that's merged since duplicates it.

Full rate_add_io_slots/fetch_config_options/octopus_slots/octopus_misc/octopus_free test modules pass; run_pre_commit clean.

[chalfontchubby]

Correction: the zero-kWh scoping fix I described above wasn't actually committed when I posted that - it was still sitting in the working tree. Pushed now as d0100fb.

[Speshman]

Thank you for taking the time to rebase this and revisit the zero-kWh concern. I had another look at the current head (d0100fb6). The rebase and compatibility concerns appear resolved, and the core restoration behaviour still looks valuable.

I found three possible edge cases. I may be missing some context, so please treat these as review questions rather than firm conclusions:

1. decode_octopus_slot() converts an invalid or unparseable charge_in_kwh value to 0.0. The new kwh <= 0 and source == "SMART" condition would therefore still exempt malformed input when its source is also SMART. Test 30 covers numeric zero with a non-SMART source, but not malformed SMART input. Would it be safer to preserve whether parsing succeeded, require an exact genuine zero, or separate this exemption from this PR?

2. The "already underway" check appears to use the rounded settlement-period start. For example, at 10:05 a dispatch beginning at 10:20 is rounded to 10:00, making slot_start <= current_block true even though the dispatch has not started. If the car no longer needs it, that future dispatch would still be trusted. Could the original dispatch start time be used for this check, with a regression test for a mid-period future start?

3. Rates and io_adjusted are shared while rate_add_io_slots() is called separately for each car. With overlapping multi-car dispatches, a later car that does not need the block could restore rate_max_base and clear io_adjusted after another car accepted the same block. This probably does not affect most single-car installations, but the result appears dependent on car processing order.

There is also some overlap with #4528's dispatch-trust work, so it may be worth ensuring the eventual combined design does not let the zero-kWh path bypass that trust policy.

None of this detracts from the main fix, which still looks useful, and I appreciate the care taken to address the earlier feedback and add the restoration tests. I would be happy to help test any follow-up if useful.

[chalfontchubby]

I'm concerned that if we tweak this too much more it will get stale again - If it is better than what is there then I'd try to get it merged and fix issues in follow up rather than going round in circles again. I'm aware of the pending conflicts with 4528. I have the car connected to IOG rather than the charger so the planning has better visibility of the needed slots anyway - but the plan being based around slots when I might have had to go out in the car certainly bothers me.

[Speshman]

That sounds reasonable to me. I agree there is a risk of the branch becoming stale again, and I would not want the edge cases I raised to prevent a useful core improvement from landing.

I would be comfortable with the partial-current-block and multi-car cases being recorded and handled as follow-ups, particularly as the latter does not affect my installation.

My only remaining hesitation is the zero-kWh exemption because it changes existing behaviour by default, independently of octopus_intelligent_limit_future_slots. As a minimal way to keep this PR tightly scoped, perhaps octopus_slot_count_zero_kwh could default to True, preserving the previous cap behaviour, or the exemption could be deferred to a separate follow-up.

Other than that, I agree that merging the improvement and iterating is preferable to allowing it to go stale again. Thank you again for engaging with all the review points.

[chalfontchubby]

If I'm switching context back to this I might as well look at the other issues in stacked pr as well - starting with raising issues for the follow on points.
Note I can't merge here - needs @springfall2008's blessing.

[chalfontchubby]

Pushed 369cbfb: octopus_slot_count_zero_kwh now defaults to True, so the zero-kWh exemption is opt-in rather than a silent change to everyone's cap behaviour on upgrade. Tests 27-30 updated to explicitly set the switch state they're exercising rather than relying on which value happened to be the default; test 29 now documents the actual default path.

Agreed this is the minimal fix to land now. I'll raise separate issues for the three remaining points (malformed-SMART-input scoping, the rounded "already underway" boundary, and multi-car ordering) and pick them up as follow-up PRs stacked on this one - not holding this PR open for them.

Thanks again both for the thorough review.

[chalfontchubby]

Moving this to draft.

This PR and #4528 address two halves of the same underlying problem, and both rewrite the same
decision point in rate_add_io_slots(). They conflict with each other, so whichever landed first
would have forced the other into a rebase that changed its behaviour — and in the meantime both have
drifted well behind main. Rather than keep rebasing them against each other, I'm reworking the
approach so the interaction between the two is decided deliberately instead of by merge order.

Parking this as draft so it's clear it isn't waiting on review in its current form. The review points
raised here are not lost — thanks in particular for the cap-rejection and already-discounted-rate
findings, which were real bugs and are carried forward.

---

🤖 Written by Claude Code on @chalfontchubby's behalf.

[chalfontchubby]

Closing in favour of #4885, which combines this with its sibling PR into a single squashed commit against current main.

Both this and its counterpart rewrote the same decision point in rate_add_io_slots(), which is what deadlocked them — each needed the other's changes to be reviewable, and both went stale. #4885 carries the review outcomes from both, including the threads resolved here.

Nothing from this branch is lost; it is superseded rather than abandoned.

Posted by Claude on behalf of chalfontchubby.