Fix solar-export over-crediting + faithful control vocabulary (binary store/export, continuous reconstruction) (#145) by johanzander · Pull Request #146 · johanzander/bess-manager

johanzander · 2026-06-19T09:15:35Z

Summary

Design spec (no code yet) for the root-cause fix from #145: the optimizer over-credits solar export because it books export of surplus the control then stores instead. Fix: model surplus handling as a binary per-period choice — STORE-all (load_first, charge 100%, no phantom export) or EXPORT-all (grid_first, charge 0, credit export) — and credit export per disposition, not from a disposition-agnostic energy balance.

Forecast-robust by construction: STORE stays at 100% so bonus solar is captured and no phantom export is booked.

Stacked on PR #144 (the simulator) — it's the acceptance gate. Rebase to main once #144 lands. Supersedes #141 if the dither dissolves.

Acceptance gate (the validation loop the simulator unlocks)

Implement the DP change.
Existing suite green (savings may correctly shift down where phantom export is removed).
Simulator, same solar → R == P (cent-exact).
Simulator, more solar than planned → prove forecast-robustness + A/B old vs new on the bonus-solar case.

Spec: docs/superpowers/specs/2026-06-19-binary-solar-surplus-design.md. Implementation is gated on spec review.

🤖 Generated with Claude Code

Per-period STORE-all (charge 100%, no phantom export) vs EXPORT-all (charge 0, credit export) choice; forecast-robust; export credited per disposition not from a disposition-agnostic energy balance. Acceptance gate is the simulator (R==P same-solar + forecast-robustness A/B on more-solar-than-planned). Supersedes #141 if the dither dissolves. Implementation gated on spec review.

…ptance gate Financial safety: the binary fix ships only if simulator-verified realized savings are not reduced vs the current impl (same-solar and more-solar cases). Reported/planned savings dropping to the truthful executable figure is expected and is not a regression; reduced realized savings is, and blocks merge.

8 TDD tasks: characterize current; idle=EXPORT disposition; charge=STORE (store-all surplus); mirror in _build_period_data; re-baseline suite (intentional savings shifts); flip plan-faithfulness xfail (R==P); forecast-error A/B harness; full gate + reproduction-day re-measure. Hard merge gate: realized(new)>=realized(old).

…#145)

…nly excess (#145)

…#145)

DP change alone isn't executable: IDLE(export) and SOLAR_STORAGE(store) collapse to the same load_first command. Add tasks: 4a route power-0 surplus export to EXPORT_ARBITRAGE/grid_first; 4b confirm control mapping; 4c make simulator mode_to_power faithful (load_first=store-all, grid_first=export). Gate: R~=P on reproduction day before Tasks 6-8.

…first (#145) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…port (#145) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

When surplus > POWER_TOLERANCE_KW, set grid_to_battery = 0 in all three places (_state_transition, _compute_reward, _build_period_data) so the planner's model matches load_first hardware behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…er threshold) (#145) Closes the IDLE-with-small-surplus residual: tiny surpluses (~0.1 kWh) now route to grid_first (export) instead of staying IDLE->load_first (which stored them). Reproduction-day gap 1.17 -> 1.07 SEK; IDLE deltas eliminated.

Replace the grid-snapped path extraction (reading stored_period_data[(t,i)] which reported battery_soe_end snapped to the 0.1 kWh DP grid) with a continuous reconstruction loop that carries exact floating-point SoE and cost_basis forward using _state_transition and _build_period_data. The policy array (already returned by _run_dynamic_programming) is used to look up the chosen action; stored_period_data is no longer needed by the caller. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

#145) When there is no solar surplus, a charge action maps to battery_first (grid charging) which executes at MAX rate on hardware. The DP now matches this behaviour: grid_to_battery = remaining_rate instead of min(power*dt, remaining_rate). Applied identically in _state_transition, _compute_reward, and _build_period_data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

) Remove xfail from test_realized_equals_planned_on_controlled_scenario: the DP now plans grid charging at max rate so the simulator reproduces the plan exactly (R == P, cent-accurate). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Enables scenario tests to assert realized economics (R), not just the plan (P). For an executable plan R==P to the cent; a gap is a control-fidelity finding. Shared _scenario_inputs so P and R use identical inputs. (#145)

…#145) - realized_under_solar_error: optimize on forecast solar, execute on actual (higher) solar; assert realized never worse (bonus captured, no phantom export). - test_scenarios_are_plan_faithful: run scenarios through the simulator and assert R==P within the DP SoE-grid resolution (tests realized, not just plan). - update the obsolete SOLAR_STORAGE diagnostic to the new store-all model.

…el (#145) Phantom solar-export revenue removed → reported savings drop to the realisable figure (20 down, 3 up, 1 flat; total -28.7 SEK across the set, concentrated on high-spread sunny days e.g. 2024_08_16: 80.2->67.3). Regenerated directly from the optimizer; only expected_results changed.

- expected_results regenerated at 4dp (avoids 1dp rounding-boundary flips); economic shifts from removing phantom solar-export (24 scenarios moved, total -28.7 SEK, concentrated on high-spread sunny days). - expected_behavior updated where intent distribution shifted (SOLAR_STORAGE dropped, LOAD_SUPPORT/EXPORT reclassified) and savings_positive=false where a no-opportunity day now nets 0. - test_scenarios.py: assert R==P (realized == planned) within DP grid resolution; 4 high-consumption deep-peak scenarios xfail pending the discharge-pacing fix (#147).

…guide The closed-loop simulator is the key verification tool: it executes the optimizer's plan through the inverter and checks realized economics (R) match planned (P). Required for any optimizer/control change. Documents the core API, when to use it, what it caught (#145, #147), and the xfail-with-tracking-issue convention for known gaps.

* docs: design spec for binary solar-surplus handling (#145) Per-period STORE-all (charge 100%, no phantom export) vs EXPORT-all (charge 0, credit export) choice; forecast-robust; export credited per disposition not from a disposition-agnostic energy balance. Acceptance gate is the simulator (R==P same-solar + forecast-robustness A/B on more-solar-than-planned). Supersedes #141 if the dither dissolves. Implementation gated on spec review. * docs: add hard merge condition realized(new) >= realized(old) to acceptance gate Financial safety: the binary fix ships only if simulator-verified realized savings are not reduced vs the current impl (same-solar and more-solar cases). Reported/planned savings dropping to the truthful executable figure is expected and is not a regression; reduced realized savings is, and blocks merge. * docs: implementation plan for binary solar-surplus handling (#145) 8 TDD tasks: characterize current; idle=EXPORT disposition; charge=STORE (store-all surplus); mirror in _build_period_data; re-baseline suite (intentional savings shifts); flip plan-faithfulness xfail (R==P); forecast-error A/B harness; full gate + reproduction-day re-measure. Hard merge gate: realized(new)>=realized(old). * test(dp): characterize current solar-surplus disposition (#145) * feat(dp): idle is EXPORT disposition — exports surplus, holds battery (#145) * feat(dp): charge is STORE disposition — stores all surplus, exports only excess (#145) * feat(dp): mirror STORE/EXPORT disposition flows in _build_period_data (#145) * docs: extend plan with control-vocabulary tasks 4a-4c (#145) DP change alone isn't executable: IDLE(export) and SOLAR_STORAGE(store) collapse to the same load_first command. Add tasks: 4a route power-0 surplus export to EXPORT_ARBITRAGE/grid_first; 4b confirm control mapping; 4c make simulator mode_to_power faithful (load_first=store-all, grid_first=export). Gate: R~=P on reproduction day before Tasks 6-8. * feat(dp): power-0 surplus export classifies as EXPORT_ARBITRAGE/grid_first (#145) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * feat(sim): mode_to_power models load_first store-all vs grid_first export (#145) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(dp): SOLAR_STORAGE charges from solar only, no grid top-up (#145) When surplus > POWER_TOLERANCE_KW, set grid_to_battery = 0 in all three places (_state_transition, _compute_reward, _build_period_data) so the planner's model matches load_first hardware behaviour. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(dp): classify any power-0 surplus export as EXPORT_ARBITRAGE (lower threshold) (#145) Closes the IDLE-with-small-surplus residual: tiny surpluses (~0.1 kWh) now route to grid_first (export) instead of staying IDLE->load_first (which stored them). Reproduction-day gap 1.17 -> 1.07 SEK; IDLE deltas eliminated. * feat(dp): reconstruct reported plan with continuous SoE (R==P) (#145) Replace the grid-snapped path extraction (reading stored_period_data[(t,i)] which reported battery_soe_end snapped to the 0.1 kWh DP grid) with a continuous reconstruction loop that carries exact floating-point SoE and cost_basis forward using _state_transition and _build_period_data. The policy array (already returned by _run_dynamic_programming) is used to look up the chosen action; stored_period_data is no longer needed by the caller. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * fix(dp): GRID_CHARGING charges at max rate (binary, mirrors solar fix) (#145) When there is no solar surplus, a charge action maps to battery_first (grid charging) which executes at MAX rate on hardware. The DP now matches this behaviour: grid_to_battery = remaining_rate instead of min(power*dt, remaining_rate). Applied identically in _state_transition, _compute_reward, and _build_period_data. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test(sim): controlled-scenario R==P now holds with grid-charge fix (#145) Remove xfail from test_realized_equals_planned_on_controlled_scenario: the DP now plans grid charging at max rate so the simulator reproduces the plan exactly (R == P, cent-accurate). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * test: add run_scenario_realized helper (executes plan via simulator → R) Enables scenario tests to assert realized economics (R), not just the plan (P). For an executable plan R==P to the cent; a gap is a control-fidelity finding. Shared _scenario_inputs so P and R use identical inputs. (#145) * test(sim): forecast-robustness harness + scenario R==P checks (Task 7, #145) - realized_under_solar_error: optimize on forecast solar, execute on actual (higher) solar; assert realized never worse (bonus captured, no phantom export). - test_scenarios_are_plan_faithful: run scenarios through the simulator and assert R==P within the DP SoE-grid resolution (tests realized, not just plan). - update the obsolete SOLAR_STORAGE diagnostic to the new store-all model. * test: re-baseline 24 scenario expected_results for binary surplus model (#145) Phantom solar-export revenue removed → reported savings drop to the realisable figure (20 down, 3 up, 1 flat; total -28.7 SEK across the set, concentrated on high-spread sunny days e.g. 2024_08_16: 80.2->67.3). Regenerated directly from the optimizer; only expected_results changed. * test: complete scenario re-baseline + R==P check (#145, #147) - expected_results regenerated at 4dp (avoids 1dp rounding-boundary flips); economic shifts from removing phantom solar-export (24 scenarios moved, total -28.7 SEK, concentrated on high-spread sunny days). - expected_behavior updated where intent distribution shifted (SOLAR_STORAGE dropped, LOAD_SUPPORT/EXPORT reclassified) and savings_positive=false where a no-opportunity day now nets 0. - test_scenarios.py: assert R==P (realized == planned) within DP grid resolution; 4 high-consumption deep-peak scenarios xfail pending the discharge-pacing fix (#147). * docs: document the plan-faithfulness simulator (R==P) in the testing guide The closed-loop simulator is the key verification tool: it executes the optimizer's plan through the inverter and checks realized economics (R) match planned (P). Required for any optimizer/control change. Documents the core API, when to use it, what it caught (#145, #147), and the xfail-with-tracking-issue convention for known gaps. * ci: add Merge gate required-check + tighten merge-gate policy Adds a single always-running `ci-gate` job ("Merge gate") that depends on every CI job and fails only if one genuinely failed or was cancelled (skipped allowed). This is the one status check to mark required in branch protection on `main`: it transitively enforces the rest — including the slow Algorithm job whenever core/bess/ changed — without the conditional-job deadlock that naming `Algorithm tests` directly would cause. Also tightens docs/agents/workflow.md step 5: merge only after (a) all CI green incl. the slow job and (b) explicit user approval, and bans `gh pr merge --auto` (which bypassed the human gate on #146). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>

… (binary store/export, continuous reconstruction) (#145) (#146)" (#151) This reverts commit 71413a1.

…ion fix (#152) * Revert "Revert "Fix solar-export over-crediting + faithful control vocabulary (binary store/export, continuous reconstruction) (#145) (#146)" (#151)" This reverts commit b8ae0a6. * fix(dp): guarantee IDLE (power=0) action so value iteration can hold #146's "any positive power charges at max rate" turned the smallest positive power level (previously a ~negligible hold) into a full-rate grid charge, leaving the DP with no way to represent holding the battery. The value function then fell below the always-achievable all-IDLE floor and the profitability gate bailed to an all-IDLE schedule — breaking no-solar price arbitrage (test_defers_charging_to_cheaper_overnight_window). Fix: ensure power=0 is in the discretized action set. V[0,start] restored (-130.2 -> -91.3), zero IDLE-floor violations. Preserves #146's binary store/export over-crediting fix and R==P. The regression test's EXPORT_ARBITRAGE assertion was economically stale (it encoded the pre-#145 over-crediting model — exporting grid-charged energy at that spread is a loss); corrected to assert the faithful behaviour (battery used to serve load in the expensive window, savings > 5). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * test: re-baseline evening-peak-with-solar expected_results for IDLE-action fix The IDLE (do-nothing) action fix lets the optimizer hold instead of being forced to charge, improving this scenario: battery_solar_cost 0.52 -> 0.40 (savings 1.84 -> 1.96, 78.0% -> 83.1%). Adding an action is monotonic — cost can only drop — so this is a genuine improvement, not a regression. Regenerated from the optimizer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

…doc (#150) Spec/plan lifecycle policy: the spec is the durable design record; a feature's plan is execution scaffolding that drifts once shipped. Delete the plan when the feature is implemented, keep the spec. Documented in workflow.md and applied retroactively — all 5 plans map to shipped features (demo mode v9.5.0, add-inverter-platform skill, issue-integrate Stage C, simulator #144, binary store/export #146), so they are removed; all 4 specs kept. Also removes the transient investigation note (content lives in #145/CHANGELOG), and moves the plan-faithfulness simulator reference out of the buried testing.md section into its own discoverable docs/agents/simulator.md, with a short human-facing overview added to DEVELOPMENT.md (the tool's primary audience). Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>

The corrected #146 re-land (#152) included the DP IDLE-action fix that restores no-solar price arbitrage. Add it to the 9.6.0 Fixed section. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

The IDLE-action bug only existed transiently on main between #146 and #151 (revert) — it was never shipped in any released version. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

…ander#140/johanzander#136 The optimizer fix (johanzander#146), production-safety hardening (johanzander#140), and the installation-guide rework (johanzander#136) landed on main since v9.5.0 without the per-merge CHANGELOG/version-bump steps (the johanzander#146 merge skipped the gate). This backfills the user-facing CHANGELOG and bumps config.yaml to 9.6.0. MINOR bump: johanzander#146 visibly changes optimizer schedules and corrects the savings figures users see (binary store/export, no phantom solar export). Agent/CI/test-infra PRs (johanzander#137/johanzander#138/johanzander#139/johanzander#143/johanzander#144) are intentionally not in the user-facing changelog; the simulator (johanzander#144) is noted as the verification mechanism inside the johanzander#146 entry. No code change — CHANGELOG.md + version only. Does NOT tag or release. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

johanzander and others added 20 commits June 20, 2026 11:06

test(dp): characterize current solar-surplus disposition (#145)

4863f70

feat(dp): idle is EXPORT disposition — exports surplus, holds battery (…

a3ac9e9

…#145)

feat(dp): charge is STORE disposition — stores all surplus, exports o…

91e35d8

…nly excess (#145)

feat(dp): mirror STORE/EXPORT disposition flows in _build_period_data (…

b8a154f

…#145)

feat(dp): power-0 surplus export classifies as EXPORT_ARBITRAGE/grid_…

8f8687a

…first (#145) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

feat(sim): mode_to_power models load_first store-all vs grid_first ex…

977777e

…port (#145) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

johanzander force-pushed the fix/solar-surplus-binary-store-export branch from f4e91b7 to 8e99da3 Compare June 20, 2026 09:07

johanzander changed the base branch from feat/closed-loop-savings-simulator to main June 20, 2026 09:07

johanzander changed the title ~~Fix solar-export over-crediting: binary store/export surplus handling (#145)~~ Fix solar-export over-crediting + faithful control vocabulary (binary store/export, continuous reconstruction) (#145) Jun 20, 2026

johanzander marked this pull request as ready for review June 20, 2026 09:08

johanzander merged commit 71413a1 into main Jun 20, 2026

This was referenced Jun 20, 2026

ci: add Merge gate required-check + enforce human merge gate #148

Merged

release: changelog + version bump to 9.6.0 #149

Merged

This was referenced Jun 20, 2026

docs: keep specs, drop completed plans; promote simulator to its own doc #150

Merged

Revert #146: binary store/export broke no-solar price arbitrage #151

Merged

johanzander added a commit that referenced this pull request Jun 20, 2026

Revert "Fix solar-export over-crediting + faithful control vocabulary…

b8ae0a6

… (binary store/export, continuous reconstruction) (#145) (#146)" (#151) This reverts commit 71413a1.

johanzander mentioned this pull request Jun 20, 2026

Re-land corrected #146: solar-export over-crediting fix + DP IDLE-action fix #152

Merged

johanzander mentioned this pull request Jul 4, 2026

bug: DP backward-induction cost-basis threading is dead code — anti-cycling floor evaluates every discharge against frozen initial cost basis #234

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix solar-export over-crediting + faithful control vocabulary (binary store/export, continuous reconstruction) (#145) - #146

Fix solar-export over-crediting + faithful control vocabulary (binary store/export, continuous reconstruction) (#145)#146
johanzander merged 20 commits into
mainfrom
fix/solar-surplus-binary-store-export

johanzander commented Jun 19, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Conversation

johanzander commented Jun 19, 2026

Summary

Acceptance gate (the validation loop the simulator unlocks)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant