feat(debug): rolling debug-history snapshot buffer with automation hooks

[chalfontchubby]

Summary

Fixes #4417.

create_debug_yaml() (what bug reporters attach so a maintainer can replay their exact plan computation via unit_test.py --debug_file --redo) only ever wrote when switch.predbat_debug_enable was already on, on every ~5-minute cycle with no pruning at all - unbounded disk growth if left on, and no history to look back at for anyone who didn't already have it on before something went wrong (originally raised in #1745, never actually built).

Replaces that with a rolling buffer:

• debug_history.py (new) - capture/list/load/prune snapshots, mirroring the existing annual_store.py ring-buffer pattern and going through the Storage component rather than the filesystem directly. Snapshots are saved with format="text" (the payload is already-rendered YAML text from create_debug_yaml(write_file=False), not a Python object - format="yaml" would double-serialise it into something --debug_file couldn't parse).
• Three new config items: debug_history_count (default 15, 0 disables the routine capture), debug_history_interval (default 3h - 15x3h matches the History tab's worst-case "yesterday + today so far" coverage), and debug_history_force_capture - a self-resetting switch so an automation that notices something worth investigating can trigger an immediate, precisely-timed capture rather than waiting on the routine interval, then fetch it via ?id=latest without a separate lookup round-trip. Still honoured even when debug_history_count=0 (an explicit "give me one right now" is a different intent than "keep a rolling background history").
• predbat.py: _capture_debug_history(), throttled in-memory, wired alongside the existing debug_enable capture call. Independent of that switch entirely.
• web.py: /debug_history_list and /debug_history_download routes (reusing the existing html_file() attachment-serving helper the manual predbat_debug.yaml download already uses), plus a dropdown on the dashboard's Debug panel.
• web_helper.py: a Debug column on the plan's History/Yesterday view only (its rows are the ones with a real past timestamp to match a snapshot against - the live Plan view is mostly future predictions with nothing to match). Fetched and matched client-side by wall-clock time rather than threaded through publish_html_plan(), which is shared by three other call sites (live plan, Annual results viewer, Compare) with no relationship to when a snapshot was actually captured.
• Docs - customisation.md (new "Debug history" subsection), web-interface.md (dashboard dropdown), predbat-plan-card.md (new Debug column in the plan column reference).

Test plan

• apps/predbat/tests/test_debug_history.py - capture/list/load round-trip, pruning to a configured count on both Storage eviction paths (delete-capable and expiry-fallback), annotate_steps_back, corrupt-index handling, latest/falsy-id resolution, and a real-StorageLocalFiles round-trip of genuine YAML-syntax text (guards the format="text" vs "yaml" trap directly)
• apps/predbat/tests/test_debug_history_capture.py - interval throttling, count<=0 disabling routine capture, force-capture bypassing both the count and interval, the switch resetting itself, and graceful handling with no storage component available
• ./run_all --quick passes with no regressions
• ./run_pre_commit passes (ruff, black, cspell, markdownlint, full test suite)

🤖 Generated with Claude Code

[chalfontchubby]

Adds the ability to download debug files for a defined subset of points in the history without the ever expanding set of debug files enabled with "debug". This screenshot covers a period with multiple restarts, so the snapshot files are not at the expected regular intervals.

A follow on branch adds a new chart showing how these "historical plans" evolve over time (the planned soc overlaid for each historical snapshot) so it is possible to find out "what caused that weird thing I see in the history" and "why did the plan suddenly do something different"

[springfall2008]

Review

Nice feature — went through the diff in detail. A few correctness issues worth fixing before merge, plus some lower-priority cleanup.

Correctness bugs

1. predbat.py _capture_debug_history() — self.debug_history_last_capture = self.now_utc runs unconditionally after the try/except. A failed capture (exception caught and logged as a "Warning") still resets the throttle clock, so the next attempt is silently deferred for a full debug_history_interval (default 3h) instead of retrying next cycle.
2. Same function, storage-unavailable path — if not storage: return exits before that line, so the throttle timestamp never advances at all. If Storage is ever down, every ~5-minute cycle re-enters and re-logs the warning indefinitely.
3. Force-capture switch resets before the outcome is known — expose_config("debug_history_force_capture", False) fires immediately on trigger, before storage is checked or the capture is attempted. docs/customisation.md says "Predbat resets the switch back off itself once the snapshot has been taken" — the code resets it on trigger, not completion. An automation polling the switch for success has no reliable signal if the capture actually failed.
4. web.py html_debug_history_download, id=latest case — load_snapshot(storage, "latest") resolves "latest" internally and discards the resolved id; the route then calls list_snapshots() a second, independent time just to rebuild the filename. A capture landing between the two calls can serve one snapshot's bytes under another snapshot's filename.
5. debug_history.py capture_snapshot() max_age pruning — the try/except (KeyError, ValueError, TypeError) only wraps the fromisoformat parse, not the later existing_time < cutoff comparison. A naive (non-tz-aware) stored timestamp would raise an uncaught TypeError, aborting the index write and orphaning the just-captured snapshot in storage. Low likelihood today since now_utc is always tz-aware, but the guard doesn't cover what its comment claims it does.
6. web.py — "Snapshot {} not found".format(snapshot_id) reflects the raw ?id= query param unescaped into a text/html response. Reflected-XSS shape on a newly-added, unauthenticated route (this mirrors an existing pattern in html_file(), but it's new attack surface worth escaping while we're here).
7. Same-slot force-captures silently overwrite each other — the snapshot id is floored to the plan-slot grid, so firing debug_history_force_capture twice within one slot produces the same id and the second storage.save() silently clobbers the first — no log, no dedup/discard path.

Docs no longer match shipped behavior

8. docs/web-interface.md and docs/customisation.md describe a dropdown snapshot picker on the dashboard Debug panel. The merged code only adds one link, Download all (.tgz) — an earlier commit in this PR replaced the dropdown, docs weren't updated to match.
9. docs/predbat-plan-card.md still says snapshots are matched "nearest" to a slot and labelled by "how many snapshots ago" — the shipped JS does exact floored-slot matching and labels with absolute time instead.

Efficiency

10. create_debug_yaml() runs twice per cycle when debug_enable is on and a capture is due — update_pred() calls it directly, then _capture_debug_history() calls it again internally.
11. html_debug_history_download's id=latest path does two full index round-trips (see solcast entity nomenclature  #4).
12. load_all_snapshots awaits snapshot bodies sequentially instead of asyncio.gather (already used elsewhere, e.g. annual_job.py).
13. The bulk .tgz route buffers every snapshot body plus the full archive in memory rather than streaming — worth a thought given Predbat commonly runs on Pi-class hardware.
14. findNearestDebugSnapshot does an O(rows × snapshots) linear scan with repeated Date parsing per row, on every 5s poll of the Yesterday view — could build a Map once when debugHistoryData is fetched.
15. The plan table renders once synchronously, then again after loadDebugHistoryData() resolves — a visible flicker on first entry to the History/Yesterday view.

Reuse / duplication

16. web.py's new _storage() helper duplicates ComponentBase.storage, which WebInterface already inherits — could just call self.storage.
17. debug_history.py's ring-buffer (list/discard/prune) logic duplicates annual_store.py's equivalent rather than sharing it, and has already diverged (dedup + max_age only added to one side).
18. The plan-slot flooring math in _capture_debug_history() re-derives from scratch what output.py already computes via self.minutes_now.

Simplification

19. annotate_steps_back()/steps_back is computed and shipped on every /debug_history_list response, but the final JS only reads .id/.timestamp — looks like a leftover from an earlier design (before the move from "N steps back" labels to absolute time).
20. Three near-identical prune loops (same-minute dedup, max_age, max_count) could collapse into one parametrized helper.

Test coverage

Per this repo's usual bar, a couple of gaps:

21. The three new web.py routes and _storage() have no test coverage — tests/test_web_annual.py has an established pattern for this (build a FakeRequest, call the handler directly) that isn't applied here.
22. The new client-side JS (findNearestDebugSnapshot, loadDebugHistoryData, the Debug column rendering) is untested — tests/test_plan_why_reason.py has a precedent for testing this kind of embedded JS via a Python mirror of the function.

Minor

23. Dead .cspell dictionary entry (onfocus) left over from the removed dropdown markup.
24. Same-minute dedup keys on local-time strings, so a UK DST fall-back's repeated HH:MM could in theory collide and silently drop a real, distinct capture.

Most worth fixing before merge: #1–#4 and #6 (correctness/security), #8–#9 (docs actively mislead users right now), and #21–#22 (test coverage). The rest is good-to-have cleanup, happy to see it split into a follow-up if you'd rather keep this PR focused.

[springfall2008]

Suggestion: split debug_enable's file-write behavior from this feature, and give the rotating buffer its own switch

Two related follow-ups worth doing in this same PR while we're touching this area:

1. debug_enable still causes unbounded disk growth — the original problem isn't actually fixed yet

predbat.py's if self.debug_enable: self.create_debug_yaml() (still present, right next to the new _capture_debug_history() call) writes a brand-new predbat_debug_HH_MM_SS.yaml straight to config_root via raw os.makedirs/open() every ~5-minute cycle, with no cap and no pruning, whenever a user leaves the switch on. That's exactly the "unbounded disk growth if left on" issue this PR's own commit message cites as the motivation for the rotating buffer — but this PR adds the new bounded buffer alongside it rather than replacing it, so the original bug ships unchanged.

Suggest removing that call site entirely. Nothing is lost:

• The /debug_yaml route already serves the same content on demand without writing to disk (write_file=False).
• The new rotating buffer now covers "what was Predbat doing recently" automatically.

debug_enable's other ~40 usages (verbose logging in plan.py/execute.py/octopus.py, and the cache/kernel-reuse bypass in prediction.py/prediction_kernel.py for more accurate step-by-step debugging) are a separate concern and should stay as-is — only the disk-write coupling should go.

2. Give the rotating buffer its own on/off switch, rather than overloading debug_history_count == 0

Right now the feature's only off-switch is setting the debug_history_count input_number to 0. That's workable but inconsistent with how this codebase normally does on/off — debug_enable and debug_history_force_capture are both real switch CONFIG_ITEMS. Suggest adding switch.predbat_debug_history_enable (defaulting either way, your call) so "is this feature on" and "how many snapshots do I keep" are independent settings, matching the existing pattern instead of a magic-zero convention.

[chalfontchubby]

Thanks for the thorough review - went through it in priority order. Everything flagged as "most worth fixing before merge" is addressed, split into separate commits so each is reviewable on its own:

Correctness (items 1-4, 6)

• 50d33564 - debug_enable still caused unbounded disk growth even after this PR - the rotating buffer shipped alongside the original raw write, not in place of it. Rather than remove the write outright (it, and the verbose logging/kernel-bypass path it also gates, are genuinely useful at finer resolution than the history buffer's 1-hour floor for someone actively watching a live issue), it now auto-disables after 2 hours if left on.
• f1de7ed0 - items 1-3: debug_history_last_capture and the force-capture switch now only reset on a genuine successful capture, not unconditionally. A failed attempt no longer silently defers the next routine retry a full interval, the storage-unavailable warning is throttled instead of re-logging every ~5 minutes, and a failed forced capture leaves the switch on to retry rather than resetting before the snapshot was actually taken.
• 50706e09 - items 4 and 6: added resolve_and_load_snapshot() so id=latest resolves and loads in one call instead of two independent ones (closing the race where a capture landing in between could serve mismatched bytes/filename), and escaped the reflected ?id= query param in the 404 body.

Docs (items 8-9)

• 60b6af40 - updated web-interface.md, predbat-plan-card.md, customisation.md to describe the single "Download all (.tgz)" link and exact floored-slot matching that actually ships, not the earlier dropdown/nearest-match design.

Tests (items 21-22)

• 50706e09 also added test_web_debug_history_routes.py (all three routes, including the item 4/6 fixes, against a real StorageLocalFiles backend).
• 1134c514 added test_debug_history_client_js.py for findNearestDebugSnapshot/loadDebugHistoryData/the Debug column - following test_plan_why_reason.py's existing precedent for this kind of embedded-JS test (asserting on the JS source structure, since there's no JS engine in this suite).

Design suggestions in your second comment

• Went with your switch.predbat_debug_history_enable suggestion (7e790fc2) - also raised debug_history_count's minimum from 0 to 1, so it can no longer double as a second, conflicting off-switch.
• Kept the raw debug_enable disk write rather than removing it outright, for the reason in 50d33564 above - open to discussing further if you'd still rather see it gone entirely.

Full suite + pre-commit clean after every commit. The remaining items (#5, #7, #10-20, #23-24) are left as follow-up per your own note that they're good-to-have cleanup rather than blocking - happy to pick any of them up next if you'd like them in this PR instead.