Cache parent directory HTML for fast missing-file pre-check

[nick-gorman]

Summary

Caches each archive directory's HTML index for ~1 hour and uses it to answer "does this file exist?" locally instead of firing a guaranteed-404 request. Speedup hits whenever a single dynamic_data_compiler call scans multiple missing months in the same parent directory (e.g. start_time well before nem_data_model_start_time).

Smoke-tested against real nemweb:

• 1st missing-file check: ~570ms (parent HTML fetch)
• 2nd+ in same parent: ~10ms (cached)

Effectively the cost of N missing-file checks drops from N×(404 round-trip) to one HTML fetch + N local lookups.

What's new

• download_html(url) / download_html_as_soup(url) — TTL-cached helpers (1hr, 1024 entries). Also adopted by _get_matching_link and download_elements_file for the same caching benefit on the live HTML pages NEMOSIS already fetches.
• _pre_check_file_is_missing(url) — returns True/False/None against the parent HTML listing. None means "fall through to a real request" (covers non-nemweb URLs, non-archive extensions, parent HTML unreachable, etc).
• download_to_path calls the pre-check; a True answer raises the same requests.HTTPError(404) shape as a real 404, so all existing missing-month warning paths keep working unchanged.
• cachetools>=5.5.0 added to dependencies.
• Autouse fixture in tests/conftest.py clears _html_cache between tests for hermeticity.

Tests

8 new tests in tests/test_downloader.py:

• test_download_html_caches_across_calls — verifies the TTL cache by counting session.get calls
• test_pre_check_returns_none_for_non_nemweb_urls — protects every existing test that targets the local mock server
• test_pre_check_returns_none_for_non_archive_extensions — only .zip / .csv participate
• test_pre_check_returns_false_when_file_is_listed — present in parent HTML → False
• test_pre_check_returns_true_when_file_is_not_listed — absent → True
• test_pre_check_returns_none_when_parent_html_unreachable — transient infra failure falls through, doesn't suppress real downloads
• test_download_to_path_raises_http_404_when_pre_check_says_missing — end-to-end, with a session.get counter to prove the wire fetch is skipped
• test_download_to_path_proceeds_when_pre_check_says_present — False answer must not short-circuit a real download

Departures from PR #67

Adapted from Matt's 2a5812f. Differences:

• Exception type: PR Skip/Speed up tests, resolve most open issues #67 raised ValueError from the pre-check. This PR raises requests.HTTPError with a synthetic 404 response, matching the contract PR Adopt requests.Session, bump UA to Chrome 130, raise_for_status everywhere #85 established (every download function surfaces a 404 as HTTPError). All existing 404-warning callers keep working unchanged.
• Dropped the assert file_url.startswith("https://") that PR Skip/Speed up tests, resolve most open issues #67 had in the pre-check. The downloader is exercised against http:// URLs by the offline test fixtures, so an unconditional HTTPS assertion would break the test suite. The pre-check now simply returns None for any URL outside the configured nemweb prefixes — same effect for real callers.
• Module-level _html_cache exposed so tests can clear it between cases. PR Skip/Speed up tests, resolve most open issues #67's @cached(cache=TTLCache(...)) decorator-inline pattern would have leaked state across tests.
• _NEMWEB_HTML_PRECHECK_PREFIXES as a module-level constant (vs. PR Skip/Speed up tests, resolve most open issues #67's inline literal). Lets tests monkeypatch the prefix list to point at the local mock server.

Part of the PR #67 breakdown effort (#8c of ~10).

Test plan

• uv run pytest tests/test_downloader.py — 16/16 pass
• uv run pytest — full suite 397 passed / 1 skipped (no regressions)
• Smoke test against real nemweb confirms speed-up
• CI matrix (3.10–3.14 × ubuntu/windows/macos)

🤖 Generated with Claude Code