Cache hardening: auto-create dir, keep_csv default, partial-file cleanup

[nick-gorman]

Summary

Bundles three independent cache-related fixes cherry-picked from #67, plus a follow-up that fixes a latent NameError in the cleanup path and adds regression coverage.

Closes #55 (partial-file cleanup).

#	Commit	Author	What
1	Create raw_data_cache if it does not yet exist	@mdavis-xyz	cache_compiler now auto-creates the destination directory if it's missing, and raises UserInputError if the path exists as a regular file. Scope is narrow — dynamic_data_compiler and static_table (read-side) still require the directory to exist. Required conflict resolution: master's None-check (from #80) collided with PR67's adjacent error-message rewording; took PR67's logic for the directory-vs-file check, left master's None check unchanged.
2	Set keep_csv=True by default for cache_compiler	@mdavis-xyz	Behavior change — see below.
3	Tidy up incomplete feather/parquet (#55)	@mdavis-xyz	Wrap the feather/parquet write in try/except so a mid-write failure (disk full, interrupted writer, etc) doesn't leave an unreadable partial file on disk that breaks subsequent reads. Re-raises the original exception unchanged.
4	Fix os/_os shadowing in cleanup branch and add regression tests	me	See below.

⚠️ Behavior change — cache_compiler(keep_csv=…) default

cache_compiler's keep_csv default flips from False to True. Callers that didn't specify it explicitly will now retain the raw AEMO CSVs in raw_data_location alongside the typed feather/parquet. Rationale: keeping the source CSV is useful for downstream consumers and for re-reading without re-fetching from AEMO. Users who specifically don't want CSVs (e.g. disk pressure) can still pass keep_csv=False explicitly.

dynamic_data_compiler's default already was True on master — only cache_compiler is affected.

Bug found and fixed in commit 4

Commit 3's cleanup block, as cherry-picked from #67, used bare os.path.exists and os.remove:

except Exception:
    # tidy up incomplete file
    if os.path.exists(full_filename):   # ← NameError
        os.remove(full_filename)         # ← NameError
    raise

But data_fetch_methods.py imports the stdlib module as os as _os (line 2). So whenever a feather/parquet write actually failed:

1. The user-visible exception became NameError("name 'os' is not defined") instead of the real write error (the original error survives only as __context__).
2. The partial file was never cleaned up.

That silently regressed issue #55 to worse-than-before. Commit 4 swaps to _os.path.isfile / _os.unlink to match the convention on lines 781-782 of the same function.

Regression tests (commit 4)

All four new tests live in tests/end_to_end_table_tests/test_cache_compiler.py:

• test_creates_cache_directory_when_missing
• test_raises_when_cache_path_is_a_file
• test_keep_csv_true_by_default
• test_write_to_format_cleans_up_partial_file_on_failure[feather] + [parquet] — parametrized. Authored before applying the source fix and confirmed to fail with the exact NameError at line 792 on the buggy version, then to pass after the fix. Real regression coverage, not coincidental.

Test plan

• uv run pytest tests/end_to_end_table_tests/test_cache_compiler.py — 8/8 pass (3 pre-existing + 5 new)
• uv run pytest tests/ — 376 pass, 1 skipped, 1 pre-existing unrelated warning
• CI green