fix(sof-export): clean up partial results on cancel, plus a TTL reaper (#144)

[mauripunzueta]

Summary

Closes #144 and addresses the adjacent storage-hygiene gaps it surfaced in the SQL-on-FHIR async export controller ($viewdefinition-export / $sqlquery-export).

Previously, cancelling a job transitioned it to Cancelled but its already-written output shards were never deleted and stayed downloadable; and finished jobs were never reclaimed at all — the in-process jobs map and the sink output grew unbounded for the process lifetime, despite the completion manifest advertising a 24h Expires.

What changed

1. Cancellation cleanup (#144 as scoped)

• New delete_job(&job_id) on the ExportSink trait, implemented for all three sinks:
  • FilesystemSink — remove_dir_all on the job dir (missing dir is not an error)
  • InMemorySink — drops entries under the {job_id}/ key prefix
  • S3Sink — paginated list_objects_v2 + delete_object under the job key prefix (idempotent)
• Called from the cancellation path, and again from the background task if it finishes a job that was cancelled mid-run (covers the write-after-cancel race).
• read_shard is gated on terminal state so a cancelled job's files 404 even while deletion is still draining.

2. TTL reaper for finished jobs

• A background reaper deletes a terminal job's output and drops its bookkeeping once it ages past the TTL, bounding both storage and the in-memory map. Aligned with the manifest's advertised 24h Expires.
• JobStatus::Cancelled now carries cancelled_at, and a terminal_at() helper lets all three terminal states age uniformly.
• New config: HFS_EXPORT_OUTPUT_TTL (default 86400) and HFS_EXPORT_CLEANUP_INTERVAL (default 300). The interval is clamped to ≥ 1s (tokio::time::interval panics on zero).

3. Failed-job partial cleanup

• A failed job's result URL returns 500 with no manifest, so its shards are unreachable but were never deleted. They're now removed on failure, and read_shard gates Failed alongside Cancelled.

Deliberately not changed

DELETE on a completed job remains a no-op that preserves the Completed state (the status URL keeps redirecting to the result manifest), per test_export_cancel_after_completion_preserves_completed_state. Completed output is reclaimed by the TTL reaper above, not by DELETE.

Testing

• New unit tests: cancel_deletes_partial_output_and_download_404s, reap_expired_reclaims_terminal_jobs_only (age threshold, running-immunity, output deletion + bookkeeping removal).
• Full helios-rest lib suite (282) and the sof_export integration suite (44) pass, including the completion-preservation test.
• cargo fmt clean; cargo clippy --features s3 --all-targets -D warnings clean.

🤖 Generated with Claude Code

[claude]

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

[codecov]

Codecov Report

❌ Patch coverage is 91.50943% with 18 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
crates/rest/src/export/sink.rs	41.66%	7 Missing ⚠️
crates/rest/src/export/in_memory.rs	96.12%	6 Missing ⚠️
crates/persistence/src/sof/emit.rs	87.50%	3 Missing ⚠️
crates/rest/src/export/controller.rs	71.42%	2 Missing ⚠️

📢 Thoughts on this report? Let us know!