Add Chrysalis E3SM v3 archive ingestion

[tomvothecoder]

Description

Adds a targeted E3SM v3 archive ingestor that runs on Chrysalis and uploads cases to SimBoard through the remote HPC endpoint. This is required because v3 performance data is stored on Chrysalis rather than a filesystem accessible from NERSC.

• Defines documented v3 simulation names and scans archive snapshots from 2024-01
• Reuses remote discovery, validation, deduplication, packaging, and upload logic
• Skips whole-snapshot checkpoints for safe filtered reconciliation
• Adds reconciliation summaries, regression tests, and operator documentation
• Closes [DevOps]: Add v3 ingestion script for Chrysalis #280

Checklist

• Code follows project style guidelines
• Self-reviewed code
• No new warnings
• Tests added or updated
• All tests pass locally
• Documentation/comments updated
• No breaking change

Deployment Notes (if any)

Run dry-run reconciliation on Chrysalis with an externally reachable SIMBOARD_API_BASE_URL and service-account token before enabling uploads.

[Copilot]

Pull request overview

Adds a Chrysalis-specific, targeted E3SM v3 archive ingestor that filters to a
static list of documented v3 simulations, scans archive snapshots from 2024-01,
packages each selected case, and uploads via the remote
/api/v1/ingestions/from-hpc-upload route (without using whole-snapshot
checkpoints).

Changes:

• Introduces chrysalis_v3_archive_ingestor.py and a v3 reconciliation summary
  that fails the run when expected simulations are missing/transient/incomplete.
• Extends the shared archive ingestion plumbing to support optional case-path
  filtering, optional archive checkpointing, and a run-report object for
  specialized runners.
• Adds operator documentation and dedicated regression tests for the targeted
  v3 backfill behavior.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
docs/deploy/hpc-api-token-authentication.md	Documents Chrysalis v3 backfill workflow and upload endpoint usage.
docs/architecture/metadata-ingestion.md	Explains how the v3 backfill specializes remote automated upload mode and why checkpointing is disabled.
backend/tests/features/ingestion/test_chrysalis_v3_archive_ingestor.py	Adds regression coverage for v3 filtering, forced config, checkpoint bypass, and reconciliation outcomes.
backend/app/scripts/README.md	Documents the new Chrysalis v3 backfill script entry point and operational expectations.
backend/app/scripts/ingestion/nersc_archive_ingestor.py	Adds IngestorRunReport, configurable env overrides, and optional case filtering + report capture during archive scans/ingest.
backend/app/scripts/ingestion/hpc_upload_archive_ingestor.py	Adds hooks for case filtering, disabling archive checkpointing, and emitting run-report details.
backend/app/scripts/ingestion/chrysalis_v3_archive_ingestor.py	Implements the targeted v3 archive scan/upload runner with reconciliation logging and non-checkpoint behavior.

[tomvothecoder]

Need clarification on what directories to process v3 data. The v3 simulations were ran on Chrysalis, which means the performance data is there too.

On the E3SM Data Docs, there are HPSS links to the output as well. We can either

1. Parse the original performance data on Chrysalis
2. See if the required metadata is available on NERSC

[tomvothecoder]

Need clarification on what directories to process v3 data. The v3 simulations were ran on Chrysalis, which means the performance data is there too.

On the E3SM Data Docs, there are HPSS links to the output as well. We can either

1. Parse the original performance data on Chrysalis

2. See if the required metadata is available on NERSC

Hey @chengzhuzhang, can you clarify where we should be ingesting v3 data?