Report piped species as converged when their sp/freq jobs succeed

[calvinp0]

The bug

A species_sp or species_freq task routed through pipe mode computes correctly, writes valid output, passes determine_ess_status and is ingested — and the species is then reported as failed, aborting the rate coefficient:

Species CH4 failed with status:
{'sp': False, 'conf_opt': True, 'freq': False, 'opt': True}
...
Error: Cannot compute a rate coefficient for CH4 + OH <=> CH3 + H2O.
       Species ['CH4', 'OH', 'CH3', 'H2O'] did not converge.

The data is all present. From restart.yml of that run: e_elect: -106300.678 and paths['freq']: True, sitting next to job_types: {'freq': False, 'sp': False}.

There are two independent causes.

1. Nothing sets the success flags. _ingest_species_sp() sets species.e_elect and _ingest_species_freq() sets output[label]['paths']['freq'], but neither sets output[label]['job_types']['sp'|'freq']. Those are only ever set on the non-piped path in scheduler.py. _ingest_species_sp() also never sets paths['sp'].

2. The species stops being reachable. check_all_done() is only called from the main loop while iterating running_jobs, and a piped species holds no entries there. Its empty entry was deleted on the first iteration after the batch was submitted — even though has_pending_pipe_work() had just reported it busy — so when the pipe run finished the label was gone and convergence stayed False.

Why the flags aren't just set directly

job_types['freq'] is not a "job finished" marker. It stands for the imaginary-frequency QA — exactly zero for a stable species, exactly one for a TS, with switch_ts() on failure. job_types['sp'] is set only at the end of post_sp_actions(), after the T1 diagnostic, solvation-scheme follow-ups and the reaction-E0 check.

Setting the booleans directly would convert a fail-safe bug, where good results are discarded, into a fail-open one where a stable species carrying an imaginary frequency is accepted as converged.

So piped tasks are routed through the same Scheduler methods the individually submitted ones use. This is cheap: post_sp_actions() is already path-based and needs no adaptation, and check_negative_freq() reads only local_path_to_output_file from its job argument, so a minimal PipedJobView supplies it.

Finalization runs per task inside the ingestion loop rather than in _post_ingest_pipe_run(), because a cross-species batch carries one species per task and because _post_ingest_pipe_run() is skipped entirely when any sibling task is ejected for troubleshooting. A task can reach COMPLETED while still holding a non-converged ESS output, so the convergence gate is repeated before any flag can be set.

Why this went unnoticed

The path is only reached when a species_sp/species_freq batch actually forms, which needs at least pipe_settings['min_tasks'] (default 10) homogeneous species-level tasks in a single flush. Ordinary runs have far fewer species, fall through to run_sp_job() / run_freq_job() per label, and are unaffected. Existing production runs only ever formed ts_opt batches, which have their own post-ingestion handler and never consult job_types.

It becomes reachable in large campaigns — which is exactly what pipe mode is for.

Nothing here is engine-specific; the ingestion helpers contain no engine branching.

Verification

End-to-end A/B on four species (CH4, OH, CH3, H2O), identical input, toggling only pipe_settings['enabled'], with min_tasks: 2 so the batches actually form:

	verdict
pipe off (control)	converged successfully ×4
pipe on, before	failed with status ×4
pipe on, after	converged successfully ×4

Confirmed in the log that the batches genuinely formed (Routing 4 species SP jobs to pipe mode), not that pipe mode was skipped.

Tests: 6 new for the coordinator (the two positive ones fail without the fix; guard cases cover non-converged ESS, missing output, unknown species, and other families) and 1 for has_pending_pipe_work() covering all five branches. arc/job/pipe/ 149 passed; arc/scheduler_test.py 30 passed serially.

Note: arc/scheduler_test.py has pre-existing order-dependent failures under xdist (TestScheduler shares class-level fixtures and project directories across its tests). Under CI's -n 6 --dist worksteal, main already fails test_initialize_output_dict and test_check_rxn_e0_by_spc without any of these changes; all pass with --dist loadscope or -n 0. The new test is its own TestCase to avoid perturbing that class further.

_check_ess_convergence is renamed to check_ess_convergence now that it has a caller in a sibling module.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 63.48%. Comparing base (9787770) to head (0e0355b).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #934      +/-   ##
==========================================
+ Coverage   63.45%   63.48%   +0.03%     
==========================================
  Files         114      114              
  Lines       38325    38358      +33     
  Branches    10030    10033       +3     
==========================================
+ Hits        24319    24352      +33     
- Misses      11089    11096       +7     
+ Partials     2917     2910       -7     

Flag	Coverage Δ
functionaltests	63.48% <ø> (+0.03%)	⬆️
unittests	63.48% <ø> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.