Profile pipeline & write integrity: parse crashes, blanking writes, version races (3 PRs)

[ahueb]

Verified defects across the profile generation pipeline and every writer that touches ResearcherProfile.

Originally verified at origin/main @ b7edcbc (2026-07-30). Re-verified 2026-08-11 against the open PR-stack tip (issue-29-authorship-grounding @ b1d54da = main + #30/#31/#32); line numbers below refer to that tree.

Priority (triage 2026-08-11)

V1 (with the (user_id, pmid) constraint) is now Tier 1: the publications table became ground truth for issue #29's fail-closed authorship guard (_load_publication_records, simulation.py:4565-4596), and the backfill that unmutes the 11 zero-publication labs runs straight through COR-15/COR-16 (and issue #23's V9 transport — coordinate). V6-remainder and C1 are Tier 3.

These three must still ship together, in order: V1 → V6 → C1. C1 rewrites profile_version sites inside blocks V6 rewrites; splitting them across PRs guarantees a merge conflict. (The old note to land issue #23's V10 first is obsolete — V10 is fixed in the stack, so C1 can rewrite invite.py:241-244 directly.)

PR V1 — ORCID/PubMed/pipeline null-&-type safety + publication dedup (small-medium)

• COR-15 — still present. orcid.py:127 summary.get("external-ids", {}).get("external-id", []) runs outside the try (the try wraps only the HTTP fetch, :103-107); ORCID returns "external-ids": null for DOI/PMID-less works → AttributeError → the step-3 catch zeroes the whole works list. Same pattern at :115 for "title": null, and the same unguarded chain exists in fetch_orcid_grants (:92) and fetch_orcid_profile (:29-39). The null-container shape is demonstrably real in these payloads: the adjacent field is defensively guarded for exactly this (:122-123, publication-date) — one field was hardened, its siblings weren't. Also reachable via int(pub_date["year"]["value"]) on a null/non-numeric year (:124). Damage is now partly contained by the stack — profile_pipeline.py:391's lost_evidence gate keeps a previously-grounded stored profile instead of overwriting — but first-run victims still get an ungrounded profile stored. Fix: null-safe parse at all sites; a test pinning null tolerance (none exists — the contract test uses a fully-populated external-ids).
• COR-16 — still present, and reproducible in a single run: no (user_id, pmid) unique constraint (models/publication.py:13-35 has no __table_args__; 0001 created only non-unique indexes; no later migration adds one), the pmids list is built with no dedup (profile_pipeline.py:115), and the existing_pubs guard dict (:182) is never updated with rows added in the loop (:212-217) — a PMID appearing twice in one ORCID works list inserts two rows. Downstream, all live: scalar_one_or_none() raises MultipleResultsFound, swallowed at logger.debug (:272-281, methods text silently lost); duplicated citation lines in the exported profile (profile_export.py:78-105); inflated admin counts (admin.py:92). Issue GoodBot claimed authorship of a paper its PI did not co-author (inverse of #7) #29's guard itself is dedup-immune (it accumulates into a set), but _load_publication_records is an unbounded full join re-run on every roster sync — duplicates are a straight multiplier on it.
  pubmed.py:207-208 reads .text, which truncates at the first child element ("Role of <i>TP53</i>…" → "Role of "); the abstract has the identical defect (:213-218), and no itertext() exists anywhere in src/. This propagates to the export, the synthesis prompt, and — post-Authorship grounding: fix issue #29 across emit, memory, tools, and prompts #32 — to fetch_abstract/fetch_full_text, the very tools that now return authors + DOI for authorship grounding (DOIs are unaffected; titles arrive truncated).
  _validate_profile (:561-571) does research_summary.split() on an LLM null → AttributeError, and the call at :317 is not in a try — it escapes into the worker's retry-to-dead path; the techniques check uses len() not isinstance ("PCR" scores len == 3, passes, then :409 assigns a str to an ARRAY(String) column). The word-count gate (<100 or >350) also disagrees with both the log message and the retry prompt ("150-250"). Fix: (user_id, pmid) unique constraint + in-run dedup; itertext(); null/type-safe validation.
  • Note: the new unique constraint needs a de-dup step in the same migration, and makes PR V5's IntegrityError discipline (issue Web request-path robustness: concurrent-insert 500s and a 5-minute event-loop freeze (2 PRs) #24) relevant to any writer of this table.

PR V6 — Profile-write integrity + atomic writes (medium)

• COR-22 — partially fixed in stack. The pipeline now gates the overwrite on validated/lost_evidence (profile_pipeline.py:383-406) and persists synthesis_validated (:414). Residual: (a) no web save route ever resets synthesis_validated — once a run stores an unvalidated draft, a later refresh treats the PI's hand-edited profile as not worth protecting and overwrites it; (b) the first-ever run still stores an unvalidated / zero-evidence synthesis unconditionally (:407-419, deliberate per :341-366); (c) raw_abstracts_hash is written even on the discard path (:372), so change-detection records an input whose output was thrown away. pending_profile is still read-but-never-written (sole reader admin.py:105, an unreachable branch; three specs still document the flow as existing). The Form("") blanking is still present at three routes — agent_page.py:1233-1262, profile.py:113-166, onboarding.py:111-154 — each overwrites every profile field unconditionally; profile.py guards the user fields in the same function (if name:) and not the profile fields, good evidence the omission is unintentional. Fix: reset synthesis_validated on PI save; wire or delete pending_profile; don't overwrite fields absent from a POST.
• COR-24 (write-safety slice) — still present, minus one sub-item. No os.replace/tempfile/flock anywhere in src/; truncate-then-write at profile_export.py:118/:142, agent_page.py:1131, agent/agent.py:711/:738. Ordering: the public writers are now consistently DB→disk; the one inversion left is the private-profile save at agent_page.py:1129-1140 (disk first; the if profile: guard makes a missing ResearcherProfile row a permanent disk-only write; write_text there omits encoding=). The pipeline writes disk at :482 under a flush-only transaction (the commit happens later in worker/main.py:97), so a rollback leaves disk ahead of the DB. create_revision has no content-dedup — fixed in stack (profile_versioning.py:84-92 returns the latest revision unchanged on a byte-identical body). Fix: atomic temp+rename writes; fix the one remaining ordering inversion. (Consolidating all writers into one save service remains deliberately deferred.)
• COR-23 — still present, and worse than filed: the pipeline writes private_profile_seed to the DB but never exports it, the agent reads disk only (agent.py:119-126, default "No private instructions yet."), and export_private_profile exports private_profile_md — returning None when it's empty (profile_export.py:136) — so even calling it at generation time would no-op. Fix: export the seed itself at generation time (or fall back to the seed in the exporter).

PR C1 — Atomic RMW for the two proven races (small)

All 7 sites still present; with_for_update appears once in src/ (job claiming) and there is no SQL-side increment anywhere.

• profile_version = (x or 0)+1: profile_pipeline.py:417 (worst — row loaded at :286 with dozens of awaits between load and write), profile.py:166, onboarding.py:154, agent_page.py:1262.
• delegate_slack_ids whole-column reassign: agent_page.py:1423-1426, agent_page.py:1609-1612, invite.py:241-244 — each reads the ARRAY before an awaited Slack lookup, so concurrent delegate accepts drop ids.
  Fix: atomic SQL (= col + 1, array_append/array_remove) or a version_id_col.

Definition of done: each PR ships a test that fails against the pre-fix code, including a migration test for the new unique constraint against a table that already contains duplicates.