fix(provenance): stop size limits silently dropping secret provenance - #6867
Conversation
A bundle-selection cap counted cells rather than rows, so a 25-column table insert lost secret provenance for every row past 400 — the whole batch was stamped unknown with nothing logged. The same number lived in the sender, the runtime type guard, and the route contract. Consolidate every provenance limit into one definition: an 8MB serialized envelope and 10,000 distinct secrets. The pair had been copied into seven modules under fourteen names, and several copies had drifted into bounding inputs — rows, cells, files, chunks — rather than the envelope. Remove every limit that could refuse a legal payload, add write-side cause logging and a workspace-visible audit entry when a read proceeds on unrecorded provenance, and repair the existing unknown rows.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
PR SummaryMedium Risk Overview Batch provenance export adds Fail-open durable reads now emit one workspace audit entry ( Operational fixes: derived row transformation SQL treats demoted rows like the read path (untracked without requiring absent sidecar); workspace file unknown-marking is paged past 20 files; script migration 0005 relabels backlog Reviewed by Cursor Bugbot for commit a8e6233. Configure here. |
Greptile SummaryThe PR removes input-count limits that could silently discard secret provenance, centralizes the remaining envelope limits, and adds audit visibility for reads with unrecorded provenance.
Confidence Score: 5/5The PR appears safe to merge. No blocking failure remains; the status recheck preserves concurrently written exact sidecars, and parent-first ordered locking resolves the previously reported deadlock path.
|
| Filename | Overview |
|---|---|
| packages/db/script-migrations/0005_repair_unknown_table_row_provenance.ts | The migration now locks parent rows before sidecars, rechecks unknown status before deletion, advances keyset pagination even when concurrent writers repair a page, and addresses both previously reported concurrency failures. |
| apps/sim/executor/utils/resolved-secret-trace-registry.ts | Adds indexed grouped provenance export to remove the wide-write selection cap while preserving per-group narrowing and incompleteness behavior. |
| apps/sim/lib/execution/model-input-provenance.ts | Removes the duplicated selection-count cap, uses centralized envelope limits, and records concrete causes when a bundle cannot be vouched for. |
| apps/sim/lib/table/rows/secret-provenance.ts | Consolidates table provenance classification and paging behavior while adding aggregated reporting for unrecorded reads. |
| apps/sim/lib/execution/durable-secret-provenance-enforcement.ts | Extends unrecorded-provenance reporting with workspace-visible audit entries without changing the enforcement feature flag. |
Reviews (3): Last reviewed commit: "fix(provenance): take the repair's locks..." | Re-trigger Greptile
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
The repair matched sidecars by the id its page captured, so a provenance-aware write committing between the snapshot and the delete had its fresh exact sidecar removed and its marker cleared behind it — a secret-bearing row left reading as legacy. The delete now re-checks status, which under READ COMMITTED re-evaluates against the writer's committed row so it no longer matches. Walk the candidate set by keyset over row_id. A page whose rows were all repaired concurrently clears nothing, and terminating on "cleared nothing" ended the walk with the rest of the backlog untouched. Memory reported unrecorded provenance twice, and counted records even when the surface was enforced — auditing a fail-open read that had actually failed closed.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 06f5e76. Configure here.
The repair deleted the sidecar and only then updated its parent row, while mutateTableRowsWithSecretProvenance locks user_table_rows up front and upserts the sidecar inside the same transaction. Opposite orders, so an overlapping write deadlocked and Postgres resolved it by aborting either the deployment or somebody's table write. Lock the parent first, in id order, matching lockTableRows. Holding that lock is also what makes the status re-check decisive rather than racy: the writer commits its sidecar and its marker under the same lock, so once it is held the write is either wholly done or has not begun.
|
@cursor review |
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit a8e6233. Configure here.
1 participant
Summary
unknown, with nothing logged. The same number lived in three layers (sender, runtime type guard, route contract); all three are gone, and a cross-layer test pins the sender's output against the contract so they can't drift apart again.secret_provenance.unrecorded) alongside the log line, aggregated per read rather than per record.unknownnow names why, through a single chokepoint.unknownby the derived-transformation SQL, which turned ordinary column operations into bulk producers of unknowns.unknownrows to the untracked state — the state the read path already tolerates, and which passes even once a surface is closed.Type of Change
Testing
Tested manually.
bun run lint,check:audits, andcheck-block-registrypass; type-check clean acrossapps/sim,packages/audit,packages/testing,packages/db. 4,837 tests pass across the provenance, table, knowledge, memory, uploads, contracts, and audit suites. One unrelated failure (unreadable-document.test.ts, OCR error-message assertion) reproduces onstagingwithout these changes.Notes for review:
Checklist