fix(provenance): stop size limits silently dropping secret provenance (#6867)

* fix(provenance): stop size limits silently dropping secret provenance

A bundle-selection cap counted cells rather than rows, so a 25-column
table insert lost secret provenance for every row past 400 — the whole
batch was stamped unknown with nothing logged. The same number lived in
the sender, the runtime type guard, and the route contract.

Consolidate every provenance limit into one definition: an 8MB
serialized envelope and 10,000 distinct secrets. The pair had been
copied into seven modules under fourteen names, and several copies had
drifted into bounding inputs — rows, cells, files, chunks — rather than
the envelope.

Remove every limit that could refuse a legal payload, add write-side
cause logging and a workspace-visible audit entry when a read proceeds
on unrecorded provenance, and repair the existing unknown rows.

* fix(provenance): close a repair race and stop memory double-reporting

The repair matched sidecars by the id its page captured, so a
provenance-aware write committing between the snapshot and the delete
had its fresh exact sidecar removed and its marker cleared behind it —
a secret-bearing row left reading as legacy. The delete now re-checks
status, which under READ COMMITTED re-evaluates against the writer's
committed row so it no longer matches.

Walk the candidate set by keyset over row_id. A page whose rows were
all repaired concurrently clears nothing, and terminating on "cleared
nothing" ended the walk with the rest of the backlog untouched.

Memory reported unrecorded provenance twice, and counted records even
when the surface was enforced — auditing a fail-open read that had
actually failed closed.

* fix(provenance): take the repair's locks in the writer's order

The repair deleted the sidecar and only then updated its parent row,
while mutateTableRowsWithSecretProvenance locks user_table_rows up front
and upserts the sidecar inside the same transaction. Opposite orders, so
an overlapping write deadlocked and Postgres resolved it by aborting
either the deployment or somebody's table write.

Lock the parent first, in id order, matching lockTableRows. Holding that
lock is also what makes the status re-check decisive rather than racy:
the writer commits its sidecar and its marker under the same lock, so
once it is held the write is either wholly done or has not begun.

Author: icecrasher321

apps/sim/app/api/memory/secret-provenance.test.ts

@@ -5,7 +5,19 @@
 import { memorySecretProvenance } from '@sim/db/schema'
 import { dbChainMockFns, queueTableRows, resetDbChainMock } from '@sim/testing'
 import { NextRequest } from 'next/server'
-import { beforeEach, describe, expect, it } from 'vitest'
+import { beforeEach, describe, expect, it, vi } from 'vitest'
+
+const { mockIsEnforced, mockReport } = vi.hoisted(() => ({
+  mockIsEnforced: vi.fn(() => false),
+  mockReport: vi.fn(),
+}))
+
+vi.mock('@/lib/execution/durable-secret-provenance-enforcement', () => ({
+  DURABLE_SECRET_PROVENANCE_SURFACES: ['memory', 'table-row', 'knowledge'],
+  isDurableSecretProvenanceEnforced: mockIsEnforced,
+  reportUnrecordedDurableProvenance: mockReport,
+}))
+
 import { AuthType } from '@/lib/auth/hybrid'
 import {
   PRIVATE_SECRET_PROVENANCE_BUNDLE_V1,
@@ -47,6 +59,8 @@ function privateMemoryWrite(
 describe('memory write secret provenance', () => {
   beforeEach(() => {
     resetDbChainMock()
+    mockReport.mockClear()
+    mockIsEnforced.mockReturnValue(false)
   })
   it('classifies a headerless external write as exact-empty', () => {
     const request = new NextRequest('http://localhost/api/memory', { method: 'POST' })
@@ -200,7 +214,12 @@ describe('memory write secret provenance', () => {
     if (!result.success) expect(result.response.status).toBe(400)
   })
 
-  it('bounds only requested private response provenance without querying sidecars', async () => {
+  /**
+   * A read of this width used to be refused outright on the record count alone. How many memories
+   * crossed said nothing about whether their provenance could be established, so the read now
+   * vouches for them and the page size is what keeps the statement count bounded.
+   */
+  it('vouches for a very wide crossing instead of refusing on the record count', async () => {
     const request = new NextRequest('http://localhost/api/memory', {
       headers: {
         [PRIVATE_TOOL_METADATA_REQUEST_HEADER]: RESOLVED_SECRET_PROVENANCE_METADATA_V1,
@@ -220,12 +239,18 @@ describe('memory write secret provenance', () => {
     })
 
     await expect(response.json()).resolves.toMatchObject({
-      [RESOLVED_SECRET_PROVENANCE_FIELD]: { version: 1, complete: false, entries: [] },
+      [RESOLVED_SECRET_PROVENANCE_FIELD]: { version: 1, complete: true, entries: [] },
     })
-    expect(dbChainMockFns.select).not.toHaveBeenCalled()
+    /** Eleven pages of a thousand, not one statement per handful of memories. */
+    expect(dbChainMockFns.select.mock.calls.length).toBeLessThanOrEqual(11)
   })
 
-  it('marks oversized aggregate sidecar provenance incomplete before importing it', async () => {
+  /**
+   * A sidecar too large to carry reads as unrecorded, not as a reason to fail the read. The run
+   * keeps its other provenance and proceeds without this memory's — best effort, with the risk
+   * recorded — rather than refusing every projection for the rest of the run.
+   */
+  it('treats a sidecar too large to carry as unrecorded rather than failing the read', async () => {
     queueTableRows(memorySecretProvenance, [
       {
         memoryId: 'memory-1',
@@ -253,7 +278,66 @@ describe('memory write secret provenance', () => {
     })
 
     await expect(response.json()).resolves.toMatchObject({
-      [RESOLVED_SECRET_PROVENANCE_FIELD]: { version: 1, complete: false, entries: [] },
+      [RESOLVED_SECRET_PROVENANCE_FIELD]: { version: 1, complete: true, entries: [] },
+    })
+  })
+  /**
+   * One entry for the read, not one per record: the per-record import knows no workspace, so its
+   * report can only ever be a log line, and passing the workspace down instead would write
+   * thousands of audit rows for a single event.
+   */
+  it('reports one aggregated entry for a read that proceeded unvouched', async () => {
+    const request = new NextRequest('http://localhost/api/memory', {
+      headers: {
+        [PRIVATE_TOOL_METADATA_REQUEST_HEADER]: RESOLVED_SECRET_PROVENANCE_METADATA_V1,
+      },
     })
+
+    await createMemoryResponse({
+      request,
+      authType: AuthType.INTERNAL_JWT,
+      userId: 'user-1',
+      workspaceId: 'workspace-1',
+      body: { success: true },
+      memories: [
+        { id: 'memory-1', data: 'value', secretProvenanceVersion: 1 },
+        { id: 'memory-2', data: 'value', secretProvenanceVersion: 1 },
+      ],
+    })
+
+    expect(mockReport).toHaveBeenCalledTimes(1)
+    expect(mockReport).toHaveBeenCalledWith(
+      expect.objectContaining({
+        surface: 'memory',
+        cause: 'durable-provenance-unknown',
+        affectedCount: 2,
+        workspaceId: 'workspace-1',
+      })
+    )
+  })
+
+  /**
+   * Under enforcement the import fails the registry closed rather than proceeding, so there is no
+   * fail-open read to record. Counting those records anyway would audit something that never
+   * happened, in the one trail whose whole purpose is to say a read went ahead unvouched.
+   */
+  it('records nothing when the surface is enforced and the read fails closed', async () => {
+    mockIsEnforced.mockReturnValue(true)
+    const request = new NextRequest('http://localhost/api/memory', {
+      headers: {
+        [PRIVATE_TOOL_METADATA_REQUEST_HEADER]: RESOLVED_SECRET_PROVENANCE_METADATA_V1,
+      },
+    })
+
+    await createMemoryResponse({
+      request,
+      authType: AuthType.INTERNAL_JWT,
+      userId: 'user-1',
+      workspaceId: 'workspace-1',
+      body: { success: true },
+      memories: [{ id: 'memory-1', data: 'value', secretProvenanceVersion: 1 }],
+    })
+
+    expect(mockReport).not.toHaveBeenCalled()
   })
 })

apps/sim/app/api/memory/secret-provenance.ts

@@ -1,4 +1,3 @@
-import { Buffer } from 'buffer'
 import { db } from '@sim/db'
 import { memorySecretProvenance } from '@sim/db/schema'
 import { inArray } from 'drizzle-orm'
@@ -10,6 +9,10 @@ import {
   EXACT_EMPTY_DURABLE_SECRET_PROVENANCE,
   importDurableSecretProvenance,
 } from '@/lib/execution/durable-secret-provenance'
+import {
+  isDurableSecretProvenanceEnforced,
+  reportUnrecordedDurableProvenance,
+} from '@/lib/execution/durable-secret-provenance-enforcement'
 import {
   inspectPrivateSecretProvenanceRequest,
   isPrivateSecretProvenanceBundleV1,
@@ -22,10 +25,15 @@ import {
 import { readBoundMemorySecretProvenance } from '@/lib/memory/secret-provenance'
 import { ResolvedSecretTraceRegistry } from '@/executor/utils/resolved-secret-trace-registry'
 
-const MAX_PRIVATE_MEMORY_CROSSINGS = 10_000
-const PRIVATE_MEMORY_QUERY_CHUNK_SIZE = 8
-const MAX_PRIVATE_MEMORY_PROVENANCE_ENTRIES = 10_000
-const MAX_PRIVATE_MEMORY_PROVENANCE_BYTES = 8 * 1024 * 1024
+/**
+ * Ids per sidecar lookup, matching how the rest of the codebase chunks an `inArray`.
+ *
+ * It was 8, which only worked because a cap refused any read over ten thousand memories — at that
+ * width the page loop would otherwise have issued more than a thousand sequential statements. The
+ * cap is gone because refusing on a record count told us nothing about the data, so the page size
+ * now has to be the thing that keeps the read bounded.
+ */
+const PRIVATE_MEMORY_QUERY_CHUNK_SIZE = 1_000
 
 interface MemoryCrossing {
   id: string
@@ -97,52 +105,76 @@ export async function createMemoryResponse(options: {
     userId: options.userId,
     workspaceId: options.workspaceId,
   })
-  if (options.memories.length > MAX_PRIVATE_MEMORY_CROSSINGS) {
-    registry.markIncomplete('memory-crossing-capacity-exceeded')
-  } else {
-    const ids = [...new Set(options.memories.map((record) => record.id))]
-    const memoriesById = new Map<string, MemoryCrossing[]>()
-    for (const memory of options.memories) {
-      const matching = memoriesById.get(memory.id) ?? []
-      matching.push(memory)
-      memoriesById.set(memory.id, matching)
-    }
-    let provenanceEntryCount = 0
-    let provenanceBytes = 0
-    for (let index = 0; index < ids.length; index += PRIVATE_MEMORY_QUERY_CHUNK_SIZE) {
-      const pageIds = ids.slice(index, index + PRIVATE_MEMORY_QUERY_CHUNK_SIZE)
-      const sidecars = await db
-        .select()
-        .from(memorySecretProvenance)
-        .where(inArray(memorySecretProvenance.memoryId, pageIds))
-      for (const sidecar of sidecars) {
-        provenanceEntryCount += Array.isArray(sidecar.entries) ? sidecar.entries.length : 0
-        provenanceBytes += Buffer.byteLength(JSON.stringify(sidecar.entries ?? []), 'utf8')
-      }
-      if (
-        provenanceEntryCount > MAX_PRIVATE_MEMORY_PROVENANCE_ENTRIES ||
-        provenanceBytes > MAX_PRIVATE_MEMORY_PROVENANCE_BYTES
-      ) {
-        registry.markIncomplete('memory-crossing-capacity-exceeded')
-        break
-      }
-      const sidecarById = new Map(sidecars.map((sidecar) => [sidecar.memoryId, sidecar]))
-      for (const memoryId of pageIds) {
-        for (const record of memoriesById.get(memoryId) ?? []) {
-          const sidecar = sidecarById.get(record.id)
-          const provenance = readBoundMemorySecretProvenance({
-            secretProvenanceVersion: record.secretProvenanceVersion,
-            data: record.data,
-            provenanceContentHash: sidecar?.contentHash ?? null,
-            status: sidecar?.status ?? null,
-            entries: sidecar?.entries,
-          })
-          await importDurableSecretProvenance(registry, provenance, record.data, 'memory')
-        }
+  /**
+   * No cap on how many memories may cross, and no separate accounting of what they carry.
+   *
+   * There was both: a refusal past ten thousand records, and a running total of every sidecar's
+   * entries. The first said nothing about the data. The second summed entries *before* they were
+   * folded, so a page of memories sharing a handful of secrets counted once per mention and could
+   * refuse a read whose envelope would have held a dozen entries.
+   *
+   * Neither is needed, because the registry already bounds the only thing that has a real limit —
+   * the serialized envelope — as each entry is added, at the granularity it actually dedupes on.
+   * A second estimate in front of it could only ever be wrong in one of two directions.
+   */
+  const ids = [...new Set(options.memories.map((record) => record.id))]
+  const memoriesById = new Map<string, MemoryCrossing[]>()
+  for (const memory of options.memories) {
+    const matching = memoriesById.get(memory.id) ?? []
+    matching.push(memory)
+    memoriesById.set(memory.id, matching)
+  }
+  /**
+   * Counted only while the surface is open. Under enforcement the import fails the registry closed
+   * instead of proceeding, so counting those records would audit a fail-open read that never
+   * happened — and this entry exists precisely to say a read went ahead unvouched.
+   */
+  const memoryEnforced = isDurableSecretProvenanceEnforced('memory')
+  let unrecordedMemoryCount = 0
+  for (let index = 0; index < ids.length; index += PRIVATE_MEMORY_QUERY_CHUNK_SIZE) {
+    const pageIds = ids.slice(index, index + PRIVATE_MEMORY_QUERY_CHUNK_SIZE)
+    const sidecars = await db
+      .select()
+      .from(memorySecretProvenance)
+      .where(inArray(memorySecretProvenance.memoryId, pageIds))
+    const sidecarById = new Map(sidecars.map((sidecar) => [sidecar.memoryId, sidecar]))
+    for (const memoryId of pageIds) {
+      for (const record of memoriesById.get(memoryId) ?? []) {
+        const sidecar = sidecarById.get(record.id)
+        const provenance = readBoundMemorySecretProvenance({
+          secretProvenanceVersion: record.secretProvenanceVersion,
+          data: record.data,
+          provenanceContentHash: sidecar?.contentHash ?? null,
+          status: sidecar?.status ?? null,
+          entries: sidecar?.entries,
+        })
+        if (provenance.status === 'unknown' && !memoryEnforced) unrecordedMemoryCount += 1
+        await importDurableSecretProvenance(registry, provenance, record.data, 'memory', {
+          reportUnrecorded: false,
+        })
       }
     }
   }
 
+  /**
+   * Counted here and reported once, rather than left to the per-record import.
+   *
+   * That import reports without a workspace, so the workspace-visible half of the trail never
+   * reached the people it concerns — the audit entry is skipped when it cannot name one. Passing
+   * the workspace down instead would have written one row per record, which on a wide read is
+   * thousands of fire-and-forget inserts for a single event. One read is one thing that happened,
+   * so it is one entry carrying how many records it covered — the shape the table surface uses.
+   */
+  if (unrecordedMemoryCount > 0) {
+    reportUnrecordedDurableProvenance({
+      surface: 'memory',
+      cause: 'durable-provenance-unknown',
+      affectedCount: unrecordedMemoryCount,
+      workspaceId: options.workspaceId,
+      actorUserId: options.userId,
+    })
+  }
+
   const envelope = serializePrivateToolMetadataResponseEnvelope(
     options.body,
     RESOLVED_SECRET_PROVENANCE_METADATA_V1,

apps/sim/content/blog/secret-provenance/index.mdx

@@ -64,7 +64,9 @@ Results returning from tools and Function blocks are checked before they become
 
 Storage is the boundary that is easiest to miss. Workspace files, table cells, knowledge documents, and agent memory can all survive the execution that created them. Without durable provenance, a secret written during one run would look like ordinary data when another run read it back. Sim stores an encrypted provenance record and version marker alongside that content, allowing a later read to distinguish data known to be clean from data whose history is unknown.
 
-When that history cannot be established, we do not treat `unknown` as `clean`. The payload is withheld, while safe parts of the record remain available: block identity, timing, status, and error type. A tool result can be kept out of model context without hiding the fact that the tool ran. The refusal also records where it happened and what made the registry incomplete, so failing closed does not mean debugging blind.
+When that history cannot be established, we do not treat `unknown` as `clean`. At a run's own boundaries the payload is withheld, while safe parts of the record remain available: block identity, timing, status, and error type. A tool result can be kept out of model context without hiding the fact that the tool ran. The refusal also records where it happened and what made the registry incomplete, so failing closed does not mean debugging blind.
+
+Stored data is different, because a record written months ago outlives the run that could explain it. Refusing to read it would strand a team on its own tables and memory to protect against a risk we cannot confirm is there. So for that content the read proceeds and the workspace gets an audit entry naming the surface and what could not be vouched for. The judgment is deliberate: withholding is right when a run is asking to send something out, and visibility is right when the alternative is a workspace that can no longer read what it stored.
 
 ## Exact Matching, With One Limit
 

apps/sim/executor/utils/resolved-secret-trace-registry.test.ts

@@ -416,6 +416,72 @@ describe('ResolvedSecretTraceRegistry', () => {
     })
   })
 
+  it('narrows each grouped export to its own root', () => {
+    const scope = { userId: 'user-1', workspaceId: 'workspace-1' }
+    const registry = new ResolvedSecretTraceRegistry(
+      [
+        { name: 'FIRST', plaintext: 'alpha', encryptedValue: 'encrypted-first' },
+        { name: 'SECOND', plaintext: 'beta', encryptedValue: 'encrypted-second' },
+      ],
+      scope
+    )
+    registry.recordResolvedAtInputPath('FIRST', 'alpha', ['rows', '0', 'a'])
+    registry.recordResolvedAtInputPath('SECOND', 'beta', ['rows', '1', 'b'])
+    registry.recordResolvedAtInputPath('FIRST', 'alpha', ['rows', '2', 'c', 'nested'])
+
+    const exported = registry.exportCommittedProvenanceForInputPathGroups([
+      [['rows', '0', 'a']],
+      [['rows', '1', 'b']],
+      [['rows', '2', 'c']],
+      [['rows']],
+      [['rows', '3', 'untouched']],
+      [],
+      [
+        ['rows', '0', 'a'],
+        ['rows', '1', 'b'],
+      ],
+    ])
+
+    expect(exported.map((provenance) => provenance.entries)).toEqual([
+      [{ name: 'FIRST', encryptedValue: 'encrypted-first' }],
+      [{ name: 'SECOND', encryptedValue: 'encrypted-second' }],
+      [{ name: 'FIRST', encryptedValue: 'encrypted-first' }],
+      [
+        { name: 'FIRST', encryptedValue: 'encrypted-first' },
+        { name: 'SECOND', encryptedValue: 'encrypted-second' },
+      ],
+      [],
+      [],
+      [
+        { name: 'FIRST', encryptedValue: 'encrypted-first' },
+        { name: 'SECOND', encryptedValue: 'encrypted-second' },
+      ],
+    ])
+    expect(exported.every((provenance) => provenance.complete)).toBe(true)
+  })
+
+  it('fails only the grouped exports an incomplete input path overlaps', async () => {
+    const scope = { userId: 'user-1', workspaceId: 'workspace-1' }
+    const registry = new ResolvedSecretTraceRegistry(
+      [{ name: 'FIRST', plaintext: 'alpha', encryptedValue: 'encrypted-first' }],
+      scope
+    )
+    registry.recordResolvedAtInputPath('FIRST', 'alpha', ['rows', '0', 'a'])
+    await registry.importProvenanceForValueAtInputPath(null, 'alpha', ['rows', '1', 'b'], {
+      trusted: false,
+    })
+
+    const exported = registry.exportCommittedProvenanceForInputPathGroups([
+      [['rows', '1', 'b']],
+      [['rows', '1']],
+      [['rows', '1', 'b', 'deeper']],
+      [['rows', '0', 'a']],
+    ])
+
+    expect(exported.map((provenance) => provenance.complete)).toEqual([false, false, false, true])
+    expect(exported[3].entries).toEqual([{ name: 'FIRST', encryptedValue: 'encrypted-first' }])
+  })
+
   it('fails closed when independent secret paths collapse into one transformed string', () => {
     const registry = new ResolvedSecretTraceRegistry([
       { name: 'FIRST', plaintext: 'first', encryptedValue: 'encrypted-first' },
@@ -1598,8 +1664,6 @@ describe('incompleteness diagnostics', () => {
     'client-tool-execution-untrusted',
     'client-tool-content-unavailable',
     'knowledge-result-provenance-unavailable',
-    'knowledge-response-capacity-exceeded',
-    'memory-crossing-capacity-exceeded',
     'table-result-provenance-unavailable',
     'mounted-file-provenance-unavailable',
     'workspace-file-provenance-unknown',