Skip to content

Commit 3a03774

Browse files
fix(forks): stop copying connector-managed knowledge base documents (#6818)
* fix(forks): stop copying connector-managed knowledge base documents A fork copies a KB's documents but never its connectors, so a connector-sourced document arrives with `connector_id` nulled and its `external_id` intact. The sync engine keys every existing/tombstone/ exclusion lookup off `connector_id`, so that copy is invisible to it - never updated, reconciled, or purged - and `doc_connector_external_id_idx` does not constrain it either, since its `connector_id` is NULL. Attaching a connector in the child then re-ingests every page as a NEW row on top of the snapshot. Each fork hop re-copies the previous hop's orphans and adds one more generation, so a prod -> UAT -> staging chain leaves three rows per page and a knowledge search returns the same page three times, one of them serving content frozen at the fork date. Exclude connector-managed documents from all four doors a document can enter a fork through: the whole-KB content copy, the in-transaction placeholder pre-creation, the sync-only copy into an already-mapped KB, and the content fill (guarded for payloads planned by a pre-change worker mid-rollout). The placeholder path matters as much as the copy loop - filtering only the content phase would leave a permanently archived row behind a persisted `knowledge_document` mapping. Skipped on both sides, the reference clears like any other uncopied document's. A document whose connector was deleted already has a null `connector_id` (the FK is ON DELETE SET NULL) and is static in the source too, so it still copies. One count(*) per copied KB logs what was left behind, since a fully connector-synced KB now forks to zero documents. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(forks): keep the skipped-document count from failing a copied KB The connector-managed count feeds a log line, but it sat inside the KB's try block, so a transient failure on a COUNT(*) would roll back a copy that had otherwise succeeded and clear every reference to it. Move it into a helper that swallows its own error. Counting is not copying: only the copy itself may fail a resource. Test proven red by removing the catch - the mutation reports a knowledge-base failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(forks): clean up full-KB placeholders planned before the exclusion The mapped-KB fill guarded a pre-change plan, but the full-KB path did not: a placeholder planned by an old worker for a connector-managed document is simply no longer returned by the page query, so nothing fills it and it stays archived behind a live mapping that a remapped document-selector still resolves to. Report those child ids as failed documents so the shared cleanup clears their references and drops the rows, and delete their persisted identity so a later sync does not resolve to a row cleanup removes. Keyed on the SOURCE being connector-managed, which can never become copyable, so it cannot race a concurrent attempt mid-fill the way a "source is gone" check could. The mapping drop is now one helper shared with the mapped-KB catch. Test proven red by removing the reconciliation block. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(forks): make the stale-plan probe best-effort The probe ran inside the KB try, so a transient SELECT would reach the catch, roll back a complete copy, delete the child base, and clear every reference to it. Weighing it as "load-bearing, so fail closed" was wrong: the probe runs on EVERY copied KB that has referenced documents, while the state it repairs exists only inside a rollout window. Failing closed traded a common-path outage against a rare-squared one. It now swallows its own failure with a loud error log, leaving that pre-existing state in place rather than destroying a good copy. Test proven red by removing the catch - the mutation reports the KB failure. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
1 parent d1e3eee commit 3a03774

3 files changed

Lines changed: 356 additions & 24 deletions

File tree

apps/docs/content/docs/en/platform/enterprise/forks.mdx

Lines changed: 20 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -174,7 +174,7 @@ How each resource behaves at **fork** time vs **sync** time. Use this when you a
174174
| [Excluded workflows](#excluded-workflows) | Never | Never — not sent, not overwritten, not archived |
175175
| Files | Optional copy (default on) | Map or copy |
176176
| Tables | Optional copy (default on) | Map or copy |
177-
| Knowledge bases (+ documents) | Optional copy; referenced docs come with the KB | Map or copy; documents follow the KB |
177+
| Knowledge bases (+ documents) | Optional copy; uploaded documents come with the KB, [connector-synced ones do not](#connector-synced-documents-are-not-copied) | Map or copy; documents follow the KB |
178178
| Custom tools | Optional copy (default on) | Map or copy |
179179
| Skills | Optional copy (default on) | Map or copy |
180180
| External MCP servers | Optional copy (config only; sign-in cleared) | Map or copy (config only; sign-in cleared) |
@@ -227,10 +227,27 @@ Only **deployed** workflows move. Deploy is the commit; sync is the force push/p
227227

228228
| | Behavior |
229229
|---|----------|
230-
| **Fork** | Optional copy (default on). Tag definitions come with the knowledge base. Documents that the forked workflows actually reference are included. Deselect → knowledge base / document fields clear. |
230+
| **Fork** | Optional copy (default on). Tag definitions come with the knowledge base, along with every **uploaded** document in it. Deselect → knowledge base / document fields clear. |
231231
| **Sync** | Map or copy the knowledge base. Documents are not mapped by themselves — they follow the knowledge base (copied with it, or re-picked when you map to an existing one). |
232232

233-
**Example:** An agent searches knowledge base “Product docs.” Fork with that knowledge base selected → the child gets the base, tags, and the documents the agent used. On sync, mapping to the child’s existing “Product docs” means re-picking which document the tool should use.
233+
**Example:** An agent searches knowledge base “Product docs.” Fork with that knowledge base selected → the child gets the base, tags, and the uploaded documents. On sync, mapping to the child’s existing “Product docs” means re-picking which document the tool should use.
234+
235+
#### Connector-synced documents are not copied
236+
237+
Connectors themselves never cross a fork edge — the child gets no Confluence, Notion, Google Drive, or other sync running against it. Documents that a **connector** put in the knowledge base are therefore not copied either. Only documents you **uploaded** come across.
238+
239+
<Callout type="warn">
240+
Fork a knowledge base whose content is entirely connector-synced and the child gets the base, its tags, and its settings — but **no documents**. Add the connector in the child to fill it.
241+
</Callout>
242+
243+
This is deliberate. A copied connector document would arrive detached from any connector, so nothing would ever update, re-sync, or remove it — and when you added the connector in the child it would ingest every page again *alongside* the stale copy. Chain a few forks (prod → UAT → staging) and each hop leaves another dead generation behind, so one page comes back several times in a single knowledge search. Skipping them keeps the child’s own connector the single owner of that content.
244+
245+
| To get connector content into the child | Do this |
246+
|---|---|
247+
| Keep it live | Add the same connector in the child and let it sync. It re-ingests everything, so nothing is lost. |
248+
| Keep a frozen snapshot | Download the documents from the source and upload them to the child’s knowledge base — uploaded documents copy on every later fork. |
249+
250+
A document whose connector was **deleted** in the source is no longer connector-managed, so it copies like any other uploaded document.
234251

235252
---
236253

apps/sim/ee/workspace-forking/lib/copy/copy-resources.test.ts

Lines changed: 186 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -6,7 +6,9 @@ import { folder as folderTable } from '@sim/db/schema'
66
import { sha256Hex } from '@sim/security/hash'
77
import {
88
dbChainMockFns,
9+
flattenMockConditions,
910
resetDbChainMock,
11+
schemaMock,
1012
storageServiceMock,
1113
storageServiceMockFns,
1214
} from '@sim/testing'
@@ -311,6 +313,106 @@ describe('copyForkResourceContent', () => {
311313
expect(mockPersistCopiedResourceMappings).not.toHaveBeenCalled()
312314
})
313315

316+
it('never copies a connector-managed document out of the source knowledge base', async () => {
317+
dbChainMockFns.limit.mockResolvedValueOnce([])
318+
319+
const result = await copyForkResourceContent({
320+
contentPlan: basePlan({
321+
knowledgeBases: [{ sourceId: 'src-kb', childId: 'child-kb', documentIdMap: {} }],
322+
}),
323+
requestId: 'test',
324+
})
325+
326+
expect(result).toEqual({ copied: 1, failed: 0, failures: [] })
327+
// The row queue returns whatever is enqueued regardless of the predicate, so the exclusion
328+
// is only observable in the condition tree. Pinned to the column so the assertion keeps its
329+
// meaning if another nullable filter joins the same clause.
330+
const pageWhere = dbChainMockFns.where.mock.calls.at(-1)?.[0]
331+
expect(
332+
flattenMockConditions(pageWhere).some(
333+
(node) => node.type === 'isNull' && node.column === schemaMock.document.connectorId
334+
)
335+
).toBe(true)
336+
})
337+
338+
it('drops a full-KB placeholder a pre-change worker planned for a connector-managed doc', async () => {
339+
// Rolling deploy: the fork tx ran on the old code and planned a placeholder for a
340+
// connector-managed document, which this worker's page query no longer returns. Nothing
341+
// would ever fill it, so it must be reported for cleanup rather than left archived behind a
342+
// live mapping that a remapped document-selector still resolves to.
343+
dbChainMockFns.where.mockImplementationOnce(() => ({
344+
// The skipped-document count.
345+
then: (resolve: (rows: unknown[]) => unknown) => resolve([{ total: 1 }]),
346+
}))
347+
dbChainMockFns.where.mockImplementationOnce(() => ({
348+
// The stale-plan probe: the planned source is connector-managed.
349+
then: (resolve: (rows: unknown[]) => unknown) => resolve([{ id: 'doc-1' }]),
350+
}))
351+
dbChainMockFns.limit.mockResolvedValueOnce([])
352+
353+
const result = await copyForkResourceContent({
354+
contentPlan: basePlan({
355+
knowledgeBases: [
356+
{ sourceId: 'src-kb', childId: 'child-kb', documentIdMap: { 'doc-1': 'child-doc-1' } },
357+
],
358+
documentMappingContext: { edgeChildWorkspaceId: 'edge-child-ws', sourceIsParent: false },
359+
}),
360+
requestId: 'test',
361+
})
362+
363+
expect(result.failed).toBe(1)
364+
expect(result.failures).toEqual([{ kind: 'knowledge-document', childId: 'child-doc-1' }])
365+
// The persisted identity goes too, or a later sync resolves to the row cleanup deletes.
366+
expect(mockDeleteCopiedResourceMappingsByTargets).toHaveBeenCalledWith({
367+
executor: expect.anything(),
368+
edgeChildWorkspaceId: 'edge-child-ws',
369+
sourceIsParent: false,
370+
targets: [{ resourceType: 'knowledge_document', resourceId: 'child-doc-1' }],
371+
})
372+
})
373+
374+
it('keeps a copied KB alive when the stale-plan probe fails', async () => {
375+
// The probe runs on every KB with referenced documents, but the state it repairs only exists
376+
// inside a rollout window. Letting it reach the KB catch would delete a complete copy and
377+
// clear every reference to it over a transient SELECT.
378+
dbChainMockFns.where.mockImplementationOnce(() => ({
379+
then: (resolve: (rows: unknown[]) => unknown) => resolve([{ total: 0 }]),
380+
}))
381+
dbChainMockFns.where.mockImplementationOnce(() => {
382+
throw new Error('stale-plan probe failed')
383+
})
384+
dbChainMockFns.limit.mockResolvedValueOnce([])
385+
386+
const result = await copyForkResourceContent({
387+
contentPlan: basePlan({
388+
knowledgeBases: [
389+
{ sourceId: 'src-kb', childId: 'child-kb', documentIdMap: { 'doc-1': 'child-doc-1' } },
390+
],
391+
}),
392+
requestId: 'test',
393+
})
394+
395+
expect(result).toEqual({ copied: 1, failed: 0, failures: [] })
396+
})
397+
398+
it('keeps a copied KB alive when the skipped-document count fails', async () => {
399+
// The count only feeds a log line. Letting it throw into the KB's catch would roll back a
400+
// perfectly good copy and clear every reference to it over a failed COUNT(*).
401+
dbChainMockFns.where.mockImplementationOnce(() => {
402+
throw new Error('count failed')
403+
})
404+
dbChainMockFns.limit.mockResolvedValueOnce([])
405+
406+
const result = await copyForkResourceContent({
407+
contentPlan: basePlan({
408+
knowledgeBases: [{ sourceId: 'src-kb', childId: 'child-kb', documentIdMap: {} }],
409+
}),
410+
requestId: 'test',
411+
})
412+
413+
expect(result).toEqual({ copied: 1, failed: 0, failures: [] })
414+
})
415+
314416
it('uses the blob content digest so a retry cannot adopt an older failed snapshot', async () => {
315417
dbChainMockFns.limit
316418
.mockResolvedValueOnce([sourceDoc])
@@ -1051,6 +1153,25 @@ describe('copyForkResourceContent', () => {
10511153
})
10521154
})
10531155

1156+
it('U-docs: refuses a connector-managed source planned before the exclusion existed', async () => {
1157+
// A payload queued by a pre-change worker during a rolling deploy: the planner would no
1158+
// longer emit this entry, so the fill must drop the placeholder rather than detach a copy
1159+
// of a connector-managed document into the existing target KB.
1160+
dbChainMockFns.limit
1161+
.mockResolvedValueOnce([])
1162+
.mockResolvedValueOnce([{ ...sourceDoc, connectorId: 'connector-1' }])
1163+
1164+
const result = await copyForkResourceContent({
1165+
contentPlan: mappedDocumentPlan(),
1166+
requestId: 'test',
1167+
})
1168+
1169+
expect(result.copied).toBe(0)
1170+
expect(result.failures).toEqual([{ kind: 'knowledge-document', childId: 'child-doc-1' }])
1171+
expect(storageServiceMockFns.mockDownloadFile).not.toHaveBeenCalled()
1172+
expect(mockIncrementStorageUsageInTx).not.toHaveBeenCalled()
1173+
})
1174+
10541175
it('U-docs: refuses to charge when the target knowledge base moved workspaces', async () => {
10551176
queueMappedDocumentCopy()
10561177
dbChainMockFns.for.mockResolvedValueOnce([{ workspaceId: 'other-workspace' }])
@@ -1359,10 +1480,12 @@ describe('copyForkResourceContainers knowledge-base tag definitions', () => {
13591480
// would make every source folder look already-present and suppress the mirroring.
13601481
let folderCall = 0
13611482
const inserts: Array<Array<Record<string, unknown>>> = []
1483+
const wheres: Array<{ table: unknown; condition: unknown }> = []
13621484
const tx = {
13631485
select: () => ({
13641486
from: (table: unknown) => ({
1365-
where: () => {
1487+
where: (condition: unknown) => {
1488+
wheres.push({ table, condition })
13661489
if (table === folderTable) {
13671490
return Promise.resolve(folderCall++ === 0 ? sourceFolders : [])
13681491
}
@@ -1377,7 +1500,7 @@ describe('copyForkResourceContainers knowledge-base tag definitions', () => {
13771500
},
13781501
}),
13791502
}
1380-
return { tx: tx as unknown as DbOrTx, inserts }
1503+
return { tx: tx as unknown as DbOrTx, inserts, wheres }
13811504
}
13821505

13831506
const kbSelection = {
@@ -1458,6 +1581,35 @@ describe('copyForkResourceContainers knowledge-base tag definitions', () => {
14581581
expect(inserts).toHaveLength(1)
14591582
})
14601583

1584+
it('does not pre-create a placeholder for a referenced connector-managed document', async () => {
1585+
const { tx, wheres } = makeKbTx([[sourceBase], [], []])
1586+
1587+
const result = await copyForkResourceContainers({
1588+
tx,
1589+
sourceWorkspaceId: 'src-ws',
1590+
childWorkspaceId: 'child-ws',
1591+
userId: 'user-1',
1592+
now: new Date(),
1593+
selection: kbSelection,
1594+
workflowIdMap: new Map(),
1595+
referencedDocumentIds: ['doc-1'],
1596+
documentMappingContext: { edgeChildWorkspaceId: 'child-ws', sourceIsParent: true },
1597+
})
1598+
1599+
// Must agree with the content phase's exclusion: a placeholder with no content copy behind
1600+
// it would stay archived forever while its persisted mapping pointed at it.
1601+
const placeholderWhere = wheres.find(({ table }) => table === schemaMock.document)?.condition
1602+
expect(
1603+
flattenMockConditions(placeholderWhere).some(
1604+
(node) => node.type === 'isNull' && node.column === schemaMock.document.connectorId
1605+
)
1606+
).toBe(true)
1607+
expect(result.mappingEntries.some((entry) => entry.resourceType === 'knowledge_document')).toBe(
1608+
false
1609+
)
1610+
expect(result.contentPlan.knowledgeBases[0].documentIdMap).toEqual({})
1611+
})
1612+
14611613
it('mirrors the source knowledge-base folder and copies the KB into it, not the target root', async () => {
14621614
const foldered = { ...sourceBase, folderId: 'kb-folder' }
14631615
const { tx, inserts } = makeKbTx(
@@ -1510,7 +1662,9 @@ describe('planForkMappedKbDocumentCopies', () => {
15101662
fileSize: 123,
15111663
filename: `${id}.pdf`,
15121664
mimeType: 'application/pdf',
1513-
connectorId: 'connector-1',
1665+
// Hand-uploaded: connector-managed documents are filtered out by the candidate query and
1666+
// can never reach the placeholder insert.
1667+
connectorId: null,
15141668
deletedAt: null,
15151669
archivedAt: null,
15161670
})
@@ -1526,11 +1680,19 @@ describe('planForkMappedKbDocumentCopies', () => {
15261680
}> = []
15271681
) {
15281682
const inserted: Array<Record<string, unknown>> = []
1683+
const wheres: unknown[] = []
15291684
let selectCalls = 0
15301685
const tx = {
15311686
select: () => {
15321687
const rows = selectCalls++ === 0 ? docs : existingTargets
1533-
return { from: () => ({ where: () => Promise.resolve(rows) }) }
1688+
return {
1689+
from: () => ({
1690+
where: (condition: unknown) => {
1691+
wheres.push(condition)
1692+
return Promise.resolve(rows)
1693+
},
1694+
}),
1695+
}
15341696
},
15351697
insert: () => ({
15361698
values: (rows: Array<Record<string, unknown>>) => {
@@ -1539,7 +1701,7 @@ describe('planForkMappedKbDocumentCopies', () => {
15391701
},
15401702
}),
15411703
}
1542-
return { tx: tx as unknown as DbOrTx, inserted, selectCalls: () => selectCalls }
1704+
return { tx: tx as unknown as DbOrTx, inserted, wheres, selectCalls: () => selectCalls }
15431705
}
15441706

15451707
const mappedKbResolver: ForkReferenceResolver = (kind, id) =>
@@ -1584,6 +1746,25 @@ describe('planForkMappedKbDocumentCopies', () => {
15841746
])
15851747
})
15861748

1749+
it('never considers a connector-managed doc as a candidate for the mapped target KB', async () => {
1750+
const { tx, wheres } = makeTx([])
1751+
await planForkMappedKbDocumentCopies({
1752+
tx,
1753+
resolver: mappedKbResolver,
1754+
referencedDocumentIds: ['doc-1'],
1755+
alreadyCopiedSourceDocIds: new Set(),
1756+
now,
1757+
})
1758+
1759+
// The tx mock returns its rows regardless of the predicate, so the exclusion is only
1760+
// observable in the condition tree.
1761+
expect(
1762+
flattenMockConditions(wheres[0]).some(
1763+
(node) => node.type === 'isNull' && node.column === schemaMock.document.connectorId
1764+
)
1765+
).toBe(true)
1766+
})
1767+
15871768
it('skips a referenced doc whose parent KB is not mapped (reference is left to be cleared)', async () => {
15881769
const { tx, inserted } = makeTx([sourceRow('doc-1', 'unmapped-kb')])
15891770
const result = await planForkMappedKbDocumentCopies({

0 commit comments

Comments
 (0)