Commit 3a03774
fix(forks): stop copying connector-managed knowledge base documents (#6818)
* fix(forks): stop copying connector-managed knowledge base documents
A fork copies a KB's documents but never its connectors, so a
connector-sourced document arrives with `connector_id` nulled and its
`external_id` intact. The sync engine keys every existing/tombstone/
exclusion lookup off `connector_id`, so that copy is invisible to it -
never updated, reconciled, or purged - and `doc_connector_external_id_idx`
does not constrain it either, since its `connector_id` is NULL.
Attaching a connector in the child then re-ingests every page as a NEW
row on top of the snapshot. Each fork hop re-copies the previous hop's
orphans and adds one more generation, so a prod -> UAT -> staging chain
leaves three rows per page and a knowledge search returns the same page
three times, one of them serving content frozen at the fork date.
Exclude connector-managed documents from all four doors a document can
enter a fork through: the whole-KB content copy, the in-transaction
placeholder pre-creation, the sync-only copy into an already-mapped KB,
and the content fill (guarded for payloads planned by a pre-change
worker mid-rollout). The placeholder path matters as much as the copy
loop - filtering only the content phase would leave a permanently
archived row behind a persisted `knowledge_document` mapping. Skipped on
both sides, the reference clears like any other uncopied document's.
A document whose connector was deleted already has a null `connector_id`
(the FK is ON DELETE SET NULL) and is static in the source too, so it
still copies. One count(*) per copied KB logs what was left behind, since
a fully connector-synced KB now forks to zero documents.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(forks): keep the skipped-document count from failing a copied KB
The connector-managed count feeds a log line, but it sat inside the KB's
try block, so a transient failure on a COUNT(*) would roll back a copy
that had otherwise succeeded and clear every reference to it.
Move it into a helper that swallows its own error. Counting is not
copying: only the copy itself may fail a resource. Test proven red by
removing the catch - the mutation reports a knowledge-base failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(forks): clean up full-KB placeholders planned before the exclusion
The mapped-KB fill guarded a pre-change plan, but the full-KB path did
not: a placeholder planned by an old worker for a connector-managed
document is simply no longer returned by the page query, so nothing fills
it and it stays archived behind a live mapping that a remapped
document-selector still resolves to.
Report those child ids as failed documents so the shared cleanup clears
their references and drops the rows, and delete their persisted identity
so a later sync does not resolve to a row cleanup removes. Keyed on the
SOURCE being connector-managed, which can never become copyable, so it
cannot race a concurrent attempt mid-fill the way a "source is gone"
check could.
The mapping drop is now one helper shared with the mapped-KB catch.
Test proven red by removing the reconciliation block.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
* fix(forks): make the stale-plan probe best-effort
The probe ran inside the KB try, so a transient SELECT would reach the
catch, roll back a complete copy, delete the child base, and clear every
reference to it. Weighing it as "load-bearing, so fail closed" was wrong:
the probe runs on EVERY copied KB that has referenced documents, while
the state it repairs exists only inside a rollout window. Failing closed
traded a common-path outage against a rare-squared one.
It now swallows its own failure with a loud error log, leaving that
pre-existing state in place rather than destroying a good copy. Test
proven red by removing the catch - the mutation reports the KB failure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>1 parent d1e3eee commit 3a03774
3 files changed
Lines changed: 356 additions & 24 deletions
File tree
- apps
- docs/content/docs/en/platform/enterprise
- sim/ee/workspace-forking/lib/copy
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
174 | 174 | | |
175 | 175 | | |
176 | 176 | | |
177 | | - | |
| 177 | + | |
178 | 178 | | |
179 | 179 | | |
180 | 180 | | |
| |||
227 | 227 | | |
228 | 228 | | |
229 | 229 | | |
230 | | - | |
| 230 | + | |
231 | 231 | | |
232 | 232 | | |
233 | | - | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
234 | 251 | | |
235 | 252 | | |
236 | 253 | | |
| |||
Lines changed: 186 additions & 5 deletions
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
6 | 6 | | |
7 | 7 | | |
8 | 8 | | |
| 9 | + | |
9 | 10 | | |
| 11 | + | |
10 | 12 | | |
11 | 13 | | |
12 | 14 | | |
| |||
311 | 313 | | |
312 | 314 | | |
313 | 315 | | |
| 316 | + | |
| 317 | + | |
| 318 | + | |
| 319 | + | |
| 320 | + | |
| 321 | + | |
| 322 | + | |
| 323 | + | |
| 324 | + | |
| 325 | + | |
| 326 | + | |
| 327 | + | |
| 328 | + | |
| 329 | + | |
| 330 | + | |
| 331 | + | |
| 332 | + | |
| 333 | + | |
| 334 | + | |
| 335 | + | |
| 336 | + | |
| 337 | + | |
| 338 | + | |
| 339 | + | |
| 340 | + | |
| 341 | + | |
| 342 | + | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
| 346 | + | |
| 347 | + | |
| 348 | + | |
| 349 | + | |
| 350 | + | |
| 351 | + | |
| 352 | + | |
| 353 | + | |
| 354 | + | |
| 355 | + | |
| 356 | + | |
| 357 | + | |
| 358 | + | |
| 359 | + | |
| 360 | + | |
| 361 | + | |
| 362 | + | |
| 363 | + | |
| 364 | + | |
| 365 | + | |
| 366 | + | |
| 367 | + | |
| 368 | + | |
| 369 | + | |
| 370 | + | |
| 371 | + | |
| 372 | + | |
| 373 | + | |
| 374 | + | |
| 375 | + | |
| 376 | + | |
| 377 | + | |
| 378 | + | |
| 379 | + | |
| 380 | + | |
| 381 | + | |
| 382 | + | |
| 383 | + | |
| 384 | + | |
| 385 | + | |
| 386 | + | |
| 387 | + | |
| 388 | + | |
| 389 | + | |
| 390 | + | |
| 391 | + | |
| 392 | + | |
| 393 | + | |
| 394 | + | |
| 395 | + | |
| 396 | + | |
| 397 | + | |
| 398 | + | |
| 399 | + | |
| 400 | + | |
| 401 | + | |
| 402 | + | |
| 403 | + | |
| 404 | + | |
| 405 | + | |
| 406 | + | |
| 407 | + | |
| 408 | + | |
| 409 | + | |
| 410 | + | |
| 411 | + | |
| 412 | + | |
| 413 | + | |
| 414 | + | |
| 415 | + | |
314 | 416 | | |
315 | 417 | | |
316 | 418 | | |
| |||
1051 | 1153 | | |
1052 | 1154 | | |
1053 | 1155 | | |
| 1156 | + | |
| 1157 | + | |
| 1158 | + | |
| 1159 | + | |
| 1160 | + | |
| 1161 | + | |
| 1162 | + | |
| 1163 | + | |
| 1164 | + | |
| 1165 | + | |
| 1166 | + | |
| 1167 | + | |
| 1168 | + | |
| 1169 | + | |
| 1170 | + | |
| 1171 | + | |
| 1172 | + | |
| 1173 | + | |
| 1174 | + | |
1054 | 1175 | | |
1055 | 1176 | | |
1056 | 1177 | | |
| |||
1359 | 1480 | | |
1360 | 1481 | | |
1361 | 1482 | | |
| 1483 | + | |
1362 | 1484 | | |
1363 | 1485 | | |
1364 | 1486 | | |
1365 | | - | |
| 1487 | + | |
| 1488 | + | |
1366 | 1489 | | |
1367 | 1490 | | |
1368 | 1491 | | |
| |||
1377 | 1500 | | |
1378 | 1501 | | |
1379 | 1502 | | |
1380 | | - | |
| 1503 | + | |
1381 | 1504 | | |
1382 | 1505 | | |
1383 | 1506 | | |
| |||
1458 | 1581 | | |
1459 | 1582 | | |
1460 | 1583 | | |
| 1584 | + | |
| 1585 | + | |
| 1586 | + | |
| 1587 | + | |
| 1588 | + | |
| 1589 | + | |
| 1590 | + | |
| 1591 | + | |
| 1592 | + | |
| 1593 | + | |
| 1594 | + | |
| 1595 | + | |
| 1596 | + | |
| 1597 | + | |
| 1598 | + | |
| 1599 | + | |
| 1600 | + | |
| 1601 | + | |
| 1602 | + | |
| 1603 | + | |
| 1604 | + | |
| 1605 | + | |
| 1606 | + | |
| 1607 | + | |
| 1608 | + | |
| 1609 | + | |
| 1610 | + | |
| 1611 | + | |
| 1612 | + | |
1461 | 1613 | | |
1462 | 1614 | | |
1463 | 1615 | | |
| |||
1510 | 1662 | | |
1511 | 1663 | | |
1512 | 1664 | | |
1513 | | - | |
| 1665 | + | |
| 1666 | + | |
| 1667 | + | |
1514 | 1668 | | |
1515 | 1669 | | |
1516 | 1670 | | |
| |||
1526 | 1680 | | |
1527 | 1681 | | |
1528 | 1682 | | |
| 1683 | + | |
1529 | 1684 | | |
1530 | 1685 | | |
1531 | 1686 | | |
1532 | 1687 | | |
1533 | | - | |
| 1688 | + | |
| 1689 | + | |
| 1690 | + | |
| 1691 | + | |
| 1692 | + | |
| 1693 | + | |
| 1694 | + | |
| 1695 | + | |
1534 | 1696 | | |
1535 | 1697 | | |
1536 | 1698 | | |
| |||
1539 | 1701 | | |
1540 | 1702 | | |
1541 | 1703 | | |
1542 | | - | |
| 1704 | + | |
1543 | 1705 | | |
1544 | 1706 | | |
1545 | 1707 | | |
| |||
1584 | 1746 | | |
1585 | 1747 | | |
1586 | 1748 | | |
| 1749 | + | |
| 1750 | + | |
| 1751 | + | |
| 1752 | + | |
| 1753 | + | |
| 1754 | + | |
| 1755 | + | |
| 1756 | + | |
| 1757 | + | |
| 1758 | + | |
| 1759 | + | |
| 1760 | + | |
| 1761 | + | |
| 1762 | + | |
| 1763 | + | |
| 1764 | + | |
| 1765 | + | |
| 1766 | + | |
| 1767 | + | |
1587 | 1768 | | |
1588 | 1769 | | |
1589 | 1770 | | |
| |||
0 commit comments