Commit 31521d6
authored
fix(connectors): validate and repair the knowledge-base connector fleet (#6757)
* fix(connectors): validate and repair all 61 knowledge-base connectors
Audits every KB connector against its provider's live API documentation and
fixes what the audit found. The dominant defect class is deletion
reconciliation: the sync engine hard-deletes any stored document absent from a
"full" listing, and most connectors had a path where a truncated or errored
listing failed to set `syncContext.listingCapped`.
Highest-impact fixes:
- linear: `getDocument` was dead code. The query declared `$id: ID!` where the
schema is `issue(id: String!)`, so every call failed variable validation.
- salesforce: `v62.0` was substituted into a `{version}` template that already
contains the `v`, so every REST call 404'd. SOQL `LIMIT` was also used as a
page size, silently capping every sync at 200 records.
- notion: only the first level of blocks was fetched, so tables indexed empty.
- microsoft-teams: `/messages` returns messages without replies, so no threaded
content was ever indexed.
- gmail: an empty page discarded `nextPageToken`, which reads as a complete
empty listing and hard-deletes every stored thread.
- confluence: the CQL path paginated with `start` and `totalSize`, neither of
which exists on that endpoint, so label-filtered syncs stopped after one page.
- box: `supportsRefreshTokenRotation` was unset, so Box's rotated refresh token
was discarded and every credential died on its second refresh.
Removes the Evernote integration entirely: the classic EDAM API is deprecated,
its sandbox is decommissioned, and developer tokens are no longer obtainable.
Makes SFTP host-key verification mandatory, and adds an attendee-PII opt-out to
google-calendar and google-meet (default on, so existing sources are unchanged).
Bumps the contentHash namespace for notion, google-docs and hubspot so existing
documents re-hydrate once and actually receive the content fixes above.
* fix(connectors): close swallow-into-empty and cursor-taper regressions
Ship-gate pass over the connector audit. Every finding was re-verified
against the provider's live documentation or machine-readable spec before
being acted on; several pass-3 edits were reverted rather than extended.
Correctness fixes:
- fireflies: a 2xx with an unparseable body returned an empty listing
instead of throwing. fireflies runs a full sync every time, so a fault
persisting across two syncs would have tombstoned all indexed docs.
- linear: same shape via `data.issues || {}` on a non-nullable connection.
- greenhouse: a 403 from a key without scorecard permission was treated as
transient, appending `:partial` to the hash. That never matches the list
stub, forcing full re-hydration of every candidate on every sync forever.
- google-meet: `fetchParticipants` carried a 404 swallow copied from its
transcript siblings, freezing every speaker as "Unknown".
- airtable, asana, ashby: reverted page-size tapers applied over opaque
cursor tokens. The cap was already enforced server-side.
- google-docs: response byte cap resolved to 800MB and could never fire.
- google-forms, google-vault, notion, sharepoint, dropbox: `getDocument`
now throws on transient failure instead of returning null, which the
engine reads as absence.
Security:
- Retry headers are attached non-enumerably. TypeScript `private` is
compile-time only, so `SecureFetchHeaders.setCookies` was an own
enumerable property that the logger serialized into sync logs.
Docs and dead code:
- Corrected six fabricated doc citations (github, jira, jsm, linear,
google-meet, dropbox) and removed the Evernote integration entirely.
* refactor(jira): type the ADF node helpers with unknown instead of any
* fix(connectors): throw on misconfigured typeform/zendesk sources instead of returning null
A null from getDocument reads as documented absence, so on an add the
document is dropped with neither a failure counter nor a log. Both
listDocuments paths already throw on the same missing config.
* fix(connectors): keep confluence CQL page size constant; unify monday API version
The CQL search endpoint paginates by opaque cursor, and Atlassian does not
document that a cursor issued against one limit survives a request asking
for a different one. Narrowing limit to the remaining budget was the same
pattern reverted on airtable, asana, and ashby. The page size is now
constant and the cap is applied by trimming the returned page, which keeps
the cap exact without varying the request.
Monday OAuth getUserInfo hardcoded API-Version 2024-10 while every other
monday surface reads MONDAY_API_VERSION, defeating the single-source pin.
* fix(connectors): act on the final validation sweep
Findings from a read-only /validate-connector pass over all 59 changed
connectors, verified against provider specs before acting.
Silent-drop fixes (a fulfilled null from getDocument records no failure and
no log, so the document vanishes):
- ashby: candidate.info returning success with an unusable payload. Ashby
sets contentDeferred, so this path is live.
- azure-devops: an unresolvable branch, likewise live.
- dropbox: 409 covers the whole LookupError union, and restricted_content
and locked both mean the file still exists. Only not_found is absence.
- docusign: fetchFormValues swallowed every non-OK status, baking a
permanently incomplete document since the hash is metadata-only.
typeform: 'all' sent response_type=started,partial,completed, but Typeform
documents only partial and completed. An unknown enum member risks a 400
that fails the whole sync, and staging omitted the parameter entirely, so
this shipped as a regression. Now requests the widest documented set.
github: removes a utf-8 blob branch justified by a misattributed quote —
that sentence describes the encoding REQUEST parameter of Create a blob;
the GET response is documented as always base64. Also corrects two comments
that hid a real drop: >1 MB files under vnd.github+json 403 rather than
returning encoding: none.
hubspot: routes HTML detection through a shared anchored helper. The loose
pattern matched angle-bracketed prose such as an email address, and
htmlToPlainText deletes the span and collapses line structure. This matters
now because the hubspot:v2: bump rewrites every live document once.
youtube: drops an invented channel-ID format quote.
* fix(connectors): converge incidentio partial hash on settled statuses
A 403 from a key without incident_updates permission, or a 404, returns on
every sync. Marking those incomplete appended :partial to a hash that then
never matched the listing stub, so the incident re-hydrated forever without
converging. Only a transient failure may mark content incomplete now, which
matches how greenhouse already treats the same class.
* fix(connectors): flag WIQL truncation unconditionally; stop swallowing docusign form-data failures
azure-devops: the 20,000-item WIQL ceiling was probed by asking for a matching
item with an id beyond the largest returned. That probe is unsound — buildWiql
orders by ChangedDate DESC while ids are assigned in creation order, so the
highest-id match is almost always inside the returned window. The probe came
back empty for genuinely truncated projects, left the listing unflagged, and
let deletion reconciliation remove every indexed item outside it. Flag
unconditionally instead; the cost is a project sitting exactly at the ceiling
not reconciling deletions until a full resync.
docusign: fetchFormValues threw on a non-404 status and then caught its own
throw, returning []. The earlier fix was a no-op. The catch now rethrows, so a
transient failure produces a failed row instead of a permanently incomplete
document under a metadata-only hash.
* fix(connectors): restore airtable AI Text indexing; floor sentry maxIssues
airtable: staging rendered object cells with a JSON.stringify fallback, so AI
Text values and nested lookup arrays reached the index. This branch replaced
that with a fixed key-probe list to stop attachment-URL hash churn, but the
probe list has no fallback — aiText ({state,isStale,value}) and nested arrays
rendered to the empty string and vanished from every document, and the
content-derived hash never moved when the text regenerated. Read `value` last,
after the existing probes, and recurse on nested arrays. The generated text is
stable rather than an expiring signed URL, so this does not reintroduce churn.
sentry: maxIssues now feeds the request limit, and Sentry rejects a non-integer.
validateConfig accepts a fractional entry, so a config that saved cleanly would
fail every sync at listing time. Staging was immune only because it sent a
hardcoded page size.1 parent 557a474 commit 31521d6
203 files changed
Lines changed: 13633 additions & 8096 deletions
File tree
- apps
- docs
- components
- ui
- content/docs/en
- integrations
- knowledgebase
- sim
- app/api/tools
- docusign
- evernote
- copy-note
- create-notebook
- create-note
- create-tag
- delete-note
- get-notebook
- get-note
- lib
- list-notebooks
- list-tags
- search-notes
- update-note
- monday
- boards
- groups
- blocks
- blocks
- components
- connectors
- airtable
- asana
- ashby
- azure-devops
- box
- confluence
- discord
- docusign
- dropbox
- evernote
- fathom
- fireflies
- github
- gitlab
- gmail
- gong
- google-calendar
- google-docs
- google-drive
- google-forms
- google-meet
- google-sheets
- google-slides
- google-vault
- grain
- granola
- greenhouse
- hubspot
- incidentio
- intercom
- jira
- jsm
- linear
- microsoft-excel
- microsoft-teams
- mintlify
- monday
- notion
- obsidian
- onedrive
- outlook
- pagerduty
- reddit
- rootly
- s3
- salesforce
- sentry
- servicenow
- sftp
- slack
- trello
- typeform
- webflow
- wordpress
- x
- youtube
- zendesk
- zoho-desk
- zoom
- lib
- api/contracts/tools
- auth/connectors
- core/config
- integrations
- knowledge/documents
- oauth
- webhooks/providers
- tools
- evernote
- generated
- jira
- monday
- reddit
Some content is hidden
Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
2769 | 2769 | | |
2770 | 2770 | | |
2771 | 2771 | | |
2772 | | - | |
2773 | | - | |
2774 | | - | |
2775 | | - | |
2776 | | - | |
2777 | | - | |
2778 | | - | |
2779 | | - | |
2780 | 2772 | | |
2781 | 2773 | | |
2782 | 2774 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
72 | 72 | | |
73 | 73 | | |
74 | 74 | | |
75 | | - | |
76 | 75 | | |
77 | 76 | | |
78 | 77 | | |
| |||
334 | 333 | | |
335 | 334 | | |
336 | 335 | | |
337 | | - | |
338 | 336 | | |
339 | 337 | | |
340 | 338 | | |
| |||
This file was deleted.
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
73 | 73 | | |
74 | 74 | | |
75 | 75 | | |
76 | | - | |
77 | 76 | | |
78 | 77 | | |
79 | 78 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
12 | 12 | | |
13 | 13 | | |
14 | 14 | | |
15 | | - | |
| 15 | + | |
16 | 16 | | |
17 | | - | |
| 17 | + | |
18 | 18 | | |
19 | 19 | | |
20 | 20 | | |
| |||
28 | 28 | | |
29 | 29 | | |
30 | 30 | | |
31 | | - | |
| 31 | + | |
32 | 32 | | |
33 | 33 | | |
34 | 34 | | |
| |||
48 | 48 | | |
49 | 49 | | |
50 | 50 | | |
51 | | - | |
52 | 51 | | |
53 | 52 | | |
54 | 53 | | |
| |||
57 | 56 | | |
58 | 57 | | |
59 | 58 | | |
60 | | - | |
| 59 | + | |
61 | 60 | | |
62 | 61 | | |
63 | 62 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
13 | 13 | | |
14 | 14 | | |
15 | 15 | | |
| 16 | + | |
16 | 17 | | |
17 | 18 | | |
18 | 19 | | |
| |||
79 | 80 | | |
80 | 81 | | |
81 | 82 | | |
82 | | - | |
| 83 | + | |
83 | 84 | | |
84 | 85 | | |
85 | 86 | | |
| |||
0 commit comments