v0.8.8: mothership improvements, new connectors, knowledgebase hardening, v2 extension

Author: waleedlatif1

.claude/rules/sim-caching.md

@@ -0,0 +1,99 @@
+---
+paths:
+  - "apps/sim/lib/**/*.ts"
+  - "apps/sim/providers/**/*.ts"
+  - "apps/sim/executor/**/*.ts"
+  - "apps/sim/tools/**/*.ts"
+---
+
+# In-Process Caching
+
+**Never hand-roll TTL arithmetic.** `lru-cache` is a direct dependency of `apps/sim` and owns
+expiry, the ceiling, and — through `fetchMethod` — request coalescing. A
+`Map` plus `Date.now() - entry.fetchedAt < TTL` re-implements all three, badly.
+
+## First decide whether it is a cache at all
+
+Most module-level `Map`s in this codebase are **not** caches, and forcing them into one is worse
+than leaving them alone.
+
+| Shape | Key dies when | Right tool |
+| --- | --- | --- |
+| **Lifecycle map** — `activeStreams`, `pendingChildRuns`, `memoryStreams`, `handlerRegistry` | the tracked thing ends, and the code deletes it there | plain `Map`. No TTL, no ceiling. |
+| **TTL cache** — a remote read keyed by tenant (org id, user id, workspace id) | time passes | `LRUCache` |
+
+A lifecycle map's key space is unbounded and that is fine, because every key has a defined death.
+Adding a TTL to one introduces an expiry that races the lifecycle. Adding a ceiling silently drops
+live state.
+
+## TTL caches: always set `max`
+
+```ts
+const policyCache = new LRUCache<string, ResolvedSessionPolicy>({
+  max: 20_000,
+  ttl: SESSION_POLICY_CACHE_TTL_MS,
+})
+```
+
+`ttl` alone does **not** bound memory. Without `ttlAutopurge` (itself expensive — one timer per
+entry) an expired entry lingers until something touches its key or the ceiling evicts it. `max` is
+what actually caps the process, which is why a tenant-keyed `Map` grew for the life of the process
+before this rule existed.
+
+**The ceiling is a memory backstop, not an operating limit.** Exceeding it makes the LRU evict
+*inside* the TTL, so each miss becomes one more read — never a wrong answer, it degrades to exactly
+the pre-cache behavior, but it is a hit-rate cliff on whatever path the cache sits on. Entries are
+tens of bytes, so set the cap far above any plausible per-instance working set within the TTL
+window and let it stay a backstop.
+
+**Reads test `!== undefined`, not truthiness**, whenever the value can be `false`, `0`, or `null`.
+`if (cached)` on a cached `false` re-queries on every single call, for exactly the tenants the
+cache exists to protect.
+
+## Async read-through: prefer `fetchMethod`
+
+`fetchMethod` + `cache.fetch(key)` gives TTL, coalescing (concurrent callers share one promise),
+and eviction-on-rejection (`noDeleteOnFetchRejection` defaults to `false`) in one primitive. Reach
+for it before composing anything yourself.
+
+**The one reason to compose instead: a hung producer.** `fetchMethod` has no settle deadline, and
+the app pool sets no `statement_timeout` (`packages/db/db.ts` sets only `connect_timeout` /
+`idle_timeout`, neither of which bounds a query already in flight). Where a wedged read would hold
+every caller for the whole TTL, wrap `coalesceLocally` from `@/lib/concurrency/singleflight` around
+a read-through `LRUCache` instead — it evicts and rejects at its deadline. See
+`lib/api-key/byok-entitlement.ts`, and `lib/oauth/credential-service.ts` for the same shape.
+
+Do **not** build a house wrapper over `lru-cache`. Call sites differ in ways a thin helper cannot
+hold (synchronous memoization with `updateAgeOnGet` in `providers/client-cache.ts`, per-entry TTLs
+in `lib/auth/security-policy.ts`), so a wrapper covering the common case just adds a fourth pattern.
+
+## Cache the gate, never the credential
+
+Entitlements, plans, and policies tolerate bounded staleness **in the safe direction** — a lapsed
+organization keeping its own provider key for another minute costs a little metering and charges
+nobody wrongly. Key material does not: revocation has to be immediate, so
+`getBYOKKey` reads key rows fresh on every call and caches only the entitlement around them.
+
+An outage must not be cached as a negative answer. A resolver that maps a failed read to `false`
+makes an outage indistinguishable from a real lapse, so give it an `onError: 'throw'` option and
+write the cache only on the success path — see `resolveOrganizationPlan`.
+
+**Where a human is waiting, read fresh.** Keep two entry points rather than one cached function:
+the settings surfaces and management use cases must not tell an organization that just upgraded
+that it still lacks a plan, while the execution path underneath can serve from cache
+(`isOrganizationBYOKEntitled` vs `isOrganizationBYOKEntitledCached`).
+
+## React `cache()` does nothing in a worker
+
+`cache()` is request-scoped. Workflows run in Trigger.dev workers, which have no React request
+scope, so a `cache()`-wrapped gate that looks free on a settings page is uncached and per-block on
+the execution path. Anything reached from the executor needs a real cache — see
+`.claude/rules/sim-architecture.md`'s app/worker runtime boundary.
+
+## Invalidation
+
+Add a per-key invalidator only when the code that mutates the value runs in the **same process**
+that reads it. `invalidateSessionPolicyCache` works because the route writing the policy is the one
+serving the reads. An entitlement change arriving on a Stripe webhook lands in one process while
+the readers are per-worker, so an invalidator there would imply an immediacy it cannot deliver —
+the TTL is the real mechanism, and the absence of an invalidator should say so.

.github/workflows/test-build.yml

@@ -123,6 +123,9 @@ jobs:
       - name: Repo audits
         run: bun run check:audits
 
+      - name: Verify docs manifest is in sync
+        run: bun run docs-manifest:check
+
       - name: Migration safety (zero-downtime) audit
         run: |
           if [ "${{ github.event_name }}" = "pull_request" ]; then

CLAUDE.md

@@ -472,6 +472,22 @@ describe('my route', () => {
 
 Use `@sim/testing` mocks/factories over local test data.
 
+## Caching
+
+Never hand-roll TTL arithmetic — a `Map` plus `Date.now() - fetchedAt < TTL` re-implements expiry,
+the ceiling, and coalescing badly. Use `lru-cache` (a direct dependency), and always set `max`:
+`ttl` alone does not bound memory, so a tenant-keyed cache without a ceiling grows for the life of
+the process. Prefer `fetchMethod` + `cache.fetch(key)` for async read-through — it gives TTL,
+coalescing, and eviction-on-rejection in one primitive — and compose `coalesceLocally` around an
+`LRUCache` only when a hung producer would otherwise wedge callers for the whole TTL.
+
+First check the thing is a cache at all: a lifecycle map whose entry is deleted when the tracked
+thing ends (`activeStreams`, `pendingChildRuns`) is a plain `Map`, and giving it a TTL or a ceiling
+introduces an expiry that races the lifecycle. Cache the gate, never the credential — entitlements
+tolerate bounded staleness in the safe direction, key material must stay fresh so revocation is
+immediate. Full decision tree, sizing, `!== undefined` reads, and the invalidation rule are in
+`.claude/rules/sim-caching.md`.
+
 ## Utils Rules
 
 - Never create `utils.ts` for single consumer - inline it

apps/desktop/build/entitlements.mac.plist

@@ -9,5 +9,15 @@
          even after the user grants access in System Settings. -->
     <key>com.apple.security.device.audio-input</key>
     <true/>
+    <!-- The agent browser joins real meetings (Google Meet, Zoom web): its
+         getUserMedia grant is gated on the OS grant, and without this key the
+         Hardened Runtime denies the camera no matter what the user allowed. -->
+    <key>com.apple.security.device.camera</key>
+    <true/>
+    <!-- WebAuthn hybrid transport (passkey on the user's phone via QR) rides
+         Bluetooth proximity; without this the QR option silently never
+         completes in signed builds. -->
+    <key>com.apple.security.device.bluetooth</key>
+    <true/>
   </dict>
 </plist>

apps/desktop/electron-builder.yml

@@ -58,7 +58,9 @@ mac:
   # macOS refuses to show the microphone prompt at all — it kills the process —
   # unless the bundle declares why it wants the device.
   extendInfo:
-    NSMicrophoneUsageDescription: Sim uses your microphone for voice input in Chat.
+    NSMicrophoneUsageDescription: Sim uses your microphone for voice input in Chat and for meetings you join in the built-in browser.
+    NSCameraUsageDescription: Sim uses your camera for meetings you join in the built-in browser, such as Google Meet.
+    NSBluetoothAlwaysUsageDescription: Sim uses Bluetooth to complete passkey sign-ins with a nearby phone in the built-in browser.
   entitlements: build/entitlements.mac.plist
   entitlementsInherit: build/entitlements.mac.plist
   notarize: true

apps/desktop/src/main/browser-agent/cdp.test.ts

@@ -2,8 +2,9 @@ import { describe, expect, it, vi } from 'vitest'
 
 vi.mock('electron', () => import('@/test/electron-mock'))
 
-import { WebContentsView, type WebFrameMain } from 'electron'
+import { nativeImage, type WebContents, WebContentsView, type WebFrameMain } from 'electron'
 import {
+  captureScreenshot,
   clickAt,
   ensureInstrumented,
   evaluateInIsolatedFrame,
@@ -482,3 +483,89 @@ describe('browser-agent CDP theme', () => {
     })
   })
 })
+
+/**
+ * The browser panel shows a LIVE view, so a capture must not perturb the page.
+ * Chromium serves `clip` by applying device-emulation params to the widget and
+ * syncing visual properties, which the user sees as the page rescaling and
+ * snapping back. Resolution is bounded on the returned image instead.
+ */
+describe('browser-agent screenshot capture', () => {
+  function captureFixture(imageSize: { width: number; height: number } | null) {
+    const contents = new WebContentsView().webContents
+    vi.mocked(contents.debugger.sendCommand).mockImplementation((method: string) => {
+      if (method === 'Page.getLayoutMetrics') {
+        return Promise.resolve({ cssLayoutViewport: { clientWidth: 2048, clientHeight: 1024 } })
+      }
+      if (method === 'Page.captureScreenshot') return Promise.resolve({ data: 'c2lt' })
+      return Promise.resolve(undefined)
+    })
+    const resized = {
+      toJPEG: vi.fn(() => Buffer.from('resized')),
+    }
+    // Shared module-level mock: without this, a later fixture reads the
+    // earlier test's decoded image.
+    vi.mocked(nativeImage.createFromBuffer).mockReset()
+    vi.mocked(nativeImage.createFromBuffer).mockReturnValue({
+      isEmpty: vi.fn(() => imageSize === null),
+      getSize: vi.fn(() => imageSize ?? { width: 0, height: 0 }),
+      resize: vi.fn(() => resized),
+      toJPEG: vi.fn(() => Buffer.alloc(0)),
+    } as unknown as ReturnType<typeof nativeImage.createFromBuffer>)
+    return { contents, resized }
+  }
+
+  function screenshotParams(contents: WebContents): Record<string, unknown> {
+    const call = vi
+      .mocked(contents.debugger.sendCommand)
+      .mock.calls.find(([method]) => method === 'Page.captureScreenshot')
+    if (!call) throw new Error('no capture was requested')
+    return call[1] as Record<string, unknown>
+  }
+
+  it('never sends a clip, which would emulate the live page for the capture', async () => {
+    const { contents } = captureFixture({ width: 4096, height: 2048 })
+
+    await captureScreenshot(contents)
+
+    expect(screenshotParams(contents)).not.toHaveProperty('clip')
+  })
+
+  /**
+   * A 2048px CSS viewport bounded to 1024px is scale 0.5, and the capture
+   * arrives at device resolution (4096px on a 2x display). The resize is what
+   * lands the image on the CSS-relative size the coordinate contract
+   * (cssX = imageX / scale) assumes.
+   */
+  it('downscales the returned image to the CSS-relative size', async () => {
+    const { contents, resized } = captureFixture({ width: 4096, height: 2048 })
+
+    const shot = await captureScreenshot(contents)
+
+    const image = vi.mocked(nativeImage.createFromBuffer).mock.results[0].value
+    expect(image.resize).toHaveBeenCalledWith({ width: 1024, height: 512, quality: 'good' })
+    expect(resized.toJPEG).toHaveBeenCalled()
+    expect(shot).toEqual({
+      dataUrl: `data:image/jpeg;base64,${Buffer.from('resized').toString('base64')}`,
+      scale: 0.5,
+    })
+  })
+
+  it('skips the re-encode when the capture already matches the target size', async () => {
+    const { contents } = captureFixture({ width: 1024, height: 512 })
+
+    const shot = await captureScreenshot(contents)
+
+    const image = vi.mocked(nativeImage.createFromBuffer).mock.results[0].value
+    expect(image.resize).not.toHaveBeenCalled()
+    expect(shot).toEqual({ dataUrl: 'data:image/jpeg;base64,c2lt', scale: 0.5 })
+  })
+
+  it('returns the raw capture when the image cannot be decoded', async () => {
+    const { contents } = captureFixture(null)
+
+    const shot = await captureScreenshot(contents)
+
+    expect(shot).toEqual({ dataUrl: 'data:image/jpeg;base64,c2lt', scale: 0.5 })
+  })
+})