Skip to content

Commit 1372977

Browse files
feat(setup): publish standalone self-hosting package (#6849)
* feat(setup): publish standalone self-hosting package * fix(setup): refresh discovered compose installs * improvement(setup): unify repository command * fix(setup): harden standalone package launch * Update README.md * fix(setup): isolate standalone compose installs * fix(setup): restore default stopped installs
1 parent 04e0fe0 commit 1372977

108 files changed

Lines changed: 4384 additions & 2417 deletions

File tree

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

.agents/skills/add-block/SKILL.md

Lines changed: 20 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -172,8 +172,8 @@ Optional companions: `credentialLabels` (override the picker's section/connect-r
172172
### OAuth deployment availability (required for integration blocks)
173173

174174
A visible tools-category block with OAuth is deployment-gated. Its `oauth-input.serviceId` is
175-
projected into `apps/sim/lib/integrations/integrations.json`, then resolved through
176-
`resolveOAuthClientCapabilityId()` in `apps/sim/lib/core/config/env-capabilities.ts`.
175+
projected into `packages/deployment-config/src/integrations.json`, then resolved through
176+
`resolveOAuthClientCapabilityId()` in `packages/deployment-config/src/env-capabilities.ts`.
177177

178178
When adding or changing an OAuth integration block:
179179

@@ -184,13 +184,14 @@ When adding or changing an OAuth integration block:
184184
3. For a new capability, add its required client fields to `OAUTH_CLIENT_CAPABILITIES` and ensure
185185
every referenced field exists in the env schema in `apps/sim/lib/core/config/env.ts`. Then add
186186
the matching `text` or `secret` input modes to `OAUTH_CLIENT_SETUP_FIELDS` in
187-
`scripts/setup/capability-config.ts`. The CLI catalog is exhaustively typed and checked against
188-
the runtime field list; do not infer secrecy from the field name.
189-
4. If the canonical OAuth service declares `serviceAccountProviderId`, keep
190-
`SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` in
191-
`apps/sim/lib/integrations/service-account-metadata.ts` aligned. Set
192-
`deploymentRequirement` only when the service-account path is preview-gated or depends on the
193-
OAuth client fields; otherwise omit it.
187+
`packages/sim-setup/src/capability-config.ts`. The CLI catalog is exhaustively typed and checked
188+
against the runtime field list; do not infer secrecy from the field name.
189+
4. If the canonical OAuth service declares `serviceAccountProviderId`, run
190+
`bun run deployment-config:generate`; this regenerates the provider-ID facts in
191+
`packages/deployment-config/src/service-account-providers.generated.ts`. Never hand-edit that
192+
generated map. Add `deploymentRequirement` policy in
193+
`packages/deployment-config/src/service-account-metadata.ts` only when the service-account path
194+
is preview-gated or depends on the OAuth client fields; otherwise omit it.
194195

195196
Missing capability metadata is a runtime configuration error, not a reason to make the integration
196197
silently available.
@@ -992,16 +993,21 @@ After adding or changing one, run:
992993

993994
```bash
994995
bun run scripts/generate-docs.ts
996+
bun run deployment-config:generate
995997
bun run integration-catalog:check
998+
bun run deployment-config:check
996999
bun run docs:check
9971000
```
9981001

9991002
The catalog check independently derives deployment metadata from the executable block registry and
1000-
compares it with the committed `apps/sim/lib/integrations/integrations.json`. `docs:check` re-renders
1001-
every generated docs artifact in memory and fails on any committed file that differs — it runs in CI
1002-
via `check:audits`, so commit the full generator output. If the generator also trues up pages an
1003-
earlier PR left stale, commit that catch-up too; reverting it as "unrelated drift" makes `docs:check`
1004-
fail.
1003+
compares it with the committed `packages/deployment-config/src/integrations.json`. The deployment
1004+
config check verifies the generated service-account facts against the canonical OAuth registry and
1005+
catalog. `docs:check` re-renders every generated docs artifact in memory and fails on any committed
1006+
file that differs — it runs in CI via `check:audits`, so commit the full generator output. If the
1007+
generator also trues up pages an earlier PR left stale, commit that catch-up too; reverting it as
1008+
"unrelated drift" makes `docs:check` fail. Review the generated diff and keep only intentional
1009+
changes.
1010+
10051011
## Checklist Before Finishing
10061012

10071013
- [ ] `integrationType` is set to the correct `IntegrationType` enum value

.agents/skills/add-integration/SKILL.md

Lines changed: 19 additions & 12 deletions
Original file line numberDiff line numberDiff line change
@@ -538,16 +538,18 @@ the OAuth service configuration, deployment availability, and the setup CLI.
538538
1. Ensure the block has exactly one distinct OAuth `serviceId` and that it matches the canonical
539539
service entry in `apps/sim/lib/oauth/oauth.ts`.
540540
2. Confirm `resolveOAuthClientCapabilityId(serviceId)` resolves to the intended provider entry in
541-
`OAUTH_CLIENT_CAPABILITIES` in `apps/sim/lib/core/config/env-capabilities.ts`. Google and
541+
`OAUTH_CLIENT_CAPABILITIES` in `packages/deployment-config/src/env-capabilities.ts`. Google and
542542
Microsoft service IDs deliberately share provider-level capabilities.
543543
3. For a new OAuth provider, add the required client fields to `OAUTH_CLIENT_CAPABILITIES`, add
544544
every referenced field to the env schema in `apps/sim/lib/core/config/env.ts`, and add the
545545
matching `text` or `secret` entries to `OAUTH_CLIENT_SETUP_FIELDS` in
546-
`scripts/setup/capability-config.ts`. Do not create integration-specific setup logic or infer
547-
secret fields from naming; the CLI mapping is exhaustively checked against the runtime fields.
548-
4. If the canonical OAuth service has `serviceAccountProviderId`, add the matching projection to
549-
`SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID` in
550-
`apps/sim/lib/integrations/service-account-metadata.ts`. Use:
546+
`packages/sim-setup/src/capability-config.ts`. Do not create integration-specific setup logic or
547+
infer secret fields from naming; the CLI mapping is exhaustively checked against the runtime
548+
fields.
549+
4. If the canonical OAuth service has `serviceAccountProviderId`, run
550+
`bun run deployment-config:generate` to refresh
551+
`packages/deployment-config/src/service-account-providers.generated.ts`; never hand-edit the
552+
generated provider-ID map. In `packages/deployment-config/src/service-account-metadata.ts`, use:
551553
- no `deploymentRequirement` when the service-account path works independently of OAuth client fields;
552554
- `'oauth-client'` when it requires the same deployment OAuth client fields;
553555
- `'preview-gated'` when availability is controlled by the service-account preview block.
@@ -560,15 +562,18 @@ a resolvable capability must fail validation.
560562
Run the documentation generator:
561563
```bash
562564
bun run scripts/generate-docs.ts
565+
bun run deployment-config:generate
563566
bun run integration-catalog:check
567+
bun run deployment-config:check
564568
bun run docs:check
565569
```
566570

567571
This creates `apps/docs/content/docs/en/integrations/{service}.mdx` — one page per service carrying the block's Actions and, if it has one, its Triggers section. Never hand-edit generated pages; the only editable region is the `{/* MANUAL-CONTENT */}` block (see `scripts/README.md`).
568572

569-
The same generator refreshes `apps/sim/lib/integrations/integrations.json`. The catalog check then
570-
derives the deployment-relevant fields from the executable block registry and compares them with the
571-
committed projection. Review the generated diff and keep only intentional changes.
573+
The docs generator refreshes `packages/deployment-config/src/integrations.json`, and the deployment
574+
config generator projects service-account provider IDs from that catalog plus the canonical OAuth
575+
registry. The checks compare both committed projections with their sources. Review the generated
576+
diff and keep only intentional changes.
572577

573578
## V2 Integration Pattern
574579

@@ -647,14 +652,16 @@ If creating V2 versions (API-aligned outputs):
647652
- [ ] Created `index.ts` barrel export
648653
- [ ] Registered all triggers in `triggers/registry.ts`
649654

650-
### Docs
655+
### Docs and deployment metadata
651656
- [ ] Ran `bun run scripts/generate-docs.ts`
657+
- [ ] Ran `bun run deployment-config:generate` for OAuth or service-account changes
652658
- [ ] Verified docs file created
653-
- [ ] Reviewed and committed the generated `apps/sim/lib/integrations/integrations.json` change
659+
- [ ] Reviewed and committed the generated `packages/deployment-config/src/integrations.json` change
654660
- [ ] `bun run integration-catalog:check` passes
655661
- [ ] `bun run docs:check` passes — CI fails on stale generated docs, so commit the full generator
656662
output, including catch-up regeneration for pages another PR left stale (never revert it as
657663
"unrelated drift")
664+
- [ ] `bun run deployment-config:check` passes
658665

659666
### Final Validation (Required)
660667
- [ ] Read every tool file and cross-referenced inputs/outputs against the API docs
@@ -1002,4 +1009,4 @@ requiredScopes: getScopesForService('{service}'),
10021009
11. **Never hardcode scopes** - Use `getScopesForService()` in blocks and `getCanonicalScopesForProvider()` in auth.ts
10031010
12. **Always add scope descriptions** - New scopes must have entries in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts`
10041011
13. **OAuth service IDs need deployment capabilities** - Every visible OAuth integration must resolve through `OAUTH_CLIENT_CAPABILITIES`; shared Google/Microsoft aliases map to their provider capability
1005-
14. **Keep runtime and presentation separate** - Runtime OAuth fields live in `env-capabilities.ts`; CLI input modes live in the exhaustively checked `scripts/setup/capability-config.ts` mapping
1012+
14. **Keep runtime and presentation separate** - Runtime OAuth fields live in `packages/deployment-config/src/env-capabilities.ts`; CLI input modes live in the exhaustively checked `packages/sim-setup/src/capability-config.ts` mapping

.agents/skills/validate-integration/SKILL.md

Lines changed: 19 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -29,12 +29,13 @@ apps/sim/blocks/registry-maps.ts # Block + meta registry entry (BLOCK_REGISTR
2929
apps/sim/components/icons.tsx # Icon definition
3030
apps/sim/lib/auth/auth.ts # OAuth config — should use getCanonicalScopesForProvider()
3131
apps/sim/lib/oauth/oauth.ts # OAuth provider config — single source of truth for scopes
32-
apps/sim/lib/oauth/utils.ts # Scope utilities, SCOPE_DESCRIPTIONS for modal UI
33-
apps/sim/lib/core/config/env-capabilities.ts # OAuth client runtime capability source of truth
32+
apps/sim/lib/oauth/utils.ts # Scope utilities, SCOPE_DESCRIPTIONS for modal UI
33+
packages/deployment-config/src/env-capabilities.ts # OAuth client runtime capability source of truth
3434
apps/sim/lib/core/config/env.ts # Runtime env schema for capability fields
35-
scripts/setup/capability-config.ts # Exhaustive CLI input-mode mapping for OAuth fields
36-
apps/sim/lib/integrations/integrations.json # Generated client-safe integration catalog
37-
apps/sim/lib/integrations/service-account-metadata.ts # Lightweight service-account projection
35+
packages/sim-setup/src/capability-config.ts # Exhaustive CLI input-mode mapping for OAuth fields
36+
packages/deployment-config/src/integrations.json # Generated client-safe integration catalog
37+
packages/deployment-config/src/service-account-providers.generated.ts # Generated provider-ID facts
38+
packages/deployment-config/src/service-account-metadata.ts # Handwritten deployment policy
3839
```
3940

4041
## Step 2: Pull API Documentation
@@ -291,17 +292,17 @@ Scopes are centralized — the single source of truth is `OAUTH_PROVIDERS` in `l
291292
## Step 6: Validate Deployment Availability (if OAuth service)
292293

293294
The deployment UI and setup CLI do not infer OAuth client fields from scopes. They resolve the
294-
block's generated `oauthServiceId` through the application-owned capability catalog.
295+
block's generated `oauthServiceId` through the shared deployment capability catalog.
295296

296297
- [ ] The visible integration block has exactly one distinct `oauth-input.serviceId`
297298
- [ ] `resolveOAuthClientCapabilityId(serviceId)` returns the intended provider capability
298299
- [ ] The resolved provider exists in `OAUTH_CLIENT_CAPABILITIES`
299300
- [ ] Every field listed by that capability exists in `apps/sim/lib/core/config/env.ts`
300301
- [ ] Every capability field has the correct `text` or `secret` entry in `OAUTH_CLIENT_SETUP_FIELDS`; no CLI naming heuristic is required
301302
- [ ] Shared Google/Microsoft service IDs resolve to their provider capability rather than duplicate entries
302-
- [ ] `bun run setup integration <capabilityId>` is the command emitted by availability; the CLI has only the exhaustive input-mode projection, not a second runtime provider definition
303+
- [ ] `npx @sim/setup add integration <capabilityId>` is the command emitted by availability; the CLI has only the exhaustive input-mode projection, not a second runtime provider definition
303304
- [ ] If the canonical OAuth service declares `serviceAccountProviderId`,
304-
`SERVICE_ACCOUNT_METADATA_BY_OAUTH_SERVICE_ID[serviceId]` has the same provider ID
305+
the generated `SERVICE_ACCOUNT_PROVIDER_BY_OAUTH_SERVICE_ID[serviceId]` has the same provider ID
305306
- [ ] The service-account `deploymentRequirement` matches how that credential actually works:
306307
omitted for an independent path, `'oauth-client'` when it needs the OAuth client fields, or
307308
`'preview-gated'` when controlled by a preview block
@@ -386,13 +387,16 @@ Several files are generated from tool and block definitions. Editing a tool or b
386387

387388
```bash
388389
bun run tool-metadata:generate # repo root — apps/sim/tools/generated/*
389-
bun run scripts/generate-docs.ts # docs .mdx + lib/integrations/integrations.json + docs icons
390+
bun run scripts/generate-docs.ts # docs .mdx + deployment-config/integrations.json + docs icons
391+
bun run deployment-config:generate # canonical OAuth registry + catalog → provider-ID facts
390392
bun run integration-catalog:check # registry ↔ committed deployment metadata drift
391393
bun run docs:check # committed docs ↔ what the generator renders today
394+
bun run deployment-config:check # OAuth registry/catalog ↔ provider-ID fact drift
392395
```
393396

394397
- **`tool-metadata:generate`** — required whenever a tool's `outputs`, `params`, or descriptions change. CI enforces this with `bun run tool-metadata:check`, which fails with *"Generated tool metadata is stale"*. This is the easiest gate to miss, because nothing in the tool file hints that a generated artifact mirrors it.
395-
- **`generate-docs`** — required whenever block metadata changes (`bgColor`, `name`, `description`, operations, outputs). Regenerates the integration `.mdx`, `integrations.json`, and the docs copy of `components/icons.tsx`.
398+
- **`generate-docs`** — required whenever block metadata changes (`bgColor`, `name`, `description`, operations, outputs). Regenerates the integration `.mdx`, `packages/deployment-config/src/integrations.json`, and the docs copy of `components/icons.tsx`.
399+
- **`deployment-config:generate`** — required for OAuth or service-account changes. Regenerates provider-ID facts from the canonical OAuth registry and integration catalog; special deployment requirements remain handwritten policy.
396400
- **`integration-catalog:check`** — loads the executable block registry, derives visible integration
397401
deployment fields, and compares them with the committed catalog. It catches missing/unexpected
398402
entries and stale auth/service IDs without loading the executable registry in client code.
@@ -419,9 +423,10 @@ After fixing, confirm:
419423
4. Derived artifacts regenerated and their diffs reviewed (see above)
420424
5. `bun run integration-catalog:check` passes
421425
6. `bun run docs:check` passes
422-
7. For OAuth or service-account changes, `bun test apps/sim/lib/integrations/availability.server.test.ts` passes
423-
8. Re-read all modified files to verify fixes are correct
424-
9. Any remaining unknown response schemas were explicitly reported to the user instead of guessed
426+
7. For OAuth or service-account changes, `bun run deployment-config:check` passes
427+
8. For OAuth or service-account changes, `bun run --cwd apps/sim test lib/integrations/availability.server.test.ts` passes
428+
9. Re-read all modified files to verify fixes are correct
429+
10. Any remaining unknown response schemas were explicitly reported to the user instead of guessed
425430

426431
## Checklist Summary
427432

@@ -437,7 +442,7 @@ After fixing, confirm:
437442
- [ ] Validated scope descriptions exist in `SCOPE_DESCRIPTIONS` within `lib/oauth/utils.ts` for all scopes
438443
- [ ] Validated OAuth `serviceId` resolves to the intended `OAUTH_CLIENT_CAPABILITIES` entry and all capability fields exist in the env schema
439444
- [ ] Validated service-account projection and deployment requirement against the canonical OAuth service config
440-
- [ ] Regenerated `integrations.json` when block metadata changed and ran `bun run integration-catalog:check`
445+
- [ ] Regenerated deployment config when block/OAuth metadata changed and ran both catalog checks
441446
- [ ] Validated pagination consistency across tools and block
442447
- [ ] Validated memory load safety using `.agents/skills/memory-load-check/SKILL.md` when tools list/search/download/import/export/batch data
443448
- [ ] Validated error handling (error checks, meaningful messages)

0 commit comments

Comments
 (0)