diff --git a/packages/codemode/README.md b/packages/codemode/README.md index 0be6d769c7cb..afd178a36cff 100644 --- a/packages/codemode/README.md +++ b/packages/codemode/README.md @@ -179,14 +179,14 @@ Supported bearer, basic, header, and query authentication follows OpenAPI `secur ## Discovery -The agent-tool instructions use a budgeted catalog. Every tool namespace is always listed with its tool count regardless of budget, and as many complete tool signatures (each with a one-line description) as fit an estimated-token budget are inlined. Selection is round-robin across namespaces for fairness: in each round (namespaces alphabetical), every namespace still holding un-inlined tools attempts to place its next-cheapest signature line against the shared budget, and a namespace whose next line does not fit drops out while the others keep going - so every namespace gets some representation before any namespace gets everything. The instructions state exactly how comprehensive the list is, both overall (`COMPLETE list` vs `PARTIAL - N of M shown`) and per namespace (`(3 tools)`, `(3 tools, 1 shown)`, `(3 tools, none shown)`). +The agent-tool instructions use a budgeted catalog. Every tool namespace is always listed with its tool count regardless of budget, and as many complete, JSDoc-annotated tool signatures (each with a one-line description) as fit an estimated-token budget are inlined. Schema field descriptions and tags are part of each signature's measured cost. Selection is round-robin across namespaces for fairness: in each round (namespaces alphabetical), every namespace still holding un-inlined tools attempts to place its next-cheapest signature against the shared budget, and a namespace whose next signature does not fit drops out while the others keep going - so every namespace gets some representation before any namespace gets everything. The instructions state exactly how comprehensive the list is, both overall (`COMPLETE list` vs `PARTIAL - N of M shown`) and per namespace (`(3 tools)`, `(3 tools, 1 shown)`, `(3 tools, none shown)`). -The default budget is 2,000 estimated tokens (characters / 4, the same heuristic OpenCode uses). Override it when constructing a runtime: +The catalog-entry budget defaults to 2,000 estimated tokens (characters / 4, the same heuristic OpenCode uses). It applies only to full tool entries shown in the catalog; fixed instructions and namespace summaries are not counted. Override it when constructing a runtime: ```ts const runtime = CodeMode.make({ tools, - discovery: { maxInlineCatalogTokens: 6_000 }, + discovery: { catalogBudget: 6_000 }, }) ``` @@ -199,30 +199,37 @@ const matches = await tools.$codemode.search({ query: "order status", namespace: "orders", // optional: scope to one top-level namespace limit: 10, + offset: 0, }) ``` -`search` performs deterministic, additive field-weighted matching. The query is tokenized (camelCase boundaries split; every non-alphanumeric character is a separator; empties and `*` are dropped), and each term scores every tool: exact path or path-segment match (20), path substring (8), description substring (4), and searchable-text substring (2). Each term also carries naive singular variants (trailing `s`/`es` stripped), and a field check passes when the term or any variant matches - so a plural query term (`issues`) still finds a tool whose text only says `issue`, without changing the weights. The searchable text also includes the input schema's property names and their description strings, so a query naming a parameter finds its tool, and substring matching means partial words match. Scores sum across terms; matches are sorted by score (ties broken alphabetically by path) and capped at `limit` results (default 10). +`search` performs deterministic, additive field-weighted matching. The query is tokenized (camelCase boundaries split; every non-alphanumeric character is a separator; empties and `*` are dropped), and each term scores every tool: exact path or path-segment match (20), path substring (8), description substring (4), and searchable-text substring (2). Each term also carries naive singular variants (trailing `s`/`es` stripped), and a field check passes when the term or any variant matches - so a plural query term (`issues`) still finds a tool whose text only says `issue`, without changing the weights. The searchable text also includes the input schema's property names and their description strings, so a query naming a parameter finds its tool, and substring matching means partial words match. Scores sum across terms; matches are sorted by score (ties broken alphabetically by path), then sliced from the zero-based `offset` (default 0) to the configured `limit` (default 10). `remaining` counts matches after the current page. `next` is `{ offset }` when another page exists and `null` on the final page; spread it into the original request to preserve its query, namespace, and limit. -Each result contains the path, description, and generated TypeScript signature, so no second lookup is needed. The result signature is the pretty, JSDoc-annotated multiline form: each described input/output field carries its schema `description` as a `/** ... */` comment, and constraints TypeScript cannot express ride along as tags (`@deprecated`, `@default`, `@format`, `@minItems`, `@maxItems`). The inline catalog in the instructions keeps the compact single-line form. +```ts +const request = { query: "order status", namespace: "orders", limit: 10 } +const page = await tools.$codemode.search(request) +const nextPage = page.next ? await tools.$codemode.search({ ...request, ...page.next }) : undefined +``` + +Each result contains the path, description, and the same generated TypeScript signature used by the inline catalog, so no second lookup is needed. Signatures use the JSDoc-annotated multiline form: each described input/output field carries its schema `description` as a `/** ... */` comment, and constraints TypeScript cannot express ride along as tags (`@deprecated`, `@default`, `@format`, `@minItems`, `@maxItems`). ```ts tools.github.list_issues(input: { /** Repository owner */ - owner: string + owner: string, /** Cursor from the previous response's pageInfo */ - after?: string + after?: string, /** * Results per page * @default 30 */ - perPage?: number + perPage?: number, }): Promise ``` Result paths are rendered as JavaScript expressions rooted at `tools` (`tools.orders.lookup`, or `tools.context7["resolve-library-id"]` for non-identifier segments), so each `path` is directly usable as the call site. An empty query browses the catalog alphabetically by path; combined with `namespace` (`{ query: "", namespace: "orders" }`) it lists everything in that namespace. A query that names one tool path exactly (canonical path, `tools.`-prefixed path, or rendered JavaScript expression) is treated as a lookup and returns that tool alone. -The instructions are structured markdown, ordered so the workflow sits at the top and the catalog at the bottom: a `## Workflow` section with numbered steps (find a tool via search when the catalog is partial, or pick from the inlined list when it is complete; call the exact path as-is; `JSON.parse` string results; return only the needed fields), a `## Rules` section holding only guidance the workflow does not already cover (only listed/search-result tools exist inside `tools`; filter and aggregate collections in code; treat `Promise` results as shapeless until verified; run independent calls through `Promise.all`; enumerate `tools` with `Object.keys`/`for...in`; browse a namespace via search when it is advertised), a short `## Syntax` section that assumes standard JavaScript and names only what is unusual (TypeScript annotations stripped; the data-boundary serialization of Date/Map/Set/RegExp) or missing (classes, generators, `for await...of`, `.then`/`.catch`/`.finally`), and the budgeted `## Available tools` catalog. Example call forms use explicit `.`/`` placeholders - never a real or fabricated tool name. +The instructions are structured markdown, ordered so the workflow sits at the top and the catalog at the bottom: a `## Workflow` section with numbered steps (find a tool via search when the catalog is partial, or pick from the inlined list when it is complete; call the exact path as-is; return only the needed fields), a `## Rules` section holding only guidance the workflow does not already cover (only listed/search-result Code Mode tools and internal runtime tools exist inside `tools`; filter and aggregate collections in code; narrow `Promise` results at runtime; run independent calls through `Promise.all`; enumerate `tools` with `Object.keys`/`for...in`; browse a namespace and paginate search results when search is advertised), a short `## Language` section that identifies the runtime as a restricted JavaScript orchestration language and names its major unavailable capabilities, and the budgeted `## Available tools` catalog. Example call forms use explicit `.`/`` placeholders - never a real or fabricated tool name. A host cannot define its own `$codemode` top-level namespace. @@ -234,7 +241,7 @@ CodeMode executes a deliberately bounded JavaScript subset. It supports: - `if`, conditional expressions, `switch`, `for`, `for...of` (arrays, strings, Maps, Sets), `for...in` (own keys of plain objects, index strings of arrays, and namespace/tool names of `tools` references - anything else is an error suggesting `for...of` or `Object.keys`, rather than real JS's surprising behavior of indices for strings and zero iterations for Maps/Sets), `while`, and `do...while`. - Arrow functions and function declarations with closures, defaults, rest parameters, and destructuring. - Optional chaining, nullish coalescing, templates, spread (arrays, strings, Maps, Sets), and `try`/`catch`. -- Common array, string, number, `Object`, `Math`, and `JSON` operations. Mutating array methods include `push`/`pop`/`shift`/`unshift`/`splice` (removes in place and returns the removed elements)/`fill`/`copyWithin`; array `keys`/`values`/`entries` return **arrays** (matching the Map/Set convention) and work with `for...of` and spread. String methods include `localeCompare` (locale/options arguments ignored), `normalize`, and the `trimLeft`/`trimRight` aliases. `Object.keys` also accepts arrays (index strings, as in JS) and tool references: `Object.keys(tools)` lists the top-level namespaces and `Object.keys(tools.ns)` the names at that node (a callable tool enumerates as `[]`; an unknown path is an `UnknownTool` diagnostic). `Object.values`/`Object.entries` on a tool reference fail with a pointer at `Object.keys(tools)` and `tools.$codemode.search`. +- Common array, string, number, `Object`, `Math`, and `JSON` operations. Mutating array methods include `push`/`pop`/`shift`/`unshift`/`splice` (removes in place and returns the removed elements)/`fill`/`copyWithin`; array `keys`/`values`/`entries` return **arrays** (matching the Map/Set convention) and work with `for...of` and spread. String methods include `localeCompare` (locale/options arguments ignored), `normalize`, and the `trimLeft`/`trimRight` aliases. `Object.keys` also accepts arrays (index strings, as in JS) and tool references: `Object.keys(tools)` lists the top-level namespaces, including `$codemode`, and `Object.keys(tools.ns)` lists the names at that node (a callable tool enumerates as `[]`; an unknown path is an `UnknownTool` diagnostic). `Object.values`/`Object.entries` on a tool reference fail with a pointer at `Object.keys(tools)` and `tools.$codemode.search`. - `Date` - `Date.now()`/`Date.parse()`/`Date.UTC()`, `new Date(...)`, the getter methods, and date arithmetic/comparison via the time value. Dates stringify as ISO (`toString` included, for determinism across host timezones). - Regular expressions - `/literals/` and `new RegExp(...)` with `test`/`exec` (stateful `lastIndex` for `g`), plus string `match`/`matchAll`/`replace`/`replaceAll`/`split`/`search` with patterns. Match results are arrays carrying `index` and named `groups` as own properties (`input` is omitted). Invalid patterns, invalid flags, and missing-`g` calls fail with catchable errors that say what was wrong and how to fix it (escaping hints, the exact `/pattern/g` to write). Patterns run on the host engine, so pathological backtracking is bounded only by the execution timeout. Function replacers are not supported. - `Map` and `Set` - construction from entries/arrays/strings, `get`/`set`/`add`/`has`/`delete`/`clear`/`size`/`forEach`, and `keys`/`values`/`entries` returning **arrays** (not iterators). diff --git a/packages/codemode/codemode.md b/packages/codemode/codemode.md index 732c565b72dc..f42c160cad4e 100644 --- a/packages/codemode/codemode.md +++ b/packages/codemode/codemode.md @@ -65,17 +65,24 @@ From issue #34787 and design discussion. Do not relitigate these casually. ### Discovery / search - **Search only - no separate `describe`.** `tools.$codemode.search({ query?, namespace?, -limit? })` over the final tool tree, owned by this package. -- Search result item shape: `{ path, description, signature }` in an `{ items, total }` - wrapper. The `signature` string embeds the full input/output TypeScript types - in search - results it is the pretty, JSDoc-annotated multiline form (Fix 7), so per-field schema - `description`s and constraints (`@default`, `@format`, `@deprecated`, `@minItems`, - `@maxItems`) ride along as field comments. The original spec's separate `input`/`output` +limit?, offset? })` over the final tool tree, owned by this package. +- Search result item shape: `{ path, description, signature }` in an + `{ items, remaining, next }` + wrapper. The `signature` string embeds the full input/output TypeScript types and uses the + same pretty, JSDoc-annotated multiline form in inline catalogs and search results, so + per-field schema `description`s and constraints (`@default`, `@format`, `@deprecated`, + `@minItems`, `@maxItems`) ride along as field comments. The original spec's separate `input`/`output` raw-schema fields are deliberately NOT added: shapes are already fully expressed in the TypeScript signature and schema annotations now arrive as JSDoc - intent satisfied, letter deviated. Result `path`s render a JavaScript expression rooted at `tools` (for example `tools.github.list_issues` or `tools.context7["resolve-library-id"]`) so each is directly usable as the call site; the internal `ToolDescription.path` stays unprefixed. +- `offset` is zero-based and defaults to 0. `remaining` counts matches after the current page; + `next` is `{ offset }` when another page exists and `null` on the final page. +- Search is an internal `Tool.make` definition backed by Effect input/output schemas. Its + validation, output checking, call observation, and TypeScript signature use the same path as + host-provided schema tools. Host-only catalog preparation keeps internal tools out of their + own search index; only conditional advertisement remains special. - Default limit: **10** (done). Exact-path lookup goes through search too: a query equal to a canonical tool path, `tools.`-prefixed path, or rendered JavaScript expression returns that tool alone (done). @@ -251,7 +258,7 @@ output limit; return a smaller value]`; logs keep leading lines within the remai truncation is now the only result-size mechanism.) - **Search polish**: default limit 12 -> **10** (`defaultSearchLimit`); exact-path lookup - a trimmed query equal to one tool path (optionally `tools.`-prefixed) returns that tool alone - (`total: 1`), bypassing ranking. Tokenization/ranking/shape unchanged. + (`remaining: 0`, `next: null`), bypassing ranking. Tokenization/ranking/shape unchanged. ### Wave 3 - OpenCode MCP adapter (done) @@ -312,10 +319,10 @@ Instructions are now the budgeted-catalog + prompting-guidance form; verified e2 real MCP config. Package still 101 tests / 0 fail; opencode adapter suites still 34 + 16; both packages typecheck clean. -- **Budgeted catalog** (`discoveryPlan` in `tool-runtime.ts`): the all-or-nothing +- **Budgeted catalog** (`prepare` in `tool-runtime.ts`): the all-or-nothing inline/search modes are gone - `DiscoveryMode` deleted, `CodeMode.DiscoveryOptions` is just `{ maxInlineCatalogBytes? }` (default 16,000 UTF-8 bytes; later converted to - `maxInlineCatalogTokens`, default 4,000 estimated tokens - see Post-wave fixes). Port of + `catalogBudget`, default 4,000 estimated tokens - see Post-wave fixes). Port of the old opencode `describe()` `PREVIEW_BUDGET` algorithm, adapted to `ToolDescription`: every namespace is ALWAYS listed with its tool count; full signature lines @@ -439,9 +446,9 @@ adapter needed **no changes**. defeating discovery. Fixes, all in this package: - `ToolRuntime.make` now returns a `keys(path)` capability (`namespaceKeys` in `tool-runtime.ts`) threaded into the `Interpreter` alongside `invoke` - the interpreter - still never holds the host tool tree. `Object.keys(tools)` yields the top-level namespace - names (never `$codemode`, which is virtual - but `Object.keys(tools.$codemode)` yields - `["search"]`), `Object.keys(tools.ns)` the names at that node; a callable tool leaf + still never holds the callable tool tree. `Object.keys(tools)` yields the top-level namespace + names, including the internally registered `$codemode`; `Object.keys(tools.$codemode)` yields + `["search"]`, and `Object.keys(tools.ns)` yields names at that node; a callable tool leaf enumerates as `[]` (like `Object.keys` of a JS function); an unknown path throws an `UnknownTool` diagnostic suggesting `Object.keys(tools)` and `$codemode.search` (matching call-time unknown-tool behavior rather than silently returning `[]`). @@ -459,7 +466,7 @@ adapter needed **no changes**. - `supportedSyntaxMessage`, the instructions loops line, and README "Supported Programs" mention the new surface; tests in `test/enumeration.test.ts` (14, incl. the exact transcript program) plus one adapter-level assertion that `Object.keys(tools)` returns - MCP server names. + MCP server and CodeMode namespace names. - **Search ranking, namespace scoping, prefixed result paths (done).** Motivation: the Wave 4 e2e run showed a model retrying calls because search-result paths @@ -480,7 +487,7 @@ adapter needed **no changes**. An empty query now browses ALPHABETICALLY by path (was declaration order). Kept: `{ path, description, signature }` result items, default limit 10, exact-path instant lookup, input validation errors. - - **Namespace scoping**: `tools.$codemode.search({ query?, namespace?, limit? })` - + - **Namespace scoping**: `tools.$codemode.search({ query?, namespace?, limit?, offset? })` - `namespace` (validated as a string when provided) filters `SearchEntry`s to one top-level namespace before ranking; `{ query: "", namespace: "github" }` lists that namespace alphabetically. `searchSignature` updated. @@ -489,7 +496,7 @@ adapter needed **no changes**. segments), directly usable as the call site. Internal `ToolDescription.path` stays unprefixed; only the search RESULT items are rendered this way. Exact-path queries accept canonical paths and rendered expressions. - - **Instructions** (`discoveryPlan`): an explicit calling-convention line and a browse + - **Instructions** (`prepare`): an explicit calling-convention line and a browse hint on the search advertisement (both since absorbed into the `## Rules` section by the instructions restructure below). - **Tests**: package search/discovery tests updated (prefixed paths, alphabetical browse) @@ -499,28 +506,23 @@ adapter needed **no changes**. - **Instructions restructure: markdown sections, placeholder-only call forms (done).** The flat prose instructions (which mixed a real catalog tool with fabricated result - fields in the worked example) are replaced by structured markdown in `discoveryPlan`, + fields in the worked example) are replaced by structured markdown in `prepare`, ordered so the workflow sits at the top (the least likely part of a long description to be truncated or skimmed away) and the catalog at the bottom (the per-section content - described here was later condensed by Fix 8 - Workflow/Rules deduped, Syntax inverted): - - **Intro** (2 lines): "Write a CodeMode program... Return code only." + "Execute - JavaScript in a confined runtime with access to the tools listed below under - `tools.*`." (the second line drops the tools clause when the tree is empty). - - **`## Workflow`**: numbered steps - find a tool via `tools.$codemode.search` -> read - the `{ path, description, signature }` matches -> call by path -> `typeof res === -"string" ? JSON.parse(res) : res` -> return only the needed fields. When the catalog is - COMPLETE the search/read steps collapse into "Pick a tool from the list under - `## Available tools`" and the steps renumber (4 instead of 5). - - **`## Rules`**: call-by-exact-path; TEXT-is-JSON -> JSON.parse; return small (never raw - payloads); filter/aggregate large collections in code instead of per-item round-trips; + described here was later condensed by Fix 8 and the language-accuracy pass): + - **Intro**: identifies the language as restricted JavaScript for calling tools rather than + a general-purpose runtime. + - **`## Workflow`**: with a partial catalog, return search results from one execution, then + copy a selected path into the next execution. With a complete catalog, pick and call an + inlined signature, then return only the needed fields. + - **`## Rules`**: narrow unknown results at runtime; filter/aggregate large collections in code instead of per-item round-trips; console.log/warn/error/dir/table for intermediates; `Promise.all` parallelism (no .then/.catch - await + try/catch); `Object.keys(tools)`/`for...in` enumeration; browse-one-namespace via search (PARTIAL only); and host-side media handling (files/ images never enter the program; a media-only call yields a small text marker - wording verified against the adapter's `toSandboxResult`/`mediaMarker`). - - **`## Syntax`**: the dense syntax lines unchanged, minus the Promise.all and console - lines (moved into Rules) and the `for (const ns in tools)` fragment (redundant with - the enumeration rule). + - **`## Language`**: a concise positive capability summary plus the major unavailable + runtime capabilities and the data-boundary serialization note. - **`## Available tools`**: the budgeted catalog unchanged, with the COMPLETE/PARTIAL header merged into the section heading (no trailing colon); the search-signature advertisement follows when PARTIAL (its description-reading and browse clauses moved @@ -532,7 +534,7 @@ adapter needed **no changes**. appear anywhere in the instructions. Zero tools keep "No tools are currently available." under minimal sections (intro + Syntax + Available tools). - **Tests**: the package worked-example test replaced by section-structure/placeholder - assertions (section order; JSON.parse + return-small rules present; no + assertions (section order; unknown-result + return-small rules present; no `total_count`/`list_issues`/real-tool example lines; browse hint only when PARTIAL; zero-tool minimal sections) - 156 pass / 0 fail; adapter suites gain the same assertions on the built description (still 35 + 16, green). @@ -542,9 +544,9 @@ budget; namespaces must always be present): - `src/token.ts` added: copy of `@opencode-ai/core/util/token` (`round(chars / 4)`), so the package stays dependency-free; keep in sync if the core heuristic changes. -- `CodeMode.DiscoveryOptions.maxInlineCatalogBytes` -> `maxInlineCatalogTokens` (default 4,000 +- `CodeMode.DiscoveryOptions.maxInlineCatalogBytes` -> `catalogBudget` (default 4,000 estimated tokens ~ the old 16,000 bytes at 4 chars/token - behavior parity, not a size - reduction). `discoveryPlan` charges `estimate(catalogLine(tool))` per line; cheapest-first + reduction). `prepare` charges `estimate(catalogLine(tool))` per line; cheapest-first - stop-on-first-miss unchanged at the time (stop-on-first-miss replaced by round-robin in Fix 8). Namespace stub lines were and remain unbudgeted - every namespace always appears with its tool count, even at budget 0 (asserted in package and @@ -672,10 +674,8 @@ along). All in `tool-runtime.ts`; no interpreter changes. interfaces/type aliases are stripped and TS **enums actually work** (transpileModule compiles them to an IIFE the interpreter runs), hence enums deliberately unmentioned. `supportedSyntaxMessage` (the in-diagnostic text in `codemode.ts`) is untouched. -- **Workflow/Rules deduped**: the call-by-exact-path, JSON.parse-string-results, and - return-small content now lives ONLY in the numbered Workflow steps (with their - compliance-driving justifications inline: "most tools return JSON as a string", "raw - payloads get truncated and waste context"); Rules keeps only bullets adding new +- **Workflow/Rules deduped**: the call-by-exact-path and return-small content now lives ONLY + in the numbered Workflow steps; Rules keeps only bullets adding new content - filter/aggregate collections in code, console.\* intermediates (logs ride back), Promise.all parallelism, Object.keys/for...in enumeration, browse-namespace (PARTIAL only), and the media rule compressed to one line. The no-.then/.catch @@ -690,7 +690,7 @@ along). All in `tool-runtime.ts`; no interpreter changes. (332 -> 176), Syntax 453 -> 188 (419 -> 174); fixed prose total 1,005 -> 610 (927 -> 562), ~ 40% reduction with no behavioral content dropped. Workflow grew slightly because it absorbed the deduped parse/return-small justifications. -- **Round-robin namespace inlining** (`discoveryPlan`): the ported stop-on-first-miss +- **Round-robin namespace inlining** (`prepare`): the ported stop-on-first-miss behavior (alphabetically-late namespaces starved to "none shown" while an early namespace inlines everything) is replaced by round-robin fairness - in each round (namespaces alphabetical), every namespace still holding un-inlined tools attempts to @@ -708,9 +708,9 @@ along). All in `tool-runtime.ts`; no interpreter changes. the four field checks passes when ANY form matches. Weights, exact-path lookup, and namespace scoping untouched. A true plural path match still outranks a singular-only description match (path substring 8 + searchable 2 > description 4 + searchable 2). -- **Tests**: package instruction/structure assertions updated to the new text; new - syntax-section test (leads with "Standard modern JavaScript works", names the - verified not-supported list, keeps the data-boundary note); the budget-exhaustion +- **Tests**: package instruction/structure assertions updated to the new text; the + language-section test rejects full-runtime wording, names major unavailable capabilities, + and keeps the data-boundary note; the budget-exhaustion test rewritten to assert the new fairness (alpha.expensive not fitting must NOT prevent beta.cheap from showing: PARTIAL 2 of 3, `- beta (1 tool)` fully shown); new plural/singular test (query "issues" finds a singular-only tool; ranking still @@ -724,7 +724,7 @@ along). All in `tool-runtime.ts`; no interpreter changes. **Fix 9 - prompting trims per user review of Fix 8** (user reviewed the condensed instructions and directed further cuts): -- Default `maxInlineCatalogTokens` 4,000 -> **2,000** (user wants ~2k tokens of signatures +- Default `catalogBudget` 4,000 -> **2,000** (user wants ~2k tokens of signatures auto-inlined; round-robin fairness from Fix 8 spreads it across all namespaces). - Console rule and files/images rule DROPPED from `## Rules`. Replaced by a single `unknown`-treatment warning: "A result typed `Promise` has no guaranteed @@ -733,9 +733,9 @@ instructions and directed further cuts): return the same data; the prompt stays console-neutral, neither for nor against.) The media-stripping MECHANISM is unchanged and still tested; only the prose about it is gone - the `[N images attached]` marker is self-explanatory in context. -- Kept as-is per user: the JSON.parse workflow step (maps to the original motivating - transcript failure; NOT copied from prior art - see section 5 note), the browse-namespace rule - (undecided), no no-fetch/ambient-authority rule added (proposed, not approved). +- Later revised: unconditional JSON parsing was removed because text results are not + necessarily JSON. The browse-namespace rule remains; the language section now states that + ambient `fetch` is unavailable and external operations go through Code Mode tools. - Explicitly REJECTED for now: auto-parsing JSON-looking text results at the adapter boundary ("could get weird" - type flips, program-sees vs tool-sent divergence). Logged as a next-iteration follow-up below. @@ -1004,11 +1004,19 @@ focused interpreter-surface pass rather than picked off piecemeal. - [x] Sandbox values nested inside logged containers print `[CodeMode reference]` (`console.log({ m: map })`) - could deep-format instead. +### Next iteration: optional search input boundary + +- [ ] `SearchInput` uses Effect's exact `optionalKey`, so an omitted field is accepted but an + explicitly present `undefined` field is rejected. The previous handwritten validator + treated explicit `undefined` as omission. Decide whether search should preserve that + convenience locally or whether all tool arguments should adopt JSON-style undefined + normalization; do not broaden `copyOut` semantics solely to fix search. + ### Next iteration: text-result handling (deliberate follow-up, user-directed) - [ ] Revisit how MCP text results reach the program. Today: `structuredContent` when the - server sends it, else joined text as a plain string (the program JSON.parses it, - guided by a workflow step). Considered and deferred: (a) conservative boundary + server sends it, else joined text as a plain string. Programs narrow unknown results + before use; the prompt no longer recommends unconditional JSON parsing. Considered and deferred: (a) conservative boundary auto-parse (text starting with `{`/`[` that parses cleanly becomes an object) - rejected for now as potentially confusing (type flips; program sees something other than what the tool sent); (b) raw-envelope passthrough with the envelope shape diff --git a/packages/codemode/src/codemode.ts b/packages/codemode/src/codemode.ts index b207efe46a8c..083f62e28b54 100644 --- a/packages/codemode/src/codemode.ts +++ b/packages/codemode/src/codemode.ts @@ -38,12 +38,12 @@ export type ExecutionLimits = { /** Controls how much of the tool catalog is inlined in agent instructions. */ export type DiscoveryOptions = { /** - * Estimated-token budget (chars/4, default 2000) for inlined full tool signatures in agent - * instructions. Signatures that fit are inlined round-robin across namespaces; every - * namespace is always listed with its tool count regardless of budget, and + * Approximate budget, in estimated tokens (chars/4, default 2000), for full tool entries in + * the instruction catalog. Tool entries are selected round-robin across namespaces. Fixed + * instructions and namespace summaries do not count toward the budget, and * `tools.$codemode.search` is always registered. */ - readonly maxInlineCatalogTokens?: number + readonly catalogBudget?: number } type ToolTree = { @@ -3921,8 +3921,8 @@ const executeWithLimits = >( const tools = ToolRuntime.make( (options.tools ?? {}) as HostTools>, limits.maxToolCalls, - hooks, searchIndex, + hooks, ) const logs: Array = [] const logged = () => (logs.length > 0 ? { logs: [...logs] } : {}) @@ -4061,10 +4061,10 @@ export const make = = {}>( const tools = (options.tools ?? {}) as HostTools> ToolRuntime.assertValidTools(tools) const limits = resolveExecutionLimits(options.limits) - const discovery = ToolRuntime.discoveryPlan(tools, options.discovery?.maxInlineCatalogTokens) - const executeProgram = (code: string) => executeWithLimits({ ...options, code }, limits, discovery.searchIndex) - const catalog = discovery.catalog - const instructions = discovery.instructions + const prepared = ToolRuntime.prepare(tools, options.discovery?.catalogBudget) + const executeProgram = (code: string) => executeWithLimits({ ...options, code }, limits, prepared.searchIndex) + const catalog = prepared.catalog + const instructions = prepared.instructions return { catalog: () => catalog, diff --git a/packages/codemode/src/tool-runtime.ts b/packages/codemode/src/tool-runtime.ts index f19e7a9b4d55..29b27cb5c6ca 100644 --- a/packages/codemode/src/tool-runtime.ts +++ b/packages/codemode/src/tool-runtime.ts @@ -1,4 +1,4 @@ -import { Cause, Effect } from "effect" +import { Cause, Effect, Schema } from "effect" import { ToolError, toolError } from "./tool-error.js" import { decodeInput as decodeToolInput, @@ -8,6 +8,7 @@ import { inputTypeScript, isDefinition as isToolDefinition, outputTypeScript, + Tool, type Definition, } from "./tool.js" import { SandboxDate, SandboxMap, SandboxPromise, SandboxRegExp, SandboxSet } from "./values.js" @@ -76,10 +77,26 @@ export type ToolDescription = { export type SafeObject = Record const reservedNamespace = "$codemode" -const defaultMaxInlineCatalogTokens = 2_000 +const defaultCatalogBudget = 2_000 const defaultSearchLimit = 10 -const searchSignature = - "tools.$codemode.search({ query?: string, namespace?: string, limit?: number }): Promise<{ items: Array<{ path: string; description: string; signature: string }>; total: number }>" +const PositiveInt = Schema.Int.check(Schema.isGreaterThan(0)) +const NonNegativeInt = Schema.Int.check(Schema.isGreaterThanOrEqualTo(0)) +const SearchInput = Schema.Struct({ + query: Schema.optionalKey(Schema.String), + namespace: Schema.optionalKey(Schema.String), + limit: Schema.optionalKey(PositiveInt), + offset: Schema.optionalKey(NonNegativeInt), +}) +const SearchItem = Schema.Struct({ + path: Schema.String, + description: Schema.String, + signature: Schema.String, +}) +const SearchOutput = Schema.Struct({ + items: Schema.Array(SearchItem), + remaining: NonNegativeInt, + next: Schema.NullOr(Schema.Struct({ offset: NonNegativeInt })), +}) const toolExpression = (path: string) => "tools" + path @@ -295,15 +312,17 @@ const definitions = ( return entries } +const describeDefinition = (path: string, definition: Definition): ToolDescription => ({ + path, + description: definition.description, + signature: `${toolExpression(path)}(input: ${inputTypeScript(definition, true)}): Promise<${outputTypeScript(definition, true)}>`, +}) + const visibleDefinitions = (tools: HostTools) => definitions(tools).map(({ path, definition }) => ({ path, definition, - description: { - path, - description: definition.description, - signature: `${toolExpression(path)}(input: ${inputTypeScript(definition)}): Promise<${outputTypeScript(definition)}>`, - }, + description: describeDefinition(path, definition), })) export const catalog = (tools: HostTools): ReadonlyArray => @@ -317,11 +336,6 @@ export type DiscoveryPlan = { export type SearchEntry = { readonly description: ToolDescription - /** - * JSDoc-annotated multiline signature shown on search-result items; the compact - * single-line form (inline catalog lines) stays in `description.signature`. - */ - readonly signature: string /** Top-level namespace (first path segment), matched by the search `namespace` option. */ readonly namespace: string /** Lowercased path + description + input property names/descriptions, for substring matching. */ @@ -355,8 +369,76 @@ const termForms = (term: string): Array => { return forms } +const makeSearchTool = (searchIndex: ReadonlyArray) => + Tool.make({ + description: "Search available Code Mode tools", + input: SearchInput, + output: SearchOutput, + run: (request) => + Effect.sync(() => { + const query = request.query ?? "" + const offset = request.offset ?? 0 + const scoped = + request.namespace === undefined + ? searchIndex + : searchIndex.filter((entry) => entry.namespace === request.namespace) + // A query that names one tool path exactly (canonical path or rendered JavaScript + // expression) is a lookup, not a search: return that tool alone. + const trimmed = query.trim() + const pathQuery = trimmed.startsWith("tools.") ? trimmed.slice("tools.".length) : trimmed + const exact = + pathQuery === "" + ? undefined + : scoped.find( + (entry) => + entry.description.path === pathQuery || toolExpression(entry.description.path) === trimmed, + ) + const terms = tokenize(query).map(termForms) + // Additive field-weighted scoring, summed across terms: exact path or path segment + // (20) > path substring (8) > description substring (4) > any searchable text, + // including input parameter names and descriptions (2). + const ranked = + exact !== undefined + ? [exact] + : scoped + .map((entry) => { + const path = entry.description.path.toLowerCase() + const description = entry.description.description.toLowerCase() + const score = terms.reduce( + (total, forms) => + total + + (forms.some((form) => path === form || path.endsWith(`.${form}`)) ? 20 : 0) + + (forms.some((form) => path.includes(form)) ? 8 : 0) + + (forms.some((form) => description.includes(form)) ? 4 : 0) + + (forms.some((form) => entry.searchText.includes(form)) ? 2 : 0), + 0, + ) + return { entry, score } + }) + .filter(({ score }) => terms.length === 0 || score > 0) + .sort( + (left, right) => + right.score - left.score || + left.entry.description.path.localeCompare(right.entry.description.path), + ) + .map(({ entry }) => entry) + const items = ranked.slice(offset, offset + (request.limit ?? defaultSearchLimit)).map(({ description }) => ({ + ...description, + path: toolExpression(description.path), + })) + const remaining = Math.max(0, ranked.length - offset - items.length) + return { + items, + remaining, + next: remaining > 0 ? { offset: offset + items.length } : null, + } + }), + }) + +const searchDescription = describeDefinition(`${reservedNamespace}.search`, makeSearchTool([])) + const catalogLine = (tool: ToolDescription) => { - // Inline catalog lines use only a compact first line; full text stays in search results. + // Keep the tool description concise; the full schema documentation remains in the signature. const line = tool.description.split("\n", 1)[0]!.trim() const description = line.length > 120 ? line.slice(0, 119) + "..." : line return description === "" ? ` - ${tool.signature}` : ` - ${tool.signature} // ${description}` @@ -364,7 +446,6 @@ const catalogLine = (tool: ToolDescription) => { const toSearchEntry = (path: string, definition: Definition, description: ToolDescription): SearchEntry => ({ description, - signature: `${toolExpression(path)}(input: ${inputTypeScript(definition, true)}): Promise<${outputTypeScript(definition, true)}>`, namespace: path.split(".", 1)[0]!, searchText: [ path, @@ -389,7 +470,7 @@ export const assertValidTools = (tools: HostTools): void => { /** * Budgeted catalog: every namespace is always listed with its tool count; full call - * signatures are inlined against the `maxInlineCatalogTokens` budget (estimated tokens, + * signatures are inlined against the `catalogBudget` (estimated tokens, * chars/4) round-robin across namespaces - in each round (namespaces alphabetical), every * namespace still holding un-inlined tools attempts to place its next-cheapest line, and * a namespace whose next line does not fit is done while the others keep going - so every @@ -398,12 +479,12 @@ export const assertValidTools = (tools: HostTools): void => { * namespace. Namespace stub lines are never budgeted: every namespace appears with its * tool count even at budget 0. */ -export const discoveryPlan = ( +export const prepare = ( tools: HostTools, - maxInlineCatalogTokens = defaultMaxInlineCatalogTokens, + catalogBudget = defaultCatalogBudget, ): DiscoveryPlan => { - if (!Number.isSafeInteger(maxInlineCatalogTokens) || maxInlineCatalogTokens < 0) { - throw new RangeError("discovery.maxInlineCatalogTokens must be a non-negative safe integer") + if (!Number.isSafeInteger(catalogBudget) || catalogBudget < 0) { + throw new RangeError("discovery.catalogBudget must be a non-negative safe integer") } const visible = visibleDefinitions(tools) const described = visible.map(({ description }) => description) @@ -438,7 +519,7 @@ export const discoveryPlan = ( for (const selection of active) { const tool = selection.queue[0]! const cost = estimateTokens(catalogLine(tool)) - if (used + cost > maxInlineCatalogTokens) continue + if (used + cost > catalogBudget) continue selection.queue.shift() selection.picked.add(tool) used += cost @@ -459,12 +540,11 @@ export const discoveryPlan = ( // catalog at the bottom. Example call forms use placeholders - never a real or fabricated // tool name - and show both dot and bracket notation so non-identifier names are not normalized. const intro = [ - "Write a CodeMode program to answer the request. Return code only.", empty - ? "Execute JavaScript in a confined runtime." + ? "This is a restricted JavaScript language for calling tools, not a general-purpose runtime." : complete - ? "Execute JavaScript in a confined runtime. Inside this program, `tools` contains only the host-provided tools listed below; surrounding agent tools are not available unless listed here." - : "Execute JavaScript in a confined runtime. Inside this program, `tools` contains only the host-provided tools listed or searchable below; surrounding agent tools are not available unless listed here.", + ? "This is a restricted JavaScript language for calling tools, not a general-purpose runtime. Inside the confined interpreter, `tools` contains the Code Mode tools listed below and internal runtime tools; surrounding agent tools are not available." + : "This is a restricted JavaScript language for calling tools, not a general-purpose runtime. Inside the confined interpreter, `tools` contains the Code Mode tools listed or searchable below and internal runtime tools; surrounding agent tools are not available.", ...(empty ? [] : ["Do not infer or normalize tool names; use only exact signatures shown below or returned by search."]), @@ -481,16 +561,12 @@ export const discoveryPlan = ( ...(complete ? [ "1. Pick a tool from the list under `## Available tools` - each line is the exact call signature; use it as-is rather than guessing segments.", - "2. Call it using the exact signature shown; bracket notation and quotes are part of the path.", - '3. Parse text results: `const data = typeof res === "string" ? JSON.parse(res) : res` - most tools return JSON as a string.', - "4. Return only the fields you need: `return { : data. }` - raw payloads get truncated and waste context.", + "2. Call it using the exact signature shown: `const result = await tools..(input)`; bracket notation and quotes are part of the path.", + "3. Return only the fields you need from structured results; narrow unknown results before reading fields, and avoid returning large raw payloads.", ] : [ - '1. If the exact signature is not listed below, first search: `const { items } = await tools.$codemode.search({ query: "" })`.', - "2. Read the matches: each item is `{ path, description, signature }` - read the description before using an unfamiliar tool.", - "3. Call the result's `path` as-is; bracket notation and quotes are part of the path.", - '4. Parse text results: `const data = typeof res === "string" ? JSON.parse(res) : res` - most tools return JSON as a string.', - "5. Return only the fields you need: `return { : data. }` - raw payloads get truncated and waste context.", + '1. If needed, discover tools: `return await tools.$codemode.search({ query: "" })`.', + "2. In the next execution, copy a returned path exactly, call it, and return only the needed fields.", ]), ] @@ -501,24 +577,26 @@ export const discoveryPlan = ( "## Rules", "", complete - ? "- Only tools listed here are available inside `tools`; tools from the surrounding agent/runtime are not implicitly exposed." - : "- Only tools listed here or returned by `tools.$codemode.search` are available inside `tools`; tools from the surrounding agent/runtime are not implicitly exposed.", + ? "- Only Code Mode tools listed here and internal runtime tools are available; surrounding agent tools are not implicitly exposed." + : "- Only Code Mode tools listed here or returned by `tools.$codemode.search` and internal runtime tools are available; surrounding agent tools are not implicitly exposed.", "- Filter, aggregate, and transform collections in code - never return them raw or call a tool per item across messages.", - "- A result typed `Promise` has no guaranteed shape - verify what actually came back before relying on its fields.", + "- A result typed `Promise` may be structured data or text. Before reading fields, check that it is a non-null object and not an array; otherwise handle the returned text or primitive directly.", '- Run independent calls in parallel: `await Promise.all(items.map((item) => tools..(item)))`, or use `tools.["tool-name"](item)` when the listed signature uses bracket notation.', "- `Object.keys(tools)` lists namespaces; `Object.keys(tools.)` lists its tools; `for...in` works on both.", ...(complete ? [] - : ['- Browse one namespace: `await tools.$codemode.search({ query: "", namespace: "" })`.']), + : [ + '- Browse one namespace: `await tools.$codemode.search({ query: "", namespace: "" })`.', + "- If search returns `next`, repeat the same search with `offset: next.offset`.", + ]), ] - const syntax = [ + const language = [ "", - "## Syntax", + "## Language", "", - "Standard modern JavaScript works: functions/closures, destructuring, template literals, loops, try/catch, spread, optional chaining, the usual Array/String/Object/Math/JSON methods, plus Date, RegExp, Map, Set, and Promise.all/allSettled/race/resolve/reject.", - "TypeScript type annotations are allowed and stripped before execution (decorators are not supported).", - "Not supported (each fails with a message naming the alternative): classes, generators, for await...of, .then/.catch/.finally (use await with try/catch).", + "Use common JavaScript data operations, functions, control flow, selected standard-library methods, and awaited tool calls.", + "Modules/imports, classes, generators, timers, fetch, eval, prototype access, arbitrary methods, and promise chaining are unavailable. Use Code Mode tools for external operations. Use await with try/catch.", "Dates serialize to ISO strings at data boundaries; Map/Set/RegExp serialize to `{}`.", ] @@ -547,11 +625,11 @@ export const discoveryPlan = ( for (const tool of group) if (picked.has(tool)) toolSection.push(catalogLine(tool)) } if (!complete) { - toolSection.push("", "Search returns complete callable signatures:", `- ${searchSignature}`) + toolSection.push("", "Search returns complete callable signatures:", `- ${searchDescription.signature}`) } } - const lines = [...intro, ...workflow, ...rules, ...syntax, ...toolSection] + const lines = [...intro, ...workflow, ...rules, ...language, ...toolSection] return { catalog: described, instructions: lines.join("\n"), @@ -560,7 +638,7 @@ export const discoveryPlan = ( } /** - * The enumerable names at one node of the host tool tree - namespace names at the root, + * The enumerable names at one node of the callable tool tree - namespace names at the root, * tool/namespace names below - powering `Object.keys(tools)` and `for...in` over tool * references. A callable tool is a leaf and enumerates as `[]` (like `Object.keys` of a * function in JS). An unknown path is an `UnknownTool` error pointing at the working @@ -569,11 +647,7 @@ export const discoveryPlan = ( const namespaceKeys = ( tools: HostTools, path: ReadonlyArray, - searchEnabled: boolean, ): ReadonlyArray => { - // The reserved discovery namespace is virtual (never present in the host tree); enumerate - // it explicitly so `Object.keys(tools.$codemode)` matches the callable surface. - if (searchEnabled && path.length === 1 && path[0] === reservedNamespace) return ["search"] let value: HostTool | Definition | HostTools = tools for (const segment of path) { if ( @@ -585,11 +659,9 @@ const namespaceKeys = ( throw new ToolRuntimeError( "UnknownTool", `Unknown tool namespace '${path.join(".")}'.`, - searchEnabled - ? [ - "Object.keys(tools) lists the available namespaces; tools.$codemode.search({ query }) finds described tools.", - ] - : ["Object.keys(tools) lists the available namespaces."], + [ + "Object.keys(tools) lists the available namespaces; tools.$codemode.search({ query }) finds described tools.", + ], ) } value = value[segment] as HostTool | Definition | HostTools @@ -601,7 +673,6 @@ const namespaceKeys = ( const resolve = ( tools: HostTools, path: ReadonlyArray, - searchEnabled: boolean, ): HostTool | Definition => { let value: HostTool | Definition | HostTools = tools @@ -615,7 +686,7 @@ const resolve = ( throw new ToolRuntimeError( "UnknownTool", `Unknown tool '${path.join(".")}'.`, - searchEnabled ? ["Use tools.$codemode.search({ query }) to find available described tools."] : [], + ["Use tools.$codemode.search({ query }) to find available described tools."], ) } value = value[segment] as HostTool | Definition | HostTools @@ -632,7 +703,7 @@ export type ToolRuntime = { readonly root: ToolReference readonly calls: Array readonly invoke: (path: ReadonlyArray, args: Array) => Effect.Effect - /** Enumerable namespace/tool names at one node of the host tool tree; see `namespaceKeys`. */ + /** Enumerable namespace/tool names at one node of the callable tool tree; see `namespaceKeys`. */ readonly keys: (path: ReadonlyArray) => ReadonlyArray } @@ -640,11 +711,14 @@ export const make = ( tools: HostTools, /** Undefined means unlimited tool calls. */ maxToolCalls: number | undefined, + searchIndex: ReadonlyArray, hooks?: ToolCallHooks, - searchIndex?: ReadonlyArray, ): ToolRuntime => { const calls: Array = [] - const searchEnabled = searchIndex !== undefined + const callableTools = { + ...tools, + [reservedNamespace]: { search: makeSearchTool(searchIndex) }, + } // Wraps the settling portion of a tool call so onToolCallEnd observes success and failure // symmetrically. Interruption (e.g. the execution timeout) fires neither outcome. @@ -683,7 +757,7 @@ export const make = ( return { root: new ToolReference([]), calls, - keys: (path) => namespaceKeys(tools, path, searchEnabled), + keys: (path) => namespaceKeys(callableTools, path), invoke: (path, args) => Effect.gen(function* () { const name = path.join(".") @@ -694,107 +768,7 @@ export const make = ( recordCall(call) return calls.length - 1 }).pipe(Effect.tap((index) => hooks?.onToolCallStart?.({ index, name, input }) ?? Effect.void)) - if (name === "$codemode.search") { - if (!searchEnabled) throw new ToolRuntimeError("UnknownTool", `Unknown tool '${name}'.`) - const input = externalArgs[0] - if (externalArgs.length !== 1 || input === null || typeof input !== "object" || Array.isArray(input)) { - throw new ToolRuntimeError( - "InvalidToolInput", - "tools.$codemode.search expects { query?: string; namespace?: string; limit?: number }.", - ) - } - const request = input as { query?: unknown; namespace?: unknown; limit?: unknown } - if (request.query !== undefined && typeof request.query !== "string") { - throw new ToolRuntimeError( - "InvalidToolInput", - "tools.$codemode.search query must be a string when provided.", - ) - } - if (request.namespace !== undefined && typeof request.namespace !== "string") { - throw new ToolRuntimeError( - "InvalidToolInput", - "tools.$codemode.search namespace must be a string when provided.", - ) - } - if ( - request.limit !== undefined && - (typeof request.limit !== "number" || !Number.isSafeInteger(request.limit) || request.limit <= 0) - ) { - throw new ToolRuntimeError( - "InvalidToolInput", - "tools.$codemode.search limit must be a positive safe integer when provided.", - ) - } - const query = typeof request.query === "string" ? request.query : "" - const namespace = typeof request.namespace === "string" ? request.namespace : undefined - const index = yield* recordAndObserve(request) - return yield* observeEnd( - Effect.try({ - try: () => { - const limit = typeof request.limit === "number" ? request.limit : defaultSearchLimit - const scoped = - namespace === undefined ? searchIndex : searchIndex.filter((entry) => entry.namespace === namespace) - // A query that names one tool path exactly (canonical path or rendered - // JavaScript expression) is a lookup, not a search: return that tool alone. - const trimmed = query.trim() - const pathQuery = trimmed.startsWith("tools.") ? trimmed.slice("tools.".length) : trimmed - const exact = - pathQuery === "" - ? undefined - : scoped.find( - (entry) => - entry.description.path === pathQuery || toolExpression(entry.description.path) === trimmed, - ) - const terms = tokenize(query).map(termForms) - // Additive field-weighted scoring, summed across terms: exact path or path - // segment (20) > path substring (8) > description substring (4) > any - // searchable text, incl. input parameter names/descriptions (2). Each term - // matches a field when any of its forms (the term or a singular variant) - // does. An empty query browses everything, alphabetical by path. - const ranked = - exact !== undefined - ? [exact] - : scoped - .map((entry) => { - const path = entry.description.path.toLowerCase() - const description = entry.description.description.toLowerCase() - const score = terms.reduce( - (total, forms) => - total + - (forms.some((form) => path === form || path.endsWith(`.${form}`)) ? 20 : 0) + - (forms.some((form) => path.includes(form)) ? 8 : 0) + - (forms.some((form) => description.includes(form)) ? 4 : 0) + - (forms.some((form) => entry.searchText.includes(form)) ? 2 : 0), - 0, - ) - return { entry, score } - }) - .filter(({ score }) => terms.length === 0 || score > 0) - .sort( - (left, right) => - right.score - left.score || - left.entry.description.path.localeCompare(right.entry.description.path), - ) - .map(({ entry }) => entry) - // Result paths are rendered as JavaScript expressions so each `path` is - // directly usable as the call site (`await tools.github.list({ ... })` or - // `await tools.ns["dashed-name"]({ ... })`). The signature is the pretty, - // JSDoc-annotated form (schema descriptions and constraints ride along as - // field comments). - const items = ranked.slice(0, limit).map(({ description, signature }) => ({ - ...description, - path: toolExpression(description.path), - signature, - })) - return copyIn({ items, total: ranked.length }, "Result from tool '$codemode.search'") - }, - catch: (cause) => cause, - }), - { index, name, input: request }, - ) - } - - const tool = resolve(tools, path, searchEnabled) + const tool = resolve(callableTools, path) let describedInput: unknown if (isDefinition(tool)) { if (externalArgs.length !== 1) diff --git a/packages/codemode/src/tool.ts b/packages/codemode/src/tool.ts index e89f3d39d6c5..4ed2ecbcf29e 100644 --- a/packages/codemode/src/tool.ts +++ b/packages/codemode/src/tool.ts @@ -244,9 +244,9 @@ const renderSchema = ( if (properties.length === 0 && indexType === undefined) return "{}" const pad = " ".repeat(depth + 1) const lines = properties.map( - (entry) => `${jsdoc(entry[1].description, docTags(entry[1]), pad)}${pad}${field(entry)}`, + (entry) => `${jsdoc(entry[1].description, docTags(entry[1]), pad)}${pad}${field(entry)},`, ) - if (indexType !== undefined) lines.push(`${pad}[key: string]: ${indexType}`) + if (indexType !== undefined) lines.push(`${pad}[key: string]: ${indexType},`) return `{\n${lines.join("\n")}\n${" ".repeat(depth)}}` } return "unknown" diff --git a/packages/codemode/test/codemode.test.ts b/packages/codemode/test/codemode.test.ts index 087678c778cb..168dc93d5e26 100644 --- a/packages/codemode/test/codemode.test.ts +++ b/packages/codemode/test/codemode.test.ts @@ -426,7 +426,7 @@ describe("CodeMode schema flexibility", () => { { path: "adapter.call", description: "Call an adapter-described tool", - signature: "tools.adapter.call(input: { id: string; count?: number }): Promise", + signature: "tools.adapter.call(input: {\n id: string,\n count?: number,\n}): Promise", }, ]) @@ -459,7 +459,7 @@ describe("CodeMode schema flexibility", () => { { path: "users.lookup", description: "Look up a user", - signature: "tools.users.lookup(input: { login: string }): Promise<{ login: string; id: number }>", + signature: "tools.users.lookup(input: {\n login: string,\n}): Promise<{\n login: string,\n id: number,\n}>", }, ]) @@ -475,7 +475,7 @@ describe("CodeMode schema flexibility", () => { run: () => Effect.succeed("pong"), }) const runtime = CodeMode.make({ tools: { net: { ping } } }) - expect(runtime.catalog()[0]?.signature).toBe("tools.net.ping(input: { host: string }): Promise") + expect(runtime.catalog()[0]?.signature).toBe("tools.net.ping(input: {\n host: string,\n}): Promise") const result = await Effect.runPromise(runtime.execute(`return await tools.net.ping({ host: "example.test" })`)) expect(result.ok).toBe(true) @@ -512,19 +512,19 @@ describe("CodeMode public contract", () => { { path: "orders.lookup", description: "Look up an order by ID", - signature: "tools.orders.lookup(input: { id: string }): Promise<{ id: string; status: string }>", + signature: "tools.orders.lookup(input: {\n id: string,\n}): Promise<{\n id: string,\n status: string,\n}>", }, ]) expect(runtime.instructions()).toContain("Available tools (COMPLETE list") expect(runtime.instructions()).toContain("- orders (1 tool)") expect(runtime.instructions()).toContain( - " - tools.orders.lookup(input: { id: string }): Promise<{ id: string; status: string }> // Look up an order by ID", + " - tools.orders.lookup(input: {\n id: string,\n}): Promise<{\n id: string,\n status: string,\n}> // Look up an order by ID", ) // A fully inlined catalog does not advertise search in the instructions... expect(runtime.instructions()).not.toMatch(/\$codemode/) - // ...but the search tool stays registered, so a speculative call still works. Search - // results carry the pretty multiline signature; the inline catalog stays compact. + // ...but the search tool stays registered, so a speculative call still works with the + // same signature as the inline catalog. const result = await Effect.runPromise(runtime.execute(`return await tools.$codemode.search({ query: "order" })`)) expect(result.ok).toBe(true) if (result.ok) { @@ -533,10 +533,11 @@ describe("CodeMode public contract", () => { { path: "tools.orders.lookup", description: "Look up an order by ID", - signature: "tools.orders.lookup(input: {\n id: string\n}): Promise<{\n id: string\n status: string\n}>", + signature: "tools.orders.lookup(input: {\n id: string,\n}): Promise<{\n id: string,\n status: string,\n}>", }, ], - total: 1, + remaining: 0, + next: null, }) } }) @@ -554,11 +555,11 @@ describe("CodeMode public contract", () => { { path: "context7.resolve-library-id", description: "Resolve a library ID", - signature: 'tools.context7["resolve-library-id"](input: { libraryName: string }): Promise', + signature: 'tools.context7["resolve-library-id"](input: {\n libraryName: string,\n}): Promise', }, ]) expect(runtime.instructions()).toContain( - 'tools.context7["resolve-library-id"](input: { libraryName: string }): Promise', + 'tools.context7["resolve-library-id"](input: {\n libraryName: string,\n}): Promise', ) const search = await Effect.runPromise( @@ -571,10 +572,11 @@ describe("CodeMode public contract", () => { { path: 'tools.context7["resolve-library-id"]', description: "Resolve a library ID", - signature: 'tools.context7["resolve-library-id"](input: {\n libraryName: string\n}): Promise', + signature: 'tools.context7["resolve-library-id"](input: {\n libraryName: string,\n}): Promise', }, ], - total: 1, + remaining: 0, + next: null, }) } @@ -588,7 +590,7 @@ describe("CodeMode public contract", () => { runtime.execute(`return await tools.$codemode.search({ query: 'tools.context7["resolve-library-id"]' })`), ) expect(exact.ok).toBe(true) - if (exact.ok) expect((exact.value as { total: number }).total).toBe(1) + if (exact.ok) expect(exact.value).toMatchObject({ remaining: 0, next: null }) }) test("instructions use markdown sections with placeholder-only call forms", () => { @@ -597,23 +599,26 @@ describe("CodeMode public contract", () => { // Sections in order: workflow at the top, catalog at the bottom. expect(instructions).toContain("## Workflow") expect(instructions).toContain("## Rules") - expect(instructions).toContain("## Syntax") + expect(instructions).toContain("## Language") expect(instructions.indexOf("## Workflow")).toBeLessThan(instructions.indexOf("## Rules")) - expect(instructions.indexOf("## Rules")).toBeLessThan(instructions.indexOf("## Syntax")) - expect(instructions.indexOf("## Syntax")).toBeLessThan(instructions.indexOf("\n## Available tools (COMPLETE list")) - // The workflow carries the result-shape guidance; Rules only add content beyond it. - expect(instructions).toContain( - '`const data = typeof res === "string" ? JSON.parse(res) : res` - most tools return JSON as a string', + expect(instructions.indexOf("## Rules")).toBeLessThan(instructions.indexOf("## Language")) + expect(instructions.indexOf("## Language")).toBeLessThan( + instructions.indexOf("\n## Available tools (COMPLETE list"), ) + expect(instructions).not.toContain("JSON.parse(res)") expect(instructions).toContain("Return only the fields you need") - expect(instructions).toContain("raw payloads get truncated and waste context") + expect(instructions).toContain("avoid returning large raw payloads") expect(instructions).toContain("Do not infer or normalize tool names") expect(instructions).toContain("bracket notation and quotes are part of the path") - expect(instructions).toContain("surrounding agent tools are not available unless listed here") - expect(instructions).toContain("Only tools listed here are available inside `tools`") + expect(instructions).toContain("surrounding agent tools are not available") + expect(instructions).toContain("Only Code Mode tools listed here and internal runtime tools") // Placeholders use generic namespace/tool/field names only - no fabricated real tools // and no real catalog tools cherry-picked into example lines. - expect(instructions).toContain("`return { : data. }`") + expect(instructions).toContain("`const result = await tools..(input)`") + expect(instructions).toContain("Return only the fields you need from structured results") + expect(instructions).toContain("check that it is a non-null object and not an array") + expect(instructions).not.toContain("result.") + expect(instructions).not.toContain("data.") expect(instructions).not.toContain("total_count") expect(instructions).not.toContain("list_issues") expect(instructions).not.toContain("tools.orders.lookup({") @@ -621,36 +626,35 @@ describe("CodeMode public contract", () => { expect(instructions).toContain("1. Pick a tool from the list under `## Available tools`") expect(instructions).not.toContain("Browse one namespace") - const partial = CodeMode.make({ tools, discovery: { maxInlineCatalogTokens: 0 } }).instructions() + const partial = CodeMode.make({ tools, discovery: { catalogBudget: 0 } }).instructions() // PARTIAL: the workflow starts with search (with query-style guidance that is clearly // a query string, never a tool name) and the browse-namespace rule appears. expect(partial).toContain( - '1. If the exact signature is not listed below, first search: `const { items } = await tools.$codemode.search({ query: "" })`.', + '1. If needed, discover tools: `return await tools.$codemode.search({ query: "" })`.', ) + expect(partial).toContain("In the next execution, copy a returned path exactly") expect(partial).toContain( - "Only tools listed here or returned by `tools.$codemode.search` are available inside `tools`", + "Only Code Mode tools listed here or returned by `tools.$codemode.search` and internal runtime tools", ) expect(partial).toContain( '- Browse one namespace: `await tools.$codemode.search({ query: "", namespace: "" })`.', ) + expect(partial).toContain("repeat the same search with `offset: next.offset`") + expect(partial).toContain(" limit?: number,\n offset?: number,") expect(partial).not.toContain("total_count") expect(partial).not.toContain("tools.orders.lookup({") }) - test("the syntax section names what is unusual or missing, not an allowlist", () => { + test("the language section describes the restricted runtime without overclaiming", () => { const instructions = CodeMode.make({ tools }).instructions() - // Models already know JavaScript; the section leads with that. - expect(instructions).toContain("Standard modern JavaScript works") - expect(instructions).toContain("TypeScript type annotations are allowed and stripped before execution") - // The not-supported list is derived from (and verified against) the interpreter. - expect(instructions).toContain("Not supported") - for (const missing of ["classes", "generators", "for await...of", ".then/.catch/.finally"]) { + expect(instructions).toContain("restricted JavaScript language for calling tools") + expect(instructions).toContain("not a general-purpose runtime") + expect(instructions).not.toContain("Standard modern JavaScript works") + expect(instructions).not.toContain("TypeScript type annotations") + for (const missing of ["Modules/imports", "classes", "generators", "fetch", "promise chaining"]) { expect(instructions).toContain(missing) } - // Implemented by the DSL-expansion pass, so no longer listed as missing. - expect(instructions).not.toContain("instanceof Error") - expect(instructions).not.toContain("splice") - // The data-boundary note survives. + expect(instructions).toContain("Use Code Mode tools for external operations") expect(instructions).toContain( "Dates serialize to ISO strings at data boundaries; Map/Set/RegExp serialize to `{}`.", ) @@ -660,7 +664,7 @@ describe("CodeMode public contract", () => { const runtime = CodeMode.make({}) const instructions = runtime.instructions() expect(instructions).toContain("No tools are currently available.") - expect(instructions).toContain("## Syntax") + expect(instructions).toContain("## Language") expect(instructions).toContain("## Available tools") expect(instructions).not.toContain("## Workflow") expect(instructions).not.toContain("## Rules") @@ -682,7 +686,7 @@ describe("CodeMode public contract", () => { }) const runtime = CodeMode.make({ tools: { thread: { uploadFile: upload, generateImage: generate }, orders: { lookup } }, - discovery: { maxInlineCatalogTokens: 0 }, + discovery: { catalogBudget: 0 }, }) expect(runtime.instructions()).toContain( "Available tools (PARTIAL - 0 of 3 shown; find the rest with tools.$codemode.search)", @@ -707,15 +711,16 @@ describe("CodeMode public contract", () => { { path: "tools.thread.uploadFile", description: "Upload one readable local file to the current Discord thread", - signature: "tools.thread.uploadFile(input: {\n path: string\n}): Promise<{\n sent: boolean\n}>", + signature: "tools.thread.uploadFile(input: {\n path: string,\n}): Promise<{\n sent: boolean,\n}>", }, { path: "tools.thread.generateImage", description: "Generate an image and upload it to the current Discord thread", - signature: "tools.thread.generateImage(input: {\n prompt: string\n}): Promise<{\n sent: boolean\n}>", + signature: "tools.thread.generateImage(input: {\n prompt: string,\n}): Promise<{\n sent: boolean,\n}>", }, ], - total: 2, + remaining: 0, + next: null, }) expect(result.toolCalls).toStrictEqual([{ name: "$codemode.search" }]) @@ -761,9 +766,14 @@ describe("CodeMode public contract", () => { const browse = await Effect.runPromise(runtime.execute(`return await tools.$codemode.search({})`)) expect(browse.ok).toBe(true) if (browse.ok) { - const value = browse.value as { items: Array<{ path: string }>; total: number } + const value = browse.value as { + items: Array<{ path: string }> + remaining: number + next: { offset: number } | null + } expect(value.items).toHaveLength(10) - expect(value.total).toBe(14) + expect(value.remaining).toBe(4) + expect(value.next).toStrictEqual({ offset: 10 }) } for (const query of ["many.tool13", "tools.many.tool13"]) { @@ -777,10 +787,11 @@ describe("CodeMode public contract", () => { { path: "tools.many.tool13", description: "Numbered tool 13", - signature: "tools.many.tool13(input: {\n id: string\n}): Promise", + signature: "tools.many.tool13(input: {\n id: string,\n}): Promise", }, ], - total: 1, + remaining: 0, + next: null, }) } } @@ -807,8 +818,8 @@ describe("CodeMode public contract", () => { ) expect(browse.ok).toBe(true) if (browse.ok) { - const value = browse.value as { items: Array<{ path: string }>; total: number } - expect(value.total).toBe(2) + const value = browse.value as { items: Array<{ path: string }>; remaining: number } + expect(value.remaining).toBe(0) expect(value.items.map((item) => item.path)).toStrictEqual([ "tools.github.create_issue", "tools.github.list_issues", @@ -821,8 +832,8 @@ describe("CodeMode public contract", () => { ) expect(scoped.ok).toBe(true) if (scoped.ok) { - const value = scoped.value as { items: Array<{ path: string }>; total: number } - expect(value.total).toBe(1) + const value = scoped.value as { items: Array<{ path: string }>; remaining: number } + expect(value.remaining).toBe(0) expect(value.items[0]?.path).toBe("tools.linear.list_issues") } @@ -858,8 +869,8 @@ describe("CodeMode public contract", () => { ) expect(byParameter.ok).toBe(true) if (byParameter.ok) { - const value = byParameter.value as { items: Array<{ path: string }>; total: number } - expect(value.total).toBe(1) + const value = byParameter.value as { items: Array<{ path: string }>; remaining: number } + expect(value.remaining).toBe(0) expect(value.items[0]?.path).toBe("tools.files.upload") } @@ -869,8 +880,8 @@ describe("CodeMode public contract", () => { ) expect(bySubstring.ok).toBe(true) if (bySubstring.ok) { - const value = bySubstring.value as { items: Array<{ path: string }>; total: number } - expect(value.total).toBe(1) + const value = bySubstring.value as { items: Array<{ path: string }>; remaining: number } + expect(value.remaining).toBe(0) expect(value.items[0]?.path).toBe("tools.files.upload") } }) @@ -898,8 +909,8 @@ describe("CodeMode public contract", () => { ) expect(plural.ok).toBe(true) if (plural.ok) { - const value = plural.value as { items: Array<{ path: string }>; total: number } - expect(value.total).toBe(1) + const value = plural.value as { items: Array<{ path: string }>; remaining: number } + expect(value.remaining).toBe(0) expect(value.items[0]?.path).toBe("tools.tracker.fetch_all") } @@ -907,8 +918,8 @@ describe("CodeMode public contract", () => { const ranked = await Effect.runPromise(runtime.execute(`return await tools.$codemode.search({ query: "issues" })`)) expect(ranked.ok).toBe(true) if (ranked.ok) { - const value = ranked.value as { items: Array<{ path: string }>; total: number } - expect(value.total).toBe(2) + const value = ranked.value as { items: Array<{ path: string }>; remaining: number } + expect(value.remaining).toBe(0) expect(value.items.map((item) => item.path)).toStrictEqual([ "tools.github.list_issues", "tools.tracker.fetch_all", @@ -934,13 +945,33 @@ describe("CodeMode public contract", () => { const browse = await Effect.runPromise(runtime.execute(`return await tools.$codemode.search({})`)) expect(browse.ok).toBe(true) if (browse.ok) { - const value = browse.value as { items: Array<{ path: string }>; total: number } + const value = browse.value as { items: Array<{ path: string }>; remaining: number; next: unknown } expect(value.items.map((item) => item.path)).toStrictEqual([ "tools.alpha.aardvark", "tools.alpha.beta", "tools.zeta.last", ]) + expect(value.remaining).toBe(0) + expect(value.next).toBeNull() + } + + const middle = await Effect.runPromise( + runtime.execute(`return await tools.$codemode.search({ limit: 1, offset: 1 })`), + ) + expect(middle.ok).toBe(true) + if (middle.ok) { + expect(middle.value).toMatchObject({ + items: [{ path: "tools.alpha.beta" }], + remaining: 1, + next: { offset: 2 }, + }) } + + const exhausted = await Effect.runPromise( + runtime.execute(`return await tools.$codemode.search({ limit: 1, offset: 3 })`), + ) + expect(exhausted.ok).toBe(true) + if (exhausted.ok) expect(exhausted.value).toStrictEqual({ items: [], remaining: 0, next: null }) }) test("inlines round-robin across namespaces so one expensive namespace cannot starve the rest", () => { @@ -965,7 +996,7 @@ describe("CodeMode public contract", () => { // other namespaces from inlining (beta already got its line in the same round). const runtime = CodeMode.make({ tools: { alpha: { cheap, expensive }, beta: { cheap } }, - discovery: { maxInlineCatalogTokens: 40 }, + discovery: { catalogBudget: 40 }, }) const instructions = runtime.instructions() @@ -973,14 +1004,38 @@ describe("CodeMode public contract", () => { "Available tools (PARTIAL - 2 of 3 shown; find the rest with tools.$codemode.search)", ) expect(instructions).toContain("- alpha (2 tools, 1 shown)") - expect(instructions).toContain(" - tools.alpha.cheap(input: { q: string }): Promise // Cheap") + expect(instructions).toContain(" - tools.alpha.cheap(input: {\n q: string,\n}): Promise // Cheap") expect(instructions).not.toContain("tools.alpha.expensive(") // Fully shown namespaces read cleanly (no "shown" annotation). expect(instructions).toContain("- beta (1 tool)") - expect(instructions).toContain(" - tools.beta.cheap(input: { q: string }): Promise // Cheap") + expect(instructions).toContain(" - tools.beta.cheap(input: {\n q: string,\n}): Promise // Cheap") expect(instructions).toMatch(/\$codemode\.search/) }) + test("charges inline JSDoc against the catalog token budget", () => { + const documented = Tool.make({ + description: "Look up a record", + input: { + type: "object", + properties: { + id: { type: "string", description: "A detailed identifier description. ".repeat(20) }, + }, + required: ["id"], + } as const, + run: () => Effect.succeed("ok"), + }) + const runtime = CodeMode.make({ + tools: { records: { lookup: documented } }, + discovery: { catalogBudget: 40 }, + }) + + expect(runtime.catalog()[0]?.signature).toContain("/** A detailed identifier description.") + expect(runtime.instructions()).toContain( + "Available tools (PARTIAL - 0 of 1 shown; find the rest with tools.$codemode.search)", + ) + expect(runtime.instructions()).not.toContain("tools.records.lookup(input:") + }) + test("decodes tool input and output before exposing either side", async () => { const observed: Array = [] const transformed = Tool.make({ @@ -1031,17 +1086,27 @@ describe("CodeMode public contract", () => { expect(() => CodeMode.execute({ code: "return 1", limits: { maxToolCalls: -1 } })).toThrow(RangeError) expect(() => CodeMode.execute({ code: "return 1", limits: { maxOutputBytes: -1 } })).toThrow(RangeError) - expect(() => CodeMode.make({ tools, discovery: { maxInlineCatalogTokens: -1 } })).toThrow(RangeError) + expect(() => CodeMode.make({ tools, discovery: { catalogBudget: -1 } })).toThrow(RangeError) const result = await Effect.runPromise( CodeMode.make({ tools, - discovery: { maxInlineCatalogTokens: 0 }, + discovery: { catalogBudget: 0 }, }).execute(`return await tools.$codemode.search({ query: "order", limit: 0.5 })`), ) expect(result.ok).toBe(false) if (result.ok) return expect(result.error.kind).toBe("InvalidToolInput") + + for (const offset of [-1, 0.5, Number.MAX_SAFE_INTEGER + 1, "1"]) { + const invalidOffset = await Effect.runPromise( + CodeMode.make({ tools }).execute( + `return await tools.$codemode.search({ query: "order", offset: ${JSON.stringify(offset)} })`, + ), + ) + expect(invalidOffset.ok).toBe(false) + if (!invalidOffset.ok) expect(invalidOffset.error.kind).toBe("InvalidToolInput") + } }) test("enforces the tool-call limit as a diagnostic", async () => { diff --git a/packages/codemode/test/enumeration.test.ts b/packages/codemode/test/enumeration.test.ts index 87c075a79282..f176fe2dc895 100644 --- a/packages/codemode/test/enumeration.test.ts +++ b/packages/codemode/test/enumeration.test.ts @@ -41,7 +41,7 @@ describe("Object.keys over tool references", () => { const namespaces = Object.keys(tools) return { namespaces, count: namespaces.length } `), - ).toEqual({ namespaces: ["github", "memory", "playwright"], count: 3 }) + ).toEqual({ namespaces: ["github", "memory", "playwright", "$codemode"], count: 4 }) }) test("enumerates tool names at a nested namespace", async () => { @@ -52,7 +52,7 @@ describe("Object.keys over tool references", () => { expect(await value(`return Object.keys(tools.github.list_issues)`)).toEqual([]) }) - test("the virtual discovery namespace enumerates its callable surface", async () => { + test("the internal discovery namespace enumerates its callable surface", async () => { expect(await value(`return Object.keys(tools.$codemode)`)).toEqual(["search"]) }) @@ -137,7 +137,7 @@ describe("for...in", () => { ).toBe("only") }) - test("enumerates namespaces and tools from the host tool tree", async () => { + test("enumerates namespaces and tools from the callable tool tree", async () => { expect( await value(` const names = [] @@ -146,7 +146,13 @@ describe("for...in", () => { } return names `), - ).toEqual(["github.list_issues", "github.get_issue", "memory.search", "playwright.navigate"]) + ).toEqual([ + "github.list_issues", + "github.get_issue", + "memory.search", + "playwright.navigate", + "$codemode.search", + ]) }) test("unsupported values fail with a hint at for...of and Object.keys", async () => { diff --git a/packages/codemode/test/signature.test.ts b/packages/codemode/test/signature.test.ts index 4f25645de22c..53a3b766d58a 100644 --- a/packages/codemode/test/signature.test.ts +++ b/packages/codemode/test/signature.test.ts @@ -40,21 +40,21 @@ describe("pretty signature rendering", () => { [ "{", " /** Repository owner */", - " owner: string", + " owner: string,", " /** Cursor from the previous response's pageInfo */", - " after?: string", + " after?: string,", " /**", " * Results per page", " * @default 30", " */", - " perPage?: number", + " perPage?: number,", " /**", " * Filter by labels", " * @minItems 1", " * @maxItems 10", " */", - " labels?: Array", - ' state?: "open" | "closed"', + " labels?: Array,", + ' state?: "open" | "closed",', "}", ].join("\n"), ) @@ -83,7 +83,7 @@ describe("pretty signature rendering", () => { true, ) expect(pretty).toBe( - ["{", " /** Search filter */", " filter?: {", " /** Issue state */", " state?: string", " }", "}"].join( + ["{", " /** Search filter */", " filter?: {", " /** Issue state */", " state?: string,", " },", "}"].join( "\n", ), ) @@ -91,10 +91,10 @@ describe("pretty signature rendering", () => { test("Effect Schema annotations become JSDoc on input and output fields", () => { expect(inputTypeScript(lookupOrder, true)).toBe( - ["{", " /** Order identifier */", " id: string", " verbose?: boolean", "}"].join("\n"), + ["{", " /** Order identifier */", " id: string,", " verbose?: boolean,", "}"].join("\n"), ) expect(outputTypeScript(lookupOrder, true)).toBe( - ["{", " /** Current order status */", " status: string", "}"].join("\n"), + ["{", " /** Current order status */", " status: string,", "}"].join("\n"), ) }) @@ -129,7 +129,7 @@ describe("pretty signature rendering", () => { { type: "object", properties: { size: { type: "number", default: 1n } } }, true, ) - expect(pretty).toBe(["{", " size?: number", "}"].join("\n")) + expect(pretty).toBe(["{", " size?: number,", "}"].join("\n")) }) test("neutralizes */ inside descriptions so nothing closes the comment early", () => { @@ -150,7 +150,7 @@ describe("pretty signature rendering", () => { true, ) expect(pretty).toBe( - ["{", " /**", " * First line", " *", " * Second line", " */", " query?: string", "}"].join("\n"), + ["{", " /**", " * First line", " *", " * Second line", " */", " query?: string,", "}"].join("\n"), ) }) @@ -235,12 +235,12 @@ describe("non-identifier property names render as quoted keys", () => { expect(jsonSchemaToTypeScript(rawSchema, true)).toBe( [ "{", - ' "123"?: number', - ' "foo-bar"?: string', - ' "@type": string', + ' "123"?: number,', + ' "foo-bar"?: string,', + ' "@type": string,', " /** Dotted name */", - ' "x.y"?: number', - " plain?: boolean", + ' "x.y"?: number,', + " plain?: boolean,", "}", ].join("\n"), ) @@ -259,7 +259,7 @@ describe("non-identifier property names render as quoted keys", () => { }) expect(inputTypeScript(tool)).toContain('"foo-bar"?: string') expect(outputTypeScript(tool)).toBe('{ "content-type": string }') - expect(outputTypeScript(tool, true)).toBe(["{", ' "content-type": string', "}"].join("\n")) + expect(outputTypeScript(tool, true)).toBe(["{", ' "content-type": string,', "}"].join("\n")) }) test("Effect Schema structs with non-identifier field names quote too", () => { @@ -269,7 +269,7 @@ describe("non-identifier property names render as quoted keys", () => { run: () => Effect.succeed(null), }) expect(inputTypeScript(tool)).toBe('{ "foo-bar": string; plain?: number }') - expect(inputTypeScript(tool, true)).toBe(["{", ' "foo-bar": string', " plain?: number", "}"].join("\n")) + expect(inputTypeScript(tool, true)).toBe(["{", ' "foo-bar": string,', " plain?: number,", "}"].join("\n")) }) }) @@ -332,7 +332,7 @@ describe("union schemas render every alternative", () => { }) }) -describe("pretty signatures in search results", () => { +describe("JSDoc signatures in catalogs and search results", () => { const runtime = CodeMode.make({ tools: { github: { list_issues: listIssues }, orders: { lookup: lookupOrder } } }) const search = async (query: string) => { @@ -341,7 +341,7 @@ describe("pretty signatures in search results", () => { ) expect(result.ok).toBe(true) if (!result.ok) throw new Error("search failed") - return result.value as { items: Array<{ path: string; signature: string }>; total: number } + return result.value as { items: Array<{ path: string; signature: string }>; remaining: number } } test("a raw JSON Schema (MCP-style) tool's result signature carries field JSDoc and tags", async () => { @@ -351,21 +351,21 @@ describe("pretty signatures in search results", () => { [ "tools.github.list_issues(input: {", " /** Repository owner */", - " owner: string", + " owner: string,", " /** Cursor from the previous response's pageInfo */", - " after?: string", + " after?: string,", " /**", " * Results per page", " * @default 30", " */", - " perPage?: number", + " perPage?: number,", " /**", " * Filter by labels", " * @minItems 1", " * @maxItems 10", " */", - " labels?: Array", - ' state?: "open" | "closed"', + " labels?: Array,", + ' state?: "open" | "closed",', "}): Promise", ].join("\n"), ) @@ -379,26 +379,26 @@ describe("pretty signatures in search results", () => { [ "tools.orders.lookup(input: {", " /** Order identifier */", - " id: string", - " verbose?: boolean", + " id: string,", + " verbose?: boolean,", "}): Promise<{", " /** Current order status */", - " status: string", + " status: string,", "}>", ].join("\n"), ) } }) - test("the inline catalog line for the same tool stays single-line compact", () => { + test("the inline catalog uses the same JSDoc signatures", async () => { const instructions = runtime.instructions() - expect(instructions).toContain( - ' - tools.github.list_issues(input: { owner: string; after?: string; perPage?: number; labels?: Array; state?: "open" | "closed" }): Promise // List issues in a repository', - ) - expect(instructions).toContain( - " - tools.orders.lookup(input: { id: string; verbose?: boolean }): Promise<{ status: string }> // Look up an order", - ) - expect(instructions).not.toContain("/**") + const github = (await search("list issues repository")).items.find( + ({ path }) => path === "tools.github.list_issues", + )! + const orders = (await search("look up order")).items.find(({ path }) => path === "tools.orders.lookup")! + expect(instructions).toContain(` - ${github.signature} // List issues in a repository`) + expect(instructions).toContain(` - ${orders.signature} // Look up an order`) + expect(instructions).toContain("/** Repository owner */") }) }) @@ -421,7 +421,7 @@ describe("non-identifier tool paths", () => { const instructions = runtime.instructions() expect(instructions).toContain( - 'tools.context7["resolve-library-id"](input: { query: string; libraryName: string }): Promise', + 'tools.context7["resolve-library-id"](input: {\n query: string,\n libraryName: string,\n}): Promise', ) expect(instructions).toContain("Do not infer or normalize tool names") expect(instructions).toContain("bracket notation and quotes are part of the path") diff --git a/packages/opencode/src/tool/code-mode.ts b/packages/opencode/src/tool/code-mode.ts index 0f566a49ade0..332d4b43f150 100644 --- a/packages/opencode/src/tool/code-mode.ts +++ b/packages/opencode/src/tool/code-mode.ts @@ -11,18 +11,11 @@ import { Plugin } from "@/plugin" export const CODE_MODE_TOOL = "execute" -const DESCRIPTION = [ - "Execute a JavaScript/TypeScript program that orchestrates the connected MCP tools inside a confined runtime.", - "The full usage guide and the catalog of available tools follow below.", -].join("\n") +const DESCRIPTION = "Run a confined orchestration script with access to connected MCP tools." export const Parameters = Schema.Struct({ code: Schema.String.annotate({ - description: [ - "JavaScript source to execute.", - "Inside CodeMode, `tools` contains only the MCP/CodeMode tools listed in this execute tool's description; top-level opencode tools like bash, read, or lsp are not available unless listed there.", - "Call available tools using the exact signatures shown in this execute tool's description, compose the results, and `return` the final value.", - ].join(" "), + description: "Script body executed by the confined interpreter.", }), }) diff --git a/packages/opencode/test/tool/code-mode-integration.test.ts b/packages/opencode/test/tool/code-mode-integration.test.ts index f36b6c866ef5..671acd896222 100644 --- a/packages/opencode/test/tool/code-mode-integration.test.ts +++ b/packages/opencode/test/tool/code-mode-integration.test.ts @@ -177,8 +177,10 @@ describe("code mode integration (real MCP server)", () => { test("the appended catalog inlines full signatures with real MCP schemas", () => { expect(description).toContain("Available tools (COMPLETE list") expect(description).toContain("- fixtures (4 tools)") - expect(description).toContain("tools.fixtures.add(input: { a: number; b: number }): Promise<{ sum: number }>") - expect(description).toContain("tools.fixtures.get_text(input: { name: string }): Promise") + expect(description).toContain( + "tools.fixtures.add(input: {\n a: number,\n b: number,\n}): Promise<{\n sum: number,\n}>", + ) + expect(description).toContain("tools.fixtures.get_text(input: {\n name: string,\n}): Promise") expect(description).toContain("// Add two numbers and return the structured sum") expect(description).not.toContain("$codemode") expect(description).toContain("## Workflow") diff --git a/packages/opencode/test/tool/code-mode.test.ts b/packages/opencode/test/tool/code-mode.test.ts index a220f14530fe..0a2b0b7fe424 100644 --- a/packages/opencode/test/tool/code-mode.test.ts +++ b/packages/opencode/test/tool/code-mode.test.ts @@ -98,6 +98,13 @@ describe("code mode execute", () => { const decode = Schema.decodeUnknownEffect(Parameters) await expect(Effect.runPromise(decode({ code: "return 1" }))).resolves.toEqual({ code: "return 1" }) await expect(Effect.runPromise(decode({}))).rejects.toThrow() + expect(Schema.toJsonSchemaDocument(Parameters).schema).toMatchObject({ + properties: { + code: { + description: "Script body executed by the confined interpreter.", + }, + }, + }) }) test("groups multi-underscore server names by longest matching prefix", () => { @@ -124,13 +131,13 @@ describe("code mode execute", () => { }, ["weather"], ) - expect(description).toContain("tools.weather.current(input: { city: string }): Promise<{ tempC: number }>") + expect(description).toContain("tools.weather.current(input: {\n city: string,\n}): Promise<{\n tempC: number,\n}>") }) test("the static base description carries no catalog; the registry appends it", async () => { const tool = await build({ github_list_issues: mcpTool("list_issues", () => "") }) expect(tool.id).toBe(CODE_MODE_TOOL) - expect(tool.description).toContain("confined runtime") + expect(tool.description).toBe("Run a confined orchestration script with access to connected MCP tools.") expect(tool.description).not.toContain("Available tools") expect(tool.description).not.toContain("list_issues") }) @@ -150,7 +157,7 @@ describe("code mode execute", () => { expect(description).toContain("- github (2 tools)") expect(description).toContain("- linear (1 tool)") expect(description).toContain( - "tools.github.create_issue(input: { title: string; body?: string }): Promise", + "tools.github.create_issue(input: {\n title: string,\n body?: string,\n}): Promise", ) expect(description).toContain("tools.github.list_issues(") expect(description).toContain("tools.linear.search(") @@ -159,9 +166,8 @@ describe("code mode execute", () => { expect(description).not.toContain("Browse one namespace") expect(description).toContain("## Workflow") expect(description).toContain("1. Pick a tool from the list under `## Available tools`") - expect(description).toContain( - '`const data = typeof res === "string" ? JSON.parse(res) : res` - most tools return JSON as a string', - ) + expect(description).not.toContain("JSON.parse(res)") + expect(description).toContain("check that it is a non-null object and not an array") expect(description).toContain("Return only the fields you need") expect(description).not.toContain("total_count") }) @@ -180,7 +186,7 @@ describe("code mode execute", () => { ), }) expect(description).toContain( - "tools.weather.current(input: { city: string }): Promise<{ tempC: number; summary?: string }>", + "tools.weather.current(input: {\n city: string,\n}): Promise<{\n tempC: number,\n summary?: string,\n}>", ) }) @@ -207,9 +213,16 @@ describe("code mode execute", () => { expect(description).toContain("Available tools (PARTIAL - ") expect(description).toMatch(/- alpha \(150 tools, \d+ shown\)/) expect(description).toContain("- zeta (1 tool)\n") - expect(description).toContain("tools.zeta.only_tool(input: { topic: string }): Promise") + expect(description).toContain( + "tools.zeta.only_tool(input: {\n /** Subject to look up */\n topic: string,\n}): Promise", + ) expect(description).toContain("tools.$codemode.search(") - expect(description).toContain("1. If the exact signature is not listed below, first search:") + expect(description).toContain(" limit?: number,\n offset?: number,") + expect(description).toContain(" remaining: number,\n next: {") + expect(description).toContain(" offset: number,\n } | null,") + expect(description).toContain( + '1. If needed, discover tools: `return await tools.$codemode.search({ query: "" })`.', + ) expect(description).toContain( '- Browse one namespace: `await tools.$codemode.search({ query: "", namespace: "" })`.', ) @@ -219,17 +232,18 @@ describe("code mode execute", () => { const tool = await build(tools, ["alpha", "zeta"]) const out = await Effect.runPromise( - tool.execute({ code: "return await tools.$codemode.search({ query: 'only tool', limit: 3 })" }, ctx), + tool.execute({ code: "return await tools.$codemode.search({ query: 'only tool', limit: 3, offset: 0 })" }, ctx), ) const result = JSON.parse(out.output) expect(result.items.map((i: any) => i.path)).toContain("tools.zeta.only_tool") + expect(result).toMatchObject({ remaining: 0, next: null }) expect(result.items[0].signature).toContain("tools.") const signature = result.items.find((i: any) => i.path === "tools.zeta.only_tool").signature expect(signature).toContain("tools.zeta.only_tool(input: {\n") expect(signature).toContain(" /** Subject to look up */\n topic: string") - expect(description).not.toContain("/**") + expect(description).toContain("/** Subject to look up */") expect(out.metadata.toolCalls).toEqual([ - { tool: "$codemode.search", status: "completed", input: { query: "only tool", limit: 3 } }, + { tool: "$codemode.search", status: "completed", input: { query: "only tool", limit: 3, offset: 0 } }, ]) }) @@ -240,7 +254,7 @@ describe("code mode execute", () => { expect(output.metadata.toolCalls).toEqual([]) }) - test("Object.keys(tools) enumerates the MCP server namespaces", async () => { + test("Object.keys(tools) enumerates the MCP server and CodeMode namespaces", async () => { const tool = await build({ github_list_issues: mcpTool("list_issues", () => ""), linear_search: mcpTool("search", () => ""), @@ -251,7 +265,7 @@ describe("code mode execute", () => { ctx, ), ) - expect(JSON.parse(output.output)).toEqual({ namespaces: ["github", "linear"], count: 2 }) + expect(JSON.parse(output.output)).toEqual({ namespaces: ["github", "linear", "$codemode"], count: 3 }) }) test("calls a namespaced MCP tool and flows its text result back into the program", async () => { diff --git a/packages/opencode/test/tool/registry.test.ts b/packages/opencode/test/tool/registry.test.ts index f3ccd5997c19..c8c5fac59559 100644 --- a/packages/opencode/test/tool/registry.test.ts +++ b/packages/opencode/test/tool/registry.test.ts @@ -132,7 +132,7 @@ describe("tool.registry", () => { expect(ids).toContain("execute") expect(tools.map((tool) => tool.id)).toContain("execute") - expect(execute?.description).toContain("tools.weather.current(input: { city: string })") + expect(execute?.description).toContain("tools.weather.current(input: {\n city: string,\n})") }), )