Skip to content

Support provider-defined tools#1466

Open
rosetta-livekit-bot[bot] wants to merge 2 commits into
mainfrom
moneybag-ripper-yogin
Open

Support provider-defined tools#1466
rosetta-livekit-bot[bot] wants to merge 2 commits into
mainfrom
moneybag-ripper-yogin

Conversation

@rosetta-livekit-bot
Copy link
Copy Markdown
Contributor

Summary

  • Add provider-defined tools to the shared LLM tool context while keeping function-tool execution paths filtered to callable tools.
  • Port provider tool constructors for OpenAI, Google, xAI, and Mistral, and wire provider adapters to include their native tool payloads.
  • Add a changeset for the public tool API additions.

Testing

  • pnpm exec prettier --write ...
  • pnpm --filter @livekit/agents build
  • pnpm --filter @livekit/agents-plugin-mistralai build
  • pnpm --filter @livekit/agents-plugin-openai build (JS bundling succeeds; declaration step fails because optional test workspace deps are missing in this checkout: @livekit/agents-plugins-test, @livekit/agents-plugin-silero)
  • pnpm --filter @livekit/agents-plugin-google build (JS bundling succeeds; declaration step fails because optional test workspace deps / OpenAI declarations are missing)
  • pnpm --filter @livekit/agents-plugin-xai build (JS bundling succeeds; declaration step fails because optional test workspace deps / OpenAI declarations are missing)
  • pnpm --filter @livekit/agents lint (warnings only, existing repo warnings)
  • pnpm --filter @livekit/agents-plugin-openai lint (warnings only)
  • pnpm --filter @livekit/agents-plugin-google lint (warnings only)
  • pnpm --filter @livekit/agents-plugin-xai lint (warnings only)
  • pnpm --filter @livekit/agents-plugin-mistralai lint

@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented May 12, 2026

🦋 Changeset detected

Latest commit: 86b45c0

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 31 packages
Name Type
@livekit/agents Patch
@livekit/agents-plugin-google Patch
@livekit/agents-plugin-mistralai Patch
@livekit/agents-plugin-openai Patch
@livekit/agents-plugin-xai Patch
@livekit/agents-plugin-anam Patch
@livekit/agents-plugin-assemblyai Patch
@livekit/agents-plugin-baseten Patch
@livekit/agents-plugin-bey Patch
@livekit/agents-plugin-cartesia Patch
@livekit/agents-plugin-cerebras Patch
@livekit/agents-plugin-deepgram Patch
@livekit/agents-plugin-elevenlabs Patch
@livekit/agents-plugin-fishaudio Patch
@livekit/agents-plugin-hedra Patch
@livekit/agents-plugin-hume Patch
@livekit/agents-plugin-inworld Patch
@livekit/agents-plugin-lemonslice Patch
@livekit/agents-plugin-liveavatar Patch
@livekit/agents-plugin-livekit Patch
@livekit/agents-plugin-minimax Patch
@livekit/agents-plugin-mistral Patch
@livekit/agents-plugin-neuphonic Patch
@livekit/agents-plugin-phonic Patch
@livekit/agents-plugin-resemble Patch
@livekit/agents-plugin-rime Patch
@livekit/agents-plugin-runway Patch
@livekit/agents-plugin-sarvam Patch
@livekit/agents-plugin-silero Patch
@livekit/agents-plugins-test Patch
@livekit/agents-plugin-trugen Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

Copy link
Copy Markdown
Contributor

@devin-ai-integration devin-ai-integration Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 1 potential issue.

View 6 additional findings in Devin Review.

Open in Devin Review

Comment thread agents/src/llm/utils.ts
Comment on lines 195 to +202
const tool = toolCtx[toolCall.name]!;
if (!isFunctionTool(tool)) {
return FunctionCallOutput.create({
callId: toolCall.callId,
output: `Unknown function tool: ${toolCall.name}`,
isError: true,
});
}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔴 executeToolCall crashes with TypeError when tool name is not found in toolCtx

The new isFunctionTool(tool) guard at line 196 is intended to gracefully handle non-function tools (returning an error FunctionCallOutput), but when toolCall.name doesn't exist in toolCtx, tool is undefined at runtime (the ! at line 195 is just a TS assertion). isFunctionTool(undefined) then throws a TypeError because tool_context.ts:376 accesses tool[FUNCTION_TOOL_SYMBOL] without a null guard—the short-circuit on line 375 only protects tool[TOOL_SYMBOL], not the next line. The graceful error path at lines 197–201 is never reached.

This can be triggered when an LLM hallucinates a tool name. Compare with the safe pattern in agents/src/voice/generation.ts:955-967 which has if (!tool) { ... continue; } before calling isFunctionTool.

Suggested change
const tool = toolCtx[toolCall.name]!;
if (!isFunctionTool(tool)) {
return FunctionCallOutput.create({
callId: toolCall.callId,
output: `Unknown function tool: ${toolCall.name}`,
isError: true,
});
}
const tool = toolCtx[toolCall.name];
if (!tool || !isFunctionTool(tool)) {
return FunctionCallOutput.create({
callId: toolCall.callId,
output: `Unknown function tool: ${toolCall.name}`,
isError: true,
});
}
Open in Devin Review

Was this helpful? React with 👍 or 👎 to provide feedback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants