diff --git a/api-reference/server/services/supported-services.mdx b/api-reference/server/services/supported-services.mdx index e4b6073e..81337ae3 100644 --- a/api-reference/server/services/supported-services.mdx +++ b/api-reference/server/services/supported-services.mdx @@ -7,17 +7,18 @@ description: "AI services integrated with Pipecat and their setup requirements" Transports exchange audio and video streams between the user and bot. -| Service | Setup | -| --------------------------------------------------------------------------------------- | --------------------------------- | -| [DailyTransport](/api-reference/server/services/transport/daily) | `uv add "pipecat-ai[daily]"` | -| [FastAPIWebSocketTransport](/api-reference/server/services/transport/fastapi-websocket) | `uv add "pipecat-ai[websocket]"` | -| [HeyGenTransport](/api-reference/server/services/transport/heygen) | `uv add "pipecat-ai[heygen]"` | -| [LemonSliceTransport](/api-reference/server/services/transport/lemonslice) | `uv add "pipecat-ai[lemonslice]"` | -| [LiveKitTransport](/api-reference/server/services/transport/livekit) | `uv add "pipecat-ai[livekit]"` | -| [SmallWebRTCTransport](/api-reference/server/services/transport/small-webrtc) | `uv add "pipecat-ai[webrtc]"` | -| [TavusTransport](/api-reference/server/services/transport/tavus) | `uv add "pipecat-ai[tavus]"` | -| [WebSocket Transports](/api-reference/server/services/transport/websocket-server) | `uv add "pipecat-ai[websocket]"` | -| [WhatsAppTransport](/api-reference/server/services/transport/whatsapp) | `uv add "pipecat-ai[webrtc]"` | +| Service | Setup | +| --------------------------------------------------------------------------------------- | --------------------------------------------- | +| [DailyTransport](/api-reference/server/services/transport/daily) | `uv add "pipecat-ai[daily]"` | +| [FastAPIWebSocketTransport](/api-reference/server/services/transport/fastapi-websocket) | `uv add "pipecat-ai[websocket]"` | +| [HeyGenTransport](/api-reference/server/services/transport/heygen) | `uv add "pipecat-ai[heygen]"` | +| [LemonSliceTransport](/api-reference/server/services/transport/lemonslice) | `uv add "pipecat-ai[lemonslice]"` | +| [LiveKitTransport](/api-reference/server/services/transport/livekit) | `uv add "pipecat-ai[livekit]"` | +| [SmallWebRTCTransport](/api-reference/server/services/transport/small-webrtc) | `uv add "pipecat-ai[webrtc]"` | +| [TavusTransport](/api-reference/server/services/transport/tavus) | `uv add "pipecat-ai[tavus]"` | +| [VonageVideoConnectorTransport](/api-reference/server/services/transport/vonage) | `uv add "pipecat-ai[vonage-video-connector]"` | +| [WebSocket Transports](/api-reference/server/services/transport/websocket-server) | `uv add "pipecat-ai[websocket]"` | +| [WhatsAppTransport](/api-reference/server/services/transport/whatsapp) | `uv add "pipecat-ai[webrtc]"` | ## Serializers diff --git a/api-reference/server/services/transport/vonage.mdx b/api-reference/server/services/transport/vonage.mdx new file mode 100644 index 00000000..661aa226 --- /dev/null +++ b/api-reference/server/services/transport/vonage.mdx @@ -0,0 +1,413 @@ +--- +title: "VonageVideoConnectorTransport" +description: "WebRTC transport implementation using Vonage Video API for real-time audio/video communication" +--- + +## Overview + +`VonageVideoConnectorTransport` provides real-time WebRTC audio and video communication using the Vonage Video API. It handles bidirectional media streams, participant management, and session lifecycle events, enabling you to build AI-powered applications that participate in Vonage Video sessions. + + + The Vonage Video Connector library runs on Linux AMD64 and ARM64 platforms + only (Python 3.13+). + + + + + Pipecat's API methods for Vonage Video Connector integration + + + Complete Vonage Video Connector voice agent example + + + Official Vonage Video API documentation and guides + + + Sign up for Vonage API access and application management + + + +## Installation + +To use `VonageVideoConnectorTransport`, install the required dependencies: + +```bash +uv add "pipecat-ai[vonage-video-connector]" +``` + + + The `vonage-video-connector` extra installs the Vonage Video Connector native + library, which is only available on Linux AMD64 and ARM64 with Python 3.13. + + +## Prerequisites + +### Vonage Account Setup + +Before using `VonageVideoConnectorTransport`, you need: + +1. **Vonage Account**: Sign up at [Vonage Dashboard](https://ui.idp.vonage.com/ui/auth/registration) +2. **Application**: Create a Vonage Video API application in the dashboard +3. **Session**: Create a Vonage Video session (via the server SDK or REST API) +4. **Token**: Generate a participant token for the session + + + The [Vonage Video Playground](https://tools.vonage.com/video/playground) is an + easy way to create a session and generate a token for testing — no code + required. + + +### Required Environment Variables + +- `VONAGE_APPLICATION_ID`: Your Vonage Video application ID +- `VONAGE_SESSION_ID`: The ID of the Video API session to join +- `VONAGE_TOKEN`: A valid participant token for the session + +### Key Features + +- **Vonage Video API**: Integrate with Vonage's managed WebRTC infrastructure +- **Audio and Video I/O**: Bidirectional audio and video streaming +- **Participant Management**: Stream subscription and participant lifecycle events +- **Auto-subscription**: Optionally auto-subscribe to incoming audio and video streams +- **Interruption Handling**: Automatic media buffer clearing on pipeline interruptions + +## Configuration + +### VonageVideoConnectorTransport + + + Your Vonage Video API application identifier. + + + + The ID of the Vonage Video session to connect to. + + + + A valid participant token for the session. + + + + Transport configuration parameters. See + [VonageVideoConnectorTransportParams](#vonagevideoconnectortransportparams) + below and + [TransportParams](/api-reference/server/services/transport/transport-params) + for inherited base parameters. + + +### VonageVideoConnectorTransportParams + +Inherits all parameters from [TransportParams](/api-reference/server/services/transport/transport-params) (audio, video settings) with these additional fields: + + + Display name for the bot's published stream in the session. + + + + Whether to enable Opus Discontinuous Transmission (DTX) for the publisher + audio, which reduces bandwidth when no audio is being sent. + + + + Whether to enable session migration, allowing the connection to survive + network changes. + + + + Whether to automatically subscribe to incoming audio streams from session + participants. + + + + Whether to automatically subscribe to incoming video streams from session + participants. + + + + Preferred `(width, height)` resolution for auto-subscribed video input + streams. + + + + Preferred framerate for auto-subscribed video input streams. + + + + Whether to clear media buffers in the Vonage Video Connector when an + `InterruptionFrame` is received. Enable this for conversational AI + applications that need to stop playback immediately when the user interrupts; + disable it for recording or streaming scenarios where all media should be + preserved. + + + + Log level for the Vonage Video Connector native library. Accepted values: + `"DEBUG"`, `"INFO"`, `"WARNING"`, `"ERROR"`. + + +### SubscribeSettings + +Used with `subscribe_to_stream()` to control per-stream subscription quality when `audio_in_auto_subscribe` or `video_in_auto_subscribe` are disabled. + + + Whether to subscribe to the stream's audio track. + + + + Whether to subscribe to the stream's video track. + + + + Preferred `(width, height)` resolution for the subscribed video track. The + server provides the closest available quality if the exact resolution is + unavailable. + + + + Preferred framerate for the subscribed video track. + + +## Usage + +`VonageVideoConnectorTransport` connects your Pipecat bot to a Vonage Video session where it can communicate with participants through audio and video streams. The transport handles session joining, stream subscription, and media conversion automatically. + +Use `pipecat.runner.vonage.configure()` to read credentials from environment variables: + +```python +from pipecat.runner.vonage import configure +from pipecat.transports.vonage.video_connector import ( + VonageVideoConnectorTransport, + VonageVideoConnectorTransportParams, +) + +(application_id, session_id, token) = await configure() + +transport = VonageVideoConnectorTransport( + application_id, + session_id, + token, + VonageVideoConnectorTransportParams( + audio_in_enabled=True, + audio_out_enabled=True, + publisher_name="Bot", + ), +) +``` + +See the [complete example](https://github.com/pipecat-ai/pipecat/blob/main/examples/transports/transports-vonage.py) for a full implementation including: + +- Session credential configuration via environment variables +- Transport setup with audio input and output +- Pipeline integration with an LLM service +- Event handling to trigger the bot when a participant connects + +## Event Handlers + +`VonageVideoConnectorTransport` provides event handlers for session lifecycle, participant stream management, and subscriber connectivity. Register handlers using the `@event_handler` decorator on the transport instance. + +### Events Summary + +| Event | Description | +| ----------------------------- | ---------------------------------------------------- | +| `on_joined` | Bot joined the Vonage session | +| `on_left` | Bot left the Vonage session | +| `on_error` | Transport error occurred | +| `on_first_participant_joined` | First participant stream joined the session (sync) | +| `on_participant_joined` | A participant stream joined the session (sync) | +| `on_participant_left` | A participant stream left the session | +| `on_client_connected` | Transport connected to a participant's stream (sync) | +| `on_client_disconnected` | Transport disconnected from a participant's stream | + +### Session Lifecycle + +#### on_joined + +Fired when the bot successfully connects to the Vonage Video session. + +```python +@transport.event_handler("on_joined") +async def on_joined(transport, data): + print(f"Joined session: {data['sessionId']}") +``` + +**Parameters:** + +| Parameter | Type | Description | +| ----------- | ------------------------------- | --------------------------------- | +| `transport` | `VonageVideoConnectorTransport` | The transport instance | +| `data` | `dict` | Event data containing `sessionId` | + +#### on_left + +Fired when the bot disconnects from the Vonage Video session. + +```python +@transport.event_handler("on_left") +async def on_left(transport, data): + print(f"Left session: {data['sessionId']}") +``` + +**Parameters:** + +| Parameter | Type | Description | +| ----------- | ------------------------------- | --------------------------------- | +| `transport` | `VonageVideoConnectorTransport` | The transport instance | +| `data` | `dict` | Event data containing `sessionId` | + +#### on_error + +Fired when a transport-level error occurs. + +```python +@transport.event_handler("on_error") +async def on_error(transport, error): + print(f"Transport error: {error}") +``` + +**Parameters:** + +| Parameter | Type | Description | +| ----------- | ------------------------------- | ---------------------- | +| `transport` | `VonageVideoConnectorTransport` | The transport instance | +| `error` | `str` | Error description | + +### Participants + +#### on_first_participant_joined + +Fired when the first participant stream is received in the session. Commonly used to trigger the bot's initial response. + +```python +@transport.event_handler("on_first_participant_joined") +async def on_first_participant_joined(transport, data): + print(f"First participant joined stream: {data['streamId']}") + await task.queue_frames([LLMRunFrame()]) +``` + +**Parameters:** + +| Parameter | Type | Description | +| ----------- | ------------------------------- | ------------------------------------------------------------------- | +| `transport` | `VonageVideoConnectorTransport` | The transport instance | +| `data` | `dict` | Event data containing `sessionId`, `streamId`, and `connectionData` | + + + This is a **synchronous** event — the session will not proceed until all + handlers complete. Keep handlers fast. + + +#### on_participant_joined + +Fired when any participant stream is received in the session. + +```python +@transport.event_handler("on_participant_joined") +async def on_participant_joined(transport, data): + print(f"Participant joined stream: {data['streamId']}") +``` + +**Parameters:** + +| Parameter | Type | Description | +| ----------- | ------------------------------- | ------------------------------------------------------------------- | +| `transport` | `VonageVideoConnectorTransport` | The transport instance | +| `data` | `dict` | Event data containing `sessionId`, `streamId`, and `connectionData` | + + + This is a **synchronous** event — the session will not proceed until all + handlers complete. Keep handlers fast. + + + + Both `on_first_participant_joined` and `on_participant_joined` fire for the + first stream. Use `on_first_participant_joined` if you only need to react + once. + + +#### on_participant_left + +Fired when a participant's stream is dropped from the session. + +```python +@transport.event_handler("on_participant_left") +async def on_participant_left(transport, data): + print(f"Participant left stream: {data['streamId']}") +``` + +**Parameters:** + +| Parameter | Type | Description | +| ----------- | ------------------------------- | ------------------------------------------------------------------- | +| `transport` | `VonageVideoConnectorTransport` | The transport instance | +| `data` | `dict` | Event data containing `sessionId`, `streamId`, and `connectionData` | + +### Subscribers + +#### on_client_connected + +Fired when the transport successfully connects to a participant's stream — at this point the subscription is established and media is flowing. + +```python +@transport.event_handler("on_client_connected") +async def on_client_connected(transport, data): + print(f"Client connected to stream: {data['streamId']}") + await task.queue_frames([LLMRunFrame()]) +``` + +**Parameters:** + +| Parameter | Type | Description | +| ----------- | ------------------------------- | ---------------------------------------------------------------------- | +| `transport` | `VonageVideoConnectorTransport` | The transport instance | +| `data` | `dict` | Event data containing `subscriberId`, `streamId`, and `connectionData` | + + + This is a **synchronous** event — the session will not proceed until all + handlers complete. Keep handlers fast. + + +#### on_client_disconnected + +Fired when the transport disconnects from a participant's stream, either due to an explicit unsubscription or a connection drop. + +```python +@transport.event_handler("on_client_disconnected") +async def on_client_disconnected(transport, data): + print(f"Client disconnected from stream: {data['streamId']}") +``` + +**Parameters:** + +| Parameter | Type | Description | +| ----------- | ------------------------------- | ---------------------------------------------------------------------- | +| `transport` | `VonageVideoConnectorTransport` | The transport instance | +| `data` | `dict` | Event data containing `subscriberId`, `streamId`, and `connectionData` | + +## Additional Resources + +- [Events Overview](/api-reference/server/events/overview) - Overview of all events in Pipecat +- [TransportParams](/api-reference/server/services/transport/transport-params) - Base transport parameters reference diff --git a/docs.json b/docs.json index d64aef7d..fd457a59 100644 --- a/docs.json +++ b/docs.json @@ -329,6 +329,7 @@ "api-reference/server/services/transport/livekit", "api-reference/server/services/transport/small-webrtc", "api-reference/server/services/transport/tavus", + "api-reference/server/services/transport/vonage", "api-reference/server/services/transport/websocket-server", "api-reference/server/services/transport/whatsapp", "api-reference/server/services/transport/transport-params"