Skip to content

docs: update for pipecat PR #4052 #832

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
markbackman wants to merge 1 commit into from
Closed

docs: update for pipecat PR #4052 #832

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions api-reference/server/services/supported-services.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ Transports exchange audio and video streams between the user and bot.
| [LiveKitTransport](/api-reference/server/services/transport/livekit) | `uv add "pipecat-ai[livekit]"` |
| [SmallWebRTCTransport](/api-reference/server/services/transport/small-webrtc) | `uv add "pipecat-ai[webrtc]"` |
| [TavusTransport](/api-reference/server/services/transport/tavus) | `uv add "pipecat-ai[tavus]"` |
| [VonageVideoConnectorTransport](/api-reference/server/services/transport/vonage) | `uv add "pipecat-ai[vonage-video-connector]"` |
| [WebSocket Transports](/api-reference/server/services/transport/websocket-server) | `uv add "pipecat-ai[websocket]"` |
| [WhatsAppTransport](/api-reference/server/services/transport/whatsapp) | `uv add "pipecat-ai[webrtc]"` |

Expand Down
375 changes: 375 additions & 0 deletions api-reference/server/services/transport/vonage.mdx
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,375 @@
---
title: "VonageVideoConnectorTransport"
description: "WebRTC transport implementation using Vonage Video Connector for real-time audio/video communication"
---

## Overview

`VonageVideoConnectorTransport` provides real-time audio and video communication using Vonage's Video Connector API. It handles bidirectional media streams, participant management, and session lifecycle events for conversational AI applications using Vonage's video infrastructure.

<CardGroup cols={2}>
<Card
title="VonageVideoConnectorTransport API Reference"
icon="code"
href="https://reference-server.pipecat.ai/en/latest/api/pipecat.transports.vonage.video_connector.html"
>
Transport methods and configuration options
</Card>
<Card
title="Example Implementation"
icon="play"
href="https://github.com/pipecat-ai/pipecat/blob/main/examples/transports/transports-vonage.py"
>
Complete Vonage Video Connector transport example
</Card>
<Card
title="Vonage Video API Documentation"
icon="book"
href="https://developer.vonage.com/en/video/overview"
>
Official Vonage Video API documentation
</Card>
<Card
title="Vonage Dashboard"
icon="external-link"
href="https://dashboard.vonage.com/"
>
Sign up for Vonage API access
</Card>
</CardGroup>

## Installation

To use `VonageVideoConnectorTransport`, install the required dependencies:

```bash
uv add "pipecat-ai[vonage-video-connector]"
```

<Note>
The Vonage Video Connector transport is currently only available on Python 3.13+ and Linux platforms.
</Note>

## Prerequisites

### Vonage Account Setup

Before using VonageVideoConnectorTransport, you need:

1. **Vonage Account**: Sign up at [Vonage Dashboard](https://dashboard.vonage.com/)
2. **Video API Application**: Create a Video API application in your Vonage dashboard
3. **Application ID**: Get your application ID from the dashboard
4. **Session Creation**: Sessions must be created before connecting
5. **Token Generation**: Generate tokens for authentication

### Required Environment Variables

- `VONAGE_APPLICATION_ID`: Your Vonage Video application ID
- `VONAGE_SESSION_ID`: The session ID to connect to
- `VONAGE_TOKEN`: Authentication token for the session

## Key Features

- **Real-time Audio/Video**: Bidirectional audio and video streaming
- **Multi-participant Support**: Handle multiple participants in video sessions
- **Session Management**: Automatic session connection and lifecycle handling
- **Event-Driven Architecture**: Rich event handlers for session and participant events
- **Configurable Media**: Fine-grained control over audio/video parameters
- **Buffer Management**: Automatic media buffer clearing on interruptions

## Configuration

### VonageVideoConnectorTransport

<ParamField path="application_id" type="str" required>
The Vonage Video application ID from your Vonage dashboard.
</ParamField>

<ParamField path="session_id" type="str" required>
The session ID to connect to.
</ParamField>

<ParamField path="token" type="str" required>
Authentication token for the session.
</ParamField>

<ParamField path="params" type="VonageVideoConnectorTransportParams" default="VonageVideoConnectorTransportParams()">
Transport configuration parameters. See [VonageVideoConnectorTransportParams](#vonagevideconnectortransportparams) below and
[TransportParams](/api-reference/server/services/transport/transport-params)
for inherited base parameters.
</ParamField>

### VonageVideoConnectorTransportParams

Inherits all parameters from [TransportParams](/api-reference/server/services/transport/transport-params) (audio, video, VAD settings) with these additional fields:

<ParamField path="publisher_name" type="str" default="Bot">
Name of the publisher stream displayed to participants.
</ParamField>

<ParamField path="publisher_enable_opus_dtx" type="bool" default="False">
Whether to enable OPUS DTX (Discontinuous Transmission) for publisher audio to reduce bandwidth during silence.
</ParamField>

<ParamField path="session_enable_migration" type="bool" default="False">
Whether to enable session migration for improved reliability.
</ParamField>

<ParamField path="audio_in_auto_subscribe" type="bool" default="True">
Whether to automatically subscribe to audio streams from participants.
</ParamField>

<ParamField path="video_in_auto_subscribe" type="bool" default="False">
Whether to automatically subscribe to video streams from participants.
</ParamField>

<ParamField path="video_connector_log_level" type="str" default="INFO">
Log level for the Vonage Video Connector SDK. Valid values: "DEBUG", "INFO", "WARNING", "ERROR".
</ParamField>

<ParamField path="video_in_preferred_resolution" type="tuple[int, int] | None" default="None">
Preferred resolution for video input capture (width, height).
</ParamField>

<ParamField path="video_in_preferred_framerate" type="int | None" default="None">
Preferred framerate for video input capture.
</ParamField>

<ParamField path="clear_buffers_on_interruption" type="bool" default="True">
Whether to clear media buffers when an interruption frame is received.
</ParamField>

## Usage

VonageVideoConnectorTransport connects your Pipecat bot to Vonage Video sessions where it can communicate with participants through audio and video streams. The transport handles session joining, participant events, and media streaming automatically.

### Basic Setup

```python
from pipecat.transports.vonage.video_connector import (
VonageVideoConnectorTransport,
VonageVideoConnectorTransportParams
)

transport = VonageVideoConnectorTransport(
application_id="your_application_id",
session_id="your_session_id",
token="your_token",
VonageVideoConnectorTransportParams(
audio_in_enabled=True,
audio_out_enabled=True,
publisher_name="Bot",
)
)
```

### Pipeline Integration

```python
from pipecat.pipeline.pipeline import Pipeline

pipeline = Pipeline([
transport.input(),
user_aggregator,
llm,
transport.output(),
assistant_aggregator,
])
```

See the [complete example](https://github.com/pipecat-ai/pipecat/blob/main/examples/transports/transports-vonage.py) for a full implementation including:

- Session creation and token generation
- Transport configuration with event handlers
- Pipeline integration with audio processing
- VAD and turn detection setup

## Event Handlers

VonageVideoConnectorTransport provides event handlers for session lifecycle and participant management. Register handlers using the `@event_handler` decorator on the transport instance.

### Events Summary

| Event | Description |
| ----------------------------- | ---------------------------------- |
| `on_joined` | Bot joined the session |
| `on_left` | Bot left the session |
| `on_error` | Session error occurred |
| `on_client_connected` | Client connected (sync) |
| `on_client_disconnected` | Client disconnected |
| `on_first_participant_joined` | First participant joined (sync) |
| `on_participant_joined` | A participant joined (sync) |
| `on_participant_left` | A participant left |

### Session Lifecycle

#### on_joined

Fired when the bot successfully joins the Vonage Video session.

```python
@transport.event_handler("on_joined")
async def on_joined(transport, session):
print(f"Joined session: {session['sessionId']}")
```

**Parameters:**

| Parameter | Type | Description |
| --------- | ---- | ------------------------------- |
| transport | VonageVideoConnectorTransport | The transport instance |
| session | dict | Session info with `sessionId` |

#### on_left

Fired when the bot leaves the Vonage Video session.

```python
@transport.event_handler("on_left")
async def on_left(transport, session):
print(f"Left session: {session['sessionId']}")
```

**Parameters:**

| Parameter | Type | Description |
| --------- | ---- | ------------------------------- |
| transport | VonageVideoConnectorTransport | The transport instance |
| session | dict | Session info with `sessionId` |

#### on_error

Fired when a session error occurs.

```python
@transport.event_handler("on_error")
async def on_error(transport, error):
print(f"Session error: {error}")
```

**Parameters:**

| Parameter | Type | Description |
| --------- | ---- | ------------------------------- |
| transport | VonageVideoConnectorTransport | The transport instance |
| error | str | Error description |

### Participant Events

#### on_first_participant_joined

Fired when the first participant joins the session. This is a synchronous event handler.

```python
@transport.event_handler("on_first_participant_joined")
async def on_first_participant_joined(transport, client):
print(f"First participant joined: {client['streamId']}")
# Start the conversation
await task.queue_frames([LLMRunFrame()])
```

**Parameters:**

| Parameter | Type | Description |
| --------- | ---- | -------------------------------------------------------- |
| transport | VonageVideoConnectorTransport | The transport instance |
| client | dict | Client info with `sessionId`, `streamId`, `connectionData` |

#### on_participant_joined

Fired when a participant joins the session. This is a synchronous event handler.

```python
@transport.event_handler("on_participant_joined")
async def on_participant_joined(transport, client):
print(f"Participant joined: {client['streamId']}")
```

**Parameters:**

| Parameter | Type | Description |
| --------- | ---- | -------------------------------------------------------- |
| transport | VonageVideoConnectorTransport | The transport instance |
| client | dict | Client info with `sessionId`, `streamId`, `connectionData` |

#### on_participant_left

Fired when a participant leaves the session.

```python
@transport.event_handler("on_participant_left")
async def on_participant_left(transport, client):
print(f"Participant left: {client['streamId']}")
```

**Parameters:**

| Parameter | Type | Description |
| --------- | ---- | -------------------------------------------------------- |
| transport | VonageVideoConnectorTransport | The transport instance |
| client | dict | Client info with `sessionId`, `streamId`, `connectionData` |

### Client Events

#### on_client_connected

Fired when a client (subscriber) connects. This is a synchronous event handler.

```python
@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
print(f"Client connected: {client['subscriberId']}")
```

**Parameters:**

| Parameter | Type | Description |
| --------- | ---- | -------------------------------------------------------- |
| transport | VonageVideoConnectorTransport | The transport instance |
| client | dict | Client info with `subscriberId`, `streamId`, `connectionData` |

#### on_client_disconnected

Fired when a client (subscriber) disconnects.

```python
@transport.event_handler("on_client_disconnected")
async def on_client_disconnected(transport, client):
print(f"Client disconnected: {client['subscriberId']}")
```

**Parameters:**

| Parameter | Type | Description |
| --------- | ---- | -------------------------------------------------------- |
| transport | VonageVideoConnectorTransport | The transport instance |
| client | dict | Client info with `subscriberId`, `streamId`, `connectionData` |

## Advanced Features

### Manual Stream Subscription

While auto-subscription is enabled by default for audio, you can manually control stream subscriptions:

```python
from pipecat.transports.vonage.client import SubscribeSettings

# Subscribe to a specific participant's stream
await transport.subscribe_to_stream(
stream_id="participant_stream_id",
params=SubscribeSettings(
subscribe_to_audio=True,
subscribe_to_video=True,
preferred_resolution=(1280, 720),
preferred_framerate=30
)
)
```

## Notes

- The Vonage Video Connector transport is currently only available on Python 3.13+ and Linux platforms due to native library requirements.
- Sessions must be created using the Vonage Video API before connecting with the transport.
- Authentication tokens are required for all session connections and should be generated server-side.
- Media buffers are automatically cleared on interruption to ensure responsive conversation flow.
- The `on_client_connected`, `on_first_participant_joined`, and `on_participant_joined` event handlers are synchronous (`sync=True`).
1 change: 1 addition & 0 deletions docs.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -329,6 +329,7 @@
"api-reference/server/services/transport/livekit",
"api-reference/server/services/transport/small-webrtc",
"api-reference/server/services/transport/tavus",
"api-reference/server/services/transport/vonage",
"api-reference/server/services/transport/websocket-server",
"api-reference/server/services/transport/whatsapp",
"api-reference/server/services/transport/transport-params"
Expand Down
Loading