get webrtc adm into rust by xianshijing-lk · Pull Request #1037 · livekit/rust-sdks

xianshijing-lk · 2026-04-22T19:49:23Z

Summary

This PR implements Platform Audio support for the LiveKit Rust SDK, enabling WebRTC's built-in audio device handling with microphone capture and speaker playout. The implementation introduces a handle-based PlatformAudio API that coexists with the existing NativeAudioSource for manual audio pushing.

Key Features

Two Audio Source Types:
- RtcAudioSource::Native (default): Manual audio push via NativeAudioSource for TTS, file streaming, agents
- RtcAudioSource::Device: WebRTC handles mic capture & speaker playout with echo cancellation (AEC)
Handle-based API: Create PlatformAudio instances that enable ADM recording; drop to release
Reference Counting: Multiple PlatformAudio instances share the same underlying ADM
Device Enumeration & Selection: List and select recording/playout devices
Hot-swap Device Switching: switch_recording_device() / switch_playout_device() for changing devices during active sessions
Audio Processing Configuration: AEC, AGC, NS with hardware/software preference
WebRTC Patching: external_audio_source.patch prevents audio mixing conflicts between device and manual sources

Design Document

See docs/ADM_PROXY_DESIGN.md for full architecture details including:

Recording gate pattern
WebRTC patching explanation
FFI API documentation

API Overview

use livekit::prelude::*;

// Create PlatformAudio instance (enables ADM recording)
let audio = PlatformAudio::new()?;

// Enumerate and select devices
for i in 0..audio.recording_devices() as u16 {
println!("Mic [{}]: {}", i, audio.recording_device_name(i));
}
audio.set_recording_device(0)?;

// Connect and publish
let (room, _) = Room::connect(&url, &token, RoomOptions::default()).await?;
let track = LocalAudioTrack::create_audio_track("mic", audio.rtc_source());
room.local_participant().publish_track(LocalTrack::Audio(track), opts).await?;

// Cleanup - just drop the handle
room.close().await?;
drop(audio); // ADM recording disabled when all handles released

Testing

Run Standalone Tests (no LiveKit server required)

Set custom WebRTC build path

export LK_CUSTOM_WEBRTC="/path/to/webrtc-sys/libwebrtc/mac-arm64-debug"

Run standalone PlatformAudio tests

cargo test -p livekit --test platform_audio_test test_platform_audio_standalone -- --nocapture

Run FFI request handler tests

cargo test -p livekit-ffi requests::tests -- --nocapture

Run E2E Integration Tests (requires LiveKit server)

Start a local LiveKit server first, then:

LIVEKIT_URL=ws://localhost:7880
LIVEKIT_API_KEY=devkey
LIVEKIT_API_SECRET=secret
cargo test -p livekit --test platform_audio_test --features __lk-e2e-test -- --nocapture

Test Coverage
Category │ Tests │ Description │

Standalone - Creation │ 1 │ PlatformAudio creation, device enumeration
Standalone - Ref Counting │ 1 │ Clone, sharing, drop behavior
Standalone - Device Selection │ 1 │ Set devices, invalid index handling
Standalone - Processing │ 1 │ AEC/AGC/NS configuration, hardware availability
Standalone - Reset │ 1 │ reset_platform_audio() function
Standalone - Lifecycle │ 1 │ Full create→configure→use→release cycle
FFI - Handlers │ 6 │ NewPlatformAudio, GetDevices, SetDevice, handle lifecycle
E2E - Room Connection │ 4+ │ Platform audio with room, two participants, device switching

All tests handle missing audio devices gracefully (CI-friendly).
Run the Example

List Audio Devices

cargo run -p basic_room -- --list-devices

Connect with Platform Audio (microphone capture)

LIVEKIT_URL=wss://your-server.livekit.cloud
LIVEKIT_API_KEY=your-key
LIVEKIT_API_SECRET=your-secret
cargo run -p basic_room -- --platform-audio

Connect with File Audio

cargo run -p basic_room -- --file path/to/audio.raw

Connect with Both Platform Audio and File

cargo run -p basic_room -- --platform-audio-and-file path/to/audio.raw

WebRTC Build Requirements

The external_audio_source.patch must be applied to WebRTC. The patch is automatically applied by all platform build scripts:

build_macos.sh
build_ios.sh
build_linux.sh
build_android.sh
build_windows.cmd

For local development, set LK_CUSTOM_WEBRTC to point to your patched WebRTC build.

Known Limitations

   Limitation      │                                  Description

Process-global │ Audio configuration affects all rooms in the process
Device indices │ May change on hot-plug; match by name for persistence
Single device track │ One device audio track per ADM (use NativeAudioSource for additional streams)

…o a room and thus failing the audio mode switching

ladvoc · 2026-04-24T02:14:45Z


+/// Tracks the number of active room connections.
+/// Used to prevent audio mode switching while rooms are connected.
+static ACTIVE_ROOM_COUNT: AtomicUsize = AtomicUsize::new(0);


suggestion: A potentially cleaner way to handle this is to have every room hold a Arc<()> and leverage strong_count to learn the number of active rooms—no need to manually decrement.

ladvoc · 2026-04-24T02:18:24Z

+
+/// Test setting Platform mode.
+#[test]
+#[serial]


comment (non-blocking): Currently in CI, all tests are run serially. If we switch over to Nextest (outdated PR, #816), we can configure which tests are run in serial through that config and run everything else in parallel.

…different local audio tracks

github-actions · 2026-04-27T23:37:24Z

Changeset

The following package versions will be affected by this PR:

Package	Bump
`libwebrtc`	`patch`
`livekit`	`patch`
`webrtc-sys`	`patch`

…hub.com/livekit/rust-sdks into sxian/CLT-2765/bring-webrtc-adm-to-rust

MaxHeimbrock · 2026-05-07T13:40:51Z

+- Text-to-speech (TTS) audio
+- Audio from files or network streams
+- Testing without audio hardware
+


This is the original audio input right? Existing Unity clients who want to keep the "Unity" style microphone management would also use this.

MaxHeimbrock · 2026-05-07T13:43:19Z

+
+### Hybrid Approach
+
+You can combine both approaches - use `PlatformAudio` for automatic speaker playback while also creating `NativeAudioStream` for audio processing/analysis:


Is this also possible from a Unity client? As we discussed, for the lip sync animation Unity clients might want read access to the audio data, but still want output through the platform audio.

MaxHeimbrock · 2026-05-07T13:45:25Z

+// Set recording device
+message SetRecordingDeviceRequest {
+  uint64 platform_audio_handle = 1;
+  uint32 index = 2;
+}
+
+message SetRecordingDeviceResponse {
+  optional string error = 1;
+}
+
+// Set playout device
+message SetPlayoutDeviceRequest {
+  uint64 platform_audio_handle = 1;
+  uint32 index = 2;
+}


How does it handle switching the device at runtime?

MaxHeimbrock · 2026-05-07T14:08:43Z

+**Suitable for:**
+- Server-side agents
+- Text-to-speech (TTS) audio
+- Audio from files or network streams


Or screen share audio right?

… adm

…hub.com/livekit/rust-sdks into sxian/CLT-2765/bring-webrtc-adm-to-rust

… initialized

…hub.com/livekit/rust-sdks into sxian/CLT-2765/bring-webrtc-adm-to-rust

ladvoc · 2026-05-11T18:10:04Z

 async fn main() {
    env_logger::init();

+    let args: Vec<String> = env::args().collect();


suggestion: Since we already use clap (with derive) for argument parsing in the other examples, it might be a good idea to use the same approach here. We also would get --help for free if the args have doc comments.

ladvoc · 2026-05-11T18:12:34Z


-// Connect to a room using the specified env variables
-// and print all incoming events
+// Usage:


nitpick: Other examples put usage guide in a README in the example directory.

ladvoc · 2026-05-11T18:35:59Z

+    /// let audio = PlatformAudio::new()?;
+    /// println!("Found {} microphones", audio.recording_devices());
+    /// ```
+    pub fn recording_devices(&self) -> i16 {


suggestion: Index-based property accessors like this are not typical in Rust APIs. I would recommend encapsulating these info fields in a struct and making the API iterator based:

struct RecordingDeviceInfo { pub index: u16, // or new type pub name: String, pub guid: String // or new type }

The signature of this method becomes:

pub fn recording_devices(&self) -> impl IntoIterator<Item = AudioDeviceInfo>

Usage:

let audio = PlatformAudio::new()?; for device in audio.recording_devices() { println!("{} (GUID: {})", device.name, device.guid); } // Alternatively, collect into a Vec let device_list: Vec<_> = audio.recording_devices().collect();

This same pattern would apply to playout devices.

ladvoc · 2026-05-11T18:50:06Z

+    /// ```
+    ///
+    /// [`recording_device_guid`]: Self::recording_device_guid
+    pub fn set_recording_device_by_guid(&self, guid: &str) -> AudioResult<()> {


suggestion: This is a good application for the new type pattern. The query API would provide a new type wrapper (e.g. RecordingDeviceGuid) instead of a string for device GUID, and this method would accept it. This method still has to be fallible since a device might no longer be available, but applying this pattern adds a level of type safety that enforces correct usage (e.g., providing an arbitrary string that is not a valid guid is not possible). Also applicable to playout devices.

ladvoc · 2026-05-11T18:55:46Z

+    /// [`set_recording_device`]: Self::set_recording_device
+    pub fn switch_recording_device(&self, index: u16) -> AudioResult<()> {
+        let count = self.recording_devices();
+        if index >= count as u16 {


question: What happens if the device index is invalidated between this check and the call to runtime.set_recording_device(index)?

ladvoc · 2026-05-11T19:17:10Z

 livekit = { workspace = true, features = ["rustls-tls-native-roots"] }
 livekit-api = { workspace = true, features = ["rustls-tls-native-roots"] }
 log = { workspace = true }
+hound = "3.5"


suggestion: This should be made a workspace dependency since it is also used by livekit-wakeword, the basic_room example, and soxr-sys (as a dev dependency). This will ensure we only pull down one version.

ladvoc · 2026-05-11T19:20:36Z

+        // Log audio m-lines to debug sample rate issues
+        for line in sdp.lines() {
+            if line.starts_with("m=audio") || line.contains("opus") || line.contains("a=rtpmap") {
+                log::info!("SDP audio: {}", line);


issue: This should be removed (or made debug level) before merging.

ladvoc · 2026-05-11T19:24:05Z

+use livekit::options::TrackPublishOptions;
+use livekit::{prelude::*, AudioError, AudioResult, PlatformAudio, RtcAudioSource};
+use serial_test::serial;
+use tokio::time::timeout;


nitpick: There are a few unused imports here.

ladvoc · 2026-05-11T19:25:56Z

+// ==================== Platform Audio ====================
+
+/// FFI wrapper for PlatformAudio handle.
+pub struct FfiPlatformAudio {


issue: To follow the established pattern, all request handlers and FFI wrappers for platform audio should be moved into their own module.

ladvoc · 2026-05-11T19:32:02Z

    }
+
+    // ===== Device Management Methods =====
+    // These methods are primarily for FFI use. Use PlatformAudio for the public API.


issue: I do not see these methods being called from livekit-ffi. If they are never used directly for FFI, they should be marked pub(crate).

xianshijing-lk added 3 commits April 22, 2026 12:47

get webrtc adm into rust

7e9d580

fix copyright and build

905f6ee

fix the tests where that some of them don't know if it is connected t…

7565045

…o a room and thus failing the audio mode switching

xianshijing-lk requested review from MaxHeimbrock, cloudwebrtc, ladvoc and reenboog April 22, 2026 22:50

ladvoc reviewed Apr 24, 2026

View reviewed changes

Switch over to PlatformAudio that supports different AudioSource for …

a2ac932

…different local audio tracks

xianshijing-lk added 2 commits April 27, 2026 16:45

cargo fmt

086935b

added unit tests to the new reqeusts.rs function

0ba69b0

xianshijing-lk force-pushed the sxian/CLT-2765/bring-webrtc-adm-to-rust branch from faf99f6 to 0ba69b0 Compare April 27, 2026 23:54

github-actions Bot and others added 12 commits April 27, 2026 23:54

generated protobuf

924d638

refactor some code and try integrating with Unity

64aadf9

Merge branch 'sxian/CLT-2765/bring-webrtc-adm-to-rust' of https://git…

95d1af1

…hub.com/livekit/rust-sdks into sxian/CLT-2765/bring-webrtc-adm-to-rust

generated protobuf

adb5375

update with the latest changes that make ffi work

7b4389c

Merge branch 'sxian/CLT-2765/bring-webrtc-adm-to-rust' of https://git…

0fdc716

…hub.com/livekit/rust-sdks into sxian/CLT-2765/bring-webrtc-adm-to-rust

generated protobuf

4949561

WebRTC build improvements (to be moved to separate PR)

fe98b5e

Merge branch 'sxian/CLT-2765/bring-webrtc-adm-to-rust' of https://git…

30b1a6b

…hub.com/livekit/rust-sdks into sxian/CLT-2765/bring-webrtc-adm-to-rust

fix the patch

8ecfc49

adding more ffi features to control mute / unmute of recording

11437ee

generated protobuf

67289ce

MaxHeimbrock mentioned this pull request May 6, 2026

[Draft] Integrate the PlatformAudio to unity livekit/client-sdk-unity#268

Open

MaxHeimbrock reviewed May 7, 2026

View reviewed changes

xianshijing-lk and others added 12 commits May 7, 2026 15:02

changed device enumeration to use uuid, and added reference count for…

748f93e

… adm

Merge branch 'sxian/CLT-2765/bring-webrtc-adm-to-rust' of https://git…

0309bbe

…hub.com/livekit/rust-sdks into sxian/CLT-2765/bring-webrtc-adm-to-rust

cargo fmt

a8a0424

generated protobuf

18a5150

fix the race condition that volume callbacks might come before ADM is…

143d97a

… initialized

Merge branch 'sxian/CLT-2765/bring-webrtc-adm-to-rust' of https://git…

06ab095

…hub.com/livekit/rust-sdks into sxian/CLT-2765/bring-webrtc-adm-to-rust

Merge branch 'main' into sxian/CLT-2765/bring-webrtc-adm-to-rust

497a934

improve the code from self review

345d7c7

switched over to use synthetic_adm_ (audioDevice class)

59df310

cargo fmt and added unit tests for platformAudio

d340830

addressed the comments

2279d0c

fix the tests

15753b6

ladvoc reviewed May 11, 2026

View reviewed changes


		### Hybrid Approach

		You can combine both approaches - use `PlatformAudio` for automatic speaker playback while also creating `NativeAudioStream` for audio processing/analysis:

Conversation

xianshijing-lk commented Apr 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Set custom WebRTC build path

Run standalone PlatformAudio tests

Run FFI request handler tests

Start a local LiveKit server first, then:

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changeset

Uh oh!

Choose a reason for hiding this comment

Uh oh!

MaxHeimbrock May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

xianshijing-lk commented Apr 22, 2026 •

edited

Loading

github-actions Bot commented Apr 27, 2026 •

edited

Loading

MaxHeimbrock May 7, 2026 •

edited

Loading