feat(vllm-realtime): add realtime speech transcription example#28
Open
Gideonjon wants to merge 4 commits into
Open
feat(vllm-realtime): add realtime speech transcription example#28Gideonjon wants to merge 4 commits into
Gideonjon wants to merge 4 commits into
Conversation
Implements GitHub issue livepeer#2: audio ingest via Trickle, transcript events streamed back to the client over an orchestrator-proxied WebSocket, live settings via mid-stream session.update, mock backend (no GPU), and the real vLLM Voxtral backend. Verified end-to-end offchain with both backends: mock on a GPU-free Docker host, and vLLM 0.24 serving Voxtral-Mini-4B-Realtime-2602 on an RTX 4090 with live speech transcribed as streaming word deltas. Key implementation notes: - Runner subscribes to Trickle via the channel's internal_url; the public url goes to the client (the advertised serviceAddr is not reachable from inside the runner's network). - vLLM /v1/realtime protocol (v0.24): model at the top level of session.update, generation started/flushed via input_audio_buffer .commit final=false/true, transcription.delta/done events, empty-text deltas filtered. - vllm compose service caps --max-model-len at 16384 so the KV cache fits next to the weights on 24 GB cards. Also adds platform: linux/amd64 to compose.orchestrator.yml for Apple Silicon Mac compatibility.
The SDK meters Trickle for free (TrickleSubscriber.get_stats()) but does
not meter WebSockets, so an app that streams its results over one has no
observability unless it counts them itself.
Add stats.py with the missing half, shaped like the SDK's stats:
WebSocketMeter/WebSocketStats for the transcript socket, and
TranscriptionMeter/TranscriptionStats for latency and throughput. The
runner folds in the SDK's Trickle counters and emits a final
{"type": "stats"} event; the client prints it as a summary.
Note that the SDK's frame counters (MediaOutputStats.audio_frames_decoded)
belong to the AV decode path. This example publishes raw PCM16, so there is
no decoder and no frame count -- audio duration is derived from byte totals.
Also correct the framing around --no-realtime. It cannot be used to measure
throughput: Trickle deletes unread segments when the publisher closes, so an
unpaced run drops most of its audio and reports confident nonsense (1 of 14
segments in practice). Under realtime pacing the real-time factor is pinned
near 1.0 by construction and only answers "did the pipeline keep up"; the
honest latency signals are time-to-first-word and the finalize tail.
Verified end-to-end on an RTX 4090 serving Voxtral-Mini-4B-Realtime:
6.56 s of speech, finalize tail 0.38 s (reproducible within ~10 ms),
RTF 1.12x, 14 segments with zero sequence gaps, retries or failures.
A Trickle channel keeps no backlog. TrickleSubscriber defaults to
start_seq=-2 (join at the live edge) and an earlier index answers 470
"no data at this index" rather than replaying, so any segment published
before the runner reads the previous one is destroyed, not queued.
That is the right default for video -- drop stale frames, stay current --
but audio is not droppable, and nothing reports the loss: publishing 6.5s
of speech unpaced delivered 1 of 14 segments with seq_gap_events=0, no
error, and a transcript that was simply short. Subscribing at start_seq=0
is not a fix; it delivers 0 segments and resets to the live edge.
Fix it with app-level backpressure over the WebSocket we already have. The
runner emits {"type": "progress", "audio_bytes": N} after each segment it
consumes; the client publishes one segment, waits for the runner to
acknowledge it, then publishes the next. At most one segment is ever in
flight, so the live edge is always the unread segment.
This makes --no-realtime measure real throughput instead of shredding the
audio, and the two modes now report different, honest numbers on the same
6.56s clip (RTX 4090, Voxtral-Mini-4B-Realtime):
paced finalize tail 0.37s, RTF 1.13x -- live latency
unpaced wall 1.78s, RTF 0.27x -- ~3.7x realtime throughput
Both transcribe identically, 14/14 segments, no gaps. Verified against the
mock backend too, paced and unpaced.
The performance summary metered the runner's ingest (TrickleSubscriber) and the WebSocket, but never the client's own publish path, even though TricklePublisher.get_stats() offers it for free: bytes submitted, segments started/completed/failed, post successes and retries. Report it, and order the three transport lines along the audio's actual path -- publish, ingest, websocket out. Pairing the two SDK lines is the point: publish segments=14/14 against ingest segments=14 proves nothing was dropped between them. That check is worth surfacing because the transport will not raise on loss -- a run that shredded 13 of 14 segments still reported seq_gap_events=0. The publish rate also independently confirms what each mode measures. Paced sustains 30 kB/s, exactly PCM16 mono @16kHz realtime. Unpaced sustains 1159 kB/s -- 38x faster -- and still lands 14/14, which is the backpressure doing its job at full tilt. Verified on GPU (paced, unpaced, --language en) and against the mock backend.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements GitHub issue #2: audio ingest via Trickle, transcript events streamed back to the client over an orchestrator-proxied WebSocket, live settings via mid-stream session.update, mock backend (no GPU), and the real vLLM Voxtral backend.
Verified end-to-end offchain with both backends: mock on a GPU-free Docker host, and vLLM 0.24 serving Voxtral-Mini-4B-Realtime-2602 on an RTX 4090 with live speech transcribed as streaming word deltas.
Key implementation notes:
Also adds platform: linux/amd64 to compose.orchestrator.yml for Apple Silicon Mac compatibility.