Skip to content

Feat/anam add director notes cue support#6278

Draft
sr-anam wants to merge 3 commits into
livekit:mainfrom
anam-org:feat/anam-add-director-notes-cue-support
Draft

Feat/anam add director notes cue support#6278
sr-anam wants to merge 3 commits into
livekit:mainfrom
anam-org:feat/anam-add-director-notes-cue-support

Conversation

@sr-anam

@sr-anam sr-anam commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Requires - #6267

What

Adds inline director-note cues to the Anam avatar plugin:
director_note_cue_transform, a tts_text_transforms callable that reads
bracket cues (e.g. [happy], [sad]) in the agent's speech and forwards them to
the Anam engine, keyed by where each lands in the spoken text — so the avatar's
expression changes moment-to-moment, mid-utterance, driven by the LLM's own
output.

Why

The plugin can stream an Anam avatar but gives no way to drive its expression at
runtime. Letting the LLM emit [tag] cues inline means expression follows the
content of what's being said, with no separate control channel and no extra
turn-taking.

Design

  • A standard tts_text_transforms entry — composes with other transforms and
    needs no changes to the avatar session.
  • Two independent levers cover the real cases: stripped_tags (remove the tag
    before TTS) and forwarded_tags (send it to Anam). So: strip+forward for a TTS
    that would otherwise speak the tag (ElevenLabs), forward-only for a tag both
    face and voice should act on ([laughter]), and a no-op for a TTS that handles
    the tags itself (Cartesia sonic-3.5). Non-preset brackets (e.g. ElevenLabs
    [whispers]) pass through by default.
  • Cues ride the existing LiveKit data channel on the director_note_cue topic,
    targeted only at the engine participant (discovered via lk.publish_on_behalf),
    consistent with how the framework's avatar audio output coordinates with its
    worker.
  • Offsets are resolved against the TTS-aligned timed transcript, so
    use_tts_aligned_transcript=True is required.

Scoped entirely to livekit-plugins-anam; no change to default behavior for
existing users.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant