Skip to content

Add Krisp VIVA Turn as a user turn-stop strategy - #178

Open
weiz9 wants to merge 8 commits into
mainfrom
pr/wz/krisp-viva-turn
Open

Add Krisp VIVA Turn as a user turn-stop strategy#178
weiz9 wants to merge 8 commits into
mainfrom
pr/wz/krisp-viva-turn

Conversation

@weiz9

@weiz9 weiz9 commented Jul 8, 2026

Copy link
Copy Markdown
Collaborator

Summary

Adds Krisp VIVA Turn (pipecat's KrispVivaTurn, VIVA SDK turn detection v3 / Tt API) as a selectable turn_stop_strategy in the cascade pipeline, alongside the existing LocalSmartTurnAnalyzerV3. Unlike Smart Turn (which analyzes audio in batches when VAD detects a pause), KrispVivaTurn processes audio frame-by-frame in real time, using the VAD's speech flag as an external signal. This PR also bakes the licensed SDK into the Docker image so the strategy can run on Toolkit.

Changes

  • turn_config.py — new krisp_viva_turn branch in create_turn_stop_strategy(). Forwards threshold / frame_duration_ms to KrispTurnParams and model_path / api_key / sample_rate to the KrispVivaTurn constructor (all fall back to env vars). The pipecat module is imported lazily because it hard-requires the proprietary krisp_audio SDK — environments without it are unaffected until the strategy is selected.
  • config.py + .env.example — document the new strategy and required env vars (KRISP_VIVA_TURN_MODEL_PATH, KRISP_VIVA_API_KEY).
  • Dockerfile + vendor/krisp/ + .gitignore — optionally bake the Krisp wheel + turn model into the runtime image (see Deployment below).
  • scripts/docker_build.sh — helper to build the linux/amd64 image with git-provenance build args (and optionally push). Usage: scripts/docker_build.sh <image:tag> [--push].
  • test_turn_config.py — 4 new tests covering the branch (happy path, case-insensitivity, tuning-param routing, constructor-param routing) via an injected fake krisp module.
  • Bumped simulation_version 2.0.1 → 2.0.2 (affects benchmark outputs).

Enabling it locally

Krisp is a licensed SDK, not a PyPI package, so it is intentionally not in pyproject.toml. To run this strategy you provision it at runtime:

  1. Download from the Krisp developer portal (developers.krisp.ai): the VIVA UAR Python SDK zip and the Turn-Taking models zip. Unzip both.
  2. Install the wheel matching your interpreter's Python version and architecture from the SDK's dist/ folder — e.g. macOS x86_64 / Py3.11:
    uv pip install .../krisp-viva-uar-python-sdk-<ver>/dist/krisp_audio-<ver>-cp311-cp311-macosx_11_0_x86_64.whl
    (Pick arm64 vs x86_64 to match python -c "import platform; print(platform.machine())".)
  3. Point env vars at the model + your license key, then select the strategy:
    export KRISP_VIVA_TURN_MODEL_PATH=/path/to/krisp-viva-tt-models/krisp-viva-tp-v3.kef
    export KRISP_VIVA_API_KEY=<license key>
    eva --domain airline --model.llm gpt-5-mini --model.turn-stop-strategy krisp_viva_turn

Optional tuning via EVA_MODEL__TURN_STOP_STRATEGY_PARAMSthreshold (default 0.5) and frame_duration_ms (default 20; one of 10/15/20/30/32).

Network: the SDK validates its license against sdkapi.krisp.ai and reports usage to analytics.krisp.ai. Both must be reachable; a WAF/proxy that blocks them will fail license validation. Eval keys are time-limited and metered per minute. (Locally, this requires CloudVPN.)

Deployment (Toolkit / Docker)

The image is linux/amd64, so it needs the Linux wheel from the same SDK download —
krisp_audio-<ver>-cp311-cp311-linux_x86_64.whl (ships alongside the macOS wheels).

The Dockerfile runtime stage optionally installs the SDK: it copies from vendor/krisp/
and, when a wheel is present, installs it into the venv (uv pip install --python /opt/venv/bin/python, matching the builder stage — a bare pip would miss the uv-created
venv) and copies the .kef model to /opt/krisp/models/, setting KRISP_VIVA_TURN_MODEL_PATH.
If vendor/krisp/ has no wheel, the build still succeeds and krisp_viva_turn is simply
unavailable — so existing image builds are unaffected.

  • The wheel + .kef are proprietary (not public, not on PyPI) so they are git-ignored;
    only vendor/krisp/README.md (setup instructions) is committed. Populate vendor/krisp/
    from your Krisp account before building.
  • KRISP_VIVA_API_KEY is not baked in — supply it as a runtime env var/secret in the job.
  • Runtime egress to sdkapi.krisp.ai + analytics.krisp.ai is required (confirmed reachable
    from Toolkit — see Testing).

Testing

  • 27/27 test_turn_config.py unit tests pass; ruff + version-bump checks pass; no new mypy errors.
  • Live end-to-end validated locally (macOS, cascade pipeline): a full multi-turn airline
    conversation ran with turn_stop_strategy=krisp_viva_turn; the SDK initialized, loaded the
    model, and produced 90 real end-of-turn predictions (probabilities 51–99%, e2e latencies
    ~0.1 ms–1.5 s, mostly ~100–300 ms). Turn-taking was driven entirely by Krisp.
  • Validated on Toolkit with the baked image (flux-general-en + gpt-4.1 + sonic-3.5):
    Krisp turn detection ran correctly ([KrispVivaTurn] TURN: COMPLETE, e.g. probability 88–96%,
    e2e ~99–282 ms) with no license errors — confirming Toolkit's egress reaches Krisp's
    licensing endpoints. Also verified krisp_audio imports from /opt/venv in the built image.
  • Confirmed the lazy-import boundary: when krisp_audio is absent, selecting the strategy fails
    with a clean, actionable error and no other strategy is affected.

weiz9 added 2 commits July 8, 2026 09:58
Wire pipecat's KrispVivaTurn (VIVA SDK turn detection v3) into the cascade
pipeline as a selectable turn_stop_strategy alongside the existing
LocalSmartTurnAnalyzerV3. The analyzer is imported lazily because it depends
on the proprietary krisp_audio SDK plus a .kef model and license key, so
environments without the SDK are unaffected until the strategy is selected.

- turn_config: add 'krisp_viva_turn' branch forwarding threshold/
  frame_duration_ms to KrispTurnParams and model_path/api_key/sample_rate
  to the KrispVivaTurn constructor (both fall back to env vars)
- config + .env.example: document the strategy and required env vars
  (KRISP_VIVA_TURN_MODEL_PATH, KRISP_VIVA_API_KEY)
- tests: cover the new branch via an injected fake krisp module
- bump simulation_version 2.0.1 -> 2.0.2
@weiz9
weiz9 marked this pull request as ready for review July 23, 2026 17:38
weiz9 added 5 commits July 24, 2026 11:29
Enable the krisp_viva_turn turn-stop strategy on Toolkit by installing the
licensed Krisp VIVA wheel and turn model into the runtime image when present.

The wheel + .kef are proprietary (not public, not on PyPI), downloaded per
Krisp developer account, so they are git-ignored and provisioned into
vendor/krisp/ before the build (see vendor/krisp/README.md). The install step
is optional: builds without the files still succeed and krisp_viva_turn is
simply unavailable. KRISP_VIVA_API_KEY stays a runtime secret, not baked in.

The image is linux/amd64, so the Linux x86_64 cp311 wheel is required (not the
macOS wheel used for local dev).
…caching

Krisp's cache key now only depends on vendor/krisp/ + the eva package overlay,
so editing scripts/, configs/, data/, or assets/ no longer forces an
unnecessary Krisp reinstall. Verified with a rebuild: the Krisp COPY/RUN
layers stay CACHED while only the touched layer reruns.
The merge from main switched the Docker build to uv (uv venv / uv sync), and a
uv-created /opt/venv has no seeded pip. Our Krisp step still used a bare
'pip install', which would fall through to the system pip and install krisp_audio
into the system site-packages — invisible to the /opt/venv interpreter the eva
entrypoint runs. Switch to 'uv pip install --python /opt/venv/bin/python' (matching
the builder stage) and bring uv into the runtime stage. Verified krisp_audio now
imports from /opt/venv in the built image.
…t provenance)

Takes the full image:tag as arg 1 and an optional --push, e.g.
  scripts/docker_build.sh registry.console.elementai.com/snow.core_llm/eva:krisp --push

@raghavm243512 raghavm243512 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

docker build.sh should not be in the PR

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants