[agentserver] Add azure-ai-agentserver-voice package for Voice Live Bridge Protocol

[knitvoger]

Description

Why

Developers who already run a hosted text agent (text-in / text-out, custom logic in
their own container) want to give it a voice without rewriting it as a speech-to-speech
agent. The platform puts a managed Voice Live session in front, which owns audio, STT/TTS,
VAD, turn-taking, and barge-in, and relays text turns to the agent. Today an agent must
either be rewritten for speech-to-speech or hand-roll an incompatible schema over the raw
invocations_ws relay. This PR ships the SDK side of a shared, typed text-turn contract
so a text-only handler can serve a real voice call.

What

Adds the new azure-ai-agentserver-voice package — a hand-written server-host layer that
implements Voice Live Bridge Protocol 1.0 (namespace azure.ai.agentserver.voice) over the
shipped invocations_ws WebSocket transport. It lets a voiceLiveCompatible agent that
declares bridgeProtocolVersion: "1.0" handle greeting, streamed replies, barge-in, DTMF,
handoff, proactive responses, and clean hang-up with no audio code. Also includes the
Core/Invocations support seams the voice host depends on.

How

• Voice package (_host.py, _models.py, _protocol.py, _runtime.py): a full-duplex
  host with a single receive loop over one agent WebSocket, typed inbound/outbound bridge
  messages, response-lifecycle arbitration, and callback dispatch — layered on
  @app.ws_handler, not modeled as a per-turn chat_bot.
• Core/Invocations seams (commit 1): get_tracer/get_meter observability seams, W3C
  trace-context propagation + x-platform-server identity over WebSocket upgrades, an
  enable_sensitive_data host option for a content-free default, and preservation of an
  application-selected WebSocket close code. Bumps Core→2.0.0b9, Invocations→1.0.0b8.
• Registers the voice artifact in ci.yml and the tracing_e2e target in tests.yml.

Opt-in rides existing untyped agent metadata (voiceLiveCompatible, invocations_ws,
bridgeProtocolVersion), so no TypeSpec / service-model change is required.

Spec: voice_live_bridge/spec.md · Design: hosted_text_agent_and_voice_live_bridge.md

All SDK Contribution checklist:

• The pull request does not introduce [breaking changes]
• CHANGELOG is updated for new features, bug fixes or other significant changes.
• I have read the contribution guidelines.

General Guidelines and Best Practices

• Title of the pull request is clear and informative.
• There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• Pull request includes test coverage for the included changes.

[github-actions]

Thank you for your contribution @knitvoger! We will review the pull request and get back to you soon.

[azure-pipelines]

Azure Pipelines:
Successfully started running 1 pipeline(s).
8 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

[Copilot]

Pull request overview

Introduces the preview azure-ai-agentserver-voice package and extends AgentServer Core/Invocations with WebSocket tracing, identity, and close-code support.

Changes:

• Adds the typed Voice Live Bridge Protocol host, models, runtime, documentation, and samples.
• Adds comprehensive protocol, host, and live tracing tests.
• Extends shared WebSocket observability and package identity infrastructure.

Reviewed changes

Copilot reviewed 41 out of 42 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
sdk/agentserver/tests.yml	Adds Voice package testing.
sdk/agentserver/ci.yml	Registers Voice in CI.
sdk/agentserver/azure-ai-agentserver-voice/tests/test_voice_host.py	Tests host lifecycle and protocol behavior.
sdk/agentserver/azure-ai-agentserver-voice/tests/test_tracing_e2e.py	Tests live trace export.
sdk/agentserver/azure-ai-agentserver-voice/tests/test_protocol.py	Tests protocol parsing and validation.
sdk/agentserver/azure-ai-agentserver-voice/tests/conftest.py	Defines shared test fixtures.
sdk/agentserver/azure-ai-agentserver-voice/samples/basic_voice_agent/requirements.txt	Declares sample dependency.
sdk/agentserver/azure-ai-agentserver-voice/samples/basic_voice_agent/basic_voice_agent.py	Adds basic Voice agent sample.
sdk/agentserver/azure-ai-agentserver-voice/README.md	Documents package usage.
sdk/agentserver/azure-ai-agentserver-voice/pyrightconfig.json	Configures type checking.
sdk/agentserver/azure-ai-agentserver-voice/pyproject.toml	Defines package metadata and dependencies.
sdk/agentserver/azure-ai-agentserver-voice/MANIFEST.in	Configures source distribution contents.
sdk/agentserver/azure-ai-agentserver-voice/LICENSE	Adds MIT license.
sdk/agentserver/azure-ai-agentserver-voice/docs/voice-live-bridge-sdk-design.md	Documents SDK architecture and protocol.
sdk/agentserver/azure-ai-agentserver-voice/dev_requirements.txt	Declares development dependencies.
sdk/agentserver/azure-ai-agentserver-voice/cspell.json	Configures spelling checks.
sdk/agentserver/azure-ai-agentserver-voice/CHANGELOG.md	Records initial preview features.
sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/voice/py.typed	Marks package as typed.
sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/voice/_version.py	Defines preview version.
sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/voice/_runtime.py	Implements session and response helpers.
sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/voice/_protocol.py	Implements protocol encoding and validation.
sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/voice/_models.py	Defines public event models.
sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/voice/__init__.py	Exports the public Voice API.
sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/__init__.py	Extends namespace package.
sdk/agentserver/azure-ai-agentserver-voice/azure/ai/__init__.py	Extends namespace package.
sdk/agentserver/azure-ai-agentserver-voice/azure/__init__.py	Extends namespace package.
sdk/agentserver/azure-ai-agentserver-voice/api.metadata.yml	Records APIView metadata.
sdk/agentserver/azure-ai-agentserver-voice/api.md	Captures the Voice API surface.
sdk/agentserver/azure-ai-agentserver-invocations/tests/test_ws_server_routes.py	Tests WebSocket identity headers.
sdk/agentserver/azure-ai-agentserver-invocations/tests/test_ws_close_event.py	Tests application close-code reporting.
sdk/agentserver/azure-ai-agentserver-invocations/CHANGELOG.md	Documents Invocations updates.
sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/_invocation.py	Registers Invocations identity.
sdk/agentserver/azure-ai-agentserver-invocations/azure/ai/agentserver/invocations/_invocation_ws.py	Tracks application-selected close codes.
sdk/agentserver/azure-ai-agentserver-core/tests/test_tracing.py	Tests tracing additions.
sdk/agentserver/azure-ai-agentserver-core/tests/test_server_routes.py	Tests WebSocket platform headers.
sdk/agentserver/azure-ai-agentserver-core/CHANGELOG.md	Documents Core observability changes.
sdk/agentserver/azure-ai-agentserver-core/azure/ai/agentserver/core/_tracing.py	Adds protocol tracing and meter seams.
sdk/agentserver/azure-ai-agentserver-core/azure/ai/agentserver/core/_base.py	Adds WebSocket identity and privacy options.
sdk/agentserver/azure-ai-agentserver-core/azure/ai/agentserver/core/__init__.py	Exports observability helpers.
sdk/agentserver/azure-ai-agentserver-core/api.metadata.yml	Updates Core API metadata.
sdk/agentserver/azure-ai-agentserver-core/api.md	Updates the Core API surface.

Comments suppressed due to low confidence (1)

sdk/agentserver/azure-ai-agentserver-core/azure/ai/agentserver/core/_base.py:235

• This unconditionally changes the established custom configure_observability callback contract from two keyword arguments to three. Existing callbacks that accept only connection_string and log_level now raise TypeError; the broad handler below suppresses it and silently disables observability. Preserve compatibility by detecting whether the custom callable accepts enable_sensitive_data (or **kwargs) before passing the new keyword, and add a test with the prior two-argument callback shape.

                configure_observability(
                    connection_string=_conn_str,
                    log_level=log_level,
                    enable_sensitive_data=_sensitive_data,
                )

[Copilot]

Pull request overview

Copilot reviewed 41 out of 42 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/voice/_runtime.py:807

• This measures the raw UTF-8 text, not its encoded JSON frame size. Quotes, backslashes, and control characters expand during json.dumps (up to six bytes per input byte), so text accepted under the 900 KiB check can still exceed the documented 1 MB WebSocket limit and be disconnected instead of raising locally. Validate the complete encoded frame size, including JSON escaping, envelope fields, and the optional voice patch.

def _text_size(value: str) -> int:
    size = len(value.encode("utf-8"))
    if size > _MAX_OUTPUT_ITEM_BYTES:
        raise ValueError("An output item exceeds the maximum encoded text size")

[Copilot]

Pull request overview

Copilot reviewed 42 out of 43 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (4)

sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/voice/_host.py:545

• This correlation entry is never removed after a normal response; it is only popped if a late pre-response timeout arrives. Consequently every completed VoiceResponse, including all accumulated output chunks, remains reachable for the entire connection, bypassing the bounded _recent_responses cache and causing memory to grow with every turn. Use bounded lightweight prefix tombstones and release response text/state once the reconciliation window ends.

            self._resolved_input_prefixes[in_reply_to] = (active, True)

sdk/agentserver/azure-ai-agentserver-core/azure/ai/agentserver/core/_base.py:235

• Passing this new keyword unconditionally breaks existing custom configure_observability callbacks that implement the prior (connection_string, log_level) contract. Their TypeError is caught by the broad handler below, so host construction continues with observability silently disabled. Add a backward-compatible invocation path or a versioned callback contract rather than treating an unsupported new keyword as setup failure.

                configure_observability(
                    connection_string=_conn_str,
                    log_level=log_level,
                    enable_sensitive_data=_sensitive_data,
                )

sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/voice/_host.py:1567

• Every inbound payload's full canonical JSON is retained for the connection. With the documented 1 MiB frame limit and 4,096-entry allowance, a peer can make this cache consume several GiB and exhaust the process. Store a fixed-size digest of the canonical payload (or otherwise enforce a byte budget) so changed-payload detection does not retain all caller content.

            if len(self._seen_messages) >= _MAX_SEEN_MESSAGES:
                raise VoiceBridgeProtocolError("Message dedupe limit exceeded", close_code=1008)
            self._seen_messages[message_id] = canonical

sdk/agentserver/azure-ai-agentserver-voice/azure/ai/agentserver/voice/_runtime.py:807

• This limit measures the raw UTF-8 text, not its encoded JSON representation. JSON escaping can expand control characters sixfold, so a value well below 900 KiB (for example, 200 KiB of \u0000) passes this check but produces a frame over the documented 1 MiB transport limit. Account for json.dumps(..., ensure_ascii=False) bytes plus envelope overhead for both complete and accumulated streamed output.

def _text_size(value: str) -> int:
    size = len(value.encode("utf-8"))
    if size > _MAX_OUTPUT_ITEM_BYTES:
        raise ValueError("An output item exceeds the maximum encoded text size")

[microsoft-github-policy-service]

@knitvoger please read the following Contributor License Agreement(CLA). If you agree with the CLA, please reply with the following information.

@microsoft-github-policy-service agree [company="{your company}"]

Options:

• (default - no company specified) I have sole ownership of intellectual property rights to my Submissions and I am not making Submissions in the course of work for my employer.

@microsoft-github-policy-service agree

• (when company given) I am making Submissions in the course of work for my employer (or my employer has intellectual property rights in my Submissions by contract or applicable law). I have permission from my employer to make Submissions and enter into this Agreement on behalf of my employer. By signing below, the defined term “You” includes me and my employer.

@microsoft-github-policy-service agree company="Microsoft"

Contributor License Agreement

Contribution License Agreement

This Contribution License Agreement (“Agreement”) is agreed to by the party signing below (“You”),
and conveys certain license rights to Microsoft Corporation and its affiliates (“Microsoft”) for Your
contributions to Microsoft open source projects. This Agreement is effective as of the latest signature
date below.

1. Definitions.
   “Code” means the computer software code, whether in human-readable or machine-executable form,
   that is delivered by You to Microsoft under this Agreement.
   “Project” means any of the projects owned or managed by Microsoft and offered under a license
   approved by the Open Source Initiative (www.opensource.org).
   “Submit” is the act of uploading, submitting, transmitting, or distributing code or other content to any
   Project, including but not limited to communication on electronic mailing lists, source code control
   systems, and issue tracking systems that are managed by, or on behalf of, the Project for the purpose of
   discussing and improving that Project, but excluding communication that is conspicuously marked or
   otherwise designated in writing by You as “Not a Submission.”
   “Submission” means the Code and any other copyrightable material Submitted by You, including any
   associated comments and documentation.
2. Your Submission. You must agree to the terms of this Agreement before making a Submission to any
   Project. This Agreement covers any and all Submissions that You, now or in the future (except as
   described in Section 4 below), Submit to any Project.
3. Originality of Work. You represent that each of Your Submissions is entirely Your original work.
   Should You wish to Submit materials that are not Your original work, You may Submit them separately
   to the Project if You (a) retain all copyright and license information that was in the materials as You
   received them, (b) in the description accompanying Your Submission, include the phrase “Submission
   containing materials of a third party:” followed by the names of the third party and any licenses or other
   restrictions of which You are aware, and (c) follow any other instructions in the Project’s written
   guidelines concerning Submissions.
4. Your Employer. References to “employer” in this Agreement include Your employer or anyone else
   for whom You are acting in making Your Submission, e.g. as a contractor, vendor, or agent. If Your
   Submission is made in the course of Your work for an employer or Your employer has intellectual
   property rights in Your Submission by contract or applicable law, You must secure permission from Your
   employer to make the Submission before signing this Agreement. In that case, the term “You” in this
   Agreement will refer to You and the employer collectively. If You change employers in the future and
   desire to Submit additional Submissions for the new employer, then You agree to sign a new Agreement
   and secure permission from the new employer before Submitting those Submissions.
5. Licenses.

• Copyright License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license in the
  Submission to reproduce, prepare derivative works of, publicly display, publicly perform, and distribute
  the Submission and such derivative works, and to sublicense any or all of the foregoing rights to third
  parties.
• Patent License. You grant Microsoft, and those who receive the Submission directly or
  indirectly from Microsoft, a perpetual, worldwide, non-exclusive, royalty-free, irrevocable license under
  Your patent claims that are necessarily infringed by the Submission or the combination of the
  Submission with the Project to which it was Submitted to make, have made, use, offer to sell, sell and
  import or otherwise dispose of the Submission alone or with the Project.
• Other Rights Reserved. Each party reserves all rights not expressly granted in this Agreement.
  No additional licenses or rights whatsoever (including, without limitation, any implied licenses) are
  granted by implication, exhaustion, estoppel or otherwise.

6. Representations and Warranties. You represent that You are legally entitled to grant the above
   licenses. You represent that each of Your Submissions is entirely Your original work (except as You may
   have disclosed under Section 3). You represent that You have secured permission from Your employer to
   make the Submission in cases where Your Submission is made in the course of Your work for Your
   employer or Your employer has intellectual property rights in Your Submission by contract or applicable
   law. If You are signing this Agreement on behalf of Your employer, You represent and warrant that You
   have the necessary authority to bind the listed employer to the obligations contained in this Agreement.
   You are not expected to provide support for Your Submission, unless You choose to do so. UNLESS
   REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING, AND EXCEPT FOR THE WARRANTIES
   EXPRESSLY STATED IN SECTIONS 3, 4, AND 6, THE SUBMISSION PROVIDED UNDER THIS AGREEMENT IS
   PROVIDED WITHOUT WARRANTY OF ANY KIND, INCLUDING, BUT NOT LIMITED TO, ANY WARRANTY OF
   NONINFRINGEMENT, MERCHANTABILITY, OR FITNESS FOR A PARTICULAR PURPOSE.
7. Notice to Microsoft. You agree to notify Microsoft in writing of any facts or circumstances of which
   You later become aware that would make Your representations in this Agreement inaccurate in any
   respect.
8. Information about Submissions. You agree that contributions to Projects and information about
   contributions may be maintained indefinitely and disclosed publicly, including Your name and other
   information that You submit with Your Submission.
9. Governing Law/Jurisdiction. This Agreement is governed by the laws of the State of Washington, and
   the parties consent to exclusive jurisdiction and venue in the federal courts sitting in King County,
   Washington, unless no federal subject matter jurisdiction exists, in which case the parties consent to
   exclusive jurisdiction and venue in the Superior Court of King County, Washington. The parties waive all
   defenses of lack of personal jurisdiction and forum non-conveniens.
10. Entire Agreement/Assignment. This Agreement is the entire agreement between the parties, and
    supersedes any and all prior agreements, understandings or communications, written or oral, between
    the parties relating to the subject matter hereof. This Agreement may be assigned by Microsoft.