Add cascaded agent-framework init template; make all init templates importable + type-checked by alexkroman · Pull Request #170 · AssemblyAI/cli

alexkroman · 2026-06-16T02:04:55Z

Summary

Adds a fourth assembly init starter — agent-framework, a server-orchestrated cascaded voice agent (Streaming STT → LLM Gateway → sandbox TTS) — and refactors all four init templates into importable, in-tree, strictly type-checked packages.

New template: `agent-framework`

Same browser UI/UX as voice-agent, but the FastAPI backend wires the three AssemblyAI primitives together itself. All three credentials stay server-side; the browser holds one /ws.
Conversation memory — per-session sliding window (MAX_HISTORY) threaded into every LLM call.
Per-sentence TTS streaming — flushes each . ! ?-terminated sentence to TTS as the LLM streams, cutting time-to-first-audio.
Sandbox-only (streaming TTS has no prod host): scaffold with assembly --sandbox init agent-framework; deploy via the shipped Procfile/Dockerfile (long-lived WS, not Vercel-serverless).
Live-QA fixes: TTS-readable system prompt (no markdown/emoji), valid TTS preset voice, Flush (not ForceFlushTextBuffer), and a guard for the gateway's choice-less final chunk.

Other template changes

live-captions — mints its streaming token via the SDK's StreamingClient.create_temporary_token instead of raw httpx2.
assembly init — the interactive picker now shows a one-line description per template.

Template architecture refactor

Templates are now importable packages and type-checked in-tree instead of excluded scaffold blobs:

Dirs renamed to legal module names (agent-framework → agent_framework, …); kebab IDs unchanged via templates.dir_for().
api/ modules use relative imports, so the same code works as the shipped top-level api/ (uvicorn api.index:app) and as aai_cli.init.templates.<name>.api.* in-tree. The registry templates.py merged into templates/__init__.py; scaffold._copy_tree skips repo-only package markers.
Dropped the mypy + pyright template excludes; every template's api/ code is strict-clean against the real SDK types (websockets ClientConnection, openai ChatCompletionMessageParam, fastapi WebSocket, explicit TranscriptionConfig kwargs) with zero new # type: ignore / Any / cast. deptry's exclude stays (templates carry their own requirements.txt).

Tooling / editor

Added a validate-pyproject gate step (via uvx; no dev-dep/lock change).
.vscode/settings.json + a pyright tests execution-environment so the editor (Pylance/Mypy) matches the gate.

Test plan

./scripts/check.sh passes end-to-end (ruff, mypy, pyright src+tests, vulture, deptry, import-linter, xenon, contract gates, 90% + 100% patch coverage, mutation, escape-hatch, CodeQL, build + twine check).

Notes for reviewers

Large branch spanning a few themes (new template, SDK migration, init UX, importability/type-check refactor, tooling). Happy to split into stacked PRs.
Design/plan docs: docs/superpowers/specs/2026-06-15-agent-framework-template-design.md, docs/superpowers/plans/2026-06-15-agent-framework-template.md.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…s.real Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…lture) Yield one delta before raising/blocking so no line is unreachable and no pragma: no cover is needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Extract shared loaders/fakes into tests/_agent_framework.py; split tests into core + api modules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…var) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Starter apps funnel any leg failure into one user-facing error event, so the blind-except lint doesn't apply to scaffolds; move it to the template per-file ignore (with S105/TID251) rather than adding net-new inline noqa hatches. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

… + tests Replace Any with _Settings/_Socket/_Browser Protocols in the template (mirrors tts/session.py) and route dynamic template-module access through untyped test helpers, so the no-new-escape-hatches Any gate stays at the merge-base count. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…xtBuffer' The sandbox streaming-TTS server's tagged union accepts Generate/Flush/Terminate/ KeepAlive/Cancel; the old ForceFlushTextBuffer tag is rejected (union_tag_invalid), which surfaced as a session error on first synthesis during manual QA. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Replace the raw httpx2 GET /v3/token with StreamingClient.create_temporary_token; drop httpx2, add assemblyai. Update the dedicated + serve tests to mock the SDK. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…n (<1) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ot ivy) 'ivy' is a Voice Agent voice; the streaming-TTS server rejects it. Default to 'jane' (the CLI's default English voice, a valid TTS preset). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tration - Display transcript.user only for the finalized (formatted) turn; interim turns just barge in, no partial display. - Replace the manual mic/listen race (asyncio.wait + cancel/gather) with an asyncio.TaskGroup; a _SessionClosed sentinel unwinds it when either side hangs up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

The Anthropic-backed LLM gateway ends a streamed completion with a usage/final chunk that has an empty choices list; chunk.choices[0] then IndexErrors, which _generate_reply turns into session.error -> the browser hangs up. The greeting is TTS-only so it's unaffected, which is why the session died right after the first STT result, before the first reply's TTS. Skip choice-less chunks. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

- Add templates.DESCRIPTIONS + description_for(); show it under each choice in the interactive picker (questionary Choice description). - agent-framework SYSTEM_PROMPT now tells the model to write plain spoken prose (no markdown/emoji/lists/code) since the reply is read aloud by TTS. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Reading choices back via a dict[str, object] tripped pyright ('object is not iterable'); record them on the returned SimpleNamespace instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Keep a per-session sliding-window history (MAX_HISTORY) so the agent remembers prior turns. Stream the LLM reply into TTS sentence-by-sentence (flush on . \! ?) instead of buffering the whole reply, cutting perceived latency to first audio. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Pylance reads [tool.pyright] (strict, include=aai_cli) and also analyzes open test files, flooding them with strict-only fixture/mock diagnostics the gate never raises (it checks tests in standard mode via pyrightconfig.tests.json). Add a tests/ execution environment that disables the strict-only 'unknown type'/private-usage/unused/bare- generic family so the editor matches the gate. Editor-facing only — the gate's pyright run never analyzes tests/ (not in include). Add .vscode/settings.json pointing at .venv so imports resolve. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…ple id↔dir Templates become importable-shaped (legal module dirs) while the init IDs stay kebab. templates.dir_for() maps id->dir; scaffold resolves the source via it; the output dir stays the kebab id. No import or type-check-exclude changes yet (P2/P3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…+ markers) Merge the registry templates.py into templates/__init__.py (resolving the templates.py-vs-templates/ name collision), add package markers to each template dir, and switch the api/ modules to relative imports — so each template imports as aai_cli.init.templates.<name>.api.* for in-tree type-checking, while still shipping as a self-contained top-level api/ package. scaffold._copy_tree skips the repo-only root __init__.py (keeping api/__init__.py). Allow TID252 (relative imports) for templates. No type-check-exclude changes yet (P3). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…des) Remove the mypy + pyright template excludes and make every template's api/ code strict-clean against the real SDK types — zero new Any / type: ignore / cast: - agent_framework: annotate settings constants as str/int (satisfy _Settings), type STT/TTS sockets as websockets ClientConnection, messages as ChatCompletionMessageParam, FastAPIBrowser via fastapi.WebSocket. - audio_transcription: replace the untypable **dict[str,bool] TranscriptionConfig unpack with explicit boolean feature kwargs; serialize the transcript via a small Protocol. (live_captions/voice_agent were already clean.) deptry keeps its templates exclude (they carry their own requirements.txt). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

…tor noise Add a validate-pyproject gate step (uvx, like twine/codespell — no dev-dep/uv.lock entry) so a malformed [build-system]/[project] table fails the gate, not just the build. Disable Even Better TOML's schema validation in .vscode: its bundled pyproject schema's [tool.ruff] definition is stale and wrongly rejects our valid config (ruff and validate-pyproject both accept it); the gate now validates pyproject authoritatively. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

pyright warns when a config defines exclude without its built-in defaults, so list **/node_modules, **/__pycache__, **/.* explicitly (templates stay IN scope — proven by an injected error still being caught). Add mypy-type-checker.importStrategy=fromEnvironment so the VS Code Mypy extension uses the project's mypy + keyring stubs and matches the gate (which is clean on conftest.py — the bundled-mypy false positive doesn't reflect our config). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

app.js: the server speaks the greeting the instant the WebSocket opens, so `reply.audio` can arrive before `startMic()`'s getUserMedia permission prompt resolves — at which point `player` was still null and `player.playBase64Chunk` threw (Aikido finding). Create the PcmPlayer synchronously before the mic await so it always exists when the first audio frame is handled, and guard the `reply.audio` branch defensively. cascade.py: give the `_Browser` Protocol's `send`/`recv` stubs docstring bodies instead of `...`, clearing the CodeQL "statement has no effect" notes (and documenting the protocol). The `await <name>` notes are CodeQL false positives — their `_ = await …` autofix breaks our strict mypy (func-returns-value on a None-returning await), so those stay as-is. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

+        if task is not None and not task.done():
+            task.cancel()
+            with contextlib.suppress(asyncio.CancelledError, Exception):
+                await task


+        task = self.reply_task
+        if task is not None:
+            with contextlib.suppress(Exception):
+                await task


Addresses the /code-review findings on cascade.py's reply lifecycle — the same silent-failure / premature-close bug family as the earlier greeting fix, now on the greeting, barge-in, and conversation-memory paths: - _speak (greeting) now wraps connect/synthesize in try/except like _generate_reply, so a TTS failure surfaces as one session.error instead of a swallowed, never-retrieved task exception (silent dead greeting). - A finalized user turn now calls maybe_barge_in() instead of a bare cancel_reply(), so the browser gets input.speech.started and flushes the still-playing reply's queued audio (server-side cancel alone left old TTS playing). The dead reply.done "interrupted" branch in app.js is removed. - The greeting is seeded into history, so the model has a record of its opening line on every subsequent turn. - _generate_reply records the spoken-so-far text when cancelled mid-reply, so a barge-in can't leave history with two consecutive user turns. - _synthesize iterates the TTS socket (like _pump_stt) so a close before the final Audio frame ends the loop cleanly instead of raising ConnectionClosed. - _pump_mic skips a malformed-base64 frame instead of crashing the session. Splits the reply-path tests into test_init_template_agent_framework_reply.py to stay under the 500-line file gate. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

+
+async def _until_closed(pump: Awaitable[None]) -> None:
+    """Run a pump to its natural end, then raise to close the session TaskGroup."""
+    await pump


+        await asyncio.sleep(0)
+        task.cancel()
+        with pytest.raises(asyncio.CancelledError):
+            await task


+            await asyncio.sleep(0)  # let it stream + synthesize the sentence, then block
+        task.cancel()
+        with pytest.raises(asyncio.CancelledError):
+            await task


alexkroman-assembly and others added 30 commits June 15, 2026 11:50

docs: design for agent-framework init template

9d2bb81

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

docs: implementation plan for agent-framework template

88a32f2

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(init): register agent-framework template + inject TTS host

b71fc29

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(agent-framework): skeleton + shared static assets

7f432cf

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(agent-framework): settings module

43a8945

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(agent-framework): assert settings defaults (mutation coverage)

78bd66e

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(agent-framework): cascade pure helpers

e5dbecf

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(agent-framework): assert speech_model param (mutation coverage)

2667921

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(agent-framework): cascade orchestrator

58671d6

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(agent-framework): cover live-client adapters + cancel/skip branches

d21ade0

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(agent-framework): FastAPI app + websocket adapter

c515e18

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(agent-framework): assert ws handler wires real settings into Dep…

0d78e7f

…s.real Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(agent-framework): frontend (cascade UI + /ws client)

409d2d9

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

feat(agent-framework): deploy, docs, and dependency scaffold

d998b75

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(init): refresh --help snapshot for agent-framework template

b347f6c

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(agent-framework): satisfy mypy on dynamic template-module imports

e7a0df5

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(agent-framework): allow None queue elements in FakeBrowser inbound

9e9e2f3

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(agent-framework): make async-generator fakes fully reachable (vu…

2d6b099

…lture) Yield one delta before raising/blocking so no line is unreachable and no pragma: no cover is needed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(agent-framework): split oversized test file under the 500-line gate

d99d277

Extract shared loaders/fakes into tests/_agent_framework.py; split tests into core + api modules. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

chore(docs-gate): allowlist ASSEMBLYAI_TTS_HOST (template-deployment …

da30455

…var) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

refactor(live-captions): rename shadowing local + align assemblyai pi…

f151a6b

…n (<1) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

test(init): record picker choices on the fake namespace (pyright-clean)

ea2707d

Reading choices back via a dict[str, object] tripped pyright ('object is not iterable'); record them on the returned SimpleNamespace instead. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

alexkroman-assembly and others added 9 commits June 15, 2026 16:34

fix(template-gate): skip __pycache__ now that template dirs are packages

379e262

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

update

7607a9e

aikido-pr-checks Bot reviewed Jun 16, 2026

View reviewed changes

Comment thread aai_cli/init/templates/agent_framework/static/app.js Outdated

alexkroman enabled auto-merge June 16, 2026 02:06

github-code-quality Bot found potential problems Jun 16, 2026

View reviewed changes

alexkroman disabled auto-merge June 16, 2026 02:08

github-code-quality Bot found potential problems Jun 16, 2026

View reviewed changes

alexkroman enabled auto-merge June 16, 2026 03:23

alexkroman added this pull request to the merge queue Jun 16, 2026

Merged via the queue into main with commit 3561950 Jun 16, 2026
19 checks passed

alexkroman deleted the feat-agent-framework-template branch June 16, 2026 03:29

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add cascaded agent-framework init template; make all init templates importable + type-checked#170

Add cascaded agent-framework init template; make all init templates importable + type-checked#170
alexkroman merged 41 commits into
mainfrom
feat-agent-framework-template

alexkroman commented Jun 16, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

alexkroman commented Jun 16, 2026

Summary

New template: agent-framework

Other template changes

Template architecture refactor

Tooling / editor

Test plan

Notes for reviewers

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

New template: `agent-framework`