diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 2a3a73e2..37c13529 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -33,10 +33,16 @@ ordered by latency cost (lowest first) and detection authority (highest first): 1. **SDK layer (in-process)** — *this repo*. The SDK applies pre-execution allow/deny on tool calls via the native shim over `aa-sdk-client`. Fastest path; requires SDK - adoption. It does **not** emit audit events: the adapters offer every governed - outcome to an audit hook, but on every interceptor this SDK ships that hook does not - resolve, so nothing is recorded — for allowed calls as much as denied ones - (AAASM-5731). Do not describe this layer as producing an audit trail. + adoption. It hands audit events to the runtime **only over a connected runtime**: + the adapters offer a governed outcome to an audit hook, and + `RuntimeQueryInterceptor` writes it to the native event channel. Two limits are + load-bearing and must not be dropped when describing this: the send is + unacknowledged, so a handoff is **not** evidence and never ADR 0033 §6 + *Observed* — AAASM-5783 is open on the downstream half and must land before + that changes; and only `google_adk`, `pydantic_ai` and `openai_agents` record on the + **denied** path — the other eight governed adapters return or raise first. With no + reachable runtime nothing is recorded at all (AAASM-5750). Never describe this + layer as producing an audit trail. 2. **Sidecar proxy (`aa-proxy`)** — MitM of outbound HTTPS; enforces network-egress policy with no code changes. (Lives in the monorepo.) 3. **eBPF (`aa-ebpf*`)** — kernel uprobes; catches everything, including bypass diff --git a/README.md b/README.md index 0bdaf6d1..206bc972 100644 --- a/README.md +++ b/README.md @@ -13,13 +13,15 @@ Python SDK for **AI Agent Assembly** — a governance-native runtime for AI agents. One `init_assembly()` call wires your agent into the policy gateway and applies pre-execution allow/deny on tool calls, without changing how the agent itself is written. -> **The SDK layer produces no audit evidence of its own.** The framework adapters offer the outcome of every governed call to an audit hook on the governance interceptor — but on every interceptor this SDK ships, that hook **does not resolve**, so nothing is emitted. This covers **allowed** calls as much as denied ones. Enforcement is unaffected: a policy DENY still blocks the tool. `init_assembly()` warns about it and reports `audit_sink` on the returned context; supply your own handler with a `record_result` or `on_tool_end` to retain the record ([AAASM-5731](https://lightning-dust-mite.atlassian.net/browse/AAASM-5731)). +> **The SDK hands records to the runtime; it does not give you an audit trail.** The framework adapters offer a governed call's outcome to an audit hook on the governance interceptor, and over a connected runtime that hook writes it to the native event channel — the same one agent registration uses. That is a handoff, **not** evidence: the send is unacknowledged, so this SDK cannot tell you the record arrived, and does not claim it did. Downstream, [AAASM-5783](https://lightning-dust-mite.atlassian.net/browse/AAASM-5783) is open on `report_event` payloads reaching neither the live stream nor the durable entry — until it lands, no SDK can claim ADR 0033 §6 *Observed*. Without a reachable runtime there is no channel at all and nothing is emitted. +> +> **Denied calls are mostly not covered.** Only `google_adk`, `pydantic_ai` and `openai_agents` build a record on the denied path. The other eight governed adapters — `crewai`, `llamaindex`, `haystack`, `agno`, `smolagents`, `microsoft_agent_framework`, `mcp` and `langchain` — return or raise before their record helper, so a deny there produces no record for any sink to carry. Enforcement is unaffected either way: a policy DENY still blocks the tool. `init_assembly()` warns when no record can be sent and reports `audit_sink` on the returned context ([AAASM-5750](https://lightning-dust-mite.atlassian.net/browse/AAASM-5750)). ## Why use it - **Framework adapters** for LangChain, LangGraph, CrewAI, OpenAI Agents, Pydantic AI, Google ADK, Haystack, Smolagents, Agno, LlamaIndex, Microsoft Agent Framework, and MCP servers — drop in, no SDK rewrites required. - **Pre-execution policy enforcement** via the `FrameworkAdapter` ABC — block disallowed tool calls before they hit the LLM. -- **Agent lineage** — parent / root / team identity is registered with the gateway and carried on every policy check. (An audit *trail* is not part of what this SDK layer delivers — see the note above.) +- **Agent lineage** — parent / root / team identity is registered with the gateway and carried on every policy check. (An audit *trail* from this SDK layer depends on a reachable runtime — see the note above.) - **Native PyO3 fast path** (optional) — drop into a Rust runtime client when you need sub-millisecond policy checks. - **Typed throughout** — Pydantic models for every gateway payload, mypy strict on adapter base and registry. diff --git a/agent_assembly/adapters/_shared/tool_governance.py b/agent_assembly/adapters/_shared/tool_governance.py index 0bc9f1f2..affba47b 100644 --- a/agent_assembly/adapters/_shared/tool_governance.py +++ b/agent_assembly/adapters/_shared/tool_governance.py @@ -6,9 +6,10 @@ verdict, honour a ``pending`` approval round-trip, deny by raising when the verdict is ``deny``, otherwise run the original inside a spawn-context scope. Either way the outcome is *offered* to the audit hook before the flow ends -(AAASM-5665) — offered, not recorded: on every interceptor this SDK ships the hook -does not resolve, so nothing is retained on either path. See -:func:`_record_async_tool_result` for the measurement. That shared body — previously duplicated verbatim in both +(AAASM-5665). Whether the hook resolves depends on the interceptor: over a +connected runtime it does, and the record is forwarded to the runtime's evidence +pipeline (AAASM-5750); without one it does not, and nothing is retained on either +path. See :func:`_record_async_tool_result` for the measurement. That shared body — previously duplicated verbatim in both adapters (the cross-file duplication SonarCloud flagged on PR #269, AAASM-4746) — lives here so each adapter keeps only its framework-specific glue. @@ -178,22 +179,22 @@ async def _record_async_tool_result( apart from a tool that ran and returned that same text. Whether anything is recorded depends entirely on the ``callback_handler``. - Both hooks are duck-typed, and on every interceptor the SDK ships *neither - resolves*: ``RuntimeQueryInterceptor`` defines only ``check_tool_start`` and - delegates the rest to ``GatewayClient``, whose surface has no - ``record_result`` and no ``on_tool_end``. Measured against the native and - HTTP boundaries, this function therefore finds no hook and emits nothing on - the shipped path — for allowed calls as much as denied ones. - - Under ADR 0033 §6 that makes SDK-side recording **Planned** (AAASM-5750), - not *Unmeasured*: §6 reserves ``Unmeasured`` for an action no control - inspected, where nothing is known, and here exactly where the record stops - has been measured. It is certainly not *Observed*, which needs a durable - event attributed to the action. Every handler the SDK ships declares this in - ``audit_sink`` (see :mod:`agent_assembly.core.audit_sink`), ``init_assembly`` - warns about it, and a caller that supplies its own handler does get the - record. Wiring a sink into the SDK's own interceptor is a separate - capability. + Both hooks are duck-typed. Over a connected runtime + ``RuntimeQueryInterceptor.record_result`` resolves and writes the record to + the native event channel (AAASM-5750). That is a handoff and **not** ADR 0033 + §6 *Observed*: the send is unacknowledged, so nothing here establishes a + durable event attributed to the action, and AAASM-5783 is open on the + downstream half of that. Without a runtime neither hook resolves and this + function emits nothing, measured against the native and HTTP boundaries. + + Note the scope of "allowed or denied" here: *this* shared flow calls the hook + on both paths, which is why ``google_adk`` and ``pydantic_ai`` cover denies. + Most adapters do not route through it and return or raise before their own + record helper, so their denied calls produce no record at all. + + Which of the two a run is in is declared in ``audit_sink`` (see + :mod:`agent_assembly.core.audit_sink`) and warned about by ``init_assembly``; + a caller that supplies its own handler gets the record either way. """ denial_flag = {"denied": denied} if denied else {} @@ -295,9 +296,9 @@ async def run_governed_async_tool( # Offer the deny to the audit hook before raising (AAASM-5665). # Previously this raised straight past the record call below, so a # denied call could not reach an audit sink even when the caller had - # supplied one. See _record_async_tool_result on why the SDK's own - # interceptor still resolves no hook, so the shipped path emits nothing - # here either (AAASM-5731). + # supplied one. See _record_async_tool_result for where the record then + # goes: the SDK's own interceptor resolves the hook over a connected + # runtime and forwards it, and resolves nothing without one (AAASM-5750). # # Best-effort, and the guard is load-bearing: the hook is duck-typed # from caller-supplied code, and inserting a call here where none used diff --git a/agent_assembly/adapters/haystack/patch.py b/agent_assembly/adapters/haystack/patch.py index 40605361..2b011441 100644 --- a/agent_assembly/adapters/haystack/patch.py +++ b/agent_assembly/adapters/haystack/patch.py @@ -13,8 +13,10 @@ The interceptor contract mirrors the other tool-call adapters (CrewAI, Pydantic AI): a ``check_tool_start`` pre-execution gate that returns ``allow`` / ``deny`` / ``pending``, an optional ``wait_for_tool_approval`` for the pending flow, and a -post-execution ``record_result`` / ``on_tool_end`` audit hook — which no interceptor -this SDK ships resolves, so the outcome is offered and not recorded (AAASM-5731). +post-execution ``record_result`` / ``on_tool_end`` audit hook — which the SDK's own +interceptor resolves over a connected runtime, handing the outcome to the +runtime's event channel, and does not resolve without one. Note this adapter does +not reach that hook on the denied path: it raises first (AAASM-5750). Under the fail-closed ``enforce`` posture an unknown or malformed verdict denies (AAASM-3107). """ diff --git a/agent_assembly/adapters/langchain/callback_handler.py b/agent_assembly/adapters/langchain/callback_handler.py index 2f2cbf18..94cbb2e2 100644 --- a/agent_assembly/adapters/langchain/callback_handler.py +++ b/agent_assembly/adapters/langchain/callback_handler.py @@ -59,10 +59,14 @@ def audit_sink(self) -> AuditSinkDisposition: wrapped, and this handler sits on the *other* side of the split from the interceptors it wraps. ``on_tool_end`` is defined here, so the adapters' audit-hook lookup **does** resolve on this object — the record is built - and handed over. It is then forwarded to the interceptor's own - ``on_tool_end``, and on every interceptor this SDK ships there is none, - so the record stops here: accepted and dropped, which is ``discarded``, - not ``absent``. + and handed over — and it is then forwarded to the interceptor's own + ``on_tool_end``. + + So the answer is the wrapped interceptor's, with one substitution: when + the wrapped one resolves no hook at all, the record still reaches *this* + object before stopping, which is ``discarded`` and not ``absent``. The + distinction is not cosmetic — ``absent`` says nothing constructs the + event, and here something does. A caller-supplied interceptor that really records is reported as such: this SDK does not claim anything about a handler it did not build, in diff --git a/agent_assembly/adapters/llamaindex/adapter.py b/agent_assembly/adapters/llamaindex/adapter.py index dc39f8f4..d2cd4020 100644 --- a/agent_assembly/adapters/llamaindex/adapter.py +++ b/agent_assembly/adapters/llamaindex/adapter.py @@ -11,8 +11,10 @@ class LlamaIndexAdapter(FrameworkAdapter): Wires the SDK-layer pre-execution allow/deny onto the LlamaIndex tool-execution path (``FunctionTool.call`` / ``acall``), and offers each - outcome to the audit hook — which no interceptor this SDK ships resolves, so - nothing is recorded from here (AAASM-5731). The + outcome to the audit hook — which the SDK's own interceptor resolves over a + connected runtime, handing the record to the runtime's event channel, and + does not resolve without one. Only the *allowed* path reaches it here: a deny + raises before the record helper, so it produces no record (AAASM-5750). The framework package is imported as ``llama_index.core``; the patch targets the concrete tool methods the agent loop actually invokes (the base methods are abstract). diff --git a/agent_assembly/core/assembly.py b/agent_assembly/core/assembly.py index 2a3e1c55..82f6ba65 100644 --- a/agent_assembly/core/assembly.py +++ b/agent_assembly/core/assembly.py @@ -124,12 +124,16 @@ class AssemblyContext: # (AAASM-4547, mirroring the Node SDK's ``ctx.registered``). registered: bool = True # What the governance interceptor the adapters were handed does with the - # hook-layer audit record for a governed tool call (AAASM-5731). Anything - # other than ``"caller-supplied"`` means governed actions produce NO audit - # evidence from this SDK, so no claim of attributability or after-the-fact - # review holds on the SDK path. The programmatic counterpart of the stderr - # warning ``_warn_audit_not_recorded`` emits, so the gap is detectable in - # code and not only by reading stderr. + # hook-layer audit record for a governed tool call (AAASM-5731). + # ``"forwarded"`` means the record is handed to the runtime's event channel + # (a handoff, not evidence — the send is unacknowledged); + # ``"absent"`` and ``"discarded"`` both mean governed actions produce NO + # audit evidence from this SDK, so no claim of attributability or + # after-the-fact review holds on that path; ``"caller-supplied"`` means this + # SDK makes no claim. The programmatic counterpart of the stderr warning + # ``_warn_audit_not_recorded`` emits for the two evidence-free values, so + # which case a run is in is detectable in code and not only by reading + # stderr. audit_sink: AuditSinkDisposition = AUDIT_SINK_ABSENT _lock: Lock = field(default_factory=Lock, init=False, repr=False) _is_shutdown: bool = field(default=False, init=False, repr=False) @@ -325,7 +329,14 @@ def init_assembly( # AAASM-5731 — surface an audit path that retains nothing, on the # default path with nothing opted into. Emitted after registration so it # reflects the interceptor the adapters were actually handed. - if audit_sink != AUDIT_SINK_CALLER_SUPPLIED: + # + # The condition enumerates the dispositions that warrant a warning rather + # than excluding the one that does not (AAASM-5750). Written as + # ``!= AUDIT_SINK_CALLER_SUPPLIED`` it warned about every value that was + # not the caller's own, which silently included ``forwarded`` the moment + # that value existed — telling a caller whose records do reach the runtime + # that they produce none. + if audit_sink in (AUDIT_SINK_ABSENT, AUDIT_SINK_DISCARDED): _warn_audit_not_recorded(audit_sink) context = AssemblyContext( @@ -397,15 +408,18 @@ def _warn_agent_unregistered(detail: str) -> None: def _warn_audit_not_recorded(disposition: AuditSinkDisposition) -> None: - """Emit a loud, unconditional stderr warning that no audit record is kept. + """Emit a loud stderr warning that no audit record is kept on this run. The framework adapters offer the outcome of every governed tool call to an - audit hook on the interceptor they were handed. On every interceptor this SDK - ships that hook does not resolve, so nothing is emitted — for **allowed** - calls as much as denied ones — and the caller had no way to learn that short - of reading the interceptor. Enforcement is genuinely unaffected, which is - exactly why the gap is easy to miss: denies still deny, and the governed call - returns normally. + audit hook on the interceptor they were handed. Over a connected runtime that + hook resolves and forwards the record (AAASM-5750); without one it does not + resolve at all, so nothing is emitted — for **allowed** calls as much as + denied ones — and the caller had no way to learn that short of reading the + interceptor. Enforcement is genuinely unaffected, which is exactly why the gap + is easy to miss: denies still deny, and the governed call returns normally. + + Fires only for the dispositions that leave no evidence; see the enumeration at + the call site in :func:`init_assembly`. Written straight to ``sys.stderr`` for the same reason as :func:`_warn_agent_unregistered`: ``logging`` configuration cannot silence it. @@ -425,7 +439,8 @@ def _warn_audit_not_recorded(disposition: AuditSinkDisposition) -> None: "the interceptor it forwards to exposes no on_tool_end" if disposition == AUDIT_SINK_DISCARDED else "no audit hook (record_result / on_tool_end) resolves on the governance " - "interceptor this SDK builds, so no record is even attempted" + "interceptor this SDK built, because no runtime is reachable for it to " + "send a record to, so no record is even attempted" ) sys.stderr.write( "[agent-assembly] WARNING: hook-layer audit records are NOT retained " @@ -433,10 +448,11 @@ def _warn_audit_not_recorded(disposition: AuditSinkDisposition) -> None: "ones as well as denied ones — therefore produce NO audit evidence from " "this SDK, and nothing on this path can be attributed or reviewed after " "the fact. Enforcement is unaffected: a policy DENY still blocks a tool " - "call, and the proxy / eBPF layers remain authoritative. Supply your own " - "handler with a record_result or on_tool_end to retain the record, and " - "inspect the 'audit_sink' attribute on the returned assembly context to " - "detect this programmatically (AAASM-5731).\n" + "call, and the proxy / eBPF layers remain authoritative. Connect a runtime so " + "the SDK's own sink resolves, or supply your own handler with a " + "record_result or on_tool_end, and inspect the 'audit_sink' attribute on " + "the returned assembly context to detect this programmatically " + "(AAASM-5731, AAASM-5750).\n" ) diff --git a/agent_assembly/core/audit_sink.py b/agent_assembly/core/audit_sink.py index c079cd0f..4ac0bdfa 100644 --- a/agent_assembly/core/audit_sink.py +++ b/agent_assembly/core/audit_sink.py @@ -6,31 +6,41 @@ and both return ``None``, so a handler that retains the record and one that does nothing with it are indistinguishable at the call site. -On every interceptor this SDK ships, neither hook **resolves at all**: -``RuntimeQueryInterceptor`` defines only ``check_tool_start`` and delegates the -rest to :class:`~agent_assembly.client.gateway.GatewayClient`, whose surface has -no ``record_result`` and no ``on_tool_end``. The adapters' ``getattr`` guard -therefore finds nothing and returns without emitting — for **allowed** calls as -much as denied ones. Nothing in ``agent_assembly`` calls the native -``RuntimeClient.send_event`` either, so no tool-call event reaches the runtime by -any other route. - -This module is how that stops being invisible. Every handler the SDK ships -declares its disposition; :func:`resolve_audit_sink` reads it; ``init_assembly`` -warns about it and reports it on the returned context. - -Under ADR 0033 §6 this makes SDK-side recording **Planned** (AAASM-5750), not -*Observed* — *Observed* requires a durable event attributed to the action, and -there is none. It is deliberately not *Unmeasured*: §6 reserves that for an -action no control inspected, where nothing is known, and here exactly where the -record stops has been measured against the native boundary. +``RuntimeQueryInterceptor`` defines ``record_result``, which forwards the record +to the runtime over the native event channel (AAASM-5750). Until that landed +neither hook resolved at all on any interceptor this SDK ships — the ``getattr`` +guard found nothing and returned without emitting, for **allowed** calls as much +as denied ones. + +Which of the two a given run is in still depends on the run, so the declaration +is not decoration. An interceptor built without a native runtime — the +fail-closed one, or a build with no extension — has no channel to send on and +still resolves no hook. This module is how the difference stops being invisible: +every handler the SDK ships declares its disposition; :func:`resolve_audit_sink` +reads it; ``init_assembly`` warns about it and reports it on the returned +context. + +The ADR 0033 §6 term follows the disposition rather than the SDK as a whole: + +* :data:`AUDIT_SINK_FORWARDED` — the event is handed to the runtime's event + channel. **Not** *Observed*: the send is unacknowledged, so no durable event + attributed to the action is established from this side. The downstream half of + that gap is tracked as AAASM-5783 and is unfixed — until ``report_event`` + payloads reach the live stream and the durable entry, no SDK can claim + *Observed*. +* :data:`AUDIT_SINK_ABSENT` — the control is configured and its channel is + unavailable, which is *Degraded*; deliberately not *Unmeasured*, since §6 + reserves that for an action no control inspected, and here exactly where the + record stops has been measured against the native boundary. +* :data:`AUDIT_SINK_DISCARDED` — a hook resolves and drops the record, so no + evidence exists on that path either. """ from __future__ import annotations from typing import Any, Literal, get_args -type AuditSinkDisposition = Literal["absent", "discarded", "caller-supplied"] +type AuditSinkDisposition = Literal["forwarded", "absent", "discarded", "caller-supplied"] """What a governance handler does with the hook-layer audit record. The vocabulary separates *how* a record fails to survive, not merely that it @@ -39,10 +49,32 @@ misdescribe at least one of them. """ +AUDIT_SINK_FORWARDED: AuditSinkDisposition = "forwarded" +"""An audit hook resolves and hands the record to the runtime. + +The record crosses the native event channel — the same +``RuntimeClient.send_event`` primitive and the same connected session the agent +registration already uses. + +It says "forwarded", not "recorded", on purpose, and the gap between those two +words is wider than it looks. The channel is fire-and-forget and unacknowledged, +so this SDK never learns whether the runtime received the event; what the runtime +and the gateway behind it retain is theirs to state. **This value is not evidence +and does not earn ADR 0033 §6 *Observed*.** Closing that needs AAASM-5783, which +is open: today ``report_event`` payloads reach neither the live stream nor the +durable entry. + +Its coverage is also uneven across adapters, which the disposition cannot express +because it is a property of the client, not of the call site: every governed +adapter reaches the hook on the **allowed** path, but only ``google_adk``, +``pydantic_ai`` and ``openai_agents`` build a record on the **denied** one. +""" + AUDIT_SINK_ABSENT: AuditSinkDisposition = "absent" """No audit hook resolves on this handler, so no record is even attempted. -This is what every interceptor this SDK ships does. It is strictly worse than +This is what an interceptor built without a reachable native runtime does — the +fail-closed one, and any build with no extension. It is strictly worse than ``"discarded"``: nothing constructs the event, so supplying a sink downstream is not sufficient on its own — the call site finds no hook to call. It is also why the gap covers the **allowed** path and not only the denied one. @@ -53,16 +85,20 @@ The call site is correct and the sink is not. This is what the LangChain :class:`~agent_assembly.adapters.langchain.callback_handler.AssemblyCallbackHandler` -does when the interceptor it wraps has no ``on_tool_end`` to forward to, and it -is what the Go and Node SDKs' shipped clients do (AAASM-5731 / AAASM-5681). +does when the interceptor it wraps has no ``on_tool_end`` to forward to. + +It used to add that the Go and Node SDKs' shipped clients do the same. That was +true when AAASM-5731 / AAASM-5681 measured it and stops being true when their +AAASM-5750 counterparts land, so the cross-SDK comparison is dropped rather than +left to go stale — each SDK's own ``AuditSinkDisposition`` is the answer for it. """ AUDIT_SINK_CALLER_SUPPLIED: AuditSinkDisposition = "caller-supplied" """The handler did not come from this SDK, so this SDK claims nothing about it. -The **absence of a claim, not an assurance** that the record is retained. An -*Observed* claim for the hook layer is available only on this branch, and only -if the caller's own handler actually keeps what it is given. +The **absence of a claim, not an assurance** that the record is retained. +Whichever §6 term the caller's own handler earns is the caller's to establish, +not this SDK's to assert. """ AUDIT_HOOK_NAMES = ("record_result", "on_tool_end") @@ -97,8 +133,9 @@ def resolve_delegated_audit_sink(delegate: Any) -> AuditSinkDisposition: * no hook resolves on ``delegate`` — nothing can be attempted through this interceptor either, so :data:`AUDIT_SINK_ABSENT`; - * a hook resolves — this SDK ships no client that has one, so the hook came - from the caller and this SDK makes no claim about it: + * a hook resolves — this SDK builds no *client* that has one (the sink lives + on the interceptor, not on the wrapped client), so the hook came from the + caller and this SDK makes no claim about it: :data:`AUDIT_SINK_CALLER_SUPPLIED`. Note the failure direction if this is ever wrong: it under-claims. Reporting diff --git a/agent_assembly/core/runtime_audit.py b/agent_assembly/core/runtime_audit.py new file mode 100644 index 00000000..9933ca82 --- /dev/null +++ b/agent_assembly/core/runtime_audit.py @@ -0,0 +1,162 @@ +"""Hand a governed tool call's outcome to the runtime's event channel. + +The framework adapters build an audit record for every governed call — allowed +or denied — and offer it to an audit hook on the interceptor they were handed. +Until AAASM-5750 no interceptor this SDK ships had one, so the record was built +and never emitted. This module is the sink that closes that: it encodes the +outcome and hands it to the native ``RuntimeClient.send_event``, the same +primitive and the same connected session that ``register_agent`` already uses. + +**What the SDK can and cannot say about the result.** ``send_event`` is +fire-and-forget over a bounded IPC channel: it hands the event to the runtime and +returns, with no acknowledgement. **That handoff is the entire claim, and it is +not ADR 0033 §6 *Observed*.** §6 requires a durable event attributed to the +action; nothing observable from this side establishes one, so this module must +not be cited as evidence that a governed call was recorded. The downstream gap is +tracked as AAASM-5783 — until ``report_event`` payloads reach the live stream and +the durable entry, that will stay true no matter what this module sends. Retention past the +boundary belongs to the runtime and the gateway behind it, and asserting their +outcome from here would broaden a claim this layer cannot see (ADR 0034). + +**The identity fields are placeholders, not attribution.** The native +``GovernanceEvent`` constructor validates its argument as ``aa_core::AuditEntry`` +JSON, so the payload has to carry that struct's shape — including ``agent_id``, +``session_id`` and the two chain hashes. Only ``event_type`` is read from it (to +tag the event) and the JSON as a whole travels as an opaque ``details`` label; +none of those four fields reaches the wire as an identity or a chain link. They +are zero-filled rather than fabricated, and the agent identity the SDK actually +knows travels inside ``payload`` where it is plainly a claim by the SDK. The +authoritative attribution is the runtime's, derived from the verified identity +of the IPC connection the event arrived on — not from anything written here. +""" + +from __future__ import annotations + +import json +import time +from typing import Any, Final + +#: Zero-filled stand-ins for the `AuditEntry` fields the wire does not read. +#: See the module docstring: writing a plausible-looking value into an identity +#: or hash-chain field that nothing downstream honours is how a placeholder gets +#: mistaken for evidence later. +_UNSET_ID: Final = [0] * 16 +_UNSET_HASH: Final = [0] * 32 + +#: `aa_core::AuditEventType` variants for the two outcomes the adapters report. +#: A denied call is a policy violation, not an intercepted call that ran. +#: +#: **The tag does not currently reach anything that acts on it.** An earlier +#: version of this comment said the runtime "keys its own handling off this tag"; +#: it does not — it keys off ``action_type`` / ``detail``, and +#: ``aa_sdk_client::report_event`` puts this tag into proto ``labels`` while +#: leaving ``action_type``, ``detail`` and ``decision`` at their proto3 zero. So +#: the runtime's ``is_policy_violation`` is false for every hook-layer record and +#: a deny is batched exactly like an allow. The distinction is preserved here +#: because collapsing it would be wrong at the source too, but it is carried, not +#: honoured — do not build a claim on it. +_EVENT_TYPE_ALLOWED: Final = "ToolCallIntercepted" +_EVENT_TYPE_DENIED: Final = "PolicyViolation" + + +def build_tool_outcome_payload( + *, + tool_name: str, + result: Any, + agent_id: str | None, + run_id: str | None, + denied: bool, +) -> str: + """Encode one governed call's outcome as ``aa_core::AuditEntry`` JSON. + + ``result`` carries the tool's output on the allowed path and the + short-circuit error text on the denied one, so it is written under a + different key per branch: a reader must not have to infer from an empty + string whether a tool returned nothing or never ran. + + ``seq`` is 0 because sequencing is the runtime's — it assigns one per + connection on ingress, and a number chosen here would be a second, competing + ordering that agrees with nothing. + """ + outcome_key = "error" if denied else "result" + payload = { + "agent_id": agent_id or "", + "tool_name": tool_name, + "run_id": run_id or "", + "denied": denied, + outcome_key: "" if result is None else str(result), + } + return json.dumps( + { + "seq": 0, + "timestamp_ns": time.time_ns(), + "event_type": _EVENT_TYPE_DENIED if denied else _EVENT_TYPE_ALLOWED, + "agent_id": _UNSET_ID, + "session_id": _UNSET_ID, + "payload": json.dumps(payload), + "previous_hash": _UNSET_HASH, + "entry_hash": _UNSET_HASH, + } + ) + + +def runtime_can_record(runtime_client: Any) -> bool: + """Whether ``runtime_client`` exposes the native event channel. + + Read rather than assumed so the disposition an interceptor declares is + computed from the same fact :func:`send_tool_outcome` acts on. A build whose + native shim predates ``send_event``, or a caller-supplied stand-in that only + answers policy queries, must declare that it records nothing rather than + claim a channel it does not hold. + """ + return callable(getattr(runtime_client, "send_event", None)) + + +def send_tool_outcome( + runtime_client: Any, + *, + tool_name: str, + result: Any, + agent_id: str | None, + run_id: str | None, + denied: bool, +) -> bool: + """Send one governed call's outcome to the runtime. Returns whether it went. + + **Every failure is swallowed.** The adapters call the audit hook from inside + the governed tool path — on the allowed branch without a guard of their own + (``adapters/_shared/tool_governance.py``) — so an exception raised here would + surface as a failure of the tool call itself. A degraded audit channel must + never change what a governed call does, in either direction: it must not fail + an allowed call, and it must not turn a deny into some other error. + + The boolean is returned rather than logged so a caller (and a test) can tell + a send that happened from one that did not, without this function acquiring + an opinion about how to report it. + """ + if not runtime_can_record(runtime_client): + return False + + try: + from agent_assembly._core import GovernanceEvent + except ImportError: + # The interceptor that owns this sink only exists over a connected native + # runtime, so this is unreachable on the shipped path — but a partially + # built extension must degrade to "no record", not to an ImportError + # escaping a tool call. + return False + + try: + event = GovernanceEvent( + build_tool_outcome_payload( + tool_name=tool_name, + result=result, + agent_id=agent_id, + run_id=run_id, + denied=denied, + ) + ) + runtime_client.send_event(event) + except Exception: + return False + return True diff --git a/agent_assembly/core/runtime_interceptor.py b/agent_assembly/core/runtime_interceptor.py index 6373cd8d..3bca0095 100644 --- a/agent_assembly/core/runtime_interceptor.py +++ b/agent_assembly/core/runtime_interceptor.py @@ -44,7 +44,12 @@ from importlib import metadata from typing import Any -from agent_assembly.core.audit_sink import AuditSinkDisposition, resolve_delegated_audit_sink +from agent_assembly.core.audit_sink import ( + AUDIT_SINK_FORWARDED, + AuditSinkDisposition, + resolve_delegated_audit_sink, +) +from agent_assembly.core.runtime_audit import runtime_can_record, send_tool_outcome from agent_assembly.exceptions import OpTerminatedError ENV_RUNTIME_SOCKET = "AA_RUNTIME_SOCKET" @@ -264,27 +269,35 @@ class RuntimeQueryInterceptor: ``decision`` — maps to ``deny`` (fail closed). When ``False`` those paths proceed (fail open), preserving the observe / disabled behavior. - The "delegates everything else" clause is doing more work than it looks: - ``record_result`` and ``on_tool_end`` — the adapters' audit hook — delegate to - a ``GatewayClient`` that has neither, so neither resolves and the adapters - emit **no** audit record for a governed call, allowed or denied. That is - declared in :attr:`audit_sink` rather than left to be discovered by reading - this class (AAASM-5731). + The audit hook is the one thing besides the check that this class owns + rather than delegates. ``record_result`` hands a governed call's outcome to + the runtime over the native event channel — a write with no acknowledgement, + so it is not evidence the record survived (AAASM-5750). Whether the *denied* + path reaches it is the adapter's business, not this class's: most adapters + raise first. Before that it delegated like everything else, to a + ``GatewayClient`` that has neither ``record_result`` nor ``on_tool_end``, so + the adapters' ``getattr`` lookup found nothing and no record was emitted on + either path (AAASM-5731). """ @property def audit_sink(self) -> AuditSinkDisposition: - """What this interceptor does with the audit record (AAASM-5731). - - Computed from the wrapped client rather than fixed, because this class - owns no audit hook of its own — ``__getattr__`` hands both names - straight to the client, so the client's surface *is* the answer. With - the ``GatewayClient`` this SDK builds, neither resolves and the record is - never attempted; with a caller-supplied client that has one, this SDK - makes no claim. Declared on this class rather than inherited through - ``__getattr__`` so a test can require the interceptor to speak for - itself. + """What this interceptor does with the audit record (AAASM-5750). + + Computed from the runtime client, not fixed, because the two branches of + :meth:`record_result` genuinely differ: with the native event channel the + record is forwarded, and without it there is nothing to send on. A + constant here would be a claim about a channel this interceptor may not + hold — which is the shape of defect this whole type exists to catch. + + Falls back to the delegated answer rather than to ``discarded``: with no + channel this class contributes no sink of its own, so what the adapters + would find is again whatever the wrapped client exposes. Declared on this + class rather than inherited through ``__getattr__`` so a test can require + the interceptor to speak for itself. """ + if runtime_can_record(self._runtime_client): + return AUDIT_SINK_FORWARDED return resolve_delegated_audit_sink(self._client) def __init__( @@ -388,6 +401,82 @@ def check_tool_start( # mirroring the error-sentinel handling above (AAASM-4014). return self._on_query_failure(reason or f"runtime returned unrecognized decision {decision!r}") + def record_result( + self, + *, + tool_name: str, + result: Any = None, + agent_id: str | None = None, + run_id: str | None = None, + denied: bool = False, + **_kwargs: Any, + ) -> bool: + """Forward one governed call's outcome to the runtime (AAASM-5750). + + This is the first audit hook any interceptor this SDK ships resolves. + Before it, ``getattr(handler, "record_result", None)`` fell through + ``__getattr__`` to a ``GatewayClient`` that has neither audit hook, so the + adapters found nothing to call and every governed call — allowed as well + as denied — produced no evidence. + + The signature mirrors what the adapters pass positionally by keyword + (``adapters/_shared/tool_governance.py`` and the per-framework record + helpers), including the ``denied`` flag they only supply when the callee + accepts it. ``**_kwargs`` absorbs anything a future adapter adds so a new + keyword degrades to an unused field rather than a ``TypeError`` raised + inside a governed tool call. + + Returns whether the record was sent, which is what the declaration in + :attr:`audit_sink` is about. Nothing on the adapter path reads it — the + adapters discard the return — so it exists for a caller, and a test, that + wants the distinction. Failures never raise; see + :func:`~agent_assembly.core.runtime_audit.send_tool_outcome` for why that + is load-bearing rather than defensive. + """ + return send_tool_outcome( + self._runtime_client, + tool_name=tool_name, + result=result, + agent_id=agent_id if agent_id is not None else self._agent_id, + run_id=run_id, + denied=denied, + ) + + def on_tool_end( + self, + *, + output: Any = None, + tool_name: str = "", + agent_id: str | None = None, + run_id: Any = None, + denied: bool = False, + **_kwargs: Any, + ) -> bool: + """Second audit-hook name, for callers that only know that one. + + ``AUDIT_HOOK_NAMES`` lists ``record_result`` first and ``on_tool_end`` + second, and every adapter that looks the hook up takes the first that + resolves — so this exists for the one caller that does not look: the + LangChain callback handler forwards its own ``on_tool_end`` to the + interceptor's, by that name specifically. Without this the record built on + LangChain's callback path was accepted by the handler and dropped one hop + later, which is the drop this ticket closes rather than a separate one. + + **Named limitation:** LangChain's ``on_tool_end`` callback contract + carries the run id but not the tool name, so a record arriving by that + route names the run and not the tool unless the caller supplies one. The + pre-execution check that path performs *does* have the tool name — it + comes in on ``on_tool_start`` — so this weakens the evidence, not the + enforcement. + """ + return self.record_result( + tool_name=tool_name, + result=output, + agent_id=agent_id, + run_id=None if run_id is None else str(run_id), + denied=denied, + ) + def _on_query_failure(self, reason: str) -> dict[str, str]: """Map an unauthoritative query to deny (enforce) or allow (observe).""" if self._enforce: @@ -422,12 +511,23 @@ class _FailClosedInterceptor: every tool rather than silently allow it (AAASM-3106). Non-check attributes delegate to the wrapped ``GatewayClient``, whose surface carries no audit hook — so a call denied here produces no audit record either (see - :attr:`audit_sink`, AAASM-5731). + :attr:`audit_sink`). + + That gap is not an oversight left standing: this interceptor exists precisely + because the runtime is unreachable, and the runtime is the sink. There is no + channel for it to record on, which under ADR 0033 §6 is *Degraded* — the + control is configured and unavailable — rather than something AAASM-5750 + could have wired. """ @property def audit_sink(self) -> AuditSinkDisposition: - """See :attr:`RuntimeQueryInterceptor.audit_sink` — same delegation, same answer.""" + """See :attr:`RuntimeQueryInterceptor.audit_sink` — same delegation, same answer. + + There is deliberately no runtime branch here. This interceptor holds no + runtime client at all, so the forwarded branch is unreachable by + construction rather than by a check that could drift. + """ return resolve_delegated_audit_sink(self._client) def __init__(self, client: Any, reason: str) -> None: diff --git a/docs/concepts/architecture.md b/docs/concepts/architecture.md index c9f942a4..c65c2ca5 100644 --- a/docs/concepts/architecture.md +++ b/docs/concepts/architecture.md @@ -69,7 +69,7 @@ flowchart LR Solid arrows are install-time; dashed arrows fire on every framework call after hooks are installed. The interceptor → gateway hop is the only network boundary in the data path. -There is deliberately no audit edge on that hop. The adapters offer every governed outcome to an audit hook on the interceptor, but on every interceptor this SDK ships the hook does not resolve, so no record leaves the SDK — for allowed calls as much as denied ones ([AAASM-5731](https://lightning-dust-mite.atlassian.net/browse/AAASM-5731)). +The audit edge on that hop runs to the runtime, not to the gateway. The adapters offer a governed outcome to an audit hook on the interceptor; over a connected runtime that hook resolves and writes the record to the native event channel. It is a handoff — unacknowledged, so the SDK cannot report arrival — and it covers every governed adapter on the allowed path but only `google_adk`, `pydantic_ai` and `openai_agents` on the denied one; the other eight return or raise before their record helper. Without a reachable runtime the hook does not resolve and no record leaves the SDK ([AAASM-5750](https://lightning-dust-mite.atlassian.net/browse/AAASM-5750)). Downstream of the handoff, [AAASM-5783](https://lightning-dust-mite.atlassian.net/browse/AAASM-5783) is open on `report_event` payloads reaching neither the live stream nor the durable entry — until it lands, no SDK can claim ADR 0033 §6 *Observed*. ## PyO3 FFI layer @@ -79,8 +79,8 @@ The pure-Python adapters described above are sufficient for governing most agent The native crate lives at `native/aa-ffi-python/` in the repository and is built with [`maturin`](https://www.maturin.rs/). When installed, it exposes a private `agent_assembly._core` module with two symbols: -- `RuntimeClient` — a Rust-backed runtime client (a thin shim over the shared `aa-sdk-client` crate). `agent_assembly` uses it for `register` and `query_policy` only. It also exposes `send_event`, which **nothing in `agent_assembly` calls** — the capability exists in the shim and the SDK never reaches it, so no governance event is shipped from here (AAASM-5731). -- `GovernanceEvent` — Rust-side dataclass for the events that channel would carry. Exported from `agent_assembly`, and never constructed by it. +- `RuntimeClient` — a Rust-backed runtime client (a thin shim over the shared `aa-sdk-client` crate). `agent_assembly` uses it for `register`, `query_policy`, and — since AAASM-5750 — `send_event`, which is how a governed call's outcome is handed to the runtime (`core/runtime_audit.py`) — a write to the channel, with no acknowledgement back. +- `GovernanceEvent` — Rust-side wrapper for the events that channel carries. It validates its argument as `aa_core::AuditEntry` JSON, and `core/runtime_audit.py` constructs one per forwarded outcome. `agent_assembly/__init__.py` imports these symbols inside a `try / except ImportError` block. **If the native extension was never built, the SDK still works** — pure-Python `GatewayClient` is the fallback, and the `RuntimeClient` symbol simply is not present in `agent_assembly.__all__`. @@ -104,7 +104,7 @@ For most contributors, this is unnecessary — the pure-Python SDK is the defaul 2. **Create the gateway client** — pure-Python `GatewayClient` by default. If `mode != "sdk-only"` and the native extension is available, the assembly may switch to the Rust `RuntimeClient` (transparent to the caller). 3. **Discover adapters** via `AdapterRegistry.get_available_adapters_by_priority()`. Adapters whose underlying framework is not importable are silently skipped — no warning noise. 4. **Install hooks** by calling `adapter.register_hooks(interceptor)` for each available adapter, in priority order. Each adapter records the patches it owns so they can be reverted in step 9. -5. **Start the network layer.** This is the seam reserved for the sidecar handshake under `mode="ebpf"` / `mode="proxy"`; in this SDK all three branches currently return a no-op shutdown and start nothing, so no side-channel streams audit events from here (AAASM-5731). +5. **Start the network layer.** This is the seam reserved for the sidecar handshake under `mode="ebpf"` / `mode="proxy"`; in this SDK all three branches currently return a no-op shutdown and start nothing, so no side-channel streams audit events from here — the record path is the native event channel opened at step 3, not this seam (AAASM-5750). 6. **Register the active context** in a process-global slot under a lock — `init_assembly()` is idempotent within a process: a second call returns the active context unchanged rather than double-installing hooks. 7. Return the [`AssemblyContext`](../api-reference/index.md) to the caller. @@ -112,7 +112,7 @@ For most contributors, this is unnecessary — the pure-Python SDK is the defaul The returned `AssemblyContext` doubles as a context manager (`__enter__` / `__exit__`). On `shutdown()`: -8. **Stop the network layer** — a no-op today, matching step 5; there are no in-flight audit events to flush. +8. **Stop the network layer** — a no-op today, matching step 5; it holds no audit buffer to flush. Records already handed to the native channel are the runtime's; a send still in flight at process exit can be lost. 9. **`unregister_hooks()` on every adapter, in reverse install order** — guarantees that nested patches (e.g. LangGraph wrapping LangChain) come off in the order opposite to install. 10. **Close the gateway client** — drain the HTTP keep-alive pool. 11. **Clear the process-global active-context slot** — the next `init_assembly()` call starts clean. diff --git a/docs/examples/crewai-research-crew.md b/docs/examples/crewai-research-crew.md index 5b61dcf1..3d11f3e2 100644 --- a/docs/examples/crewai-research-crew.md +++ b/docs/examples/crewai-research-crew.md @@ -1,6 +1,6 @@ # CrewAI — multi-agent research crew -A three-agent CrewAI-style research crew (researcher → writer → critic) governed by Agent Assembly, where every governed tool call is attributed to the acting agent with the full delegation chain captured on each audit event by the demo's own `CrewPolicyEngine`. The SDK layer records nothing itself (AAASM-5731). +A three-agent CrewAI-style research crew (researcher → writer → critic) governed by Agent Assembly, where every governed tool call is attributed to the acting agent with the full delegation chain captured on each audit event by the demo's own `CrewPolicyEngine`. The delegation-aware call stack shown here is the demo's; the SDK layer's own record carries the tool, run and outcome (AAASM-5750). ## What this example demonstrates @@ -177,7 +177,7 @@ Running crew delegation trajectory: → write_file({"path": "report.md"}) ❌ BLOCKED — Approval for 'write_file' by 'critic' was rejected — the crew may not persist files without sign-off. -Delegation-aware audit events recorded this run (by the demo's own handler — the SDK layer produces none, AAASM-5731): +Delegation-aware audit events recorded this run (by the demo's own handler — this example supplies its own, so the SDK's runtime sink is not what produces these): ---------------------------------------------- ✅ allow web_search chain: researcher → web_search ✅ allow web_search chain: researcher → web_search diff --git a/docs/examples/framework-support.md b/docs/examples/framework-support.md index 6f055edb..9473ad7f 100644 --- a/docs/examples/framework-support.md +++ b/docs/examples/framework-support.md @@ -24,9 +24,9 @@ with init_assembly( mode="sdk-only", ): # Build and run your agent exactly as you normally would. - # Every tool call now passes through the policy gate. It is NOT audited by the - # SDK layer: the outcome is offered to an audit hook that does not resolve on - # any interceptor this SDK ships, so nothing is recorded (AAASM-5731). + # A call to a governed tool now passes through the policy gate, and its outcome is + # offered to an audit hook. Over a connected runtime that hook forwards the + # record to the runtime; without one nothing is recorded (AAASM-5750). ... ``` diff --git a/docs/examples/langchain-research-agent.md b/docs/examples/langchain-research-agent.md index 216d619d..6b041a19 100644 --- a/docs/examples/langchain-research-agent.md +++ b/docs/examples/langchain-research-agent.md @@ -8,7 +8,7 @@ This example initializes Agent Assembly with `init_assembly()` in `sdk-only` mod - **Network allowlist** — outbound egress is only allowed to `*.openai.com`. - **Daily budget** — tool calls are metered against a `$1.00 / day` cap. -- **Tool-call logging** — the demo appends every governed call to its own in-process `audit_log`. That log is the demo's, not the SDK's: the SDK layer produces no audit evidence (AAASM-5731). +- **Tool-call logging** — the demo appends every governed call to its own in-process `audit_log`. That log is the demo's, not the SDK's: this example supplies its own handler and runs without a runtime, so the SDK's own sink is not what fills it (AAASM-5750). - **Credential-leak block** — any tool input carrying a secret is denied. It also includes a credential-leak demo that uses a **SAFE, FAKE** key (`sk-FAKE...`) — never a real secret — to show the leak rule firing. Finally, `--mock` mode runs the whole demo offline with no API keys, so CI can run it. diff --git a/docs/guides/authoring-adapters.md b/docs/guides/authoring-adapters.md index b7a43e5b..b6e8e0a6 100644 --- a/docs/guides/authoring-adapters.md +++ b/docs/guides/authoring-adapters.md @@ -134,8 +134,8 @@ status string `"allow" | "deny" | "pending"`, or a mapping `{"status": ..., "rea | `check_tool_start` / `check_tool_call` | adapter → interceptor, returns decision | Pre-execution gate for a tool call; `deny` blocks it. | | `wait_for_tool_approval` | adapter → interceptor, returns decision | Block until a `pending` tool call is approved or rejected (human-in-the-loop). | | `get_pending_tool_approval_timeout_seconds` | adapter → interceptor | Configurable timeout for the approval wait. | -| `record_result` / `on_tool_end` | adapter → interceptor, no return | Offer a governed tool call's outcome for audit — allowed or denied. **No interceptor this SDK ships resolves either name**, so on the shipped path the `getattr` guard finds nothing and the outcome is not recorded; only a caller-supplied handler retains it (AAASM-5731). | -| `record` | adapter → interceptor, no return | Generic structured event (e.g. `action="task_start"`). **No interceptor this SDK ships resolves this name either** — the CrewAI patch looks it up for task start/complete (`crewai/patch.py:429,445`) and finds nothing, so those events are not recorded on the shipped path (AAASM-5731). | +| `record_result` / `on_tool_end` | adapter → interceptor, no return | Offer a governed tool call's outcome for audit. Over a connected runtime the SDK's interceptor resolves both names and writes the record to the runtime's event channel — a handoff, not a retention guarantee; without one neither resolves and the `getattr` guard finds nothing. **Your adapter must call this on the denied path too** — most do not: only `google_adk`, `pydantic_ai` and `openai_agents` do, and the other eight return or raise first (AAASM-5750). | +| `record` | adapter → interceptor, no return | Generic structured event (e.g. `action="task_start"`). **No interceptor this SDK ships resolves this name** — unlike the row above, `record` was not wired by AAASM-5750. The CrewAI patch looks it up for task start/complete (`crewai/patch.py:429,445`) and finds nothing, so those events are recorded only by a caller-supplied handler. | !!! note "These are conventions, not a typed contract" The names above are what today's built-in adapters happen to call (see the CrewAI patch diff --git a/docs/index.md b/docs/index.md index 88586eeb..76fb38ad 100644 --- a/docs/index.md +++ b/docs/index.md @@ -3,8 +3,8 @@ **In plain terms:** this SDK is how a Python agent asks for permission before it acts. You wrap your existing agent in one `init_assembly()` call, and from that point on every tool call your agent makes is checked against a governance policy — allowed or denied — -without you rewriting a single line of the agent itself. Recording is *not* a third -outcome: the SDK layer produces no audit evidence of its own (see below). +without you rewriting a single line of the agent itself. Over a connected runtime the +outcome of each call is also handed to the runtime's event channel (see below). It is two things in one package: @@ -47,8 +47,8 @@ flowchart LR - **Operators** who need agents to run under a policy gate they control, with identity and lineage registered against the gateway. -Note that an **audit trail of governed tool calls is not something this SDK layer -produces** — see the warning under "Why use it" below. +Note that an **audit trail of governed tool calls depends on a reachable runtime** — +see the note under "Why use it" below. ## Why use it @@ -58,19 +58,33 @@ produces** — see the warning under "Why use it" below. - **Agent lineage** — parent / root / team identity is registered with the gateway and carried on every policy check. -!!! warning "The SDK layer keeps no audit trail of its own" +!!! note "Audit evidence depends on a reachable runtime" The framework adapters offer the outcome of every governed call to an audit hook - on the governance interceptor. On every interceptor this SDK ships that hook - **does not resolve**, so nothing is emitted — for **allowed** calls as much as - denied ones — and no claim of attributability or after-the-fact review holds on - the SDK path. - - Enforcement is unaffected: a policy DENY still blocks the tool, and the proxy / - eBPF layers remain authoritative. `init_assembly()` warns at startup and reports - `audit_sink` on the returned context; supply your own handler exposing - `record_result` or `on_tool_end` to retain the record - ([AAASM-5731](https://lightning-dust-mite.atlassian.net/browse/AAASM-5731)). + on the governance interceptor. Over a connected runtime that hook resolves and + writes the record to the native event channel — the same channel agent + registration uses. On the **allowed** path that covers every governed adapter; + on the denied path it covers three of eleven (see below). + + Without a reachable runtime there is no channel to send on, the hook does not + resolve, and nothing is emitted; no claim of attributability or after-the-fact + review holds on that path. Enforcement is unaffected either way: a policy DENY + still blocks the tool, and the proxy / eBPF layers remain authoritative. + `init_assembly()` warns when no record can be sent and reports `audit_sink` on the + returned context. Delivery is best-effort: a failed send degrades to no record + rather than to a failed tool call + ([AAASM-5750](https://lightning-dust-mite.atlassian.net/browse/AAASM-5750)). + + **A handoff is not evidence, and denied calls are mostly not covered.** The send + is unacknowledged, so this SDK cannot report that a record arrived and does not + claim it did — and downstream, + [AAASM-5783](https://lightning-dust-mite.atlassian.net/browse/AAASM-5783) is open + on `report_event` payloads reaching neither the live stream nor the durable + entry, so no SDK can claim ADR 0033 §6 *Observed* until it lands. And only `google_adk`, `pydantic_ai` and `openai_agents` build a + record on the **denied** path — the other eight governed adapters (`crewai`, + `llamaindex`, `haystack`, `agno`, `smolagents`, `microsoft_agent_framework`, + `mcp`, `langchain`) return or raise before their record helper, so a deny there + produces no record for any sink to carry. - **Native PyO3 fast path** (optional) — drop into a Rust runtime client when you need sub-millisecond policy checks. - **Typed throughout** — typed models for every gateway payload; the package ships a diff --git a/examples/basic_usage.py b/examples/basic_usage.py index 5bab9f4c..c5731aae 100644 --- a/examples/basic_usage.py +++ b/examples/basic_usage.py @@ -32,8 +32,9 @@ # - Register the agent with the gateway # - Check policy compliance before executing actions # -# Note: the SDK layer does NOT log audit events. Governed outcomes are offered to an -# audit hook that no interceptor this SDK ships resolves (AAASM-5731). +# Note: governed outcomes are offered to an audit hook on the interceptor. Over a +# connected runtime the SDK's own interceptor resolves it and forwards the record; +# without one the hook does not resolve and nothing is logged (AAASM-5750). # Don't forget to shutdown the runtime when done assembly.shutdown() diff --git a/test/integration/test_native_core_runtime.py b/test/integration/test_native_core_runtime.py index 103e4cd2..1fe8d95c 100644 --- a/test/integration/test_native_core_runtime.py +++ b/test/integration/test_native_core_runtime.py @@ -319,3 +319,38 @@ def test_wired_check_fails_open_when_runtime_unreachable(native_core: Any) -> No finally: client.close() socket_dir.cleanup() + + +@pytest.mark.integration +def test_the_audit_payload_builder_is_accepted_by_the_real_governance_event(native_core: Any) -> None: + """The SDK's audit payload must satisfy the REAL native constructor (AAASM-5750). + + ``GovernanceEvent.__new__`` deserializes its argument as + ``aa_core::AuditEntry`` JSON and raises ``ValueError`` on anything else, so a + builder emitting the wrong shape produces no record at all — silently, since + ``send_tool_outcome`` swallows the failure to keep a degraded audit channel + from breaking a governed tool call. + + The unit suite substitutes a double for this constructor and can therefore + only catch an obviously wrong payload; it cannot establish that the real one + accepts a given payload. This test is where that is established, so it lives + behind the same native gate as the rest of this module. + """ + from agent_assembly.core.runtime_audit import build_tool_outcome_payload + + for denied in (False, True): + payload = build_tool_outcome_payload( + tool_name="web_search", + result="RESULT" if not denied else "blocked by policy", + agent_id="agent-1", + run_id="run-1", + denied=denied, + ) + # Constructing it IS the assertion: a rejected payload raises ValueError. + event = native_core.GovernanceEvent(payload) + assert event.payload_json == payload + + # Negative control on the same constructor: it must reject a payload that is + # not an AuditEntry, or "it accepted ours" would say nothing. + with pytest.raises(ValueError): + native_core.GovernanceEvent(json.dumps({"event_type": "ToolCallIntercepted"})) diff --git a/test/unit/core/_fake_core.py b/test/unit/core/_fake_core.py index 16c028d3..4d76b7a1 100644 --- a/test/unit/core/_fake_core.py +++ b/test/unit/core/_fake_core.py @@ -9,6 +9,7 @@ from __future__ import annotations +import json import sys import types from collections.abc import Callable @@ -26,6 +27,7 @@ def __init__(self, decision: str = "allow", reason: str = "") -> None: self.register_calls: list[tuple[str, str, str, str | None, str | None, str | None]] = [] self.query_calls: list[tuple[Any, ...]] = [] self.register_should_raise: Exception | None = None + self.sent_events: list[Any] = [] # Set by install_fake_core's connect to the (socket_path, agent_id, # sdk_version) it was called with (AAASM-3683). self.connect_args: tuple[str, str | None, str | None] | None = None @@ -54,6 +56,16 @@ def query_policy( self.query_calls.append((agent_id, action_type, tool_name, tool_args_json)) return {"decision": self._decision, "reason": self._reason} + def send_event(self, event: Any) -> None: + """The native audit channel (AAASM-5750). + + Present because the real shim has it and the SDK reads for it by name to + decide whether it can record at all: a double missing it makes every + interceptor built over it declare ``absent``, which would quietly turn + the forwarding path off in any test using this fake. + """ + self.sent_events.append(getattr(event, "payload_json", event)) + def close(self) -> None: return None @@ -84,6 +96,48 @@ def close(self) -> None: return None +#: Fields ``aa_core::AuditEntry`` requires, mirrored from the struct the real +#: ``GovernanceEvent`` constructor deserializes into. Everything else on that +#: struct is ``Option`` / ``default``; these are not, so a payload missing one is +#: rejected by the native extension. +AUDIT_ENTRY_REQUIRED_FIELDS = frozenset( + {"seq", "timestamp_ns", "event_type", "agent_id", "session_id", "payload", "previous_hash", "entry_hash"} +) + + +class FakeGovernanceEvent: + """Stand-in for the native ``GovernanceEvent`` wrapper. + + It replicates the one behaviour the SDK depends on and could get wrong: the + real constructor deserializes its argument as ``aa_core::AuditEntry`` JSON and + raises ``ValueError`` when that fails, so a payload builder that emits the + wrong shape fails at the boundary rather than silently. A double that + accepted any string would make every "the record crossed" assertion pass over + a payload the real extension rejects. + + It is a **replica of the contract, not the validator**. It checks the + required field set, not serde's full type discipline, so it cannot prove the + real constructor accepts a given payload — only that an obviously wrong one + is caught. The real constructor is exercised against this SDK's builder in + ``test/integration/test_native_core_runtime.py``, which runs only where the + extension is built. + """ + + def __init__(self, payload_json: str) -> None: + try: + decoded = json.loads(payload_json) + except ValueError as error: + raise ValueError(f"GovernanceEvent payload must be serialized aa_core::AuditEntry JSON: {error}") from error + if not isinstance(decoded, dict): + raise ValueError("GovernanceEvent payload must be serialized aa_core::AuditEntry JSON: not an object") + missing = AUDIT_ENTRY_REQUIRED_FIELDS - decoded.keys() + if missing: + raise ValueError( + f"GovernanceEvent payload must be serialized aa_core::AuditEntry JSON: missing {sorted(missing)}" + ) + self.payload_json = payload_json + + def install_fake_core( monkeypatch: pytest.MonkeyPatch, runtime_client: Any, @@ -104,6 +158,7 @@ def connect(_socket_path: str, agent_id: str | None = None, sdk_version: str | N fake_core = types.ModuleType("agent_assembly._core") fake_core.RuntimeClient = _ConnectingRuntimeClient # type: ignore[attr-defined] + fake_core.GovernanceEvent = FakeGovernanceEvent # type: ignore[attr-defined] monkeypatch.setitem(sys.modules, "agent_assembly._core", fake_core) return runtime_client @@ -128,4 +183,5 @@ def install_fake_core_with_connect( fake_core = types.ModuleType("agent_assembly._core") fake_core.RuntimeClient = runtime_client_cls # type: ignore[attr-defined] + fake_core.GovernanceEvent = FakeGovernanceEvent # type: ignore[attr-defined] monkeypatch.setitem(sys.modules, "agent_assembly._core", fake_core) diff --git a/test/unit/core/test_audit_sink_disposition.py b/test/unit/core/test_audit_sink_disposition.py index 51852aeb..a0a15673 100644 --- a/test/unit/core/test_audit_sink_disposition.py +++ b/test/unit/core/test_audit_sink_disposition.py @@ -1,30 +1,33 @@ -"""AAASM-5731 — a shipped governance handler must not swallow the audit record silently. +"""AAASM-5731 / AAASM-5750 — what a shipped governance handler does with the audit record. The adapters' audit hook is duck-typed and returns ``None``, so a handler that -retains the record, one that drops it, and one that never resolves the hook at -all are indistinguishable at the call site. On every interceptor this SDK ships -the hook does not resolve, so **nothing is emitted for an allowed call either** — -not just for a denied one — and before this suite there was no signal of that at -all. +forwards the record, one that drops it, and one that never resolves the hook at +all are indistinguishable at the call site. Over a connected runtime the SDK's +interceptor now resolves the hook and sends the record across the native +boundary; without one no hook resolves and **nothing is emitted for an allowed +call either** — not just for a denied one. -Three things are pinned separately, because any one of them alone passes while -the defect is present: +Three things are pinned separately, because any one of them alone passes while a +defect is present: 1. every handler ``build_governance_interceptor`` can return, plus the LangChain handler that replaces it, *declares* a disposition; 2. the declaration matches behaviour in **both** directions — a handler - declaring ``absent`` must resolve no hook and reach nothing, a handler - declaring ``discarded`` must resolve a hook and still reach nothing, and a - handler that genuinely records must be reported as caller-supplied; -3. ``init_assembly`` surfaces it on the DEFAULT path, with nothing opted into. + declaring ``forwarded`` must reach the native boundary with the record, one + declaring ``absent`` must resolve no hook and reach nothing, one declaring + ``discarded`` must resolve a hook and still reach nothing, and a handler that + genuinely records must be reported as caller-supplied; +3. ``init_assembly`` surfaces the gap on the DEFAULT path, with nothing opted + into, and stays quiet when there is no gap. The stubs here sit at the **downstream boundaries** — the native ``RuntimeClient`` and the ``GatewayClient``'s HTTP transport — not in place of -the code under test. The point is to prove nothing crosses them. Every -"reached nothing" assertion is paired with a positive control on the same -boundary, because otherwise it is indistinguishable from a probe that never ran, -and with a forwarding control, because otherwise it is indistinguishable from a -probe that cannot see a record at all. +the code under test. Nothing here injects a sink into the SDK's own path: a suite +that supplies its own recording handler proves that handler records and stays +green over an interceptor wired to nothing, which is the defect AAASM-5749 found +one row over. Every claim about a boundary is paired with a positive control on +the same boundary, because otherwise it is indistinguishable from a probe that +never ran. """ from __future__ import annotations @@ -47,8 +50,10 @@ AUDIT_SINK_ABSENT, AUDIT_SINK_CALLER_SUPPLIED, AUDIT_SINK_DISCARDED, + AUDIT_SINK_FORWARDED, resolve_audit_sink, ) +from agent_assembly.core.runtime_audit import build_tool_outcome_payload from agent_assembly.core.runtime_interceptor import build_governance_interceptor from ._fake_core import FakeRuntimeClient, install_fake_core @@ -83,9 +88,11 @@ def register(self, *args: Any, **kwargs: Any) -> str: return super().register(*args, **kwargs) def send_event(self, *args: Any, **kwargs: Any) -> None: - # Exposed by the native shim and never called from ``agent_assembly``. - # Recorded so a future wiring change shows up here rather than silently. - self.crossings.append(f"send_event:{args!r}") + # The audit sink. Recorded as its payload JSON rather than the wrapper's + # repr, because the assertions below look for the probe INSIDE the + # record and a default object repr would hide it. + payloads = [getattr(arg, "payload_json", arg) for arg in args] + self.crossings.append(f"send_event:{payloads!r}") def __getattr__(self, name: str) -> Any: # Any attribute the SDK reaches for that is not defined above is still an @@ -251,7 +258,7 @@ def test_every_shipped_governance_handler_declares_its_audit_sink() -> None: label for label, handler in handlers.items() if getattr(handler, "audit_sink", None) - not in {AUDIT_SINK_ABSENT, AUDIT_SINK_DISCARDED, AUDIT_SINK_CALLER_SUPPLIED} + not in {AUDIT_SINK_FORWARDED, AUDIT_SINK_ABSENT, AUDIT_SINK_DISCARDED, AUDIT_SINK_CALLER_SUPPLIED} ] assert not undeclared, ( f"handler(s) {undeclared} are shipped without declaring what they do with the " @@ -260,13 +267,22 @@ def test_every_shipped_governance_handler_declares_its_audit_sink() -> None: ) -@pytest.mark.parametrize("label", ["runtime reachable, enforce", "runtime unreachable, enforce"]) +# Only the enforce labels: under observe with no runtime the factory returns the +# bare GatewayClient, which declares 'absent' but has no check_tool_start for the +# positive control below to stand on. That branch's declaration is covered by the +# exhaustiveness sweep instead. +@pytest.mark.parametrize("label", ["runtime unreachable, enforce"]) def test_a_handler_declaring_absent_resolves_no_audit_hook(label: str) -> None: """``absent`` means the hook does not resolve — not merely that it records nothing. The distinction is load-bearing: it is why the gap covers the ALLOWED path. The controls are on the same objects, so a blanket ``getattr`` failure cannot masquerade as the finding. + + Every label here is a run with no reachable runtime, which is what makes the + branch real rather than the only branch: the reachable labels resolve the + hook and declare ``forwarded``, and + :func:`test_the_disposition_moves_with_the_runtime` compares the two. """ handlers = _shipped_handler_matrix([]) handler = handlers[label] @@ -279,32 +295,128 @@ def test_a_handler_declaring_absent_resolves_no_audit_hook(label: str) -> None: ) # Positive controls on the same objects: attribute resolution works, and - # delegation to the wrapped GatewayClient works. Without these, the four + # delegation to the wrapped GatewayClient works. Without these, the # `is None` assertions above are consistent with a broken probe. assert callable(handler.check_tool_start) assert callable(handler.report_edge) +@pytest.mark.parametrize("label", ["runtime reachable, enforce", "runtime reachable, observe"]) +def test_a_handler_declaring_forwarded_resolves_the_audit_hook(label: str) -> None: + """The other side of the same split, on the same matrix. + + The disposition is computed from the runtime client, so a constant would pass + every reachable case here. Pairing this with the ``absent`` cases above is + what shows the computation moving rather than agreeing by luck. + """ + handlers = _shipped_handler_matrix([]) + handler = handlers[label] + assert handler.audit_sink == AUDIT_SINK_FORWARDED + + for hook in _AUDIT_HOOKS: + assert callable(getattr(handler, hook, None)), ( + f"{label} declares {AUDIT_SINK_FORWARDED!r} but {hook!r} does not resolve on it; " + "the adapters look the hook up by name, so an unresolved one records nothing" + ) + + +def test_the_disposition_moves_with_the_runtime() -> None: + """Two constructions of the same code, not two constants compared. + + ``build_governance_interceptor`` is driven twice over inputs that differ in + exactly one thing — whether a runtime client is present — and the two + dispositions must differ. Without this, ``audit_sink`` could return a fixed + literal and every case above would still be green on its own side. + """ + reachable = _shipped_interceptor(_RecordingRuntimeClient(), []) + unreachable = _shipped_interceptor(None, []) + assert reachable.audit_sink == AUDIT_SINK_FORWARDED + assert unreachable.audit_sink == AUDIT_SINK_ABSENT + assert reachable.audit_sink != unreachable.audit_sink + + @pytest.mark.parametrize("decision", ["allow", "deny"]) -def test_the_shipped_path_reaches_no_boundary_with_the_record(decision: str) -> None: - native = _RecordingRuntimeClient(decision=decision, reason="policy forbids this") +def test_the_shipped_path_forwards_the_record_across_the_native_boundary( + monkeypatch: pytest.MonkeyPatch, decision: str +) -> None: + """The load-bearing measurement, end to end and on both branches (AAASM-5750). + + It is the inversion of the assertion this suite shipped with. Nothing here + injects a sink: the handler is the one ``build_governance_interceptor`` + returns, driven through the SDK's real governed-tool chain, and the only + substitution is the native extension itself — which is the boundary being + measured, not the code under test. + + Deleting the ``send_tool_outcome`` call from ``record_result`` turns both + parametrisations red. + """ + install_fake_core(monkeypatch, FakeRuntimeClient()) + native = _RecordingRuntimeClient(decision=decision, reason=f"policy forbids this {_PROBE_RESULT}") http_crossings: list[str] = [] handler = _shipped_interceptor(native, http_crossings) + assert handler.audit_sink == AUDIT_SINK_FORWARDED outcome, _value = _run_governed(handler) assert outcome == ("raised" if decision == "deny" else "returned") # Positive control: the check crossed the native boundary carrying the probe. - assert any(_PROBE in crossing for crossing in native.crossings), ( - f"nothing carrying the probe crossed the native boundary (crossings: " - f"{native.crossings}); the probe never ran, so the absence below proves nothing" + # Without it an empty event list is indistinguishable from a probe that never + # ran — and it stays a control rather than the finding because the assertion + # below looks only at the send_event channel. + assert any(crossing.startswith("query_policy") and _PROBE in crossing for crossing in native.crossings), ( + f"no policy query carrying the probe crossed the native boundary (crossings: " + f"{native.crossings}); the probe never ran, so nothing below proves anything" + ) + + # The record channel specifically. On the denied branch the discriminator is + # the deny reason rather than the tool result, because the tool never ran and + # asserting on its output there is an assertion that cannot fail. + sends = [crossing for crossing in native.crossings if crossing.startswith("send_event")] + carrying = [crossing for crossing in sends if _PROBE_RESULT in crossing] + assert carrying, ( + f"the {decision} branch sent no audit record carrying {_PROBE_RESULT!r} across the " + f"native boundary; send_event crossings were {sends}, and the handler declares " + f"{handler.audit_sink!r} — the declaration and the behaviour disagree" + ) + + # The record must be tagged as the outcome it describes, or a deny and an + # allow are the same event downstream. + expected_type = "PolicyViolation" if decision == "deny" else "ToolCallIntercepted" + assert any(expected_type in crossing for crossing in carrying), ( + f"the {decision} branch's record is not tagged {expected_type!r}: {carrying}" + ) + + # The HTTP boundary must stay out of it: the record rides the native channel, + # and a copy going out over the gateway's HTTP surface would be a second, + # unaccounted path for tool output to leave the process. + assert not [crossing for crossing in http_crossings if _PROBE_RESULT in crossing], ( + f"the record also crossed the HTTP boundary: {http_crossings}" ) - all_crossings = native.crossings + http_crossings - leaked = [crossing for crossing in all_crossings if _PROBE_RESULT in crossing] - assert not leaked, ( - f"the tool outcome reached a boundary on the shipped path: {leaked}; the " - f"handler declares {handler.audit_sink!r}, so the declaration is wrong" + +def test_a_malformed_record_payload_is_rejected_at_the_native_boundary( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The boundary double must be able to say no, or crossing it proves little. + + The real ``GovernanceEvent`` constructor validates its argument as + ``aa_core::AuditEntry`` JSON and raises on anything else, so a builder that + emits the wrong shape produces no record at all. If the double accepted + everything, the assertions above would pass over a payload the shipped + extension rejects. + """ + install_fake_core(monkeypatch, FakeRuntimeClient()) + from agent_assembly._core import GovernanceEvent + + with pytest.raises(ValueError): + GovernanceEvent(json.dumps({"event_type": "ToolCallIntercepted"})) + + # And the builder's real output is accepted, so the check above is a + # discriminator rather than a double that rejects everything. + assert GovernanceEvent( + build_tool_outcome_payload( + tool_name="web_search", result=_PROBE_RESULT, agent_id=_AGENT_ID, run_id="run-1", denied=False + ) ) @@ -341,12 +453,12 @@ def record_result(self, **kwargs: Any) -> None: def test_a_caller_supplied_recording_client_is_not_reported_as_absent() -> None: """A false ``absent`` is a claim about the caller's code that this SDK cannot make. - ``RuntimeQueryInterceptor`` owns no audit hook — ``__getattr__`` hands both - names to the wrapped client — so its disposition is the client's. When it was - a fixed class attribute, a caller-supplied client whose ``record_result`` - resolves still reported ``absent``, contradicting the very hook the adapters - would have called, and the LangChain handler on top compounded it to - ``discarded``. + Without a runtime, ``RuntimeQueryInterceptor`` owns no audit hook — + ``__getattr__`` hands both names to the wrapped client — so its disposition is + the client's. When it was a fixed class attribute, a caller-supplied client + whose ``record_result`` resolves still reported ``absent``, contradicting the + very hook the adapters would have called, and the LangChain handler on top + compounded it to ``discarded``. The direction of the old error matters and is why this is a correctness fix rather than a severity one: it under-claimed. It never reported retention @@ -360,9 +472,11 @@ def record_result(self, **_kwargs: Any) -> None: client = _RecordingClient(_GW_URL, _AGENT_ID, api_key=_API_KEY) client._client = httpx.Client(base_url=client.gateway_url, transport=_RecordingTransport(http_crossings)) - interceptor = build_governance_interceptor( - client, _AGENT_ID, None, runtime_client=_RecordingRuntimeClient(), native_available=True - ) + # No runtime client: the SDK contributes no sink of its own here, so what the + # adapters would find is the caller's hook and nothing else. With a runtime + # the SDK's own sink resolves first and the answer is 'forwarded', which is a + # claim about this SDK rather than about the caller's client. + interceptor = build_governance_interceptor(client, _AGENT_ID, None, runtime_client=None, native_available=True) # Precondition: the hook really does resolve through the delegation, or this # test is asserting about a situation that cannot arise. @@ -371,66 +485,100 @@ def record_result(self, **_kwargs: Any) -> None: assert resolve_audit_sink(interceptor) == AUDIT_SINK_CALLER_SUPPLIED assert resolve_audit_sink(AssemblyCallbackHandler(interceptor)) == AUDIT_SINK_CALLER_SUPPLIED - # Control on the same shapes: the client this SDK actually ships still reads - # 'absent', so the fix did not simply stop reporting the real gap. - shipped = _shipped_interceptor(_RecordingRuntimeClient(), http_crossings) + # Control on the same shapes: an interceptor with no runtime still reads + # 'absent', so the caller-supplied answer above is a discrimination rather + # than a blanket one. + shipped = _shipped_interceptor(None, http_crossings) assert resolve_audit_sink(shipped) == AUDIT_SINK_ABSENT assert resolve_audit_sink(AssemblyCallbackHandler(shipped)) == AUDIT_SINK_DISCARDED -def test_the_langchain_handler_resolves_the_hook_and_still_drops_the_record() -> None: - """``discarded`` is a different failure from ``absent``, and both are shipped. +def test_the_langchain_handler_forwards_or_drops_with_its_interceptor( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The LangChain callback path is a second, separate hop the record must survive. + + The handler defines ``on_tool_end``, so the adapters' lookup resolves on it — + and it then forwards to the *interceptor's* ``on_tool_end`` by that name + specifically, not by the ``record_result``-first order every other adapter + uses. Until AAASM-5750 no interceptor had one, so the record was accepted here + and dropped one hop later: ``discarded``, not ``absent``. - The handler defines ``on_tool_end``, so the adapters' lookup DOES resolve and - the record is handed over. It is then forwarded to the interceptor's own - ``on_tool_end``, which does not exist — so the record stops here. + Both directions are driven, because the handler's disposition is its + interceptor's and a single direction cannot show that. """ + install_fake_core(monkeypatch, FakeRuntimeClient()) native = _RecordingRuntimeClient() http_crossings: list[str] = [] - handler = AssemblyCallbackHandler(_shipped_interceptor(native, http_crossings)) - - assert handler.audit_sink == AUDIT_SINK_DISCARDED - assert callable(handler.on_tool_end), ( - "the LangChain handler declares 'discarded', which asserts the hook RESOLVES " - "and the record is dropped after being accepted; if no hook resolves the " - "honest declaration is 'absent'" - ) - - baseline = len(native.crossings) + len(http_crossings) - handler.on_tool_end(_PROBE_RESULT, run_id=uuid.uuid4()) - assert len(native.crossings) + len(http_crossings) == baseline, ( - f"on_tool_end crossed a boundary: native={native.crossings} http={http_crossings}" - ) + forwarding = AssemblyCallbackHandler(_shipped_interceptor(native, http_crossings)) - # Positive control on the same handler and the same boundary. - handler.on_tool_start({"name": "web_search"}, _PROBE, run_id=uuid.uuid4()) - assert any(_PROBE in crossing for crossing in native.crossings), ( - "the positive control did not cross either; the probe never ran" + assert forwarding.audit_sink == AUDIT_SINK_FORWARDED + assert callable(forwarding.on_tool_end) + forwarding.on_tool_end(_PROBE_RESULT, run_id=uuid.uuid4()) + assert any(crossing.startswith("send_event") and _PROBE_RESULT in crossing for crossing in native.crossings), ( + f"the LangChain callback path sent no record carrying the probe: {native.crossings}" ) + # The other direction, on the same class: with no runtime under it the hop + # still ends here, and the handler must say 'discarded' rather than 'absent' + # — something did construct and accept the record. + dropping_native = _RecordingRuntimeClient() + dropping = AssemblyCallbackHandler(_shipped_interceptor(None, [])) + assert dropping.audit_sink == AUDIT_SINK_DISCARDED + dropping.on_tool_end(_PROBE_RESULT, run_id=uuid.uuid4()) + assert not [c for c in dropping_native.crossings if _PROBE_RESULT in c] -def test_init_assembly_warns_and_reports_the_audit_sink_on_the_default_path( - monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] -) -> None: - """The signal must arrive with nothing opted into. - A caller who has to already suspect the problem in order to discover it has - not been told. - """ - install_fake_core(monkeypatch, FakeRuntimeClient(decision="allow")) +def _init_sdk_only(monkeypatch: pytest.MonkeyPatch) -> Any: monkeypatch.setattr( core_assembly, "_start_network_layer", lambda **_kwargs: ("sdk-only", core_assembly._noop_shutdown), ) core_assembly._ACTIVE_CONTEXT = None + return init_assembly(gateway_url=_GW_URL, api_key=_API_KEY, agent_id=_AGENT_ID, mode="sdk-only") + + +def test_init_assembly_warns_about_the_audit_gap_only_when_there_is_one( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """The signal must arrive with nothing opted into — and only when it is true. - context = init_assembly(gateway_url=_GW_URL, api_key=_API_KEY, agent_id=_AGENT_ID, mode="sdk-only") + A caller who has to already suspect the problem in order to discover it has + not been told. A caller warned on every run, including the ones whose records + do reach the runtime, stops reading the warning — which costs the real case + its signal too. So both directions are driven through the same ``init_assembly``. + """ + # No runtime: connect_runtime_client returns None, so no hook resolves. + monkeypatch.setattr(core_assembly, "connect_runtime_client", lambda _agent_id: None) + context = _init_sdk_only(monkeypatch) try: stderr = capsys.readouterr().err - assert context.audit_sink != AUDIT_SINK_CALLER_SUPPLIED + assert context.audit_sink == AUDIT_SINK_ABSENT for expected in ("audit", "NOT retained", context.audit_sink, "ALLOWED", "AAASM-5731"): assert expected in stderr, f"{expected!r} missing from init stderr: {stderr!r}" finally: context.shutdown() core_assembly._ACTIVE_CONTEXT = None + + +def test_init_assembly_stays_quiet_when_the_record_is_forwarded( + monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] +) -> None: + """The other arm of the warning, through the same ``init_assembly``. + + Without it the assertions in the test above are satisfied by a build that + warns unconditionally — which is what the condition at the call site used to + do, and what made the warning worth nothing on the run that has no gap. + """ + install_fake_core(monkeypatch, FakeRuntimeClient(decision="allow")) + context = _init_sdk_only(monkeypatch) + try: + stderr = capsys.readouterr().err + assert context.audit_sink == AUDIT_SINK_FORWARDED + assert "NOT retained" not in stderr, ( + f"init warned that records are not retained on a run that forwards them: {stderr!r}" + ) + finally: + context.shutdown() + core_assembly._ACTIVE_CONTEXT = None diff --git a/test/unit/core/test_planned_referent.py b/test/unit/core/test_planned_referent.py index 44ccb780..00478abd 100644 --- a/test/unit/core/test_planned_referent.py +++ b/test/unit/core/test_planned_referent.py @@ -13,30 +13,32 @@ cited as the ticket that measured the gap, never as the ticket that will fix it.** -The assertion is two-tier, because one tier alone fails in one direction or the -other and review caught both: - -* a **guarded** site (one of :data:`EXPECTED_SITES`) must name AAASM-5750 - exactly. Without this the gate stops asserting the thing the change made true - — repointing a guarded site to any other live ticket passed green, which is - precisely the drift the gate exists to catch. -* **any other** site must merely not name a stale referent. Asserting - AAASM-5750 repository-wide was the first version's defect: §6 scopes - ``Planned`` to any decided-but-unbuilt capability with any ticket, so an - unrelated roadmap row — including - ``docs/examples/framework-support.md``'s docs-area maturity label, a - different axis entirely — is legitimate and must not fail this gate. - -Two limits are disclosed rather than fixed, both measured as currently -unreachable: - -* the reachability check is per **file**, not per site. A guarded file that - reflowed its real site out of the scan's reach *and* gained a second, correct - claim would keep its entry. Requires two coordinated edits; today no file in - this repository carries more than one site. -* the gate file is excluded from its own scan, so it is a hiding place for a - stale referent. It is a test file that documents no SDK behaviour, and the - exclusion matches one exact path rather than a prefix. +**AAASM-5750 built the sink, so it joined the list it used to be the answer +to.** This gate previously required a fixed set of guarded files to name +AAASM-5750 as the ticket their ``Planned`` deferred to. Once the capability +exists there is nothing left to defer: a site still calling SDK-side recording +*Planned* under AAASM-5750 describes shipped behaviour as unbuilt, which is the +same stale-pointer defect one ticket later. So the rule collapsed to a single +tier — **no forward-looking claim in this repository may defer SDK-side audit +recording to any of the three tickets that are done with it** — and applies +repository-wide rather than to a named set. + +§6 still scopes ``Planned`` to any decided-but-unbuilt capability with any +ticket, so an unrelated roadmap row — including +``docs/examples/framework-support.md``'s docs-area maturity label, a different +axis entirely — is legitimate and must not fail this gate. Only the three named +referents are forbidden. + +A rule whose expected result is "no findings" needs the scan proved reachable, or +a broken walk passes as loudly as a clean tree. +:func:`test_the_deferral_scan_can_see` feeds the detector synthetic lines +carrying exactly the shapes this file forbids and requires it to find them. The +empty result is meaningful only because that control is green. + +One limit is disclosed rather than fixed: the gate file is excluded from its own +scan, so it is a hiding place for a stale referent. It is a test file that +documents no SDK behaviour, and the exclusion matches one exact path rather than +a prefix. """ from __future__ import annotations @@ -44,14 +46,17 @@ import re from pathlib import Path -#: The ticket that owns building the SDK-side audit sink. Guarded sites must -#: name it exactly. -CAPABILITY_REFERENT = "AAASM-5750" +import pytest -#: Tickets that *measured* the absence of an SDK-side audit sink. Backward -#: citations to them are correct and are left alone; what this gate forbids is -#: either one appearing as the ticket a forward-looking claim defers to. -STALE_REFERENTS = frozenset({"AAASM-5731", "AAASM-5681"}) +#: Tickets a forward-looking claim about SDK-side audit recording may no longer +#: defer to. Backward citations to any of them are correct and are left alone; +#: what this gate forbids is one of them appearing as the ticket a *deferral* +#: points at. The reason differs per entry, and the failure message says which. +STALE_REFERENTS = { + "AAASM-5731": "measured the gap and never intended to fix it", + "AAASM-5681": "measured the gap and never intended to fix it", + "AAASM-5750": "built the sink; SDK-side recording is no longer deferred", +} #: The two shapes a deferral takes: the ADR 0033 §6 term, and the plain #: "tracked as" pointer used where no term is stated. @@ -65,16 +70,6 @@ #: let a floor be satisfied by the gate quoting itself. _GATE_FILE = "test/unit/core/test_planned_referent.py" -#: Audit-sink deferrals that must remain reachable by the scan. A fixture -#: compared against a walk of the tree, not a constant compared against another -#: constant: if a site is deleted, renamed, or reflowed out of the scan's reach, -#: the walk stops finding it and this fails. -EXPECTED_SITES = ( - "agent_assembly/core/audit_sink.py", - "agent_assembly/adapters/_shared/tool_governance.py", - "test/unit/test_quickstart_negative_control.py", -) - def _repo_root() -> Path: """Walk up to the directory holding ``pyproject.toml``. @@ -123,65 +118,104 @@ def _deferral_sites() -> list[tuple[str, int, str, str]]: if _SKIPPED_DIRS.intersection(rel.parts) or str(rel) == _GATE_FILE: continue - lines = path.read_text(encoding="utf-8", errors="replace").splitlines() - for index, line in enumerate(lines): - if not _FORWARD_CLAIM.search(line): - continue + try: + body = path.read_text(encoding="utf-8", errors="replace") + except OSError: + # Removed between the directory listing and the read — a build + # artefact, an editor temp file, a package manager's scratch + # directory. Letting it raise would abort the whole scan, turning a + # gate whose verdict is "no findings" into one that produced no + # verdict; the node SDK's equivalent walk hit exactly that in CI. A + # file that no longer exists carries no claim, so skipping it is + # safe; a scan that reaches nothing is the dangerous failure, and + # test_the_deferral_scan_can_see is what catches that. + continue - # Extend to the next line only when this line carries no ticket of - # its own AND does not end a sentence. Without the sentence guard - # the window pairs a claim with a ticket belonging to the *next* - # sentence — review produced a real case where an inserted line of - # forward-looking prose was blamed for a correct backward citation - # beneath it. There are 33 such backward citations in this repo. - window = line - if not _TICKET_REF.search(line) and not _ends_sentence(line) and index + 1 < len(lines): - window = f"{line}\n{lines[index + 1]}" + sites.extend(deferrals_in_lines(str(rel), body.splitlines())) - ticket = _TICKET_REF.search(window) - if ticket is None: - continue + return sites - sites.append((str(rel), index + 1, ticket.group(0), line.strip())) - return sites +def deferrals_in_lines(path: str, lines: list[str]) -> list[tuple[str, int, str, str]]: + """The detector, split out from the walk. + + Separated so a control can drive it over input it constructs rather than over + whatever the tree happens to contain: a gate whose expected result is + "nothing found" is only as good as the proof that it can find something. + """ + sites: list[tuple[str, int, str, str]] = [] + for index, line in enumerate(lines): + if not _FORWARD_CLAIM.search(line): + continue + # Extend to the next line only when this line carries no ticket of its + # own AND does not end a sentence. Without the sentence guard the window + # pairs a claim with a ticket belonging to the *next* sentence — review + # produced a real case where an inserted line of forward-looking prose + # was blamed for a correct backward citation beneath it. There are 33 + # such backward citations in this repo. + window = line + if not _TICKET_REF.search(line) and not _ends_sentence(line) and index + 1 < len(lines): + window = f"{line}\n{lines[index + 1]}" + + ticket = _TICKET_REF.search(window) + if ticket is None: + continue -def test_forward_claims_name_the_right_ticket() -> None: - guarded = set(EXPECTED_SITES) - problems = [] - - for path, lineno, ticket, text in _deferral_sites(): - if path in guarded: - if ticket != CAPABILITY_REFERENT: - problems.append( - f"{path}:{lineno} is a guarded audit-sink deferral and must " - f"name {CAPABILITY_REFERENT}, not {ticket} — this is the site " - f"the referent change corrected, and letting it drift to any " - f"other ticket is what this gate exists to prevent: {text}" - ) - elif ticket in STALE_REFERENTS: - problems.append( - f"{path}:{lineno} defers to {ticket}, which measured the gap and " - f"will not fix it — use the ticket that builds the sink " - f"(AAASM-5750, per its own description): {text}" - ) + sites.append((path, index + 1, ticket.group(0), line.strip())) + return sites - assert not problems, "\n".join(problems) +def test_no_forward_claim_defers_to_a_finished_ticket() -> None: + problems = [ + f"{path}:{lineno} defers to {ticket}, which {STALE_REFERENTS[ticket]} — a " + f"forward-looking claim must not point at it: {text}" + for path, lineno, ticket, text in _deferral_sites() + if ticket in STALE_REFERENTS + ] + assert not problems, "\n".join(problems) -def test_every_expected_site_is_still_reachable() -> None: - """Anti-vacuity, and the reason it names paths rather than counting. - A count can be held up by an unrelated site appearing as a real one is - deleted. Naming them makes that substitution visible. +@pytest.mark.parametrize( + ("lines", "ticket"), + [ + (["# recording here is Planned (AAASM-5731), not Observed."], "AAASM-5731"), + ( + [ + "# Under ADR 0033 section 6 SDK-side recording is Planned", + "# (AAASM-5750), not Observed.", + ], + "AAASM-5750", + ), + (["# Wiring a sink that retains it is tracked as AAASM-5681."], "AAASM-5681"), + ], + ids=["one line", "wrapped onto two lines", "termless 'tracked as'"], +) +def test_the_deferral_scan_can_see(lines: list[str], ticket: str) -> None: + """Positive control for the assertion above. + + That assertion expects to find nothing, and every way of breaking the scan — + a regex that stops matching, a walk that reaches no files, a window that + never extends across a wrapped comment — produces exactly the same green. So + the detector is fed input containing each shape it is supposed to catch and + required to catch it. If it stops seeing these, the repository-wide silence + stops meaning anything. """ - seen = {site[0] for site in _deferral_sites()} - missing = [path for path in EXPECTED_SITES if path not in seen] - assert not missing, "\n".join( - f"{path} carries no forward claim the scan can pair with a ticket; it " - f"was deleted, renamed, or reflowed so the term and the ticket are more " - f"than one line apart — in which case a stale referent there would no " - f"longer be checked" - for path in missing + found = deferrals_in_lines("synthetic.py", lines) + assert len(found) == 1 and found[0][2] == ticket, ( + f"the detector found {found} in {lines}; it must find exactly one deferral naming " + f"{ticket}, or the repository-wide empty result proves nothing" ) + assert ticket in STALE_REFERENTS, f"{ticket} is not forbidden, so this control could not fail the gate" + + +def test_an_unrelated_open_deferral_is_detected_and_permitted() -> None: + """The other direction: the gate must not forbid every ticket. + + Without this it could pass by rejecting all deferrals, which would push + authors to drop the ticket reference §6 requires rather than to fix the + referent. + """ + found = deferrals_in_lines("synthetic.py", ["# A curated example is Planned (AAASM-9999)."]) + assert len(found) == 1, f"the detector missed an unrelated roadmap deferral: {found}" + assert found[0][2] not in STALE_REFERENTS diff --git a/test/unit/test_quickstart_negative_control.py b/test/unit/test_quickstart_negative_control.py index 90793b94..87a3f5a7 100644 --- a/test/unit/test_quickstart_negative_control.py +++ b/test/unit/test_quickstart_negative_control.py @@ -296,13 +296,12 @@ def test_a_denied_call_emits_an_audit_record_carrying_the_agent_and_tool( # straight past the hook, so a denied call offered it nothing. # # Scope of the evidence: the record is captured by this fixture's - # handler. The interceptor the SDK builds resolves no audit hook at all - # (RuntimeQueryInterceptor + GatewayClient expose neither - # record_result nor on_tool_end), so tool outcomes produce no audit - # evidence on the shipped path — Planned under ADR 0033 §6 - # (AAASM-5750), not Unmeasured, since where the record stops has been - # measured. What this pins is the governance flow's call — the part - # fixable without wiring a sink. + # handler, so what this pins is the governance flow's CALL and its + # contents — nothing about where the record ends up. A fixture that + # receives what it supplied cannot decide that in either direction. + # That the shipped interceptor then forwards the record across the + # native boundary is measured where it can be, against that boundary, + # in test/unit/core/test_audit_sink_disposition.py (AAASM-5750). assert len(quickstart.interceptor.records) == 1 record = quickstart.interceptor.records[0] assert record.tool_name == "write_to_disk"