diff --git a/docs/content/docs/cli/agentkit-cli.en.mdx b/docs/content/docs/cli/agentkit-cli.en.mdx index 4ab3d396..28ea681a 100644 --- a/docs/content/docs/cli/agentkit-cli.en.mdx +++ b/docs/content/docs/cli/agentkit-cli.en.mdx @@ -38,6 +38,35 @@ veadk agentkit init veadk agentkit launch ``` +## Invoke HarnessApp Runtime + +`veadk agentkit invoke` can call a HarnessApp Runtime directly and enable Harness Extension for a single request. Common flags: + +| Flag | Description | +| :-- | :-- | +| `--harness` | Agent name inside HarnessApp. | +| `--endpoint` | Runtime endpoint. | +| `--apikey` | Runtime API key. | +| `--model-id` | One-shot model override. | +| `--tools` | One-shot tool override, comma-separated. | +| `--skills` | One-shot skill override, comma-separated. | +| `--enable-harness-enhance` | Enable Harness Extension for this request. | +| `--harness-components` | Enabled Harness components for this request, comma-separated. | +| `--harness-profile` | Optional runtime profile. | +| `--harness-compression-provider` | Optional compaction provider, default `builtin`. | + +```bash +veadk agentkit invoke \ + --harness research-agent \ + --endpoint "$HARNESS_URL" \ + --apikey "$HARNESS_KEY" \ + --model-id "your-model-id" \ + --tools web_search,run_code \ + --enable-harness-enhance \ + --harness-components "invocation_context,compactor,response_verification" \ + "Summarize the tool results with evidence." +``` + ## Further reference For the full parameter details of each subcommand, see the [AgentKit CLI](https://volcengine.github.io/agentkit-sdk-python/content/2.agentkit-cli/1.overview.html). diff --git a/docs/content/docs/cli/agentkit-cli.mdx b/docs/content/docs/cli/agentkit-cli.mdx index c1494242..1bd27d86 100644 --- a/docs/content/docs/cli/agentkit-cli.mdx +++ b/docs/content/docs/cli/agentkit-cli.mdx @@ -38,6 +38,35 @@ veadk agentkit init veadk agentkit launch ``` +## 调用 HarnessApp Runtime + +`veadk agentkit invoke` 可以直接调用 HarnessApp Runtime，并通过请求参数临时开启 Harness Extension。常用参数如下： + +| 参数 | 说明 | +| :-- | :-- | +| `--harness` | HarnessApp 中的智能体名称。 | +| `--endpoint` | Runtime endpoint。 | +| `--apikey` | Runtime API Key。 | +| `--model-id` | 单次调用覆盖模型名称。 | +| `--tools` | 单次调用覆盖工具列表，逗号分隔。 | +| `--skills` | 单次调用覆盖技能列表，逗号分隔。 | +| `--enable-harness-enhance` | 本次调用启用 Harness Extension。 | +| `--harness-components` | 本次调用启用的 Harness 组件，逗号分隔。 | +| `--harness-profile` | 可选运行 profile。 | +| `--harness-compression-provider` | 可选压缩 provider，默认 `builtin`。 | + +```bash +veadk agentkit invoke \ + --harness research-agent \ + --endpoint "$HARNESS_URL" \ + --apikey "$HARNESS_KEY" \ + --model-id "your-model-id" \ + --tools web_search,run_code \ + --enable-harness-enhance \ + --harness-components "invocation_context,compactor,response_verification" \ + "Summarize the tool results with evidence." +``` + ## 更多参考 各子命令的完整参数说明请参阅 [AgentKit 命令行工具](https://volcengine.github.io/agentkit-sdk-python/content/2.agentkit-cli/1.overview.html)。 diff --git a/docs/content/docs/references/configuration/environment-variables.en.mdx b/docs/content/docs/references/configuration/environment-variables.en.mdx index 9120c02c..bf84a4cd 100644 --- a/docs/content/docs/references/configuration/environment-variables.en.mdx +++ b/docs/content/docs/references/configuration/environment-variables.en.mdx @@ -62,6 +62,23 @@ Prefix `TOOL_`, configured only when the corresponding tool is used. | VOD | `TOOL_VOD_GROUPS` | Video editing capability groups | | | `TOOL_VOD_TIMEOUT` | Connection timeout | +## Harness Extension + +Prefix `HARNESS_`, used to attach optional Harness plugins to HarnessApp Runtime or `Runner`. The Harness Extension is bundled with VeADK; install the `veadk-python[harness]` extra only when you want the optional `headroom` compaction provider. + +| Variable | Meaning | +| :- | :- | +| `HARNESS_ENHANCE_ENABLED` | Enable Harness plugins; accepts `true` / `false`. | +| `HARNESS_ENHANCE_COMPONENTS` | Enabled components, comma-separated; common value: `invocation_context,compactor,response_verification`. | +| `HARNESS_PROFILE` | Plugin runtime profile for policy selection; default `default`. | +| `HARNESS_COMPRESSION_PROVIDER` | Tool-result compaction provider, default `builtin`; optional `headroom`. | +| `HARNESS_MAX_CONTEXT_CHARS` | Context compaction threshold, default `24000`. | +| `HARNESS_MAX_TOOL_RESULT_CHARS` | Single tool-result compaction threshold, default `4000`. | +| `HARNESS_VERIFIER_MODE` | Final-response verification mode, `observe` or `block`; default `observe`. | +| `HARNESS_STORE_PATH` | Optional JSONL event store path; in-memory store is used when unset. | + +The `harness_enhance` block maps to these environment variables when deploying a HarnessApp Runtime. Prefer `harness.yaml` or `veadk agentkit invoke` flags for normal developer workflows; use environment variables for platform integration and container runtimes. + ## Databases Prefix `DATABASE_`, grouped by storage type. Memory and the knowledge base read the matching config based on the selected backend. diff --git a/docs/content/docs/references/configuration/environment-variables.mdx b/docs/content/docs/references/configuration/environment-variables.mdx index 213d5e99..3ef8ba0c 100644 --- a/docs/content/docs/references/configuration/environment-variables.mdx +++ b/docs/content/docs/references/configuration/environment-variables.mdx @@ -62,6 +62,23 @@ volcengine: | VOD | `TOOL_VOD_GROUPS` | 视频编辑能力组 | | | `TOOL_VOD_TIMEOUT` | 连接超时时长 | +## Harness Extension + +统一前缀 `HARNESS_`，用于给 HarnessApp Runtime 或 `Runner` 挂载可选的 Harness 插件能力。Harness Extension 随 VeADK 安装；只有当选择 `headroom` 压缩 provider 时，才需要安装 `veadk-python[harness]` 额外依赖。 + +| 环境变量 | 释义 | +| :- | :- | +| `HARNESS_ENHANCE_ENABLED` | 是否启用 Harness 插件，支持 `true` / `false`。 | +| `HARNESS_ENHANCE_COMPONENTS` | 启用的组件，逗号分隔；常用值为 `invocation_context,compactor,response_verification`。 | +| `HARNESS_PROFILE` | 插件运行 profile，用于区分运行策略；默认 `default`。 | +| `HARNESS_COMPRESSION_PROVIDER` | 工具结果压缩 provider，默认 `builtin`；可选 `headroom`。 | +| `HARNESS_MAX_CONTEXT_CHARS` | 上下文压缩阈值，默认 `24000`。 | +| `HARNESS_MAX_TOOL_RESULT_CHARS` | 单个工具结果压缩阈值，默认 `4000`。 | +| `HARNESS_VERIFIER_MODE` | 最终回答校验模式，`observe` 或 `block`，默认 `observe`。 | +| `HARNESS_STORE_PATH` | 可选 JSONL 事件存储路径；不设置时使用内存存储。 | + +`harness_enhance` 配置块会在 HarnessApp Runtime 部署时映射为这些环境变量。推荐开发者优先通过 `harness.yaml` 或 `veadk agentkit invoke` 参数启用，环境变量适合平台集成和镜像运行时。 + ## 数据库 统一前缀 `DATABASE_`，按存储类型分组。记忆与知识库按所选后端读取对应配置。 diff --git a/docs/extensions/harness/README.md b/docs/extensions/harness/README.md new file mode 100644 index 00000000..63345a43 --- /dev/null +++ b/docs/extensions/harness/README.md @@ -0,0 +1,281 @@ +# VeADK Harness Extension + +[中文](README.zh.md) + +VeADK Harness is a lightweight extension for tool-using VeADK agents. It adds +runtime governance around the agent call path without changing your agent, +model, or tool implementations. + +Use it when your agent handles large tool outputs, long-running multi-step +tasks, or final answers that must stay grounded in tool evidence. + +## Install + +```bash +pip install "veadk-python[harness]" +``` + +The Harness extension ships with VeADK. The `harness` extra installs optional +dependencies such as the in-process Headroom compaction provider. If you only +use the default `builtin` provider, no extra service is required. + +## Quick Start + +Attach Harness plugins when you create the `Runner`: + +```python +from veadk import Agent, Runner +from veadk.extensions.harness.plugins import build_harness_plugins + + +agent = Agent( + name="research_agent", + instruction="Answer with evidence from tool results.", +) + +runner = Runner( + agent=agent, + app_name="research_app", + plugins=build_harness_plugins( + components=[ + "invocation_context", + "compactor", + "response_verification", + ], + ), +) +``` + +Start with only compaction if you want the smallest measurable change: + +```python +plugins = build_harness_plugins(components=["compactor"]) +``` + +## Components + +| Component | Plugin | What it adds | +| --- | --- | --- | +| `invocation_context` | `HarnessInvocationContextPlugin` | Injects task anchors, recent context, and tool-use guidance before model calls. | +| `compactor` | `HarnessCompressPlugin` | Compacts oversized tool results and old function responses. | +| `response_verification` | `HarnessResponseVerificationPlugin` | Records tool receipts and checks whether final answers are supported. | +| `long_run_control` | `HarnessLongRunControlPlugin` | Adds finish-oriented guidance when a run approaches its model-call budget. | + +## Core Concepts + +| Concept | Meaning | +| --- | --- | +| Harness module | Standalone capability that can be imported and tested directly. | +| Harness plugin | VeADK runtime wrapper that connects a module to lifecycle callbacks. | +| Invocation context block | Compact context injected before a model call. | +| Tool receipt | Short record of a tool call, its status, and useful evidence. | +| Compaction report | Metrics for how much context or tool output was reduced. | +| Verification report | Result showing whether the final answer is supported by evidence. | + +## Source Layout + +| Path | Purpose | +| --- | --- | +| `veadk/extensions/harness/extension.py` | Facade for creating Harness plugins from code. | +| `veadk/extensions/harness/env.py` | Builds plugins from environment variables. | +| `veadk/extensions/harness/schemas.py` | Public Pydantic models such as `ToolReceipt`, `CompactionReport`, and `VerificationReport`. | +| `veadk/extensions/harness/modules/invocation_context/` | Atomic invocation-context builder. | +| `veadk/extensions/harness/modules/tool_result_compactor/` | Atomic compactor plus builtin and Headroom providers. | +| `veadk/extensions/harness/modules/final_response_verifier/` | Atomic final-response verifier. | +| `veadk/extensions/harness/plugins/entrypoints.py` | Public plugin entry points. | +| `veadk/extensions/harness/plugins/builder/` | Shared-store plugin bundle assembly. | +| `veadk/extensions/harness/plugins/invocation_context/` | Invocation-context callback plugin. | +| `veadk/extensions/harness/plugins/compactor/` | Tool-result and context compaction callback plugin. | +| `veadk/extensions/harness/plugins/response_verification/` | Receipt recording and final-response verification callback plugin. | +| `veadk/extensions/harness/plugins/long_run_control/` | Long-run guidance callback plugin. | +| `veadk/extensions/harness/plugins/_shared/` | Internal callback helpers shared by plugins. | +| `veadk/extensions/harness/stores/` | Store protocol and in-memory or JSONL implementations. | + +## Runtime Flow + +```text +user message + -> invocation_context builds and injects a compact context block + -> model call + -> tool call + -> compactor reduces oversized tool output + -> model continues with compacted evidence + -> response_verification checks final answer support + -> response is returned +``` + +The plugins share a store. The context plugin records messages, the compactor +writes compaction events, and the verifier reads tool receipts before adding +verification metadata or blocking unsupported answers when configured to do so. + +## Use Atomic Modules + +You can use modules directly without plugins. This is useful for unit tests, +custom runtimes, and focused integration checks. + +```python +from veadk.extensions.harness import ( + HarnessInvocationContextBuilder, + HarnessInvocationRef, +) + + +context = HarnessInvocationRef(session_id="session-1", invocation_id="run-1") +block = HarnessInvocationContextBuilder().prepare_context( + context=context, + user_input="Compare the tool results and explain the conclusion.", +) +print(block.header) +``` + +```python +from veadk.extensions.harness.modules.tool_result_compactor import ToolResultCompactor + + +tool_result = { + "title": "Search results", + "content": "important result\n" + ("raw text\n" * 2000), +} + +compacted, report = ToolResultCompactor().compress_tool_result(tool_result) +print(compacted) +print(report.model_dump()) +``` + +```python +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifier, +) +from veadk.extensions.harness.schemas import ToolReceipt + + +receipts = [ + ToolReceipt( + name="web_search", + status="success", + summary="The release date is 2026-05-01.", + ) +] + +report = FinalResponseVerifier().verify_text( + "The release date is 2026-05-01.", + receipts=receipts, +) +print(report.model_dump()) +``` + +## Runtime Configuration + +Use environment variables for deployed runtimes: + +```bash +export HARNESS_ENHANCE_ENABLED=true +export HARNESS_ENHANCE_COMPONENTS=invocation_context,compactor,response_verification +export HARNESS_COMPRESSION_PROVIDER=builtin +export HARNESS_VERIFIER_MODE=observe +``` + +Equivalent YAML: + +```yaml +harness_enhance: + enabled: true + components: [invocation_context, compactor, response_verification] + compression_provider: builtin + verifier_mode: observe +``` + +Enable per request with the CLI: + +```bash +veadk agentkit invoke \ + --harness my-agent \ + --endpoint "$HARNESS_URL" \ + --apikey "$HARNESS_KEY" \ + --enable-harness-enhance \ + --harness-components "invocation_context,compactor,response_verification" \ + "Summarize the tool results with evidence." +``` + +## Configuration Reference + +| Setting | Default | Meaning | +| --- | --- | --- | +| `HARNESS_ENHANCE_ENABLED` | `false` | Enables Harness plugins for runtime assembly. | +| `HARNESS_ENHANCE_COMPONENTS` | `invocation_context,compactor,response_verification` | Selects enabled components. | +| `HARNESS_COMPRESSION_PROVIDER` | `builtin` | Compaction provider: `builtin` or `headroom`. | +| `HARNESS_MAX_CONTEXT_CHARS` | `24000` | Context compaction threshold. | +| `HARNESS_MAX_TOOL_RESULT_CHARS` | `4000` | Tool-result compaction threshold. | +| `HARNESS_VERIFIER_MODE` | `observe` | Verification behavior: `observe` or `block`. | +| `HARNESS_STORE_PATH` | unset | Uses a JSONL event store when set. | + +## Compaction Providers + +The default `builtin` provider is generic and dependency-free. It does not rely +on a task prompt, a tool name, or a business-specific output schema. For +JSON-like results, it walks mappings and sequences with bounded depth, keeps +representative facts, replaces very long scalar values with shape information, +records omitted counts, and applies basic redaction before writing summaries. + +A compacted tool response includes a marker such as: + +```json +{ + "harness_compressed": true, + "provider": "builtin", + "summary": "...", + "original_chars": 8033 +} +``` + +When `HARNESS_COMPRESSION_PROVIDER=headroom` is configured and the optional +dependency is installed, Harness lazy-loads Headroom through Python imports and +calls it in-process. It does not start a service. If Headroom is unavailable or +returns an invalid result, Harness falls back to the builtin provider. + +## Recommended Defaults + +| Setting | Suggested value | Reason | +| --- | --- | --- | +| `components` | `invocation_context,compactor,response_verification` | Balanced default for tool-heavy agents. | +| `compression_provider` | `builtin` | Stable and dependency-free. | +| `max_tool_result_chars` | `4000` | Compresses clearly large results while leaving small results untouched. | +| `max_context_chars` | `24000` | Preserves enough multi-step context while limiting prompt growth. | +| `verifier_mode` | `observe` | Lets you inspect verification reports before blocking answers. | + +## Validate Impact + +Start with these checks: + +| Signal | What good looks like | +| --- | --- | +| Tool result compaction | Large tool responses include `harness_compressed: true` and `compressed_chars < original_chars`. | +| Prompt size reduction | Later model calls carry less raw tool context. | +| Answer support | Unsupported completion claims are detected in verification metadata. | +| Task quality | The final answer remains correct while latency or token usage improves for long-output tasks. | + +## FAQ + +### Do I need every component? + +No. `compactor` is the easiest component to measure first. Add +`invocation_context` and `response_verification` when you need stronger context +control and answer grounding. + +### Can compaction lose important details? + +Compaction is designed to preserve structured facts, titles, links, numbers, +errors, and short summaries. For tasks that require exact original text, raise +the compaction thresholds or enable compaction only for selected use cases. + +### Does verification block answers? + +The default mode is `observe`, which records verification metadata without +blocking. Use `block` only after you have validated that the rules fit your +application. + +### Does agent code need major changes? + +Usually no. Add `plugins=build_harness_plugins(...)` when constructing the +`Runner`; your tools, model settings, and agent instructions can keep +their existing structure. diff --git a/docs/extensions/harness/README.zh.md b/docs/extensions/harness/README.zh.md new file mode 100644 index 00000000..5e50ae0f --- /dev/null +++ b/docs/extensions/harness/README.zh.md @@ -0,0 +1,258 @@ +# VeADK Harness Extension 使用指南 + +[English](README.md) + +VeADK Harness 是面向工具型 VeADK Agent 的轻量扩展。它在 Agent 调用链路外增加运行时治理能力，不要求你重写 Agent、模型或工具实现。 + +当你的 Agent 会处理大工具结果、多步长任务，或者最终回答必须基于工具证据时，可以使用 Harness。 + +## 安装 + +```bash +pip install "veadk-python[harness]" +``` + +Harness Extension 随 VeADK 内置。`harness` extra 会安装可选依赖，例如进程内 Headroom 压缩 provider。如果只使用默认 `builtin` provider，不需要额外服务。 + +## 快速接入 + +创建 `Runner` 时挂载 Harness plugins： + +```python +from veadk import Agent, Runner +from veadk.extensions.harness.plugins import build_harness_plugins + + +agent = Agent( + name="research_agent", + instruction="Answer with evidence from tool results.", +) + +runner = Runner( + agent=agent, + app_name="research_app", + plugins=build_harness_plugins( + components=[ + "invocation_context", + "compactor", + "response_verification", + ], + ), +) +``` + +如果你想先做最小验证，可以只启用压缩： + +```python +plugins = build_harness_plugins(components=["compactor"]) +``` + +## 组件 + +| Component | Plugin | 能力 | +| --- | --- | --- | +| `invocation_context` | `HarnessInvocationContextPlugin` | 模型调用前注入任务锚点、近期上下文和工具使用约束。 | +| `compactor` | `HarnessCompressPlugin` | 压缩过大的工具结果和旧 function response。 | +| `response_verification` | `HarnessResponseVerificationPlugin` | 记录 tool receipt，并检查最终回答是否有证据支撑。 | +| `long_run_control` | `HarnessLongRunControlPlugin` | 当运行接近模型调用预算时，注入面向收敛的引导。 | + +## 核心概念 + +| 概念 | 含义 | +| --- | --- | +| Harness module | 可直接 import 和测试的独立原子能力。 | +| Harness plugin | 把原子能力接入 VeADK 生命周期回调的运行时包装。 | +| Invocation context block | 模型调用前注入的精简上下文块。 | +| Tool receipt | 工具调用的简短记录，包括状态和关键证据。 | +| Compaction report | 描述上下文或工具结果压缩效果的指标。 | +| Verification report | 描述最终回答是否有证据支撑的校验结果。 | + +## 源码结构 + +| 路径 | 作用 | +| --- | --- | +| `veadk/extensions/harness/extension.py` | 从代码创建 Harness plugins 的 facade。 | +| `veadk/extensions/harness/env.py` | 从环境变量组装 plugins。 | +| `veadk/extensions/harness/schemas.py` | 公共 Pydantic 模型，例如 `ToolReceipt`、`CompactionReport`、`VerificationReport`。 | +| `veadk/extensions/harness/modules/invocation_context/` | 调用上下文构造原子模块。 | +| `veadk/extensions/harness/modules/tool_result_compactor/` | 工具结果压缩原子模块，以及 builtin / Headroom providers。 | +| `veadk/extensions/harness/modules/final_response_verifier/` | 最终回答校验原子模块。 | +| `veadk/extensions/harness/plugins/entrypoints.py` | 对外 plugin 入口。 | +| `veadk/extensions/harness/plugins/builder/` | 共享 store 的 plugin bundle 组装逻辑。 | +| `veadk/extensions/harness/plugins/invocation_context/` | 调用上下文回调 plugin。 | +| `veadk/extensions/harness/plugins/compactor/` | 工具结果和上下文压缩回调 plugin。 | +| `veadk/extensions/harness/plugins/response_verification/` | Receipt 记录和最终回答校验回调 plugin。 | +| `veadk/extensions/harness/plugins/long_run_control/` | 长任务收敛引导回调 plugin。 | +| `veadk/extensions/harness/plugins/_shared/` | 多个 plugin 共享的内部回调工具。 | +| `veadk/extensions/harness/stores/` | Store 协议，以及内存 / JSONL 实现。 | + +## 运行链路 + +```text +用户消息 + -> invocation_context 构造并注入精简上下文块 + -> 模型调用 + -> 工具调用 + -> compactor 压缩过大的工具结果 + -> 模型基于压缩后的证据继续推理 + -> response_verification 校验最终回答证据 + -> 返回结果 +``` + +这些 plugins 共享一个 store。上下文 plugin 记录消息，压缩 plugin 写入压缩事件，校验 plugin 读取 tool receipt，并根据配置写入 verification metadata 或阻断缺少证据的回答。 + +## 直接使用原子模块 + +你也可以不挂 plugin，直接使用原子模块。这适合单元测试、自定义运行时或局部集成验证。 + +```python +from veadk.extensions.harness import ( + HarnessInvocationContextBuilder, + HarnessInvocationRef, +) + + +context = HarnessInvocationRef(session_id="session-1", invocation_id="run-1") +block = HarnessInvocationContextBuilder().prepare_context( + context=context, + user_input="Compare the tool results and explain the conclusion.", +) +print(block.header) +``` + +```python +from veadk.extensions.harness.modules.tool_result_compactor import ToolResultCompactor + + +tool_result = { + "title": "Search results", + "content": "important result\n" + ("raw text\n" * 2000), +} + +compacted, report = ToolResultCompactor().compress_tool_result(tool_result) +print(compacted) +print(report.model_dump()) +``` + +```python +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifier, +) +from veadk.extensions.harness.schemas import ToolReceipt + + +receipts = [ + ToolReceipt( + name="web_search", + status="success", + summary="The release date is 2026-05-01.", + ) +] + +report = FinalResponseVerifier().verify_text( + "The release date is 2026-05-01.", + receipts=receipts, +) +print(report.model_dump()) +``` + +## Runtime 配置 + +部署运行时可以使用环境变量： + +```bash +export HARNESS_ENHANCE_ENABLED=true +export HARNESS_ENHANCE_COMPONENTS=invocation_context,compactor,response_verification +export HARNESS_COMPRESSION_PROVIDER=builtin +export HARNESS_VERIFIER_MODE=observe +``` + +等价 YAML： + +```yaml +harness_enhance: + enabled: true + components: [invocation_context, compactor, response_verification] + compression_provider: builtin + verifier_mode: observe +``` + +也可以在单次请求中通过 CLI 打开： + +```bash +veadk agentkit invoke \ + --harness my-agent \ + --endpoint "$HARNESS_URL" \ + --apikey "$HARNESS_KEY" \ + --enable-harness-enhance \ + --harness-components "invocation_context,compactor,response_verification" \ + "Summarize the tool results with evidence." +``` + +## 配置速查 + +| 配置 | 默认值 | 说明 | +| --- | --- | --- | +| `HARNESS_ENHANCE_ENABLED` | `false` | 是否在运行时组装 Harness plugins。 | +| `HARNESS_ENHANCE_COMPONENTS` | `invocation_context,compactor,response_verification` | 启用哪些组件。 | +| `HARNESS_COMPRESSION_PROVIDER` | `builtin` | 压缩 provider，支持 `builtin` 或 `headroom`。 | +| `HARNESS_MAX_CONTEXT_CHARS` | `24000` | 上下文压缩阈值。 | +| `HARNESS_MAX_TOOL_RESULT_CHARS` | `4000` | 工具结果压缩阈值。 | +| `HARNESS_VERIFIER_MODE` | `observe` | 校验行为，支持 `observe` 或 `block`。 | +| `HARNESS_STORE_PATH` | 未设置 | 设置后使用 JSONL event store。 | + +## 压缩 Provider + +默认 `builtin` provider 是通用、无额外依赖的实现。它不依赖任务 prompt、工具名称或业务特定返回 schema。对于 JSON-like 结果，它会有界遍历 mapping 和 sequence，保留代表性事实，把超长标量替换为形状信息，记录省略项数量，并在写回摘要前做基础脱敏。 + +压缩后的工具响应会包含类似标记： + +```json +{ + "harness_compressed": true, + "provider": "builtin", + "summary": "...", + "original_chars": 8033 +} +``` + +当配置 `HARNESS_COMPRESSION_PROVIDER=headroom` 且安装了可选依赖时，Harness 会通过 Python import 懒加载 Headroom，并在当前进程内调用它。它不会启动服务。如果 Headroom 不可用或返回不合法结果，Harness 会回退到 builtin provider。 + +## 推荐默认值 + +| 配置 | 建议值 | 原因 | +| --- | --- | --- | +| `components` | `invocation_context,compactor,response_verification` | 适合工具型 Agent 的均衡默认组合。 | +| `compression_provider` | `builtin` | 稳定、无额外依赖。 | +| `max_tool_result_chars` | `4000` | 只压缩明显过大的结果，小结果保持原样。 | +| `max_context_chars` | `24000` | 给多步任务保留足够上下文，同时限制 prompt 增长。 | +| `verifier_mode` | `observe` | 先观察校验报告，再决定是否阻断回答。 | + +## 如何验证效果 + +建议从这些信号开始： + +| 信号 | 好的表现 | +| --- | --- | +| 工具结果压缩 | 大工具响应包含 `harness_compressed: true`，且 `compressed_chars < original_chars`。 | +| Prompt 大小下降 | 后续模型调用携带的原始工具上下文更少。 | +| 回答证据支撑 | 缺少证据的完成类声明能在 verification metadata 中被发现。 | +| 任务质量 | 最终回答仍正确，同时长输出任务的延迟或 token 使用下降。 | + +## 常见问题 + +### 需要一次开启所有组件吗？ + +不需要。`compactor` 最容易先通过上下文大小和 token 指标验证。需要更强的上下文控制和回答可靠性时，再加入 `invocation_context` 与 `response_verification`。 + +### 压缩会不会丢掉关键信息？ + +压缩目标是保留结构化事实、标题、链接、数值、错误信息和短摘要。对于强依赖原文的任务，可以调高压缩阈值，或只在特定场景启用压缩。 + +### 校验会不会直接阻断回答？ + +默认模式是 `observe`，只记录 verification metadata，不阻断。只有确认规则适合你的应用后，再使用 `block`。 + +### Agent 主代码需要大改吗？ + +通常不需要。创建 `Runner` 时增加 `plugins=build_harness_plugins(...)` 即可；工具、模型配置和 Agent 指令可以保持原有结构。 diff --git a/examples/harness_app_runtime/README.md b/examples/harness_app_runtime/README.md new file mode 100644 index 00000000..08f9df2a --- /dev/null +++ b/examples/harness_app_runtime/README.md @@ -0,0 +1,43 @@ +# HarnessApp Runtime Example + +When deploying a VeADK HarnessApp runtime, enable Harness plugins through the +runtime configuration: + +```yaml +harness_enhance: + enabled: true + components: [invocation_context, compactor, response_verification] + profile: general + compression_provider: builtin +``` + +The runtime reads this configuration as environment variables and attaches the +plugin bundle to the VeADK Runner. + +## Local Latency And Context Benchmark + +Run the local HarnessApp benchmark from the repository root: + +```bash +PYTHONPATH=. python \ + examples/harness_app_runtime/stable_latency_token_case.py \ + --repeats 3 +``` + +If your local environment needs model or runtime variables, pass an env file: + +```bash +PYTHONPATH=. python \ + examples/harness_app_runtime/stable_latency_token_case.py \ + --env-file /path/to/.env \ + --repeats 3 +``` + +The script starts a local HarnessApp Runtime, invokes it through +`veadk agentkit invoke`, and compares: + +- no enhancement headers +- `--enable-harness-enhance` with built-in compression + +It prints a JSON report with latency, prompt-context size, compression impact, +and answer consistency. diff --git a/examples/harness_app_runtime/stable_latency_token_case.py b/examples/harness_app_runtime/stable_latency_token_case.py new file mode 100644 index 00000000..34f40981 --- /dev/null +++ b/examples/harness_app_runtime/stable_latency_token_case.py @@ -0,0 +1,429 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Local HarnessApp Runtime latency/token-shape benchmark. + +This script starts a local HarnessApp Runtime and invokes it through +`veadk agentkit invoke`. It compares the same runtime in two modes: + +* no_enhance: no Harness enhancement headers. +* harness_enhance: `--enable-harness-enhance` with built-in compression. + +The model is deterministic so the test is stable on a developer laptop while +still exercising the real Runtime, Runner, Harness plugin, tool callback, HTTP +Runtime, and CLI invocation path. +""" + +from __future__ import annotations + +import argparse +import asyncio +import json +import os +import re +import socket +import statistics +import subprocess +import sys +import threading +import time +from collections.abc import AsyncGenerator +from pathlib import Path + +import uvicorn +from google.adk.models.base_llm import BaseLlm +from google.adk.models.llm_response import LlmResponse +from google.genai import types +from pydantic import BaseModel, ConfigDict, PrivateAttr + + +PROMPT = ( + "stable-latency-token-case: call comparison_payload once, then identify the " + "best candidate from the returned records. Final answer must include " + "best_model and accuracy." +) + + +class RunMetric(BaseModel): + model_config = ConfigDict(frozen=True) + + mode: str + elapsed_seconds: float + prompt_chars: int + output: str + + +class BenchmarkLlm(BaseLlm): + _delay_divisor: float = PrivateAttr() + _max_delay_seconds: float = PrivateAttr() + + def __init__( + self, + *, + model: str, + delay_divisor: float, + max_delay_seconds: float, + ) -> None: + super().__init__(model=model) + self._delay_divisor = delay_divisor + self._max_delay_seconds = max_delay_seconds + + async def generate_content_async( + self, llm_request: object, stream: bool = False + ) -> AsyncGenerator[LlmResponse, None]: + request_text = _request_text(llm_request) + has_function_response = _has_function_response(llm_request) + if PROMPT in request_text and not has_function_response: + yield LlmResponse( + content=types.Content( + role="model", + parts=[ + types.Part.from_function_call( + name="comparison_payload", args={} + ) + ], + ) + ) + return + + prompt_chars = len(request_text) + delay = min(prompt_chars / self._delay_divisor, self._max_delay_seconds) + await asyncio.sleep(delay) + compressed = "harness_compressed" in request_text + yield LlmResponse( + content=types.Content( + role="model", + parts=[ + types.Part( + text=( + "best_model: candidate-b\n" + "accuracy: 88\n" + f"prompt_chars: {prompt_chars}\n" + f"compressed_context: {str(compressed).lower()}" + ) + ) + ], + ) + ) + + +def main() -> int: + parser = argparse.ArgumentParser() + parser.add_argument("--repeats", type=int, default=3) + parser.add_argument("--rows", type=int, default=2500) + parser.add_argument("--delay-divisor", type=float, default=60000.0) + parser.add_argument("--max-delay-seconds", type=float, default=2.5) + parser.add_argument( + "--env-file", + type=Path, + default=_default_env_file(), + help=( + "Optional env file used to seed local model/runtime variables. " + "Can also be set with HARNESS_BENCHMARK_ENV_FILE." + ), + ) + args = parser.parse_args() + + _load_env_file(args.env_file) + os.environ.setdefault("LOGGING_LEVEL", "WARNING") + os.environ.setdefault("MODEL_AGENT_API_KEY", "local-test-key") + os.environ["HARNESS_ENHANCE_ENABLED"] = "false" + + runtime_port = _free_port() + runtime, server, server_thread = _start_runtime( + port=runtime_port, + rows=args.rows, + delay_divisor=args.delay_divisor, + max_delay_seconds=args.max_delay_seconds, + ) + endpoint = f"http://127.0.0.1:{runtime_port}" + + try: + baseline = [ + _invoke_cli( + mode="no_enhance", + endpoint=endpoint, + session_id=f"baseline-{index}", + enhanced=False, + ) + for index in range(args.repeats) + ] + enhanced = [ + _invoke_cli( + mode="harness_enhance", + endpoint=endpoint, + session_id=f"enhanced-{index}", + enhanced=True, + ) + for index in range(args.repeats) + ] + report = _build_report( + baseline=baseline, + enhanced=enhanced, + plugin_names=[plugin.name for plugin in runtime.plugins], + ) + print(json.dumps(report, ensure_ascii=False, indent=2)) + return 0 if report["passed"] else 1 + finally: + server.should_exit = True + server_thread.join(timeout=10) + + +def _start_runtime( + *, + port: int, + rows: int, + delay_divisor: float, + max_delay_seconds: float, +): + from veadk import Agent + from veadk.cloud.harness_app.app import HarnessApp + from veadk.memory.short_term_memory import ShortTermMemory + + def comparison_payload() -> dict[str, object]: + records = [ + {"name": "candidate-a", "accuracy": 82, "rank": 2}, + {"name": "candidate-b", "accuracy": 88, "rank": 1}, + ] + diagnostics = [ + { + "stage": "trace", + "row": index, + "trace": "diagnostic-noise-" + ("x" * 48), + } + for index in range(rows) + ] + return {"records": records, "diagnostics": diagnostics} + + runtime = HarnessApp( + Agent( + name="local_latency_token_agent", + instruction="Use tools when requested and return the benchmark markers.", + model=BenchmarkLlm( + model="benchmark-fake", + delay_divisor=delay_divisor, + max_delay_seconds=max_delay_seconds, + ), + tools=[comparison_payload], + ), + ShortTermMemory(backend="local"), + harness_name="local_latency_token_case", + max_llm_calls=6, + ) + server = uvicorn.Server( + uvicorn.Config(runtime.app, host="127.0.0.1", port=port, log_level="warning") + ) + server_thread = threading.Thread(target=server.run, daemon=True) + server_thread.start() + for _ in range(100): + if server.started: + return runtime, server, server_thread + time.sleep(0.05) + raise RuntimeError("local HarnessApp Runtime did not start") + + +def _invoke_cli( + *, + mode: str, + endpoint: str, + session_id: str, + enhanced: bool, +) -> RunMetric: + repo_root = Path(__file__).resolve().parents[2] + env = dict(os.environ) + env["PYTHONPATH"] = str(repo_root) + env.setdefault("LOGGING_LEVEL", "WARNING") + command = [ + sys.executable, + "-m", + "veadk.cli.cli", + "agentkit", + "invoke", + "--endpoint", + endpoint, + "--apikey", + "local-test-key", + "--harness", + "local_latency_token_case", + "--user-id", + "local-user", + "--session-id", + session_id, + "--max-llm-calls", + "6", + ] + if enhanced: + command.extend( + [ + "--enable-harness-enhance", + "--harness-components", + "invocation_context,compactor,response_verification", + ] + ) + command.append(PROMPT) + started = time.perf_counter() + completed = subprocess.run( + command, + cwd=repo_root, + env=env, + text=True, + capture_output=True, + check=False, + ) + elapsed = time.perf_counter() - started + output = completed.stdout + completed.stderr + if completed.returncode != 0: + raise RuntimeError(f"{mode} invoke failed:\n{output}") + prompt_chars = _extract_prompt_chars(output) + return RunMetric( + mode=mode, elapsed_seconds=elapsed, prompt_chars=prompt_chars, output=output + ) + + +def _extract_prompt_chars(output: str) -> int: + match = re.search(r"prompt_chars:\s*(\d+)", output) + if not match: + raise RuntimeError(f"prompt_chars marker not found in CLI output:\n{output}") + return int(match.group(1)) + + +def _request_text(llm_request: object) -> str: + values: list[str] = [] + for content in getattr(llm_request, "contents", []) or []: + for part in getattr(content, "parts", []) or []: + text = getattr(part, "text", None) + if text: + values.append(str(text)) + function_call = getattr(part, "function_call", None) + if function_call is not None: + values.append(_json_for_context(function_call)) + function_response = getattr(part, "function_response", None) + if function_response is not None: + values.append(_json_for_context(function_response)) + return "\n".join(values) + + +def _has_function_response(llm_request: object) -> bool: + for content in getattr(llm_request, "contents", []) or []: + for part in getattr(content, "parts", []) or []: + if getattr(part, "function_response", None) is not None: + return True + return False + + +def _json_for_context(value: object) -> str: + if hasattr(value, "model_dump"): + value = value.model_dump() + return json.dumps(value, ensure_ascii=False, default=str) + + +def _build_report( + *, + baseline: list[RunMetric], + enhanced: list[RunMetric], + plugin_names: list[str], +) -> dict[str, object]: + baseline_latency = statistics.median(item.elapsed_seconds for item in baseline) + enhanced_latency = statistics.median(item.elapsed_seconds for item in enhanced) + baseline_chars = int(statistics.median(item.prompt_chars for item in baseline)) + enhanced_chars = int(statistics.median(item.prompt_chars for item in enhanced)) + latency_saved = baseline_latency - enhanced_latency + char_saved = baseline_chars - enhanced_chars + latency_saving_ratio = latency_saved / baseline_latency if baseline_latency else 0.0 + char_saving_ratio = char_saved / baseline_chars if baseline_chars else 0.0 + compression_ratio = enhanced_chars / baseline_chars if baseline_chars else 0.0 + answers_match = all( + "best_model: candidate-b" in item.output for item in baseline + enhanced + ) + baseline_uncompressed = all( + "compressed_context: false" in item.output for item in baseline + ) + enhanced_compressed = all( + "compressed_context: true" in item.output for item in enhanced + ) + passed = ( + answers_match + and baseline_uncompressed + and enhanced_compressed + and enhanced_latency < baseline_latency + and enhanced_chars < baseline_chars + and compression_ratio < 0.1 + ) + return { + "case": "local_stable_latency_token_case", + "passed": passed, + "plugins_attached_by_default": plugin_names, + "repeats": len(baseline), + "median": { + "no_enhance_latency_seconds": round(baseline_latency, 4), + "harness_enhance_latency_seconds": round(enhanced_latency, 4), + "latency_saved_seconds": round(latency_saved, 4), + "latency_saving_pct": round(latency_saving_ratio * 100, 4), + "no_enhance_prompt_chars": baseline_chars, + "harness_enhance_prompt_chars": enhanced_chars, + "prompt_chars_saved": char_saved, + "prompt_chars_saving_pct": round(char_saving_ratio * 100, 4), + }, + "compression": { + "provider": "builtin", + "median_original_prompt_chars": baseline_chars, + "median_compressed_prompt_chars": enhanced_chars, + "compression_ratio": round(compression_ratio, 6), + }, + "answers_match": answers_match, + "baseline_uncompressed": baseline_uncompressed, + "enhanced_compressed": enhanced_compressed, + "details": { + "no_enhance": [_metric_row(item) for item in baseline], + "harness_enhance": [_metric_row(item) for item in enhanced], + }, + } + + +def _metric_row(metric: RunMetric) -> dict[str, object]: + return { + "mode": metric.mode, + "elapsed_seconds": round(metric.elapsed_seconds, 4), + "prompt_chars": metric.prompt_chars, + } + + +def _free_port() -> int: + with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as sock: + sock.bind(("127.0.0.1", 0)) + return int(sock.getsockname()[1]) + + +def _default_env_file() -> Path: + configured = os.getenv("HARNESS_BENCHMARK_ENV_FILE") + return Path(configured) if configured else Path() + + +def _load_env_file(path: Path) -> None: + if not path.is_file(): + return + for raw_line in path.read_text().splitlines(): + line = raw_line.strip() + if not line or line.startswith("#"): + continue + if line.startswith("export "): + line = line.removeprefix("export ").strip() + if "=" not in line: + continue + key, value = line.split("=", 1) + os.environ.setdefault(key.strip(), value.strip().strip("\"'")) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/examples/harness_runner_plugins/README.md b/examples/harness_runner_plugins/README.md new file mode 100644 index 00000000..afd2570d --- /dev/null +++ b/examples/harness_runner_plugins/README.md @@ -0,0 +1,14 @@ +# VeADK Runner Plugins Example + +This example shows the smallest local VeADK integration: build a regular +`Agent`, attach Harness plugins to `Runner`, and keep the agent code focused on +business behavior. + +```python +from agent import build_runner + +runner = build_runner() +``` + +The plugin bundle adds context engineering, tool-result compression, and answer +verification without changing the agent class. diff --git a/examples/harness_runner_plugins/agent.py b/examples/harness_runner_plugins/agent.py new file mode 100644 index 00000000..a6d2eda3 --- /dev/null +++ b/examples/harness_runner_plugins/agent.py @@ -0,0 +1,35 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Minimal VeADK Runner example with Harness plugins.""" + +from veadk.extensions.harness.plugins import build_harness_plugins +from veadk import Agent, Runner + + +def build_runner() -> Runner: + """Build a VeADK runner with Harness context, compression, and verification.""" + + agent = Agent( + name="harness_sample_agent", + instruction="Answer with evidence from available tools and keep responses concise.", + ) + return Runner( + agent=agent, + app_name="harness_sample", + plugins=build_harness_plugins( + components=["invocation_context", "compactor", "response_verification"], + profile="general", + ), + ) diff --git a/pyproject.toml b/pyproject.toml index 720dbc1f..4070b83b 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -84,6 +84,9 @@ eval = [ "deepeval>=3.2.6", # For DeepEval-based evaluation "google-adk[eval]", # For Google ADK-based evaluation ] +harness = [ + "headroom", +] cli = [] dev = [ "pre-commit>=4.2.0", # Format checking diff --git a/tests/cli/test_cli_agentkit_harness_invoke.py b/tests/cli/test_cli_agentkit_harness_invoke.py new file mode 100644 index 00000000..ea587951 --- /dev/null +++ b/tests/cli/test_cli_agentkit_harness_invoke.py @@ -0,0 +1,158 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from click.testing import CliRunner + +from veadk.cli import cli_agentkit +from veadk.cli.cli_agentkit import agentkit + + +class _FakeResponse: + status_code = 200 + text = "{}" + + def json(self) -> dict[str, str]: + return {"output": "ok"} + + +def test_agentkit_invoke_maps_harness_enhance_flags(monkeypatch): + calls: list[dict[str, object]] = [] + + def fake_post( + url: str, + *, + json: dict[str, object], + headers: dict[str, str], + timeout: int, + ) -> _FakeResponse: + calls.append( + { + "url": url, + "json": json, + "headers": headers, + "timeout": timeout, + } + ) + return _FakeResponse() + + monkeypatch.setattr("httpx.post", fake_post) + + result = CliRunner().invoke( + agentkit, + [ + "invoke", + "--endpoint", + "http://127.0.0.1:8000", + "--apikey", + "test-key", + "--harness", + "research-agent", + "--user-id", + "u1", + "--session-id", + "s1", + "--model-id", + "model-a", + "--tools", + "run_code", + "--enable-harness-enhance", + "--harness-components", + "context_engine,compressor", + "--harness-profile", + "research", + "--harness-compression-provider", + "headroom", + "find best model", + ], + ) + + assert result.exit_code == 0 + assert result.output.strip() == "ok" + assert calls[0]["url"] == "http://127.0.0.1:8000/harness/invoke" + body = calls[0]["json"] + headers = calls[0]["headers"] + assert body["prompt"] == "find best model" + assert body["harness_name"] == "research-agent" + assert body["run_agent_request"] == {"user_id": "u1", "session_id": "s1"} + assert body["harness"] == {"model_name": "model-a", "tools": "run_code"} + assert body["harness_enhance"] == { + "enabled": True, + "components": "context_engine,compressor", + "profile": "research", + "compression_provider": "headroom", + } + assert headers["Authorization"] == "Bearer test-key" + assert headers["X-Harness-Enhance"] == "true" + assert headers["X-Harness-Components"] == "context_engine,compressor" + assert headers["X-Harness-Profile"] == "research" + assert headers["X-Harness-Compression-Provider"] == "headroom" + assert "X-Harness-Compression-Base-Url" not in headers + assert "X-Harness-Max-Tool-Result-Chars" not in headers + assert "X-Harness-Verifier-Mode" not in headers + + +def test_agentkit_invoke_falls_back_to_upstream_click_command(monkeypatch): + calls: list[dict[str, object]] = [] + + class FakeInvokeCommand: + commands: dict[str, object] = {} + + def main( + self, + *, + args: list[str], + prog_name: str, + standalone_mode: bool, + ) -> None: + calls.append( + { + "args": args, + "prog_name": prog_name, + "standalone_mode": standalone_mode, + } + ) + + monkeypatch.setattr(cli_agentkit, "_agentkit_invoke_command", None) + monkeypatch.setattr( + cli_agentkit, + "_agentkit_invoke_click_command", + FakeInvokeCommand(), + ) + + result = CliRunner().invoke( + agentkit, + [ + "invoke", + "--endpoint", + "http://127.0.0.1:8000", + "--apikey", + "test-key", + "hello", + ], + ) + + assert result.exit_code == 0 + assert calls == [ + { + "args": [ + "--endpoint", + "http://127.0.0.1:8000", + "--apikey", + "test-key", + "hello", + ], + "prog_name": "invoke", + "standalone_mode": False, + } + ] diff --git a/tests/cli/test_cli_harness_open_source_defaults.py b/tests/cli/test_cli_harness_open_source_defaults.py new file mode 100644 index 00000000..e5225cc4 --- /dev/null +++ b/tests/cli/test_cli_harness_open_source_defaults.py @@ -0,0 +1,86 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path +from types import SimpleNamespace + +from click.testing import CliRunner + +from veadk.cli import cli_harness + + +def test_harness_create_writes_gitignore_for_local_credentials() -> None: + runner = CliRunner() + + with runner.isolated_filesystem() as temp_dir: + result = runner.invoke(cli_harness.harness, ["create", "my-harness"]) + + assert result.exit_code == 0 + harness_dir = Path(temp_dir) / "my-harness" + gitignore = harness_dir / ".gitignore" + assert gitignore.is_file() + content = gitignore.read_text() + assert ".env" in content + assert "!.env.example" in content + assert "harness.json" in content + assert "agentkit*.yaml" in content + + +def test_harness_dockerfile_uses_accelerated_source_with_official_fallback() -> None: + assert ( + "https://ghfast.top/https://github.com/volcengine/veadk-python.git" + in cli_harness._DOCKERFILE + ) + assert "https://github.com/volcengine/veadk-python.git" in cli_harness._DOCKERFILE + assert '"./src[harness]"' in cli_harness._DOCKERFILE + old_package_path = "packages/" + "agentkit" + "-harness-python" + assert old_package_path not in cli_harness._DOCKERFILE + + +def test_harness_deploy_does_not_print_runtime_api_key(monkeypatch) -> None: + runtime_key = "placeholder-runtime-key" + + def fake_launch(**_: object) -> SimpleNamespace: + return SimpleNamespace( + success=True, + error=None, + deploy_result=SimpleNamespace( + endpoint_url="https://example.invalid/runtime", + metadata={ + "runtime_id": "runtime-id", + "runtime_apikey": runtime_key, + }, + ), + ) + + monkeypatch.setattr("agentkit.toolkit.sdk.launch", fake_launch) + monkeypatch.setenv("VOLC_ACCESSKEY", "test-access-key") + monkeypatch.setenv("VOLC_SECRETKEY", "placeholder-credential") + + runner = CliRunner() + with runner.isolated_filesystem() as temp_dir: + harness_dir = Path(temp_dir) + (harness_dir / "harness.yaml").write_text("harness_name: test-harness\n") + + result = runner.invoke( + cli_harness.harness, + ["deploy", "--path", str(harness_dir)], + ) + + assert result.exit_code == 0 + assert runtime_key not in result.output + assert "saved in local harness.json (not printed)" in result.output + + record = cli_harness._load_harness_json(str(harness_dir)) + assert record["test-harness"]["key"] == runtime_key diff --git a/tests/cloud/test_harness_app_contract.py b/tests/cloud/test_harness_app_contract.py index 5383d812..f74feef4 100644 --- a/tests/cloud/test_harness_app_contract.py +++ b/tests/cloud/test_harness_app_contract.py @@ -27,10 +27,15 @@ from pathlib import Path from veadk.cloud.harness_app.types import ( + HarnessCompactionMetric, HarnessConfig, + HarnessEnhanceOverrides, HarnessOverrides, + HarnessPluginMetrics, + HarnessResponseMetrics, InvokeHarnessRequest, InvokeHarnessResponse, + LlmUsageMetrics, RunAgentRequest, ) from veadk.cloud.harness_app.env_mapping import to_runtime_env @@ -191,11 +196,20 @@ def test_run_agent_request_fields(self): "max_llm_calls", } + def test_enhance_override_defaults(self): + assert HarnessEnhanceOverrides().model_dump() == { + "enabled": False, + "components": "invocation_context,compactor,response_verification", + "profile": "default", + "compression_provider": None, + } + def test_invoke_request_fields(self): assert set(_fields(InvokeHarnessRequest)) == { "prompt", "harness_name", "harness", + "harness_enhance", "run_agent_request", } @@ -208,11 +222,69 @@ def test_invoke_request_harness_is_optional_override(self): def test_invoke_response_fields_and_defaults(self): fields = _fields(InvokeHarnessResponse) - assert set(fields) == {"harness_name", "overwrite", "output", "error"} + assert set(fields) == { + "harness_name", + "overwrite", + "output", + "metrics", + "error", + } assert fields["overwrite"].default is False + assert fields["metrics"].default is None # `error` is unset on success and carries the message verbatim on failure. assert fields["error"].default is None + def test_usage_metrics_accumulate(self): + usage = LlmUsageMetrics(prompt_tokens=10, total_tokens=12, usage_event_count=1) + usage.add( + LlmUsageMetrics( + prompt_tokens=20, + completion_tokens=5, + total_tokens=25, + cached_tokens=3, + usage_event_count=1, + ) + ) + + assert HarnessResponseMetrics(llm_usage=usage).model_dump() == { + "llm_usage": { + "prompt_tokens": 30, + "completion_tokens": 5, + "total_tokens": 37, + "cached_tokens": 3, + "usage_event_count": 2, + }, + "harness_plugins": { + "names": [], + "compaction_reports": [], + }, + } + + def test_harness_plugin_metrics_are_structured(self): + metrics = HarnessResponseMetrics( + harness_plugins=HarnessPluginMetrics( + names=["harness_compress_plugin"], + compaction_reports=[ + HarnessCompactionMetric( + provider="builtin", + original_chars=8000, + compressed_chars=400, + changed=True, + tokens_before=2000, + tokens_after=100, + tokens_saved=1900, + compression_ratio=0.05, + transforms_applied=["builtin_tool_fact_compaction"], + ) + ], + ) + ) + + report = metrics.harness_plugins.compaction_reports[0] + assert metrics.harness_plugins.names == ["harness_compress_plugin"] + assert report.changed is True + assert report.compressed_chars < report.original_chars + class TestSplitCsv: def test_splits_and_trims(self): diff --git a/tests/cloud/test_harness_enhance_env.py b/tests/cloud/test_harness_enhance_env.py new file mode 100644 index 00000000..ac2f5b25 --- /dev/null +++ b/tests/cloud/test_harness_enhance_env.py @@ -0,0 +1,45 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from veadk.cloud.harness_app.env_mapping import to_runtime_env + + +def test_harness_enhance_config_flattens_to_runtime_env(): + env = to_runtime_env( + { + "harness_enhance": { + "enabled": True, + "components": ["invocation_context", "compactor"], + "profile": "analysis", + "compression_provider": "heuristic", + "max_context_chars": 12000, + "max_tool_result_chars": 3000, + "verifier_mode": "observe", + } + } + ) + + assert env["HARNESS_ENHANCE_ENABLED"] == "true" + assert env["HARNESS_ENHANCE_COMPONENTS"] == "invocation_context,compactor" + assert env["HARNESS_ENHANCE_PROFILE"] == "analysis" + assert env["HARNESS_ENHANCE_COMPRESSION_PROVIDER"] == "heuristic" + assert env["HARNESS_ENHANCE_MAX_CONTEXT_CHARS"] == "12000" + assert env["HARNESS_ENHANCE_MAX_TOOL_RESULT_CHARS"] == "3000" + assert env["HARNESS_ENHANCE_VERIFIER_MODE"] == "observe" + assert env["HARNESS_COMPONENTS"] == "invocation_context,compactor" + assert env["HARNESS_PROFILE"] == "analysis" + assert env["HARNESS_COMPRESSION_PROVIDER"] == "heuristic" + assert env["HARNESS_MAX_CONTEXT_CHARS"] == "12000" + assert env["HARNESS_MAX_TOOL_RESULT_CHARS"] == "3000" + assert env["HARNESS_VERIFIER_MODE"] == "observe" diff --git a/tests/cloud/test_harness_plugins.py b/tests/cloud/test_harness_plugins.py new file mode 100644 index 00000000..a86f286f --- /dev/null +++ b/tests/cloud/test_harness_plugins.py @@ -0,0 +1,71 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from veadk.cloud.harness_app.harness_plugins import ( + build_harness_plugins_from_enhance, + build_harness_plugins_from_headers, + harness_env_from_enhance, + harness_env_from_headers, +) +from veadk.cloud.harness_app.types import HarnessEnhanceOverrides + + +def test_harness_env_from_headers_maps_agentkit_headers(): + env = harness_env_from_headers( + { + "X-Harness-Enable-Context": "true", + "X-Harness-Components": "invocation_context,response_verification", + "X-Harness-Profile": "analysis", + "X-Harness-Compression-Provider": "headroom", + } + ) + + assert env == { + "HARNESS_ENHANCE_ENABLED": "true", + "HARNESS_COMPONENTS": "invocation_context,response_verification", + "HARNESS_ENHANCE_COMPONENTS": "invocation_context,response_verification", + "HARNESS_PROFILE": "analysis", + "HARNESS_ENHANCE_PROFILE": "analysis", + "HARNESS_COMPRESSION_PROVIDER": "headroom", + "HARNESS_ENHANCE_COMPRESSION_PROVIDER": "headroom", + } + + +def test_build_harness_plugins_from_headers_returns_empty_when_disabled(): + assert build_harness_plugins_from_headers({}) == [] + + +def test_harness_env_from_enhance_maps_request_body_config(): + env = harness_env_from_enhance( + HarnessEnhanceOverrides( + enabled=True, + components="invocation_context,compactor", + profile="analysis", + compression_provider="builtin", + ) + ) + + assert env == { + "HARNESS_ENHANCE_ENABLED": "true", + "HARNESS_COMPONENTS": "invocation_context,compactor", + "HARNESS_ENHANCE_COMPONENTS": "invocation_context,compactor", + "HARNESS_PROFILE": "analysis", + "HARNESS_ENHANCE_PROFILE": "analysis", + "HARNESS_COMPRESSION_PROVIDER": "builtin", + "HARNESS_ENHANCE_COMPRESSION_PROVIDER": "builtin", + } + + +def test_build_harness_plugins_from_enhance_returns_empty_when_disabled(): + assert build_harness_plugins_from_enhance(HarnessEnhanceOverrides()) == [] diff --git a/tests/cloud/test_harness_skill_download.py b/tests/cloud/test_harness_skill_download.py new file mode 100644 index 00000000..a0c3b3e7 --- /dev/null +++ b/tests/cloud/test_harness_skill_download.py @@ -0,0 +1,86 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Offline tests for HarnessApp skill download resolution.""" + +from __future__ import annotations + +import io +import zipfile + +import httpx + +from veadk.cloud.harness_app import utils + + +def _skill_zip_bytes(name: str) -> bytes: + buffer = io.BytesIO() + with zipfile.ZipFile(buffer, "w", zipfile.ZIP_DEFLATED) as archive: + archive.writestr( + "SKILL.md", + f"---\nname: {name}\ndescription: Test skill.\n---\n\n# {name}\n", + ) + return buffer.getvalue() + + +def test_download_skill_resolves_short_name_to_exact_slug(monkeypatch, tmp_path): + calls: list[str] = [] + + def fake_get(url: str, **kwargs: object) -> httpx.Response: + calls.append(url) + request = httpx.Request("GET", url) + if url.endswith("/download/web-scraper"): + return httpx.Response( + 200, + request=request, + json={ + "ResponseMetadata": { + "Action": "DownloadSkill", + "Error": {"Code": "NotFound"}, + } + }, + ) + if "/v1/skills?" in url: + return httpx.Response( + 200, + request=request, + json={ + "Skills": [ + { + "Name": "web-scraper", + "Slug": "clawhub/example/web-scraper", + "SourceRepo": "clawhub/example", + } + ] + }, + ) + if url.endswith("/download/clawhub/example/web-scraper"): + return httpx.Response( + 200, + request=request, + content=_skill_zip_bytes("web-scraper"), + ) + return httpx.Response(404, request=request) + + monkeypatch.setattr(utils.httpx, "get", fake_get) + + extracted = utils._download_and_extract_skill("web-scraper", tmp_path) + + assert extracted == tmp_path / "web-scraper" + assert (extracted / "SKILL.md").is_file() + assert calls == [ + "https://skills.volces.com/v1/skills/download/web-scraper", + "https://skills.volces.com/v1/skills?query=web-scraper&pageNumber=1&pageSize=10", + "https://skills.volces.com/v1/skills/download/clawhub/example/web-scraper", + ] diff --git a/tests/extensions/harness/test_backward_compat.py b/tests/extensions/harness/test_backward_compat.py new file mode 100644 index 00000000..8c9b28e1 --- /dev/null +++ b/tests/extensions/harness/test_backward_compat.py @@ -0,0 +1,58 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from veadk.extensions.harness import ( + CapabilityReceipt, + CompactionReport, + CompactionResult, + CompressionReport, + CompressionResult, + ContextBundle, + HarnessIntervention, + HarnessInvocationRef, + HarnessRunContext, + InvocationContextBlock, + ToolReceipt, + ToolResultCompactor, + ToolResultCompressor, + VerificationDecision, +) +from veadk.extensions.harness.plugins import ( + HarnessContextPlugin, + HarnessHallucinationPlugin, + HarnessInvocationContextPlugin, + HarnessResponseVerificationPlugin, +) +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifier, + ResultVerifier, +) +from veadk.extensions.harness.modules.tool_result_compactor import ( + ContextCompactionPolicy, + ContextCompressionPolicy, +) + + +def test_renamed_public_objects_keep_backward_compatible_aliases(): + assert HarnessRunContext is HarnessInvocationRef + assert ContextBundle is InvocationContextBlock + assert CompressionReport is CompactionReport + assert CompressionResult is CompactionResult + assert CapabilityReceipt is ToolReceipt + assert HarnessIntervention is VerificationDecision + assert ToolResultCompressor is ToolResultCompactor + assert ResultVerifier is FinalResponseVerifier + assert ContextCompressionPolicy is ContextCompactionPolicy + assert HarnessContextPlugin is HarnessInvocationContextPlugin + assert HarnessHallucinationPlugin is HarnessResponseVerificationPlugin diff --git a/tests/extensions/harness/test_content_adapter.py b/tests/extensions/harness/test_content_adapter.py new file mode 100644 index 00000000..5a409f02 --- /dev/null +++ b/tests/extensions/harness/test_content_adapter.py @@ -0,0 +1,70 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tests for Runner content adapters.""" + +from google.adk.models import LlmRequest + +from veadk.extensions.harness.plugins.content_adapter import append_system_instruction + + +def test_append_system_instruction_replaces_previous_harness_context(): + request = LlmRequest() + request.config.system_instruction = ( + "Base instruction.\n\n" + "[Harness Context]\n" + "task_goal: old\n" + "[/Harness Context]\n\n" + "Keep this line." + ) + + append_system_instruction( + request, + "[Harness Context]\ntask_goal: new\n[/Harness Context]", + ) + + instruction = request.config.system_instruction + assert isinstance(instruction, str) + assert instruction.count("[Harness Context]") == 1 + assert "task_goal: old" not in instruction + assert "task_goal: new" in instruction + assert "Base instruction." in instruction + assert "Keep this line." in instruction + + +def test_append_system_instruction_replaces_only_matching_harness_block(): + request = LlmRequest() + request.config.system_instruction = ( + "Base instruction.\n\n" + "[Harness Context]\n" + "task_goal: current\n" + "[/Harness Context]\n\n" + "[Harness Long Run Control]\n" + "model_calls_so_far: 8\n" + "[/Harness Long Run Control]" + ) + + append_system_instruction( + request, + "[Harness Long Run Control]\nmodel_calls_so_far: 9\n" + "[/Harness Long Run Control]", + ) + + instruction = request.config.system_instruction + assert isinstance(instruction, str) + assert instruction.count("[Harness Context]") == 1 + assert instruction.count("[Harness Long Run Control]") == 1 + assert "task_goal: current" in instruction + assert "model_calls_so_far: 8" not in instruction + assert "model_calls_so_far: 9" in instruction diff --git a/tests/extensions/harness/test_context_engine.py b/tests/extensions/harness/test_context_engine.py new file mode 100644 index 00000000..51d3d7af --- /dev/null +++ b/tests/extensions/harness/test_context_engine.py @@ -0,0 +1,100 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from veadk.extensions.harness import ( + ContextEngine, + HarnessInvocationContextConfig, + HarnessInvocationContextBuilder, + HarnessInvocationRef, + TaskContract, +) +from veadk.extensions.harness.schemas import ToolReceipt, ConversationMessage + + +def test_invocation_context_builder_builds_task_anchor_and_receipts(): + builder = HarnessInvocationContextBuilder() + context = HarnessInvocationRef( + session_id="s1", + invocation_id="r1", + profile="research", + task=TaskContract(goal="Create a chart", acceptance_criteria=["save a PNG"]), + ) + bundle = builder.prepare_context( + context, + user_input="Create a chart from this CSV file.", + history=[ConversationMessage(role="user", content="Use the latest CSV.")], + receipts=[ + ToolReceipt(name="read_csv", status="success", summary="10 rows loaded") + ], + has_tools=True, + ) + + assert bundle.injected is True + assert "task_goal: Create a chart" in bundle.header + assert "read_csv [success]" in bundle.header + assert "[Harness Tool Protocol]" in bundle.header + assert "valid JSON object arguments" in bundle.header + + +def test_invocation_context_defaults_match_runtime_budget(): + config = HarnessInvocationContextConfig() + + assert config.max_history_messages == 12 + assert config.max_context_chars == 24000 + assert config.max_receipts == 8 + assert config.receipt_summary_chars == 500 + + +def test_invocation_context_receipts_are_bounded_and_recent(): + builder = HarnessInvocationContextBuilder() + context = HarnessInvocationRef(session_id="s1", invocation_id="r1") + receipts = [ + ToolReceipt( + name=f"tool_{index}", + status="success", + summary=f"summary {index} " + ("x" * 800), + ) + for index in range(10) + ] + + bundle = builder.prepare_context( + context, + user_input="Summarize tool evidence.", + receipts=receipts, + ) + + assert "tool_0 [success]" not in bundle.header + assert "tool_1 [success]" not in bundle.header + assert "tool_2 [success]" in bundle.header + assert "tool_9 [success]" in bundle.header + assert "x" * 700 not in bundle.header + + +def test_invocation_context_builder_injects_after_system_messages(): + builder = HarnessInvocationContextBuilder() + context = HarnessInvocationRef(session_id="s1", invocation_id="r1") + messages = [ + ConversationMessage(role="system", content="base system"), + ConversationMessage(role="user", content="hello"), + ] + + enhanced, bundle = builder.enhance_messages(messages, context, user_input="hello") + + assert bundle.injected is True + assert enhanced[0].content == "base system" + assert enhanced[1].name == "veadk_harness_context" + + +def test_context_engine_alias_remains_available(): + assert ContextEngine is HarnessInvocationContextBuilder diff --git a/tests/extensions/harness/test_env.py b/tests/extensions/harness/test_env.py new file mode 100644 index 00000000..22f67295 --- /dev/null +++ b/tests/extensions/harness/test_env.py @@ -0,0 +1,50 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from veadk.extensions.harness.env import ( + build_harness_plugins_from_env, + harness_enabled_from_env, +) + + +def test_harness_enabled_from_env(): + assert harness_enabled_from_env({"HARNESS_ENHANCE_ENABLED": "true"}) is True + assert harness_enabled_from_env({"HARNESS_ENHANCE_ENABLED": "false"}) is False + + +def test_build_harness_plugins_from_env_respects_components(): + plugins = build_harness_plugins_from_env( + { + "HARNESS_ENHANCE_ENABLED": "true", + "HARNESS_ENHANCE_COMPONENTS": "context_engine,hallucination", + "HARNESS_ENHANCE_PROFILE": "analysis", + } + ) + + assert [plugin.name for plugin in plugins] == [ + "harness_invocation_context_plugin", + "harness_response_verification_plugin", + ] + + +def test_build_harness_plugins_from_env_defaults_to_builtin_compression(): + plugins = build_harness_plugins_from_env( + { + "HARNESS_ENHANCE_ENABLED": "true", + "HARNESS_ENHANCE_COMPONENTS": "compressor", + } + ) + + assert plugins[0].name == "harness_compress_plugin" + assert plugins[0].compressor.config.provider == "builtin" diff --git a/tests/extensions/harness/test_extension.py b/tests/extensions/harness/test_extension.py new file mode 100644 index 00000000..0c2d7e99 --- /dev/null +++ b/tests/extensions/harness/test_extension.py @@ -0,0 +1,43 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from veadk.extensions.harness import HarnessExtension + + +def test_harness_extension_builds_runner_plugins() -> None: + plugins = HarnessExtension( + components=["invocation_context", "compactor", "response_verification"], + profile="test", + ).plugins() + + assert [plugin.name for plugin in plugins] == [ + "harness_invocation_context_plugin", + "harness_compress_plugin", + "harness_response_verification_plugin", + ] + + +def test_harness_extension_from_env_respects_disabled_default() -> None: + assert HarnessExtension.from_env({}).plugins() == [] + + +def test_harness_extension_from_env_builds_configured_plugins() -> None: + plugins = HarnessExtension.from_env( + { + "HARNESS_ENHANCE_ENABLED": "true", + "HARNESS_ENHANCE_COMPONENTS": "invocation_context", + } + ).plugins() + + assert [plugin.name for plugin in plugins] == ["harness_invocation_context_plugin"] diff --git a/tests/extensions/harness/test_headroom_provider.py b/tests/extensions/harness/test_headroom_provider.py new file mode 100644 index 00000000..1111e981 --- /dev/null +++ b/tests/extensions/harness/test_headroom_provider.py @@ -0,0 +1,198 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +import importlib +import sys +from types import ModuleType, SimpleNamespace + +from pytest import MonkeyPatch + +from veadk.extensions.harness.plugins import HarnessCompressPlugin +from veadk.extensions.harness.modules.headroom_provider import ( + HeadroomCompressionProvider, +) +from veadk.extensions.harness.modules.tool_result_compactor import ( + ToolResultCompactor, + ToolResultCompactorConfig, +) +from veadk.extensions.harness.schemas import CompressionRequest, ConversationMessage +from veadk.extensions.harness.stores import InMemoryHarnessStore + + +_HEADROOM_CALLS: list[dict[str, object]] = [] + + +def _install_fake_headroom(monkeypatch: MonkeyPatch) -> None: + package = ModuleType("headroom") + package.__path__ = [] + compress_module = ModuleType("headroom.compress") + + def compress( + messages: list[dict[str, object]], + *, + model: str, + optimize: bool, + ) -> object: + _HEADROOM_CALLS.append( + {"messages": messages, "model": model, "optimize": optimize} + ) + return SimpleNamespace( + messages=[ + { + "role": "tool", + "content": "HEADROOM_SUMMARY: preserved key facts only.", + "metadata": {"source": "test"}, + } + ], + tokens_before=2000, + tokens_after=40, + tokens_saved=1960, + compression_ratio=0.02, + transforms_applied=["headroom_test_compaction"], + ) + + compress_module.compress = compress + _HEADROOM_CALLS.clear() + monkeypatch.setitem(sys.modules, "headroom", package) + monkeypatch.setitem(sys.modules, "headroom.compress", compress_module) + + +def test_headroom_sdk_provider_compresses_tool_result(monkeypatch: MonkeyPatch) -> None: + _install_fake_headroom(monkeypatch) + compactor = ToolResultCompactor( + ToolResultCompactorConfig( + provider="headroom", + max_tool_result_chars=200, + ) + ) + + compressed, report = compactor.compress_tool_result({"rows": "x" * 8000}) + + assert compressed["harness_compressed"] is True + assert compressed["provider"] == "headroom" + assert "HEADROOM_SUMMARY" in str(compressed["summary"]) + assert report.provider == "headroom" + assert report.tokens_saved == 1960 + assert report.compressed_chars < report.original_chars + assert _HEADROOM_CALLS[0]["model"] == "gpt-4o" + assert _HEADROOM_CALLS[0]["optimize"] is True + + +def test_compress_plugin_after_tool_callback_uses_headroom( + monkeypatch: MonkeyPatch, +) -> None: + _install_fake_headroom(monkeypatch) + plugin = HarnessCompressPlugin( + compressor=ToolResultCompactor( + ToolResultCompactorConfig( + provider="headroom", + max_tool_result_chars=200, + ) + ), + store=InMemoryHarnessStore(), + ) + + compressed = asyncio.run( + plugin.after_tool_callback( + tool=SimpleNamespace(name="query_data"), + tool_args={}, + tool_context=SimpleNamespace( + session=SimpleNamespace(id="s1", app_name="app", user_id="u1"), + user_id="u1", + invocation_id="r1", + ), + result={"rows": "x" * 8000}, + ) + ) + + assert compressed is not None + assert compressed["provider"] == "headroom" + assert "HEADROOM_SUMMARY" in str(compressed["summary"]) + + +def test_headroom_sdk_provider_compresses_candidate_context( + monkeypatch: MonkeyPatch, +) -> None: + _install_fake_headroom(monkeypatch) + compactor = ToolResultCompactor( + ToolResultCompactorConfig( + provider="headroom", + max_context_chars=1500, + min_candidate_chars=100, + protect_recent_messages=1, + ) + ) + messages = [ + ConversationMessage(role="user", content="summarize"), + ConversationMessage(role="tool", content="a" * 1000), + ConversationMessage(role="tool", content="b" * 1000), + ] + + result = compactor.compress_messages( + CompressionRequest(messages=messages, max_context_chars=1500) + ) + + assert result.report.provider == "headroom" + assert result.report.changed is True + assert result.report.policy["candidate_count"] == 1 + assert "HEADROOM_SUMMARY" in result.messages[1].content + assert result.messages[-1].content == "b" * 1000 + + +def test_headroom_provider_does_not_install_when_unavailable( + monkeypatch: MonkeyPatch, +) -> None: + real_import_module = importlib.import_module + + def fake_import_module(name: str, package: str | None = None) -> ModuleType: + if name == "headroom.compress": + raise ImportError(name) + return real_import_module(name, package) + + monkeypatch.setattr(importlib, "import_module", fake_import_module) + provider = HeadroomCompressionProvider(auto_install=True) + + result = provider.compress( + CompressionRequest( + messages=[ConversationMessage(role="tool", content="x" * 8000)], + max_context_chars=200, + ) + ) + + assert result is None + + +def test_headroom_provider_falls_back_when_unavailable( + monkeypatch: MonkeyPatch, +) -> None: + real_import_module = importlib.import_module + + def fake_import_module(name: str, package: str | None = None) -> ModuleType: + if name == "headroom.compress": + raise ImportError(name) + return real_import_module(name, package) + + monkeypatch.setattr(importlib, "import_module", fake_import_module) + compactor = ToolResultCompactor( + ToolResultCompactorConfig(provider="headroom", max_tool_result_chars=200) + ) + + compressed, report = compactor.compress_tool_result({"rows": "x" * 1000}) + + assert compressed["harness_compressed"] is True + assert report.provider == "builtin" + assert compressed["provider"] == "builtin" + assert "headroom provider unavailable" in report.warnings[0] + assert report.compressed_chars < report.original_chars diff --git a/tests/extensions/harness/test_package_metadata.py b/tests/extensions/harness/test_package_metadata.py new file mode 100644 index 00000000..96d1b1e4 --- /dev/null +++ b/tests/extensions/harness/test_package_metadata.py @@ -0,0 +1,26 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from pathlib import Path + + +def test_harness_extra_installs_headroom_provider() -> None: + pyproject = Path(__file__).parents[3] / "pyproject.toml" + text = pyproject.read_text(encoding="utf-8") + + assert "[project.optional-dependencies]" in text + assert "harness = [" in text + assert '"headroom"' in text + old_package_name = "agentkit" + "-harness-python" + assert old_package_name not in text diff --git a/tests/extensions/harness/test_result_verifier.py b/tests/extensions/harness/test_result_verifier.py new file mode 100644 index 00000000..58e292ba --- /dev/null +++ b/tests/extensions/harness/test_result_verifier.py @@ -0,0 +1,51 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifier, + FinalResponseVerifierConfig, +) +from veadk.extensions.harness.schemas import ToolReceipt + + +def test_verifier_fails_completion_claim_without_receipt(): + verifier = FinalResponseVerifier() + + report = verifier.verify_text("Done, I created the report.") + + assert report.status == "fail" + assert report.unsupported_claims + + +def test_verifier_allows_completion_claim_with_success_receipt(): + verifier = FinalResponseVerifier() + + report = verifier.verify_text( + "Done, I created the report.", + receipts=[ + ToolReceipt(name="write_file", status="success", summary="report.md saved") + ], + ) + + assert report.status == "pass" + assert report.supported_claims + + +def test_verifier_block_intervention(): + verifier = FinalResponseVerifier(FinalResponseVerifierConfig(mode="block")) + + report = verifier.verify_text("The file was saved successfully.") + intervention = verifier.decide(report) + + assert intervention.action == "block" diff --git a/tests/extensions/harness/test_runner_plugins.py b/tests/extensions/harness/test_runner_plugins.py new file mode 100644 index 00000000..e01db1b8 --- /dev/null +++ b/tests/extensions/harness/test_runner_plugins.py @@ -0,0 +1,252 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import asyncio +from types import SimpleNamespace + +from google.adk.models import LlmRequest, LlmResponse +from google.genai import types + +from veadk.extensions.harness.plugins import ( + HarnessCompressPlugin, + HarnessInvocationContextPlugin, + HarnessLongRunControlPlugin, + HarnessResponseVerificationPlugin, + build_harness_plugins, +) +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifier, + FinalResponseVerifierConfig, +) +from veadk.extensions.harness.modules.tool_result_compactor import ( + ToolResultCompactor, + ToolResultCompactorConfig, +) +from veadk.extensions.harness.stores import InMemoryHarnessStore + + +def _callback_context(): + return SimpleNamespace( + session=SimpleNamespace(id="s1", app_name="app", user_id="u1"), + user_id="u1", + invocation_id="r1", + user_content=types.Content( + role="user", parts=[types.Part(text="Create a report")] + ), + ) + + +def _tool_context(): + return SimpleNamespace( + session=SimpleNamespace(id="s1", app_name="app", user_id="u1"), + user_id="u1", + invocation_id="r1", + ) + + +def test_context_plugin_appends_system_instruction(): + plugin = HarnessInvocationContextPlugin(store=InMemoryHarnessStore()) + request = LlmRequest( + contents=[ + types.Content(role="user", parts=[types.Part(text="Create a report")]) + ] + ) + + asyncio.run( + plugin.before_model_callback( + callback_context=_callback_context(), + llm_request=request, + ) + ) + + assert "[Harness Context]" in str(request.config.system_instruction) + + +def test_compress_plugin_replaces_large_tool_result(): + plugin = HarnessCompressPlugin( + compressor=ToolResultCompactor( + ToolResultCompactorConfig(max_tool_result_chars=1000) + ), + store=InMemoryHarnessStore(), + ) + + result = asyncio.run( + plugin.after_tool_callback( + tool=SimpleNamespace(name="query_data"), + tool_args={}, + tool_context=_tool_context(), + result={"rows": "x" * 8000}, + ) + ) + + assert result is not None + assert result["harness_compressed"] is True + assert plugin.compaction_reports + assert plugin.compaction_reports[0].compressed_chars < 8000 + + +def test_compress_plugin_compacts_model_context_function_responses(): + plugin = HarnessCompressPlugin( + compactor=ToolResultCompactor( + ToolResultCompactorConfig(max_tool_result_chars=1000) + ), + store=InMemoryHarnessStore(), + ) + request = LlmRequest( + contents=[ + types.Content( + role="user", + parts=[ + types.Part.from_function_response( + name="run_code", + response={ + "result": ( + '[{"name":"candidate-b","score":88},' + '{"stage":"trace","payload":"' + ("x" * 8000) + '"}]' + ) + }, + ) + ], + ) + ] + ) + + asyncio.run( + plugin.before_model_callback( + callback_context=_callback_context(), + llm_request=request, + ) + ) + + response = request.contents[0].parts[0].function_response.response + summary = str(response["summary"]) + assert response["harness_compressed"] is True + assert "name=candidate-b" in summary + assert "score=88" in summary + assert "payload=" in summary + assert "x" * 100 not in summary + assert plugin.compaction_reports + + +def test_compress_plugin_resets_diagnostics(): + plugin = HarnessCompressPlugin( + compactor=ToolResultCompactor( + ToolResultCompactorConfig(max_tool_result_chars=1000) + ), + store=InMemoryHarnessStore(), + ) + + asyncio.run( + plugin.after_tool_callback( + tool=SimpleNamespace(name="query_data"), + tool_args={}, + tool_context=_tool_context(), + result={"rows": "x" * 8000}, + ) + ) + plugin.reset_diagnostics() + + assert plugin.compaction_reports == [] + + +def test_response_verification_plugin_blocks_unsupported_completion_claim(): + store = InMemoryHarnessStore() + plugin = HarnessResponseVerificationPlugin( + verifier=FinalResponseVerifier(FinalResponseVerifierConfig(mode="block")), + store=store, + ) + response = LlmResponse( + content=types.Content( + role="model", parts=[types.Part(text="Done, I created it.")] + ) + ) + + blocked = asyncio.run( + plugin.after_model_callback( + callback_context=_callback_context(), + llm_response=response, + ) + ) + + assert blocked is not None + assert "cannot verify" in blocked.content.parts[0].text + + +def test_response_verification_plugin_accepts_string_tool_result(): + plugin = HarnessResponseVerificationPlugin(store=InMemoryHarnessStore()) + + result = asyncio.run( + plugin.after_tool_callback( + tool=SimpleNamespace(name="run_code"), + tool_args={}, + tool_context=_tool_context(), + result="chain_1=abc123", + ) + ) + + assert result is None + + +def test_build_harness_plugins_uses_shared_store_and_aliases(): + plugins = build_harness_plugins(components="context,compress,verifier") + + assert [plugin.name for plugin in plugins] == [ + "harness_invocation_context_plugin", + "harness_compress_plugin", + "harness_response_verification_plugin", + ] + + +def test_build_harness_plugins_accepts_long_run_control_alias(): + plugins = build_harness_plugins( + components="context_engine,compressor,verifier,long_run_control" + ) + + assert [plugin.name for plugin in plugins] == [ + "harness_invocation_context_plugin", + "harness_compress_plugin", + "harness_response_verification_plugin", + "harness_long_run_control_plugin", + ] + + +def test_long_run_control_injects_guidance_after_threshold(): + plugin = HarnessLongRunControlPlugin( + store=InMemoryHarnessStore(), + trigger_after_model_calls=2, + ) + request = LlmRequest( + contents=[types.Content(role="user", parts=[types.Part(text="Create a chart")])] + ) + + asyncio.run( + plugin.before_model_callback( + callback_context=_callback_context(), + llm_request=request, + ) + ) + assert "Harness Long Run Control" not in str(request.config.system_instruction) + + asyncio.run( + plugin.before_model_callback( + callback_context=_callback_context(), + llm_request=request, + ) + ) + + instruction = request.config.system_instruction + assert isinstance(instruction, str) + assert "[Harness Long Run Control]" in instruction + assert "model_calls_so_far: 2" in instruction + assert "generated artifacts" in instruction diff --git a/tests/extensions/harness/test_tool_compressor.py b/tests/extensions/harness/test_tool_compressor.py new file mode 100644 index 00000000..8fc65256 --- /dev/null +++ b/tests/extensions/harness/test_tool_compressor.py @@ -0,0 +1,119 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import json + +from veadk.extensions.harness.modules.tool_result_compactor import ( + ToolResultCompactor, + ToolResultCompactorConfig, +) +from veadk.extensions.harness.schemas import CompressionRequest, ConversationMessage + + +def test_tool_result_compactor_compacts_large_tool_result(): + compactor = ToolResultCompactor( + ToolResultCompactorConfig(max_tool_result_chars=200, summary_chars=80) + ) + + compressed, report = compactor.compress_tool_result({"rows": "x" * 1000}) + + assert report.changed is True + assert compressed["harness_compressed"] is True + assert report.compressed_chars < report.original_chars + + +def test_builtin_provider_preserves_bounded_structured_facts(): + compactor = ToolResultCompactor(ToolResultCompactorConfig()) + payload = { + "candidates": [ + {"name": "alpha", "score": 82, "rank": 2}, + {"name": "beta", "score": 88, "rank": 1}, + ], + "diagnostics": [ + {"stage": "debug", "row": index, "trace": "x" * 80} for index in range(200) + ], + } + + compressed, report = compactor.compress_tool_result(payload) + summary = str(compressed["summary"]) + + assert compressed["provider"] == "builtin" + assert report.provider == "builtin" + assert report.compression_ratio < 0.1 + assert "name=beta" in summary + assert "score=88" in summary + assert "trace=" in summary + assert "x" * 20 not in summary + assert "compression_policy=preserve_bounded_structured_facts" in summary + + +def test_builtin_provider_reads_nested_json_strings_generically(): + compactor = ToolResultCompactor(ToolResultCompactorConfig()) + payload = { + "result": ( + '[{"name":"candidate-b","score":88},' + '{"stage":"trace","payload":"' + ("x" * 12000) + '"}]' + ) + } + + compressed, report = compactor.compress_tool_result(payload) + summary = str(compressed["summary"]) + + assert compressed["provider"] == "builtin" + assert report.provider == "builtin" + assert "name=candidate-b" in summary + assert "score=88" in summary + assert "payload=" in summary + assert "x" * 100 not in summary + + +def test_builtin_provider_uses_representative_sequence_items(): + compactor = ToolResultCompactor(ToolResultCompactorConfig()) + rows = [ + {"item": f"row-{index}", "value": index, "payload": "x" * 12000} + for index in range(40) + ] + + compressed, report = compactor.compress_tool_result({"result": json.dumps(rows)}) + summary = str(compressed["summary"]) + + assert report.changed is True + assert "item=row-0" in summary + assert "item=row-39" in summary + assert "item=row-20" not in summary + assert "omitted_items=28" in summary + assert "x" * 100 not in summary + + +def test_policy_preserves_recent_tool_feedback(): + compactor = ToolResultCompactor( + ToolResultCompactorConfig( + max_context_chars=500, + min_candidate_chars=100, + protect_recent_messages=1, + ) + ) + messages = [ + ConversationMessage(role="user", content="summarize"), + ConversationMessage(role="tool", content="a" * 1000), + ConversationMessage(role="tool", content="b" * 1000), + ] + + result = compactor.compress_messages( + CompressionRequest(messages=messages, max_context_chars=500) + ) + + assert result.report.changed is True + assert result.messages[-1].content == "b" * 1000 + assert result.report.policy["candidate_count"] == 1 diff --git a/veadk/cli/cli_agentkit.py b/veadk/cli/cli_agentkit.py index 63453f4e..c8206718 100644 --- a/veadk/cli/cli_agentkit.py +++ b/veadk/cli/cli_agentkit.py @@ -12,10 +12,19 @@ # See the License for the specific language governing permissions and # limitations under the License. +from collections.abc import Callable +from pathlib import Path + import click from agentkit.toolkit.cli.cli import app as agentkit_typer_app from typer.main import get_command +_agentkit_invoke_command: Callable[..., object] | None +try: + from agentkit.toolkit.cli.cli import invoke_command as _agentkit_invoke_command +except ImportError: + _agentkit_invoke_command = None + @click.group() def agentkit(): @@ -29,5 +38,393 @@ def agentkit(): # duck-typing on `.commands` rather than gating on `isinstance(..., click.Group)`: # since Typer 0.26 a TyperGroup is no longer a click.Group subclass, so the # isinstance check silently evaluates False and drops every command. +_agentkit_invoke_click_command: click.Command | None = None for cmd_name, cmd in getattr(agentkit_commands, "commands", {}).items(): + if cmd_name == "invoke": + if isinstance(cmd, click.Command): + _agentkit_invoke_click_command = cmd + continue agentkit.add_command(cmd, name=cmd_name) + + +@agentkit.command("invoke") +@click.argument("message", required=False) +@click.option("--config-file", type=click.Path(), default=None) +@click.option("--payload", "-p", default=None, help="JSON payload to send.") +@click.option("--headers", "-h", default=None, help="JSON headers to send.") +@click.option("--runtime-id", "-r", default=None) +@click.option("--endpoint", "-e", default=None) +@click.option("--region", default=None) +@click.option("--a2a", is_flag=True, default=False) +@click.option("--show-reasoning", is_flag=True, default=False) +@click.option("--raw", is_flag=True, default=False) +@click.option("--apikey", "-ak", default=None) +@click.option("--harness", default=None, help="Harness name for HarnessApp Runtime.") +@click.option("--model-id", default=None, help="One-shot model override.") +@click.option("--tools", default=None, help="Comma-separated one-shot tool override.") +@click.option("--skills", default=None, help="Comma-separated one-shot skill override.") +@click.option("--system-prompt", default=None, help="One-shot system prompt override.") +@click.option("--runtime", default=None, help="One-shot runtime override.") +@click.option("--user-id", default="agentkit_user") +@click.option("--session-id", default="agentkit_sample_session") +@click.option("--max-llm-calls", type=int, default=None) +@click.option( + "--enable-harness-enhance", + is_flag=True, + default=False, + help="Enable Harness enhancement headers for this invocation.", +) +@click.option( + "--harness-components", + default=None, + help="Comma-separated Harness components.", +) +@click.option("--harness-profile", default=None) +@click.option("--harness-compression-provider", default=None) +def invoke( + message: str | None, + config_file: str | None, + payload: str | None, + headers: str | None, + runtime_id: str | None, + endpoint: str | None, + region: str | None, + a2a: bool, + show_reasoning: bool, + raw: bool, + apikey: str | None, + harness: str | None, + model_id: str | None, + tools: str | None, + skills: str | None, + system_prompt: str | None, + runtime: str | None, + user_id: str, + session_id: str, + max_llm_calls: int | None, + enable_harness_enhance: bool, + harness_components: str | None, + harness_profile: str | None, + harness_compression_provider: str | None, +) -> None: + """Invoke AgentKit or HarnessApp Runtime. + + When Harness-specific flags are present, the command sends a HarnessApp + request to `/harness/invoke` and maps enhancement options to HTTP headers. + Without those flags it delegates to the upstream AgentKit CLI unchanged. + """ + + if not _is_harness_invoke( + harness=harness, + model_id=model_id, + tools=tools, + skills=skills, + system_prompt=system_prompt, + runtime=runtime, + enable_harness_enhance=enable_harness_enhance, + harness_components=harness_components, + harness_profile=harness_profile, + harness_compression_provider=harness_compression_provider, + max_llm_calls=max_llm_calls, + ): + _delegate_agentkit_invoke( + config_file=Path(config_file) if config_file else None, + message=message, + payload=payload, + headers=headers, + runtime_id=runtime_id, + endpoint=endpoint, + region=region, + a2a=a2a, + show_reasoning=show_reasoning, + raw=raw, + apikey=apikey, + ) + return + + if config_file or runtime_id or region or a2a or show_reasoning: + raise click.ClickException( + "HarnessApp invoke supports --endpoint and Harness-specific flags; " + "do not combine it with --config-file, --runtime-id, --region, --a2a, " + "or --show-reasoning." + ) + if not endpoint: + raise click.ClickException("HarnessApp invoke requires --endpoint.") + + request_body = _build_harness_body( + message=message, + payload=payload, + harness=harness, + user_id=user_id, + session_id=session_id, + max_llm_calls=max_llm_calls, + model_id=model_id, + tools=tools, + skills=skills, + system_prompt=system_prompt, + runtime=runtime, + enable_harness_enhance=enable_harness_enhance, + harness_components=harness_components, + harness_profile=harness_profile, + harness_compression_provider=harness_compression_provider, + ) + request_headers = _build_harness_headers( + headers=headers, + apikey=apikey, + enable_harness_enhance=enable_harness_enhance, + harness_components=harness_components, + harness_profile=harness_profile, + harness_compression_provider=harness_compression_provider, + ) + response = _post_harness_invoke(endpoint, request_body, request_headers) + if raw: + click.echo(_json_dumps(response)) + return + output = response.get("output") + click.echo(output if output is not None else _json_dumps(response)) + + +def _is_harness_invoke(**values: object) -> bool: + return any(value is not None and value is not False for value in values.values()) + + +def _delegate_agentkit_invoke( + *, + config_file: Path | None, + message: str | None, + payload: str | None, + headers: str | None, + runtime_id: str | None, + endpoint: str | None, + region: str | None, + a2a: bool, + show_reasoning: bool, + raw: bool, + apikey: str | None, +) -> None: + if _agentkit_invoke_command is not None: + _agentkit_invoke_command( + config_file=config_file, + message=message, + payload=payload, + headers=headers, + runtime_id=runtime_id, + endpoint=endpoint, + region=region, + a2a=a2a, + show_reasoning=show_reasoning, + raw=raw, + apikey=apikey, + ) + return + if _agentkit_invoke_click_command is None: + raise click.ClickException( + "Installed AgentKit CLI does not expose an invoke command." + ) + _agentkit_invoke_click_command.main( + args=_agentkit_invoke_args( + config_file=config_file, + message=message, + payload=payload, + headers=headers, + runtime_id=runtime_id, + endpoint=endpoint, + region=region, + a2a=a2a, + show_reasoning=show_reasoning, + raw=raw, + apikey=apikey, + ), + prog_name="invoke", + standalone_mode=False, + ) + + +def _agentkit_invoke_args( + *, + config_file: Path | None, + message: str | None, + payload: str | None, + headers: str | None, + runtime_id: str | None, + endpoint: str | None, + region: str | None, + a2a: bool, + show_reasoning: bool, + raw: bool, + apikey: str | None, +) -> list[str]: + args: list[str] = [] + commands = getattr(_agentkit_invoke_click_command, "commands", {}) + if isinstance(commands, dict) and "run" in commands: + args.append("run") + _append_option(args, "--config-file", str(config_file) if config_file else None) + _append_option(args, "--payload", payload) + _append_option(args, "--headers", headers) + _append_option(args, "--runtime-id", runtime_id) + _append_option(args, "--endpoint", endpoint) + _append_option(args, "--region", region) + _append_flag(args, "--a2a", a2a) + _append_flag(args, "--show-reasoning", show_reasoning) + _append_flag(args, "--raw", raw) + _append_option(args, "--apikey", apikey) + if message is not None: + args.append(message) + return args + + +def _append_option(args: list[str], name: str, value: str | None) -> None: + if value is not None: + args.extend([name, value]) + + +def _append_flag(args: list[str], name: str, enabled: bool) -> None: + if enabled: + args.append(name) + + +def _build_harness_body( + *, + message: str | None, + payload: str | None, + harness: str | None, + user_id: str, + session_id: str, + max_llm_calls: int | None, + model_id: str | None, + tools: str | None, + skills: str | None, + system_prompt: str | None, + runtime: str | None, + enable_harness_enhance: bool, + harness_components: str | None, + harness_profile: str | None, + harness_compression_provider: str | None, +) -> dict[str, object]: + payload_data = _json_loads_object(payload) if payload else {} + prompt = ( + message + or _string_value(payload_data, "prompt") + or _string_value(payload_data, "message") + ) + if not prompt: + raise click.ClickException( + "Provide a prompt as MESSAGE, --payload.prompt, or --payload.message." + ) + run_request: dict[str, object] = { + "user_id": _string_value(payload_data, "user_id") or user_id, + "session_id": _string_value(payload_data, "session_id") or session_id, + } + if max_llm_calls is not None: + run_request["max_llm_calls"] = max_llm_calls + body: dict[str, object] = { + "prompt": prompt, + "harness_name": harness + or _string_value(payload_data, "harness_name") + or "default", + "run_agent_request": run_request, + } + override = {} + if model_id is not None: + override["model_name"] = model_id + if tools is not None: + override["tools"] = tools + if skills is not None: + override["skills"] = skills + if system_prompt is not None: + override["system_prompt"] = system_prompt + if runtime is not None: + override["runtime"] = runtime + if override: + body["harness"] = override + enhance = _json_object_value(payload_data, "harness_enhance") + if enable_harness_enhance: + enhance["enabled"] = True + if harness_components is not None: + enhance["components"] = harness_components + if harness_profile is not None: + enhance["profile"] = harness_profile + if harness_compression_provider is not None: + enhance["compression_provider"] = harness_compression_provider + if enhance: + body["harness_enhance"] = enhance + return body + + +def _build_harness_headers( + *, + headers: str | None, + apikey: str | None, + enable_harness_enhance: bool, + harness_components: str | None, + harness_profile: str | None, + harness_compression_provider: str | None, +) -> dict[str, str]: + request_headers = {"Content-Type": "application/json"} + request_headers.update( + {key: str(value) for key, value in _json_loads_object(headers).items()} + if headers + else {} + ) + if apikey and "Authorization" not in request_headers: + request_headers["Authorization"] = f"Bearer {apikey}" + if enable_harness_enhance: + request_headers["X-Harness-Enhance"] = "true" + if harness_components: + request_headers["X-Harness-Components"] = harness_components + if harness_profile: + request_headers["X-Harness-Profile"] = harness_profile + if harness_compression_provider: + request_headers["X-Harness-Compression-Provider"] = harness_compression_provider + return request_headers + + +def _post_harness_invoke( + endpoint: str, body: dict[str, object], headers: dict[str, str] +) -> dict[str, object]: + import httpx + + response = httpx.post( + endpoint.rstrip("/") + "/harness/invoke", + json=body, + headers=headers, + timeout=600, + ) + if response.status_code != 200: + raise click.ClickException( + f"/harness/invoke failed: HTTP {response.status_code} - {response.text}" + ) + data = response.json() + if not isinstance(data, dict): + raise click.ClickException("HarnessApp returned a non-object JSON response.") + return data + + +def _json_loads_object(value: str | None) -> dict[str, object]: + import json + + if not value: + return {} + try: + parsed = json.loads(value) + except json.JSONDecodeError as exc: + raise click.ClickException(f"Invalid JSON object: {exc}") from exc + if not isinstance(parsed, dict): + raise click.ClickException("Expected a JSON object.") + return parsed + + +def _json_dumps(value: object) -> str: + import json + + return json.dumps(value, ensure_ascii=False, indent=2) + + +def _string_value(data: dict[str, object], key: str) -> str: + value = data.get(key) + return value if isinstance(value, str) else "" + + +def _json_object_value(data: dict[str, object], key: str) -> dict[str, object]: + value = data.get(key) + return dict(value) if isinstance(value, dict) else {} diff --git a/veadk/cli/cli_harness.py b/veadk/cli/cli_harness.py index 0288fad5..91256926 100644 --- a/veadk/cli/cli_harness.py +++ b/veadk/cli/cli_harness.py @@ -19,7 +19,7 @@ * ``veadk harness create `` writes a deployable directory: a blank ``harness.yaml`` template, a ``.env.example`` (deploy credentials only), a - ``Dockerfile``, and a short ``README.md``. + ``Dockerfile``, a ``.gitignore``, and a short ``README.md``. * ``veadk harness add`` writes agent parameters into ``harness.yaml``. * ``veadk harness deploy`` flattens ``harness.yaml`` into runtime env vars and performs a cloud AgentKit build + runtime create (no local Docker). @@ -95,6 +95,19 @@ structured_tool_calls: false include_tools_every_turn: true +# --- Harness enhance (optional) --------------------------------------------- +# Enables composable Runner plugins for context engineering, tool-result +# compression, and answer verification inside the runtime. Edit this block +# directly in harness.yaml, or override it per request through AgentKit invoke. +# env: HARNESS_ENHANCE_ENABLED +# env: HARNESS_ENHANCE_COMPONENTS +# env: HARNESS_ENHANCE_PROFILE +harness_enhance: + enabled: false + components: [invocation_context, compactor, response_verification] + profile: default + compression_provider: builtin + # --- Knowledge base ---------------------------------------------------------- # type -> env: KNOWLEDGEBASE_TYPE flag: --knowledgebase-type # "" disables it. Supported: viking | opensearch | redis | tos_vector | context_search @@ -180,12 +193,25 @@ # VOLCENGINE_REGION=cn-beijing """ -# Container image for the harness server. The base image's apt mirror is an -# unreachable internal host, so apt is repointed at aliyun; the source branch is -# cloned via the ghfast proxy with a github fallback; uv installs from aliyun. +_GITIGNORE = """\ +# Local deploy credentials and generated runtime metadata. +.env +.env.* +!.env.example +harness.json +agentkit.yaml +agentkit*.yaml +.agentkit/ +""" + +# Container image for the harness server. The base image's default apt source +# can be unreachable in some environments, so apt is repointed at a public +# mirror. The source branch is cloned through an acceleration URL with an +# official GitHub fallback, and uv installs from a public Python mirror. _DOCKERFILE = """\ FROM agentkit-cn-beijing.cr.volces.com/base/py-simple:python3.12-bookworm-slim-latest ENV PYTHONUNBUFFERED=1 +ENV PYTHONPATH=/app/src RUN set -eux; \\ rm -f /etc/apt/sources.list.d/*; \\ printf 'deb http://mirrors.aliyun.com/debian bookworm main contrib non-free non-free-firmware\\n\\ @@ -196,17 +222,18 @@ apt-get install -y --no-install-recommends git ca-certificates; \\ rm -rf /var/lib/apt/lists/* WORKDIR /app +ARG VEADK_REF=main RUN set -eux; \\ for url in \\ https://ghfast.top/https://github.com/volcengine/veadk-python.git \\ https://github.com/volcengine/veadk-python.git ; do \\ for i in 1 2 3; do \\ - git clone --depth 1 -b feat/harness-runtime "$url" src && break 2 || sleep 8; \\ + git clone --depth 1 -b "$VEADK_REF" "$url" src && break 2 || sleep 8; \\ done; \\ done; \\ test -d src/veadk RUN uv pip install --system --index-url https://mirrors.aliyun.com/pypi/simple/ \\ - ./src fastapi "uvicorn[standard]" + "./src[harness]" fastapi "uvicorn[standard]" EXPOSE 8000 CMD ["python", "-m", "uvicorn", "veadk.cloud.harness_app.app:app", "--host", "0.0.0.0", "--port", "8000"] """ @@ -221,6 +248,7 @@ - `harness.yaml` — agent configuration; flattened into runtime env vars. - `.env.example` — Volcengine deploy credentials; copy to `.env` and fill in. +- `.gitignore` — keeps local credentials and generated deploy metadata out of git. - `Dockerfile` — builds the harness server image. - `README.md` — this file. @@ -250,6 +278,7 @@ Harness deployment directory created at {target}: - harness.yaml (agent configuration) - .env.example (copy to .env and set VOLCENGINE_ACCESS_KEY / SECRET_KEY) +- .gitignore (ignores local credentials and generated deploy metadata) - Dockerfile (builds the harness image) - README.md @@ -286,6 +315,7 @@ def create(dir_name: str) -> None: target.mkdir(parents=True, exist_ok=True) (target / "harness.yaml").write_text(_HARNESS_YAML) (target / ".env.example").write_text(_ENV_EXAMPLE) + (target / ".gitignore").write_text(_GITIGNORE) (target / "Dockerfile").write_text(_DOCKERFILE) (target / "README.md").write_text(_README) @@ -467,6 +497,7 @@ def add( data["system_prompt"] = system_prompt if runtime is not None: data["runtime"] = runtime + # Set only the backend `type`, preserving any connection params already set # under the component section. for type_value, section_key in ( @@ -814,7 +845,7 @@ def _create_runtime_with_harness_tag(self, request): lines.append(f"Discovery: {auth['discovery_url']}") lines.append(f"Allowed ids: {', '.join(auth['allowed_ids'])}") elif apikey: - lines.append(f"API key: {apikey}") + lines.append("API key: saved in local harness.json (not printed)") if endpoint: json_path = _record_harness( diff --git a/veadk/cloud/harness_app/app.py b/veadk/cloud/harness_app/app.py index 9262cd9e..7737edb0 100644 --- a/veadk/cloud/harness_app/app.py +++ b/veadk/cloud/harness_app/app.py @@ -39,30 +39,40 @@ from pathlib import Path from typing import Any -from fastapi import FastAPI +from fastapi import FastAPI, Request from fastapi.responses import StreamingResponse from google.adk.agents import RunConfig -from google.adk.agents.run_config import StreamingMode from google.adk.agents.base_agent import BaseAgent +from google.adk.agents.run_config import StreamingMode from google.adk.artifacts.in_memory_artifact_service import InMemoryArtifactService from google.adk.auth.credential_service.in_memory_credential_service import ( InMemoryCredentialService, ) from google.adk.cli.adk_web_server import AdkWebServer, RunAgentRequest from google.adk.cli.utils.base_agent_loader import BaseAgentLoader -from google.adk.utils.context_utils import Aclosing from google.adk.evaluation.local_eval_set_results_manager import ( LocalEvalSetResultsManager, ) from google.adk.evaluation.local_eval_sets_manager import LocalEvalSetsManager from google.adk.memory.in_memory_memory_service import InMemoryMemoryService +from google.adk.plugins import BasePlugin +from google.adk.utils.context_utils import Aclosing from typing_extensions import override from veadk import Agent from veadk.a2a.utils.agent_to_a2a import to_a2a from veadk.cloud.harness_app.agent import agent, short_term_memory +from veadk.cloud.harness_app.harness_plugins import ( + build_harness_plugins_from_enhance, + build_harness_plugins_from_headers, + build_harness_plugins_from_runtime_env, +) +from veadk.cloud.harness_app.metrics import HarnessLlmUsagePlugin from veadk.cloud.harness_app.types import ( + HarnessCompactionMetric, HarnessOverrides, + HarnessPluginMetrics, + HarnessResponseMetrics, InvokeHarnessRequest, InvokeHarnessResponse, ) @@ -83,6 +93,12 @@ DEFAULT_MAX_LLM_CALLS = ( int(os.environ["MAX_LLM_CALLS"]) if os.environ.get("MAX_LLM_CALLS") else None ) +RETURN_LLM_USAGE = os.getenv("HARNESS_APP_RETURN_LLM_USAGE", "").lower() in { + "1", + "true", + "yes", + "on", +} class _HarnessAgentLoader(BaseAgentLoader): @@ -140,10 +156,13 @@ def __init__( self.short_term_memory = short_term_memory self.harness_name = harness_name self.max_llm_calls = max_llm_calls + self.return_llm_usage = RETURN_LLM_USAGE + self.plugins = build_harness_plugins_from_runtime_env() self.runner = Runner( agent=agent, short_term_memory=short_term_memory, app_name=harness_name, + plugins=self.plugins, ) # ADK web/api server over the single in-memory agent (reuses the harness @@ -182,6 +201,7 @@ def mount(self): @self.app.post("/harness/invoke") async def invoke_harness( request: InvokeHarnessRequest, + http_request: Request, ) -> InvokeHarnessResponse: # max LLM calls: per-call override, else the harness default; if # neither is set, fall through to ADK RunConfig's own default. @@ -195,6 +215,23 @@ async def invoke_harness( ) try: + header_plugins = build_harness_plugins_from_headers( + http_request.headers + ) + body_plugins = build_harness_plugins_from_enhance( + request.harness_enhance + ) + usage_plugin = ( + HarnessLlmUsagePlugin() if self.return_llm_usage else None + ) + harness_plugins = body_plugins or header_plugins or self.plugins + self._reset_plugin_diagnostics(harness_plugins) + if harness_plugins: + logger.info( + "Harness plugins enabled for invocation: " + + ", ".join(self._plugin_names(harness_plugins)) + ) + plugins = self._plugins_for_run(harness_plugins, usage_plugin) if request.harness is not None: logger.info( f"Applying once-time harness override: {request.harness}" @@ -213,6 +250,7 @@ async def invoke_harness( agent=agent, short_term_memory=self.short_term_memory, app_name=self.harness_name, + plugins=plugins, ) output = await runner.run( messages=[request.prompt], @@ -220,6 +258,32 @@ async def invoke_harness( session_id=request.run_agent_request.session_id, run_config=run_config, ) + elif header_plugins: + runner = Runner( + agent=self.agent, + short_term_memory=self.short_term_memory, + app_name=self.harness_name, + plugins=plugins, + ) + output = await runner.run( + messages=[request.prompt], + user_id=request.run_agent_request.user_id, + session_id=request.run_agent_request.session_id, + run_config=run_config, + ) + elif usage_plugin: + runner = Runner( + agent=self.agent, + short_term_memory=self.short_term_memory, + app_name=self.harness_name, + plugins=plugins, + ) + output = await runner.run( + messages=[request.prompt], + user_id=request.run_agent_request.user_id, + session_id=request.run_agent_request.session_id, + run_config=run_config, + ) else: output = await self.runner.run( messages=[request.prompt], @@ -252,6 +316,11 @@ async def invoke_harness( harness_name=self.harness_name, overwrite=request.harness is not None, output=output, + metrics=( + self._response_metrics(harness_plugins, usage_plugin) + if usage_plugin + else None + ), ) def _mount_run_sse_override(self): @@ -354,6 +423,59 @@ async def _run_sse_events(self, req: "HarnessRunAgentRequest"): if work_dir_ctx is not None: work_dir_ctx.cleanup() + def _plugins_for_run( + self, + plugins: list[BasePlugin], + usage_plugin: HarnessLlmUsagePlugin | None, + ) -> list[BasePlugin]: + if usage_plugin is None: + return plugins + return [*plugins, usage_plugin] + + def _response_metrics( + self, + plugins: list[BasePlugin], + usage_plugin: HarnessLlmUsagePlugin, + ) -> HarnessResponseMetrics: + return HarnessResponseMetrics( + llm_usage=usage_plugin.metrics, + harness_plugins=HarnessPluginMetrics( + names=self._plugin_names(plugins), + compaction_reports=self._compaction_reports(plugins), + ), + ) + + def _plugin_names(self, plugins: list[BasePlugin]) -> list[str]: + return [ + str(getattr(plugin, "name", plugin.__class__.__name__)) + for plugin in plugins + ] + + def _reset_plugin_diagnostics(self, plugins: list[BasePlugin]) -> None: + for plugin in plugins: + reset_diagnostics = getattr(plugin, "reset_diagnostics", None) + if callable(reset_diagnostics): + reset_diagnostics() + + def _compaction_reports( + self, plugins: list[BasePlugin] + ) -> list[HarnessCompactionMetric]: + metrics: list[HarnessCompactionMetric] = [] + for plugin in plugins: + reports = getattr(plugin, "compaction_reports", None) + if not isinstance(reports, list): + continue + for report in reports: + if hasattr(report, "model_dump"): + metrics.append( + HarnessCompactionMetric.model_validate( + report.model_dump(mode="json") + ) + ) + elif isinstance(report, dict): + metrics.append(HarnessCompactionMetric.model_validate(report)) + return metrics + def serve(self, host: str = "0.0.0.0", port: int = 8000) -> None: import uvicorn diff --git a/veadk/cloud/harness_app/env_mapping.py b/veadk/cloud/harness_app/env_mapping.py index 1330140e..7b3d5d63 100644 --- a/veadk/cloud/harness_app/env_mapping.py +++ b/veadk/cloud/harness_app/env_mapping.py @@ -169,6 +169,8 @@ def to_runtime_env(spec: dict[str, Any]) -> dict[str, str]: continue env[key.upper()] = _stringify(value) + _add_harness_enhance_aliases(env, spec) + # Component sections: `type` selector + backend-specific connection params. for component, type_env in COMPONENT_TYPE_ENV.items(): section: dict[str, Any] = spec.get(component) or {} @@ -195,3 +197,43 @@ def to_runtime_env(spec: dict[str, Any]) -> dict[str, str]: env[env_name] = _stringify(value) return env + + +def _add_harness_enhance_aliases(env: dict[str, str], spec: dict[str, Any]) -> None: + """Expose harness_enhance fields under the SDK's generic env names too.""" + + section = spec.get("harness_enhance") + if not isinstance(section, dict): + return + compression = section.get("compression") + if not isinstance(compression, dict): + compression = {} + verifier = section.get("verifier") + if not isinstance(verifier, dict): + verifier = {} + + aliases = { + "components": "HARNESS_COMPONENTS", + "profile": "HARNESS_PROFILE", + "compression_provider": "HARNESS_COMPRESSION_PROVIDER", + "max_context_chars": "HARNESS_MAX_CONTEXT_CHARS", + "max_tool_result_chars": "HARNESS_MAX_TOOL_RESULT_CHARS", + "verifier_mode": "HARNESS_VERIFIER_MODE", + "store_path": "HARNESS_STORE_PATH", + } + nested_aliases = { + "provider": "HARNESS_COMPRESSION_PROVIDER", + "max_context_chars": "HARNESS_MAX_CONTEXT_CHARS", + "max_tool_result_chars": "HARNESS_MAX_TOOL_RESULT_CHARS", + } + for key, env_name in aliases.items(): + value = section.get(key) + if not _is_empty(value): + env[env_name] = _stringify(value) + for key, env_name in nested_aliases.items(): + value = compression.get(key) + if not _is_empty(value): + env[env_name] = _stringify(value) + verifier_mode = verifier.get("mode") + if not _is_empty(verifier_mode): + env["HARNESS_VERIFIER_MODE"] = _stringify(verifier_mode) diff --git a/veadk/cloud/harness_app/harness_plugins.py b/veadk/cloud/harness_app/harness_plugins.py new file mode 100644 index 00000000..731f5623 --- /dev/null +++ b/veadk/cloud/harness_app/harness_plugins.py @@ -0,0 +1,126 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Harness plugin assembly for HarnessApp Runtime.""" + +from __future__ import annotations + +import os +from collections.abc import Mapping + +from google.adk.plugins import BasePlugin + +from veadk.cloud.harness_app.types import HarnessEnhanceOverrides +from veadk.utils.logger import get_logger + +logger = get_logger(__name__) + +_TRUTHY = {"1", "true", "yes", "on"} + + +def build_harness_plugins_from_runtime_env( + env: Mapping[str, str] | None = None, +) -> list[BasePlugin]: + """Build Harness plugins from environment values.""" + + values = dict(env or os.environ) + try: + from veadk.extensions.harness.env import build_harness_plugins_from_env + except ImportError as e: + if _truthy(values.get("HARNESS_ENHANCE_ENABLED")): + logger.warning( + "HARNESS_ENHANCE_ENABLED is set but the Harness extension " + f"could not be imported: {e!r}" + ) + return [] + return build_harness_plugins_from_env(values) + + +def build_harness_plugins_from_headers( + headers: Mapping[str, str], + *, + base_env: Mapping[str, str] | None = None, +) -> list[BasePlugin]: + """Build per-invocation plugins from AgentKit/HTTP Harness headers.""" + + header_env = harness_env_from_headers(headers) + if not header_env: + return [] + values = dict(base_env or os.environ) + values.update(header_env) + return build_harness_plugins_from_runtime_env(values) + + +def build_harness_plugins_from_enhance( + enhance: HarnessEnhanceOverrides | None, + *, + base_env: Mapping[str, str] | None = None, +) -> list[BasePlugin]: + """Build per-invocation plugins from request-body Harness settings.""" + + body_env = harness_env_from_enhance(enhance) + if not body_env: + return [] + values = dict(base_env or os.environ) + values.update(body_env) + return build_harness_plugins_from_runtime_env(values) + + +def harness_env_from_headers(headers: Mapping[str, str]) -> dict[str, str]: + """Convert generic Harness headers into SDK environment keys.""" + + normalized = {str(key).lower(): str(value) for key, value in headers.items()} + enabled = normalized.get("x-harness-enhance") or normalized.get( + "x-harness-enable-context" + ) + if not _truthy(enabled): + return {} + env = {"HARNESS_ENHANCE_ENABLED": "true"} + components = normalized.get("x-harness-components") + if components: + env["HARNESS_COMPONENTS"] = components + env["HARNESS_ENHANCE_COMPONENTS"] = components + profile = normalized.get("x-harness-profile") + if profile: + env["HARNESS_PROFILE"] = profile + env["HARNESS_ENHANCE_PROFILE"] = profile + compression_provider = normalized.get("x-harness-compression-provider") + if compression_provider: + env["HARNESS_COMPRESSION_PROVIDER"] = compression_provider + env["HARNESS_ENHANCE_COMPRESSION_PROVIDER"] = compression_provider + return env + + +def harness_env_from_enhance( + enhance: HarnessEnhanceOverrides | None, +) -> dict[str, str]: + """Convert request-body Harness settings into SDK environment keys.""" + + if enhance is None or not enhance.enabled: + return {} + env = {"HARNESS_ENHANCE_ENABLED": "true"} + if enhance.components: + env["HARNESS_COMPONENTS"] = enhance.components + env["HARNESS_ENHANCE_COMPONENTS"] = enhance.components + if enhance.profile: + env["HARNESS_PROFILE"] = enhance.profile + env["HARNESS_ENHANCE_PROFILE"] = enhance.profile + if enhance.compression_provider: + env["HARNESS_COMPRESSION_PROVIDER"] = enhance.compression_provider + env["HARNESS_ENHANCE_COMPRESSION_PROVIDER"] = enhance.compression_provider + return env + + +def _truthy(value: str | None) -> bool: + return bool(value and value.strip().lower() in _TRUTHY) diff --git a/veadk/cloud/harness_app/metrics.py b/veadk/cloud/harness_app/metrics.py new file mode 100644 index 00000000..c4be40d7 --- /dev/null +++ b/veadk/cloud/harness_app/metrics.py @@ -0,0 +1,93 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Per-invocation metrics plugins for HarnessApp Runtime.""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import TYPE_CHECKING + +from google.adk.models import LlmResponse +from google.adk.plugins import BasePlugin + +from veadk.cloud.harness_app.types import LlmUsageMetrics + +if TYPE_CHECKING: + from google.adk.agents.callback_context import CallbackContext + + +class HarnessLlmUsagePlugin(BasePlugin): + """Collect aggregate LLM usage from ADK model callbacks.""" + + def __init__(self) -> None: + super().__init__(name="harness_llm_usage_collector") + self.metrics = LlmUsageMetrics() + + async def after_model_callback( + self, + *, + callback_context: "CallbackContext", + llm_response: LlmResponse, + ) -> LlmResponse | None: + usage = _usage_metrics_from_object(llm_response.usage_metadata) + if usage.has_tokens(): + self.metrics.add(usage) + return None + + +def _usage_metrics_from_object(value: object) -> LlmUsageMetrics: + return LlmUsageMetrics( + prompt_tokens=_int_value( + value, + "prompt_token_count", + "prompt_tokens", + "input_tokens", + ), + completion_tokens=_int_value( + value, + "candidates_token_count", + "completion_tokens", + "output_tokens", + ), + total_tokens=_int_value( + value, + "total_token_count", + "total_tokens", + ), + cached_tokens=_int_value( + value, + "cached_content_token_count", + "cached_tokens", + ), + usage_event_count=1, + ) + + +def _int_value(value: object, *names: str) -> int: + for name in names: + raw = _field_value(value, name) + if raw is None: + continue + try: + return int(raw) + except (TypeError, ValueError): + continue + return 0 + + +def _field_value(value: object, name: str) -> object | None: + if isinstance(value, Mapping): + return value.get(name) + return getattr(value, name, None) diff --git a/veadk/cloud/harness_app/types.py b/veadk/cloud/harness_app/types.py index a310aec5..447a1645 100644 --- a/veadk/cloud/harness_app/types.py +++ b/veadk/cloud/harness_app/types.py @@ -99,6 +99,30 @@ class HarnessConfig(HarnessOverrides): registry_poll_interval_ms: int = Field(default=5000) +class HarnessEnhanceOverrides(BaseModel): + """Per-invocation Harness enhancement options. + + This mirrors the runtime ``harness_enhance`` section but is intentionally + small: callers can enable the plugin bundle, choose components, select a + profile, and choose the compaction provider. Runtime limits keep their + deploy-time defaults. + """ + + enabled: bool = Field( + default=False, + description="Enable Harness plugins for this single invocation.", + ) + components: str = Field( + default="invocation_context,compactor,response_verification", + description="Comma-separated Harness plugin components.", + ) + profile: str = Field(default="default", description="Harness profile name.") + compression_provider: str | None = Field( + default=None, + description="Tool-result compaction provider, e.g. builtin or headroom.", + ) + + class RunAgentRequest(BaseModel): user_id: str session_id: str @@ -108,6 +132,55 @@ class RunAgentRequest(BaseModel): ) +class LlmUsageMetrics(BaseModel): + """Aggregated model usage for one HarnessApp invocation.""" + + prompt_tokens: int = 0 + completion_tokens: int = 0 + total_tokens: int = 0 + cached_tokens: int = 0 + usage_event_count: int = 0 + + def add(self, value: "LlmUsageMetrics") -> None: + self.prompt_tokens += value.prompt_tokens + self.completion_tokens += value.completion_tokens + self.total_tokens += value.total_tokens + self.cached_tokens += value.cached_tokens + self.usage_event_count += value.usage_event_count + + def has_tokens(self) -> bool: + return bool(self.total_tokens or self.prompt_tokens or self.completion_tokens) + + +class HarnessCompactionMetric(BaseModel): + """Compaction accounting exposed by HarnessApp diagnostics.""" + + provider: str = "" + original_chars: int = 0 + compressed_chars: int = 0 + changed: bool = False + tokens_before: int = 0 + tokens_after: int = 0 + tokens_saved: int = 0 + compression_ratio: float = 0.0 + transforms_applied: list[str] = Field(default_factory=list) + warnings: list[str] = Field(default_factory=list) + + +class HarnessPluginMetrics(BaseModel): + """Harness plugin diagnostics for one invocation.""" + + names: list[str] = Field(default_factory=list) + compaction_reports: list[HarnessCompactionMetric] = Field(default_factory=list) + + +class HarnessResponseMetrics(BaseModel): + """Optional machine-readable metrics returned by HarnessApp Runtime.""" + + llm_usage: LlmUsageMetrics = Field(default_factory=LlmUsageMetrics) + harness_plugins: HarnessPluginMetrics = Field(default_factory=HarnessPluginMetrics) + + class InvokeHarnessRequest(BaseModel): prompt: str harness_name: str @@ -115,6 +188,7 @@ class InvokeHarnessRequest(BaseModel): # this single call. Only the fields actually set are applied; memory and the # knowledge base are never overridable (absent from HarnessOverrides). harness: HarnessOverrides | None = None + harness_enhance: HarnessEnhanceOverrides | None = None run_agent_request: RunAgentRequest @@ -122,6 +196,10 @@ class InvokeHarnessResponse(BaseModel): harness_name: str overwrite: bool = Field(default=False) output: str + metrics: HarnessResponseMetrics | None = Field( + default=None, + description="Optional runtime metrics, enabled by HARNESS_APP_RETURN_LLM_USAGE.", + ) error: str | None = Field( default=None, description=( diff --git a/veadk/cloud/harness_app/utils.py b/veadk/cloud/harness_app/utils.py index cdef4175..f0b651a3 100644 --- a/veadk/cloud/harness_app/utils.py +++ b/veadk/cloud/harness_app/utils.py @@ -31,6 +31,7 @@ from dataclasses import replace from pathlib import Path from typing import Any +from urllib.parse import urlencode import frontmatter import httpx @@ -94,10 +95,13 @@ def _load_builtin_tool(name: str) -> Any: # Skill hub download endpoint. A skill name in a harness is the path after -# `/download/`, e.g. "clawhub/lgwventrue/system-file-handler". +# `/download/`, e.g. "namespace/owner/skill-name". SKILL_HUB_DOWNLOAD_URL = os.getenv( "SKILL_HUB_DOWNLOAD_URL", "https://skills.volces.com/v1/skills/download" ) +SKILL_HUB_SEARCH_URL = os.getenv( + "SKILL_HUB_SEARCH_URL", "https://skills.volces.com/v1/skills" +) # Maps HarnessConfig field names to their environment variables. ``app_name`` is # populated via its "name" alias. Only variables that are set are passed, so the @@ -140,7 +144,7 @@ def _download_and_extract_skill(skill: str, dest_dir: Path) -> Path: Args: skill: Skill identifier — the hub path after ``/download/`` - (e.g. ``"clawhub/lgwventrue/system-file-handler"``). + (e.g. ``"namespace/owner/skill-name"``). dest_dir: Base directory to extract into; the skill is placed in a subdirectory named after its declared name in ``SKILL.md``. @@ -148,15 +152,7 @@ def _download_and_extract_skill(skill: str, dest_dir: Path) -> Path: The directory the skill was extracted to. Its name matches the skill's declared name in ``SKILL.md`` (required by ``load_skill_from_dir``). """ - name = skill.strip("/") - url = f"{SKILL_HUB_DOWNLOAD_URL.rstrip('/')}/{name}" - logger.info(f"Downloading skill '{skill}' from {url}") - - response = httpx.get(url, timeout=60, follow_redirects=True) - if response.status_code != 200: - raise RuntimeError( - f"Failed to download skill '{skill}': HTTP {response.status_code}" - ) + name, archive = _download_skill_archive(skill) # Extract to a staging dir first; the final directory must be named after # the skill's declared name (ADK's load_skill_from_dir enforces this). @@ -165,7 +161,7 @@ def _download_and_extract_skill(skill: str, dest_dir: Path) -> Path: shutil.rmtree(staging) staging.mkdir(parents=True) staging_root = staging.resolve() - with zipfile.ZipFile(io.BytesIO(response.content)) as zf: + with zipfile.ZipFile(io.BytesIO(archive)) as zf: for member in zf.namelist(): # Guard against path traversal (zip-slip). if not (staging / member).resolve().is_relative_to(staging_root): @@ -192,6 +188,89 @@ def _download_and_extract_skill(skill: str, dest_dir: Path) -> Path: return skill_dir +def _download_skill_archive(skill: str) -> tuple[str, bytes]: + name = skill.strip("/") + response = _download_skill_response(name) + if response.status_code == 200 and _looks_like_zip(response.content): + return name, response.content + + resolved_name = _resolve_skill_download_name(name) + if resolved_name and resolved_name != name: + response = _download_skill_response(resolved_name) + if response.status_code == 200 and _looks_like_zip(response.content): + return resolved_name, response.content + + if response.status_code != 200: + raise RuntimeError( + f"Failed to download skill '{skill}': HTTP {response.status_code}" + ) + raise RuntimeError( + f"Failed to download skill '{skill}': response is not a zip archive" + ) + + +def _download_skill_response(name: str) -> httpx.Response: + url = f"{SKILL_HUB_DOWNLOAD_URL.rstrip('/')}/{name}" + logger.info(f"Downloading skill '{name}' from {url}") + return httpx.get(url, timeout=60, follow_redirects=True) + + +def _looks_like_zip(content: bytes) -> bool: + return content.startswith(b"PK\x03\x04") or content.startswith(b"PK\x05\x06") + + +def _resolve_skill_download_name(name: str) -> str | None: + if "/" in name: + return None + + query = urlencode({"query": name, "pageNumber": 1, "pageSize": 10}) + url = f"{SKILL_HUB_SEARCH_URL.rstrip('/')}?{query}" + try: + response = httpx.get(url, timeout=30, follow_redirects=True) + if response.status_code != 200: + return None + data = response.json() + except Exception: + return None + + for item in _skill_search_items(data): + slug = _skill_item_text(item, "Slug") + if slug and _skill_item_matches(name, item): + logger.info(f"Resolved skill short name '{name}' to '{slug}'") + return slug.strip("/") + return None + + +def _skill_search_items(data: object) -> list[dict[str, object]]: + if not isinstance(data, dict): + return [] + items = data.get("Skills") or data.get("Items") or data.get("skills") + if not isinstance(items, list): + return [] + return [item for item in items if isinstance(item, dict)] + + +def _skill_item_matches(name: str, item: dict[str, object]) -> bool: + normalized = _normalize_skill_token(name) + tokens = { + _normalize_skill_token(_skill_item_text(item, "Name")), + _normalize_skill_token(_skill_item_text(item, "Slug")), + _normalize_skill_token(_skill_item_text(item, "Slug").rsplit("/", 1)[-1]), + _normalize_skill_token(_skill_item_text(item, "SourceRepo")), + _normalize_skill_token(_skill_item_text(item, "SourceRepo").rsplit("/", 1)[-1]), + } + return normalized in tokens + + +def _skill_item_text(item: dict[str, object], key: str) -> str: + value = item.get(key) or item.get(key.lower()) + return value if isinstance(value, str) else "" + + +def _normalize_skill_token(value: str) -> str: + return value.strip().lower().replace("_", "-") + + class SkillLoadError(RuntimeError): """A skill failed to download or load (e.g. a malformed ``SKILL.md``). diff --git a/veadk/extensions/harness/README.md b/veadk/extensions/harness/README.md new file mode 100644 index 00000000..e6eaab8e --- /dev/null +++ b/veadk/extensions/harness/README.md @@ -0,0 +1,100 @@ +# VeADK Harness Extension + +[中文](README.zh.md) + +`veadk.extensions.harness` is a lightweight Harness extension for VeADK applications. It adds three reusable capabilities: + +- context preparation for each agent turn +- tool-result compaction +- answer verification and hallucination suppression + +The extension can be used directly as Python modules or attached to a VeADK +Runner. It does not require a separate runtime service. + +## Install + +```bash +pip install "veadk-python[harness]" +``` + +The base extension is bundled with VeADK. The `harness` extra installs the +optional in-process Headroom compaction provider. + +For local development inside this repository: + +```bash +pip install . +``` + +For local development with Headroom: + +```bash +pip install ".[harness]" +``` + +## Quick Start + +```python +from veadk.extensions.harness.plugins import build_harness_plugins +from veadk import Agent, Runner + +agent = Agent(name="research_agent") +runner = Runner( + agent=agent, + app_name="research", + plugins=build_harness_plugins( + components=["invocation_context", "compactor", "response_verification"], + profile="research", + ), +) +``` + +## Plugins + +| Plugin | Main hooks | Purpose | +| --- | --- | --- | +| `HarnessInvocationContextPlugin` | `on_user_message_callback`, `before_model_callback` | Prepares task anchors, recent context, and tool-use guardrails. | +| `HarnessCompressPlugin` | `before_model_callback`, `after_tool_callback` | Shrinks oversized tool outputs while preserving useful facts. | +| `HarnessResponseVerificationPlugin` | `after_tool_callback`, `after_model_callback`, `on_event_callback` | Records tool receipts and flags unsupported final claims. | + +## Runtime Environment + +```text +HARNESS_ENHANCE_ENABLED=true +HARNESS_ENHANCE_COMPONENTS=invocation_context,compactor,response_verification +HARNESS_PROFILE=research +HARNESS_COMPRESSION_PROVIDER=builtin +``` + +```python +from veadk.extensions.harness.env import build_harness_plugins_from_env + +plugins = build_harness_plugins_from_env() +``` + +With VeADK HarnessApp deployment, the same settings can be written in +`harness.yaml`: + +```yaml +harness_enhance: + enabled: true + components: [invocation_context, compactor, response_verification] + profile: general + compression_provider: builtin +``` + +## Direct Module Usage + +```python +from veadk.extensions.harness import HarnessInvocationContextBuilder, HarnessInvocationRef + +context = HarnessInvocationRef(session_id="session-1", invocation_id="run-1") +builder = HarnessInvocationContextBuilder() +bundle = builder.prepare_context(context, user_input="Summarize these tool results.") +``` + +## Learn More + +See [docs/extensions/harness/README.md](../../../docs/extensions/harness/README.md) +for a short zero-to-first-run guide, concepts, configuration, and integration +guidance. diff --git a/veadk/extensions/harness/README.zh.md b/veadk/extensions/harness/README.zh.md new file mode 100644 index 00000000..a03aed14 --- /dev/null +++ b/veadk/extensions/harness/README.zh.md @@ -0,0 +1,98 @@ +# VeADK Harness Extension + +[English](README.md) + +`veadk.extensions.harness` 是一个轻量级 Harness Extension，它提供三个可复用能力： + +- 为每轮 Agent 调用准备上下文 +- 压缩大体积工具结果 +- 验证最终回答，降低幻觉 + +Extension 可以作为普通 Python 模块直接使用，也可以挂载到 VeADK Runner。 +它不需要单独启动运行时服务。 + +## 安装 + +```bash +pip install "veadk-python[harness]" +``` + +基础 Harness Extension 已随 VeADK 内置。`harness` extra 会安装可选的进程内 +Headroom 压缩 provider。 + +在当前仓库内本地开发： + +```bash +pip install . +``` + +本地开发并启用 Headroom： + +```bash +pip install ".[harness]" +``` + +## 快速开始 + +```python +from veadk.extensions.harness.plugins import build_harness_plugins +from veadk import Agent, Runner + +agent = Agent(name="research_agent") +runner = Runner( + agent=agent, + app_name="research", + plugins=build_harness_plugins( + components=["invocation_context", "compactor", "response_verification"], + profile="research", + ), +) +``` + +## 插件能力 + +| 插件 | 主要 Hook | 作用 | +| --- | --- | --- | +| `HarnessInvocationContextPlugin` | `on_user_message_callback`, `before_model_callback` | 准备任务锚点、近期上下文和工具使用约束。 | +| `HarnessCompressPlugin` | `before_model_callback`, `after_tool_callback` | 压缩过大的工具输出，同时保留关键事实。 | +| `HarnessResponseVerificationPlugin` | `after_tool_callback`, `after_model_callback`, `on_event_callback` | 记录工具执行 receipt，并标记缺少证据的最终回答。 | + +## 运行时配置 + +```text +HARNESS_ENHANCE_ENABLED=true +HARNESS_ENHANCE_COMPONENTS=invocation_context,compactor,response_verification +HARNESS_PROFILE=research +HARNESS_COMPRESSION_PROVIDER=builtin +``` + +```python +from veadk.extensions.harness.env import build_harness_plugins_from_env + +plugins = build_harness_plugins_from_env() +``` + +在 VeADK HarnessApp 部署中，也可以写入 `harness.yaml`： + +```yaml +harness_enhance: + enabled: true + components: [invocation_context, compactor, response_verification] + profile: general + compression_provider: builtin +``` + +## 直接使用模块 + +```python +from veadk.extensions.harness import HarnessInvocationContextBuilder, HarnessInvocationRef + +context = HarnessInvocationRef(session_id="session-1", invocation_id="run-1") +builder = HarnessInvocationContextBuilder() +bundle = builder.prepare_context(context, user_input="Summarize these tool results.") +``` + +## 更多文档 + +请阅读 [docs/extensions/harness/README.zh.md](../../../docs/extensions/harness/README.zh.md)。 +里面包含精简的新手教程、核心概念、配置方式和接入建议。 diff --git a/veadk/extensions/harness/__init__.py b/veadk/extensions/harness/__init__.py new file mode 100644 index 00000000..37b21d30 --- /dev/null +++ b/veadk/extensions/harness/__init__.py @@ -0,0 +1,86 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Composable Agent Harness SDK.""" + +from veadk.extensions.harness.plugins import HarnessLongRunControlPlugin +from veadk.extensions.harness.extension import HarnessExtension, HarnessExtensionConfig +from veadk.extensions.harness.modules.headroom_provider import ( + HeadroomCompressionProvider, +) +from veadk.extensions.harness.modules.invocation_context import ( + ContextEngine, + HarnessInvocationContextBuilder, + HarnessInvocationContextConfig, +) +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifier, + ResultVerifier, +) +from veadk.extensions.harness.modules.tool_result_compactor import ( + ToolResultCompactor, + ToolResultCompressor, +) +from veadk.extensions.harness.schemas import ( + CapabilityReceipt, + ToolReceipt, + CompactionReport, + CompressionReport, + CompressionRequest, + CompactionResult, + CompressionResult, + ContextBundle, + InvocationContextBlock, + ConversationMessage, + EvidenceRef, + HarnessEvent, + HarnessIntervention, + VerificationDecision, + HarnessRunContext, + HarnessInvocationRef, + TaskContract, + VerificationReport, +) + +__all__ = [ + "CapabilityReceipt", + "ToolReceipt", + "CompactionReport", + "CompressionReport", + "CompressionRequest", + "CompactionResult", + "CompressionResult", + "ContextBundle", + "InvocationContextBlock", + "ContextEngine", + "ConversationMessage", + "EvidenceRef", + "HeadroomCompressionProvider", + "HarnessIntervention", + "VerificationDecision", + "HarnessEvent", + "HarnessExtension", + "HarnessExtensionConfig", + "HarnessInvocationContextBuilder", + "HarnessInvocationContextConfig", + "HarnessRunContext", + "HarnessInvocationRef", + "HarnessLongRunControlPlugin", + "FinalResponseVerifier", + "ResultVerifier", + "TaskContract", + "ToolResultCompactor", + "ToolResultCompressor", + "VerificationReport", +] diff --git a/veadk/extensions/harness/env.py b/veadk/extensions/harness/env.py new file mode 100644 index 00000000..839dee0c --- /dev/null +++ b/veadk/extensions/harness/env.py @@ -0,0 +1,118 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""VeADK helpers for Harness plugin assembly.""" + +from __future__ import annotations + +import os +from collections.abc import Mapping +from typing import Literal + +from google.adk.plugins import BasePlugin + +from veadk.extensions.harness.plugins import build_harness_plugins +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifierConfig, +) +from veadk.extensions.harness.modules.invocation_context import ( + HarnessInvocationContextConfig, +) +from veadk.extensions.harness.modules.tool_result_compactor import ( + ToolResultCompactorConfig, +) +from veadk.extensions.harness.stores import JsonlHarnessStore + + +def harness_enabled_from_env(env: Mapping[str, str] | None = None) -> bool: + """Return whether Harness plugins should be attached.""" + + values = env or os.environ + return _truthy(values.get("HARNESS_ENHANCE_ENABLED")) + + +def build_harness_plugins_from_env( + env: Mapping[str, str] | None = None, +) -> list[BasePlugin]: + """Build Harness plugins from generic runtime environment variables.""" + + values = env or os.environ + if not harness_enabled_from_env(values): + return [] + components = ( + values.get("HARNESS_ENHANCE_COMPONENTS") + or values.get("HARNESS_COMPONENTS") + or "invocation_context,compactor,response_verification" + ) + profile = ( + values.get("HARNESS_ENHANCE_PROFILE") + or values.get("HARNESS_PROFILE") + or "default" + ) + max_context_chars = _int_value( + values.get("HARNESS_MAX_CONTEXT_CHARS") + or values.get("HARNESS_ENHANCE_MAX_CONTEXT_CHARS"), + default=24000, + ) + max_tool_result_chars = _int_value( + values.get("HARNESS_MAX_TOOL_RESULT_CHARS") + or values.get("HARNESS_ENHANCE_MAX_TOOL_RESULT_CHARS"), + default=4000, + ) + store_path = values.get("HARNESS_STORE_PATH") or values.get( + "HARNESS_ENHANCE_STORE_PATH" + ) + store = JsonlHarnessStore(store_path) if store_path else None + return build_harness_plugins( + components=components, + profile=profile, + store=store, + context_config=HarnessInvocationContextConfig( + max_context_chars=max_context_chars + ), + compaction_config=ToolResultCompactorConfig( + provider=values.get("HARNESS_COMPRESSION_PROVIDER") + or values.get("HARNESS_ENHANCE_COMPRESSION_PROVIDER") + or "builtin", + max_context_chars=max_context_chars, + max_tool_result_chars=max_tool_result_chars, + ), + verifier_config=FinalResponseVerifierConfig( + mode=_verifier_mode( + values.get("HARNESS_VERIFIER_MODE") + or values.get("HARNESS_ENHANCE_VERIFIER_MODE") + ) + ), + ) + + +def _truthy(value: str | None) -> bool: + return bool(value and value.strip().lower() in {"1", "true", "yes", "on"}) + + +def _int_value(value: str | None, *, default: int) -> int: + if not value: + return default + try: + return int(value) + except ValueError: + return default + + +def _verifier_mode(value: str | None) -> Literal["observe", "block"]: + normalized = (value or "observe").strip().lower() + return "block" if normalized == "block" else "observe" + + +__all__ = ["build_harness_plugins_from_env", "harness_enabled_from_env"] diff --git a/veadk/extensions/harness/events.py b/veadk/extensions/harness/events.py new file mode 100644 index 00000000..41d5d883 --- /dev/null +++ b/veadk/extensions/harness/events.py @@ -0,0 +1,19 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Event model exports.""" + +from veadk.extensions.harness.schemas import HarnessEvent + +__all__ = ["HarnessEvent"] diff --git a/veadk/extensions/harness/extension.py b/veadk/extensions/harness/extension.py new file mode 100644 index 00000000..fed7c018 --- /dev/null +++ b/veadk/extensions/harness/extension.py @@ -0,0 +1,122 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""VeADK extension facade for Harness plugins.""" + +from __future__ import annotations + +import os +from collections.abc import Iterable, Mapping + +from google.adk.plugins import BasePlugin +from pydantic import Field + +from veadk.extensions.harness.plugins import build_harness_plugins +from veadk.extensions.harness.env import ( + build_harness_plugins_from_env, + harness_enabled_from_env, +) +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifierConfig, +) +from veadk.extensions.harness.modules.invocation_context import ( + HarnessInvocationContextConfig, +) +from veadk.extensions.harness.modules.tool_result_compactor import ( + ToolResultCompactorConfig, +) +from veadk.extensions.harness.schemas import HarnessBaseModel +from veadk.extensions.harness.stores import HarnessStoreProtocol + + +class HarnessExtensionConfig(HarnessBaseModel): + """Configuration for :class:`HarnessExtension`.""" + + enabled: bool = True + components: list[str] = Field( + default_factory=lambda: [ + "invocation_context", + "compactor", + "response_verification", + ] + ) + profile: str = "default" + + +class HarnessExtension: + """Small VeADK-facing wrapper for Harness plugin assembly. + + The extension owns no core Harness logic. It keeps the public VeADK entry + point compact while delegating atomic behavior to the modules in this + package. + """ + + def __init__( + self, + *, + enabled: bool = True, + components: Iterable[str] | str | None = None, + profile: str = "default", + store: HarnessStoreProtocol | None = None, + context_config: HarnessInvocationContextConfig | None = None, + compaction_config: ToolResultCompactorConfig | None = None, + verifier_config: FinalResponseVerifierConfig | None = None, + env: Mapping[str, str] | None = None, + ) -> None: + if components is None: + component_list = HarnessExtensionConfig().components + elif isinstance(components, str): + component_list = [ + item.strip() for item in components.split(",") if item.strip() + ] + else: + component_list = [ + str(item).strip() for item in components if str(item).strip() + ] + self.config = HarnessExtensionConfig( + enabled=enabled, + components=component_list, + profile=profile, + ) + self.store = store + self.context_config = context_config + self.compaction_config = compaction_config + self.verifier_config = verifier_config + self.env = dict(env) if env is not None else None + + @classmethod + def from_env(cls, env: Mapping[str, str] | None = None) -> HarnessExtension: + """Create an extension controlled by Harness environment variables.""" + + values = dict(env or os.environ) + return cls(enabled=harness_enabled_from_env(values), env=values) + + def plugins(self) -> list[BasePlugin]: + """Build plugins for ``Runner(..., plugins=...)``.""" + + if self.env is not None: + return build_harness_plugins_from_env(self.env) + if not self.config.enabled: + return [] + return build_harness_plugins( + components=self.config.components, + profile=self.config.profile, + store=self.store, + context_config=self.context_config, + compaction_config=self.compaction_config, + verifier_config=self.verifier_config, + ) + + +__all__ = ["HarnessExtension", "HarnessExtensionConfig"] diff --git a/veadk/extensions/harness/modules/__init__.py b/veadk/extensions/harness/modules/__init__.py new file mode 100644 index 00000000..d159dbc8 --- /dev/null +++ b/veadk/extensions/harness/modules/__init__.py @@ -0,0 +1,57 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Atomic Harness modules.""" + +from veadk.extensions.harness.modules.headroom_provider import ( + HeadroomCompressionProvider, +) +from veadk.extensions.harness.modules.invocation_context import ( + ContextEngine, + ContextEngineConfig, + HarnessInvocationContextBuilder, + HarnessInvocationContextConfig, +) +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifier, + FinalResponseVerifierConfig, + ResultVerifier, + ResultVerifierConfig, +) +from veadk.extensions.harness.modules.tool_result_compactor import ( + ContextCompressionPolicy, + ContextCompactionPolicy, + ToolResultCompactor, + ToolResultCompactorConfig, + ToolResultCompressor, + ToolResultCompressorConfig, +) + +__all__ = [ + "ContextCompactionPolicy", + "ContextEngine", + "ContextEngineConfig", + "ContextCompressionPolicy", + "HarnessInvocationContextBuilder", + "HarnessInvocationContextConfig", + "HeadroomCompressionProvider", + "FinalResponseVerifier", + "FinalResponseVerifierConfig", + "ResultVerifier", + "ResultVerifierConfig", + "ToolResultCompactor", + "ToolResultCompactorConfig", + "ToolResultCompressor", + "ToolResultCompressorConfig", +] diff --git a/veadk/extensions/harness/modules/builtin_provider.py b/veadk/extensions/harness/modules/builtin_provider.py new file mode 100644 index 00000000..70925141 --- /dev/null +++ b/veadk/extensions/harness/modules/builtin_provider.py @@ -0,0 +1,21 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Backward-compatible import for the built-in compaction provider.""" + +from veadk.extensions.harness.modules.tool_result_compactor.builtin_provider import ( + BuiltinCompressionProvider, +) + +__all__ = ["BuiltinCompressionProvider"] diff --git a/veadk/extensions/harness/modules/context_engine.py b/veadk/extensions/harness/modules/context_engine.py new file mode 100644 index 00000000..0b26cc92 --- /dev/null +++ b/veadk/extensions/harness/modules/context_engine.py @@ -0,0 +1,22 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Backward-compatible imports for invocation context preparation.""" + +from veadk.extensions.harness.modules.invocation_context import ( + HarnessInvocationContextBuilder as ContextEngine, + HarnessInvocationContextConfig as ContextEngineConfig, +) + +__all__ = ["ContextEngine", "ContextEngineConfig"] diff --git a/veadk/extensions/harness/modules/final_response_verifier/__init__.py b/veadk/extensions/harness/modules/final_response_verifier/__init__.py new file mode 100644 index 00000000..4a566bcb --- /dev/null +++ b/veadk/extensions/harness/modules/final_response_verifier/__init__.py @@ -0,0 +1,29 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Final response verifier module exports.""" + +from veadk.extensions.harness.modules.final_response_verifier.verifier import ( + FinalResponseVerifier, + FinalResponseVerifierConfig, + ResultVerifier, + ResultVerifierConfig, +) + +__all__ = [ + "FinalResponseVerifier", + "FinalResponseVerifierConfig", + "ResultVerifier", + "ResultVerifierConfig", +] diff --git a/veadk/extensions/harness/modules/final_response_verifier/verifier.py b/veadk/extensions/harness/modules/final_response_verifier/verifier.py new file mode 100644 index 00000000..18b2cbd5 --- /dev/null +++ b/veadk/extensions/harness/modules/final_response_verifier/verifier.py @@ -0,0 +1,261 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Deterministic final-response verification.""" + +from __future__ import annotations + +import ast +import json +import re +from typing import Literal + +from pydantic import Field + +from veadk.extensions.harness.schemas import ( + ToolReceipt, + EvidenceRef, + HarnessBaseModel, + VerificationDecision, + VerificationReport, +) + + +class FinalResponseVerifierConfig(HarnessBaseModel): + """Settings for deterministic verification.""" + + mode: Literal["observe", "block"] = "observe" + require_receipt_for_completion_claims: bool = True + completion_markers: list[str] = Field( + default_factory=lambda: [ + "created", + "saved", + "uploaded", + "deployed", + "verified", + "completed", + "done", + "生成", + "保存", + "部署", + "完成", + ] + ) + + +class FinalResponseVerifier: + """Checks whether final answers are grounded in available receipts.""" + + def __init__(self, config: FinalResponseVerifierConfig | None = None) -> None: + self.config = config or FinalResponseVerifierConfig() + + def verify_text( + self, + text: str, + *, + receipts: list[ToolReceipt] | None = None, + evidence: list[EvidenceRef] | None = None, + ) -> VerificationReport: + """Verify answer text against tool receipts and evidence.""" + + receipts = receipts or [] + evidence = evidence or self._evidence_from_receipts(receipts) + reasons = [] + unsupported_claims = [] + supported_claims = [] + has_success_receipt = any(receipt.status == "success" for receipt in receipts) + if ( + self.config.require_receipt_for_completion_claims + and self._has_completion_claim(text) + and not has_success_receipt + ): + reasons.append("completion claim has no successful capability receipt") + unsupported_claims.append(self._completion_sentence(text)) + + if self._looks_like_truncated_html(text): + reasons.append("response looks like truncated html") + unsupported_claims.append("html artifact is incomplete") + + if evidence and not unsupported_claims: + supported_claims.append("answer has tool evidence") + + status: Literal["pass", "warn", "fail"] = "pass" + if unsupported_claims: + status = "fail" + elif reasons: + status = "warn" + return VerificationReport( + status=status, + reasons=reasons, + supported_claims=supported_claims, + unsupported_claims=[claim for claim in unsupported_claims if claim], + evidence=evidence, + ) + + def decide(self, report: VerificationReport) -> VerificationDecision: + """Map a verification report to a plugin intervention.""" + + if report.status == "pass": + return VerificationDecision(action="allow", report=report) + if self.config.mode == "block" and report.status == "fail": + return VerificationDecision( + action="block", + reason="; ".join(report.reasons), + instruction=( + "The answer was blocked because it made unsupported " + "tool-backed completion claims." + ), + report=report, + ) + return VerificationDecision( + action="observe", + reason="; ".join(report.reasons), + report=report, + ) + + def build_repair_instruction( + self, report: VerificationReport, *, goal: str = "" + ) -> str: + """Create a compact repair instruction for callers that support retry.""" + + reason_text = "; ".join(report.reasons or ["unsupported answer"]) + parts = [ + "[Harness Repair]", + "The previous answer failed verification.", + f"Problems: {reason_text}.", + "Retry the same task with evidence-backed claims only.", + "Do not claim that files, deployments, or artifacts exist unless a tool receipt proves it.", + ] + if goal: + parts.append(f"Original goal: {goal}") + parts.append("[/Harness Repair]") + return "\n".join(parts) + + def try_repair_json_text(self, value: str) -> str: + """Repair common JSON-like tool argument text.""" + + candidate = self._strip_fences(value.strip()) + if not candidate: + return value + for repaired in self._repair_candidates(candidate): + parsed = self._loads_json_or_python(repaired) + if parsed is not None: + return json.dumps(parsed, ensure_ascii=False, separators=(",", ":")) + return value + + def _evidence_from_receipts(self, receipts: list[ToolReceipt]) -> list[EvidenceRef]: + evidence: list[EvidenceRef] = [] + for receipt in receipts: + evidence.extend(receipt.evidence) + if receipt.summary: + evidence.append( + EvidenceRef( + source=receipt.name, + content=receipt.summary, + score=1.0 if receipt.status == "success" else 0.3, + ) + ) + return evidence + + def _has_completion_claim(self, text: str) -> bool: + lowered = text.lower() + return any(marker in lowered for marker in self.config.completion_markers) + + def _completion_sentence(self, text: str) -> str: + sentences = re.split(r"(?<=[.!?。！？])\s+", text.strip()) + for sentence in sentences: + if self._has_completion_claim(sentence): + return sentence + return sentences[0] if sentences else "" + + def _looks_like_truncated_html(self, text: str) -> bool: + lowered = text.lower() + if "" not in lowered: + return True + if "" not in lowered: + return True + return "" not in lowered and len(text) > 1000 + + def _strip_fences(self, value: str) -> str: + stripped = value.strip() + if stripped.startswith("```"): + stripped = re.sub(r"^```[a-zA-Z0-9_-]*\s*", "", stripped) + stripped = re.sub(r"\s*```$", "", stripped) + return stripped.strip() + + def _repair_candidates(self, candidate: str) -> list[str]: + values = [candidate] + extracted = self._extract_outer_json_text(candidate) + if extracted and extracted not in values: + values.append(extracted) + repaired = [] + for value in values: + balanced = self._balance_brackets(value) + repaired.append(balanced) + normalized = self._normalize_json_like_text(balanced) + if normalized != balanced: + repaired.append(normalized) + return list(dict.fromkeys(repaired)) + + def _loads_json_or_python( + self, value: str + ) -> dict[str, object] | list[object] | None: + try: + parsed = json.loads(value) + if isinstance(parsed, (dict, list)): + return parsed + except json.JSONDecodeError: + pass + try: + parsed = ast.literal_eval(value) + if isinstance(parsed, (dict, list)): + return parsed + except (SyntaxError, ValueError): + pass + return None + + def _extract_outer_json_text(self, value: str) -> str: + candidates = [] + for start_char, end_char in (("{", "}"), ("[", "]")): + start = value.find(start_char) + end = value.rfind(end_char) + if start >= 0 and end > start: + candidates.append(value[start : end + 1]) + return max(candidates, key=len) if candidates else "" + + def _balance_brackets(self, value: str) -> str: + repaired = value.strip() + if repaired.count("{") > repaired.count("}"): + repaired += "}" * (repaired.count("{") - repaired.count("}")) + if repaired.count("[") > repaired.count("]"): + repaired += "]" * (repaired.count("[") - repaired.count("]")) + return repaired + + def _normalize_json_like_text(self, value: str) -> str: + repaired = value.strip() + repaired = re.sub(r",\s*([}\]])", r"\1", repaired) + repaired = repaired.replace("\u201c", '"').replace("\u201d", '"') + repaired = repaired.replace("\u2018", "'").replace("\u2019", "'") + return repaired + + +ResultVerifierConfig = FinalResponseVerifierConfig +ResultVerifier = FinalResponseVerifier + +__all__ = [ + "FinalResponseVerifier", + "FinalResponseVerifierConfig", + "ResultVerifier", + "ResultVerifierConfig", +] diff --git a/veadk/extensions/harness/modules/headroom_provider.py b/veadk/extensions/harness/modules/headroom_provider.py new file mode 100644 index 00000000..1255620a --- /dev/null +++ b/veadk/extensions/harness/modules/headroom_provider.py @@ -0,0 +1,21 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Backward-compatible import for the Headroom compaction provider.""" + +from veadk.extensions.harness.modules.tool_result_compactor.headroom_provider import ( + HeadroomCompressionProvider, +) + +__all__ = ["HeadroomCompressionProvider"] diff --git a/veadk/extensions/harness/modules/invocation_context/__init__.py b/veadk/extensions/harness/modules/invocation_context/__init__.py new file mode 100644 index 00000000..d3b200f0 --- /dev/null +++ b/veadk/extensions/harness/modules/invocation_context/__init__.py @@ -0,0 +1,29 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Invocation context module exports.""" + +from veadk.extensions.harness.modules.invocation_context.builder import ( + ContextEngine, + ContextEngineConfig, + HarnessInvocationContextBuilder, + HarnessInvocationContextConfig, +) + +__all__ = [ + "ContextEngine", + "ContextEngineConfig", + "HarnessInvocationContextBuilder", + "HarnessInvocationContextConfig", +] diff --git a/veadk/extensions/harness/modules/invocation_context/builder.py b/veadk/extensions/harness/modules/invocation_context/builder.py new file mode 100644 index 00000000..8d1084cd --- /dev/null +++ b/veadk/extensions/harness/modules/invocation_context/builder.py @@ -0,0 +1,246 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Invocation context preparation module.""" + +from __future__ import annotations + +from pydantic import Field + +from veadk.extensions.harness.schemas import ( + ToolReceipt, + InvocationContextBlock, + ConversationMessage, + HarnessBaseModel, + HarnessInvocationRef, +) +from veadk.extensions.harness.utils import summarize_text + + +class HarnessInvocationContextConfig(HarnessBaseModel): + """Settings for invocation context generation.""" + + max_history_messages: int = 12 + max_context_chars: int = 24000 + max_receipts: int = 8 + history_message_chars: int = 500 + receipt_summary_chars: int = 500 + include_history: bool = True + include_receipts: bool = True + precision_markers: list[str] = Field( + default_factory=lambda: [ + ".csv", + "ranking", + "top ", + "filter", + "outlier", + "percentage", + "selector", + "sort", + "threshold", + "日志", + "排序", + "筛选", + ] + ) + artifact_markers: list[str] = Field( + default_factory=lambda: [ + "file", + "artifact", + "chart", + "plot", + "graph", + "dashboard", + "report", + "图表", + "文件", + "报告", + ] + ) + + +class HarnessInvocationContextBuilder: + """Build compact invocation context for an Agent turn.""" + + def __init__(self, config: HarnessInvocationContextConfig | None = None) -> None: + self.config = config or HarnessInvocationContextConfig() + + def prepare_context( + self, + context: HarnessInvocationRef, + *, + user_input: str = "", + history: list[ConversationMessage] | None = None, + receipts: list[ToolReceipt] | None = None, + has_tools: bool = False, + ) -> InvocationContextBlock: + """Create an invocation context block for a model call.""" + + history = history or [] + receipts = receipts or [] + header = self.build_context_header( + context=context, + user_input=user_input, + history=history, + receipts=receipts, + has_tools=has_tools, + ) + original_chars = sum(len(message.content) for message in history) + if len(header) > self.config.max_context_chars: + header = summarize_text(header, max_chars=self.config.max_context_chars) + return InvocationContextBlock( + header=header, + messages=history[-self.config.max_history_messages :], + original_chars=original_chars, + context_chars=len(header), + injected=bool(header.strip()), + ) + + def build_context_header( + self, + *, + context: HarnessInvocationRef, + user_input: str = "", + history: list[ConversationMessage] | None = None, + receipts: list[ToolReceipt] | None = None, + has_tools: bool = False, + ) -> str: + """Build the plain-text Harness Context block.""" + + task_goal = context.task.goal if context.task else user_input + acceptance = context.task.acceptance_criteria if context.task else [] + lines = [ + "[Harness Context]", + f"profile: {context.profile}", + f"session_id: {context.session_id}", + f"task_goal: {task_goal or user_input}", + "acceptance:", + ] + if acceptance: + lines.extend(f"- {item}" for item in acceptance) + else: + lines.extend( + [ + "- Stay anchored to the current user goal.", + "- Preserve exact filenames, schemas, numbers, and dates.", + "- Verify tool-backed claims before presenting completion.", + ] + ) + + recent = (history or [])[-self.config.max_history_messages :] + if self.config.include_history and recent: + lines.append("recent_history:") + for message in recent: + text = summarize_text( + message.content, max_chars=self.config.history_message_chars + ) + lines.append(f"- {message.role}: {text}") + + if self.config.include_receipts and receipts: + lines.append("capability_receipts:") + for receipt in receipts[-self.config.max_receipts :]: + summary = summarize_text( + receipt.summary, max_chars=self.config.receipt_summary_chars + ) + lines.append(f"- {receipt.name} [{receipt.status}]: {summary}") + + mode_header = self._build_mode_header( + user_input=user_input, has_tools=has_tools + ) + if mode_header: + lines.extend(["", mode_header]) + lines.append("[/Harness Context]") + return "\n".join(lines) + + def enhance_messages( + self, + messages: list[ConversationMessage], + context: HarnessInvocationRef, + *, + user_input: str = "", + receipts: list[ToolReceipt] | None = None, + has_tools: bool = False, + ) -> tuple[list[ConversationMessage], InvocationContextBlock]: + """Return messages with a Harness context message inserted.""" + + bundle = self.prepare_context( + context, + user_input=user_input, + history=messages, + receipts=receipts, + has_tools=has_tools, + ) + if not bundle.header: + return messages, bundle + + insert_at = 0 + for index, message in enumerate(messages): + if message.role in {"system", "developer"}: + insert_at = index + 1 + injected = ConversationMessage( + role="system", + name="veadk_harness_context", + content=bundle.header, + ) + return messages[:insert_at] + [injected] + messages[insert_at:], bundle + + def _build_mode_header(self, *, user_input: str, has_tools: bool) -> str: + lowered = user_input.lower() + blocks = [] + if any(marker in lowered for marker in self.config.precision_markers): + blocks.append( + "\n".join( + [ + "[Harness Precision Mode]", + "- Treat selectors, schemas, dates, counts, and numeric thresholds as exact requirements.", + "- Prefer deterministic parsing or tool checks over unsupported estimates.", + "[/Harness Precision Mode]", + ] + ) + ) + if any(marker in lowered for marker in self.config.artifact_markers): + blocks.append( + "\n".join( + [ + "[Harness Artifact Mode]", + "- Create the requested artifact before claiming completion.", + "- Verify the artifact exists and is non-empty when tools are available.", + "[/Harness Artifact Mode]", + ] + ) + ) + if has_tools: + blocks.append( + "\n".join( + [ + "[Harness Tool Protocol]", + "- Emit tool calls with complete, valid JSON object arguments as required by the runtime.", + "- Do not truncate code, paths, strings, or JSON values inside tool arguments.", + "- If a tool fails, use the failure as evidence and choose a different path when possible.", + "[/Harness Tool Protocol]", + ] + ) + ) + return "\n\n".join(blocks) + + +ContextEngineConfig = HarnessInvocationContextConfig +ContextEngine = HarnessInvocationContextBuilder + +__all__ = [ + "ContextEngine", + "ContextEngineConfig", + "HarnessInvocationContextBuilder", + "HarnessInvocationContextConfig", +] diff --git a/veadk/extensions/harness/modules/result_verifier.py b/veadk/extensions/harness/modules/result_verifier.py new file mode 100644 index 00000000..3a46be3e --- /dev/null +++ b/veadk/extensions/harness/modules/result_verifier.py @@ -0,0 +1,22 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Backward-compatible imports for final-response verification.""" + +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifier as ResultVerifier, + FinalResponseVerifierConfig as ResultVerifierConfig, +) + +__all__ = ["ResultVerifier", "ResultVerifierConfig"] diff --git a/veadk/extensions/harness/modules/tool_compressor.py b/veadk/extensions/harness/modules/tool_compressor.py new file mode 100644 index 00000000..330c77cd --- /dev/null +++ b/veadk/extensions/harness/modules/tool_compressor.py @@ -0,0 +1,27 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Backward-compatible imports for tool-result compaction.""" + +from veadk.extensions.harness.modules.tool_result_compactor import ( + ContextCompactionPolicy as ContextCompressionPolicy, + ToolResultCompactor as ToolResultCompressor, + ToolResultCompactorConfig as ToolResultCompressorConfig, +) + +__all__ = [ + "ContextCompressionPolicy", + "ToolResultCompressor", + "ToolResultCompressorConfig", +] diff --git a/veadk/extensions/harness/modules/tool_result_compactor/__init__.py b/veadk/extensions/harness/modules/tool_result_compactor/__init__.py new file mode 100644 index 00000000..d3cbfa8c --- /dev/null +++ b/veadk/extensions/harness/modules/tool_result_compactor/__init__.py @@ -0,0 +1,41 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tool result compactor module exports.""" + +from veadk.extensions.harness.modules.tool_result_compactor.builtin_provider import ( + BuiltinCompressionProvider, +) +from veadk.extensions.harness.modules.tool_result_compactor.compactor import ( + ContextCompactionPolicy, + ContextCompressionPolicy, + ToolResultCompactor, + ToolResultCompactorConfig, + ToolResultCompressor, + ToolResultCompressorConfig, +) +from veadk.extensions.harness.modules.tool_result_compactor.headroom_provider import ( + HeadroomCompressionProvider, +) + +__all__ = [ + "BuiltinCompressionProvider", + "ContextCompactionPolicy", + "ContextCompressionPolicy", + "HeadroomCompressionProvider", + "ToolResultCompactor", + "ToolResultCompactorConfig", + "ToolResultCompressor", + "ToolResultCompressorConfig", +] diff --git a/veadk/extensions/harness/modules/tool_result_compactor/builtin_provider.py b/veadk/extensions/harness/modules/tool_result_compactor/builtin_provider.py new file mode 100644 index 00000000..92b324b9 --- /dev/null +++ b/veadk/extensions/harness/modules/tool_result_compactor/builtin_provider.py @@ -0,0 +1,258 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Built-in loss-aware compression provider.""" + +from __future__ import annotations + +import json +from collections.abc import Mapping, Sequence + +from veadk.extensions.harness.schemas import ( + CompactionReport, + CompressionRequest, + CompactionResult, + ConversationMessage, +) +from veadk.extensions.harness.utils import redact_text, summarize_text + +_UNPARSED = object() +_MAX_FACTS = 80 +_MAX_SEQUENCE_ITEMS = 12 +_MAX_DEPTH = 8 +_MAX_SCALAR_CHARS = 64 +_MAX_FACT_CHARS = 600 + + +class BuiltinCompressionProvider: + """Compress tool outputs by preserving structured facts.""" + + name = "builtin" + + def compress(self, request: CompressionRequest) -> CompactionResult | None: + """Compress eligible tool messages with bounded fact extraction.""" + + compressed: list[ConversationMessage] = [] + changed = False + for message in request.messages: + if message.role not in {"tool", "tool_result", "function"}: + compressed.append(message) + continue + facts = _extract_tool_facts( + message.content, + max_chars=max(512, request.max_context_chars - 160), + ) + if facts and len(facts) < len(message.content): + compressed.append( + message.model_copy( + update={ + "content": ( + "COMPRESSED_TOOL_OUTPUT\n" + "structured_facts:\n" + f"{facts}\n" + "compression_policy=preserve_bounded_structured_facts" + ) + } + ) + ) + changed = True + else: + compressed.append(message) + if not changed: + return None + original_chars = _messages_char_count(request.messages) + compressed_chars = _messages_char_count(compressed) + return CompactionResult( + messages=compressed, + report=CompactionReport( + provider=self.name, + original_chars=original_chars, + compressed_chars=compressed_chars, + changed=True, + tokens_before=max(1, original_chars // 4), + tokens_after=max(1, compressed_chars // 4), + tokens_saved=max(0, original_chars // 4 - compressed_chars // 4), + compression_ratio=compressed_chars / max(1, original_chars), + transforms_applied=["builtin_tool_fact_compaction"], + ), + ) + + +def _messages_char_count(messages: list[ConversationMessage]) -> int: + return sum(len(message.content) for message in messages) + + +def _extract_tool_facts(text: str, *, max_chars: int) -> str: + parsed = _parse_json_text(text) + if parsed is _UNPARSED: + return "" + facts: list[str] = [] + _collect_structured_facts( + value=parsed, + path="$", + facts=facts, + depth=0, + ) + return _join_limited_facts(facts, max_chars=max_chars) + + +def _collect_structured_facts( + *, + value: object, + path: str, + facts: list[str], + depth: int, +) -> None: + if len(facts) >= _MAX_FACTS or depth > _MAX_DEPTH: + return + if isinstance(value, Mapping): + _collect_mapping_facts(value=value, path=path, facts=facts, depth=depth) + return + if isinstance(value, Sequence) and not isinstance(value, (str, bytes, bytearray)): + _collect_sequence_facts(value=value, path=path, facts=facts, depth=depth) + return + if _is_scalar(value): + _append_fact(facts, f"{path}={_format_scalar(value)}") + + +def _collect_mapping_facts( + *, + value: Mapping[object, object], + path: str, + facts: list[str], + depth: int, +) -> None: + scalar_parts: list[str] = [] + nested_items: list[tuple[str, object]] = [] + for key, item in value.items(): + child_path = _child_path(path, key) + parsed = _parse_json_text(item) if isinstance(item, str) else _UNPARSED + if parsed is not _UNPARSED: + nested_items.append((child_path, parsed)) + elif _is_scalar(item): + scalar_parts.append(f"{_key_text(key)}={_format_scalar(item)}") + else: + nested_items.append((child_path, item)) + if scalar_parts: + _append_fact(facts, f"{path}: {' '.join(scalar_parts)}") + for child_path, item in nested_items: + _collect_structured_facts( + value=item, + path=child_path, + facts=facts, + depth=depth + 1, + ) + if len(facts) >= _MAX_FACTS: + return + + +def _collect_sequence_facts( + *, + value: Sequence[object], + path: str, + facts: list[str], + depth: int, +) -> None: + indexes = _representative_indexes(len(value)) + for index in indexes: + _collect_structured_facts( + value=value[index], + path=f"{path}[{index}]", + facts=facts, + depth=depth + 1, + ) + if len(facts) >= _MAX_FACTS: + return + omitted = len(value) - len(indexes) + if omitted > 0: + _append_fact(facts, f"{path}: omitted_items={omitted}") + + +def _representative_indexes(size: int) -> list[int]: + if size <= _MAX_SEQUENCE_ITEMS: + return list(range(size)) + head_count = max(1, _MAX_SEQUENCE_ITEMS - 2) + tail_indexes = range(max(head_count, size - 2), size) + return sorted({*range(head_count), *tail_indexes}) + + +def _parse_json_text(value: str) -> object: + stripped = value.strip() + if not stripped or stripped[0] not in "[{": + return _UNPARSED + try: + return json.loads(stripped) + except json.JSONDecodeError: + return _UNPARSED + + +def _is_scalar(value: object) -> bool: + return value is None or isinstance(value, (str, int, float, bool)) + + +def _child_path(parent: str, key: object) -> str: + key_text = _key_text(key) + if parent == "$": + return f"$.{key_text}" + return f"{parent}.{key_text}" + + +def _key_text(key: object) -> str: + text = summarize_text(str(key), max_chars=80).replace("\n", " ").strip() + return text or "" + + +def _format_scalar(value: object) -> str: + if value is None or isinstance(value, (int, float, bool)): + return json.dumps(value, ensure_ascii=False) + text = redact_text(str(value)) + normalized = " ".join(text.split()) + if len(normalized) > _MAX_SCALAR_CHARS: + return f"" + if not normalized: + return '""' + if any(char.isspace() for char in normalized) or "=" in normalized: + return json.dumps(normalized, ensure_ascii=False) + return normalized + + +def _append_fact(facts: list[str], fact: str) -> None: + if len(facts) >= _MAX_FACTS: + return + normalized = summarize_text(redact_text(fact), max_chars=_MAX_FACT_CHARS) + if normalized and normalized not in facts: + facts.append(normalized) + + +def _join_limited_facts(facts: list[str], *, max_chars: int) -> str: + lines: list[str] = [] + current_chars = 0 + for fact in facts: + next_chars = len(fact) + (1 if lines else 0) + if current_chars + next_chars > max_chars: + omitted = len(facts) - len(lines) + if omitted > 0: + lines.append(f"... omitted_facts={omitted}") + break + lines.append(fact) + current_chars += next_chars + return "\n".join(lines) + + +def _extract_facts_from_value(value: object) -> str: + """Backward-compatible private helper for older tests and callers.""" + + facts: list[str] = [] + _collect_structured_facts(value=value, path="$", facts=facts, depth=0) + return _join_limited_facts(facts, max_chars=24000) diff --git a/veadk/extensions/harness/modules/tool_result_compactor/compactor.py b/veadk/extensions/harness/modules/tool_result_compactor/compactor.py new file mode 100644 index 00000000..a5e26d50 --- /dev/null +++ b/veadk/extensions/harness/modules/tool_result_compactor/compactor.py @@ -0,0 +1,600 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tool-result compaction module.""" + +from __future__ import annotations + +import json +from typing import Literal + +from veadk.extensions.harness.modules.tool_result_compactor.builtin_provider import ( + BuiltinCompressionProvider, +) +from veadk.extensions.harness.modules.tool_result_compactor.headroom_provider import ( + HeadroomCompressionProvider, +) +from veadk.extensions.harness.schemas import ( + CompressionDecision, + CompressionPlan, + CompactionReport, + CompressionRequest, + CompactionResult, + ConversationMessage, + HarnessBaseModel, + JsonObject, +) +from veadk.extensions.harness.utils import ( + coerce_json_object, + redact_text, + stringify_json_value, + summarize_text, +) + + +class ToolResultCompactorConfig(HarnessBaseModel): + """Settings for tool-result compaction.""" + + provider: str = "builtin" + max_context_chars: int = 24000 + max_tool_result_chars: int = 4000 + min_candidate_chars: int = 4000 + protect_recent_messages: int = 2 + summary_chars: int = 900 + + +class ContextCompactionPolicy: + """Select safe historical context for compaction.""" + + def __init__(self, config: ToolResultCompactorConfig | None = None) -> None: + self.config = config or ToolResultCompactorConfig() + + def plan(self, messages: list[ConversationMessage]) -> CompressionPlan: + decisions = [ + self._classify(index=index, total=len(messages), message=message) + for index, message in enumerate(messages) + ] + candidate_indexes = [ + decision.index for decision in decisions if decision.action == "compress" + ] + by_action: dict[str, int] = {} + by_reason: dict[str, int] = {} + for decision in decisions: + by_action[decision.action] = by_action.get(decision.action, 0) + 1 + key = f"{decision.action}:{decision.reason}" + by_reason[key] = by_reason.get(key, 0) + 1 + summary: JsonObject = { + "mode": "role_and_recency_aware", + "message_count": len(decisions), + "candidate_count": len(candidate_indexes), + "candidate_indexes": list(candidate_indexes), + "by_action": by_action, + "by_reason": by_reason, + } + return CompressionPlan( + decisions=decisions, + candidate_indexes=candidate_indexes, + summary=summary, + ) + + def _classify( + self, + *, + index: int, + total: int, + message: ConversationMessage, + ) -> CompressionDecision: + role = message.role + chars = len(message.content) + messages_from_end = total - index + if role in {"system", "developer"}: + return self._decision(index, "protect", "instructions", role, chars) + if role == "user": + return self._decision(index, "protect", "user_intent", role, chars) + if role == "assistant": + return self._decision(index, "protect", "assistant_state", role, chars) + if messages_from_end <= self.config.protect_recent_messages: + return self._decision(index, "protect", "recent_feedback", role, chars) + if chars < self.config.min_candidate_chars: + return self._decision(index, "skip", "small_output", role, chars) + if role in {"tool", "tool_result", "function"}: + reason = ( + "old_large_error_or_recovery_evidence" + if self._looks_like_recovery_evidence(message.content) + else "old_large_tool_output" + ) + return self._decision(index, "compress", reason, role, chars) + return self._decision(index, "compress", "old_large_unknown_role", role, chars) + + def _decision( + self, + index: int, + action: Literal["protect", "skip", "compress"], + reason: str, + role: str, + chars: int, + ) -> CompressionDecision: + return CompressionDecision( + index=index, + action=action, + reason=reason, + role=role, + chars=chars, + ) + + def _looks_like_recovery_evidence(self, text: str) -> bool: + lowered = text.lower() + signals = ( + "traceback", + "exception", + "error:", + "failed", + "permission denied", + "no such file", + "syntaxerror", + "typeerror", + "diff --git", + ) + return any(signal in lowered for signal in signals) + + +class ToolResultCompactor: + """Dependency-free compactor for large historical tool results.""" + + def __init__(self, config: ToolResultCompactorConfig | None = None) -> None: + self.config = config or ToolResultCompactorConfig() + self.policy = ContextCompactionPolicy(self.config) + self.builtin = BuiltinCompressionProvider() + self._headroom: HeadroomCompressionProvider | None = None + + def compress_messages(self, request: CompressionRequest) -> CompactionResult: + """Compact candidate messages while preserving control-plane messages.""" + + original_chars = self._messages_char_count(request.messages) + if original_chars <= request.max_context_chars: + return CompactionResult( + messages=list(request.messages), + report=CompactionReport( + provider=self.config.provider, + original_chars=original_chars, + compressed_chars=original_chars, + changed=False, + ), + ) + + plan = self.policy.plan(request.messages) + warnings: list[str] = [] + if self._uses_headroom(): + result = self._compress_messages_with_headroom(request, plan) + if result is not None: + return result + warnings.append("headroom provider unavailable; used builtin fallback") + + if self._uses_headroom() or self._uses_builtin_or_default(): + result = self._compress_messages_with_builtin(request, plan, warnings) + if result is not None: + return result + + compressed = list(request.messages) + for index in plan.candidate_indexes: + message = compressed[index] + compressed[index] = message.model_copy( + update={"content": self._summary(message.content, index=index)} + ) + if self._messages_char_count(compressed) <= request.max_context_chars: + break + + omitted = 0 + while ( + self._messages_char_count(compressed) > request.max_context_chars + and len(compressed) > request.protected_message_count + ): + removable = self._oldest_removable_index(compressed) + if removable is None: + break + compressed.pop(removable) + omitted += 1 + + compressed_chars = self._messages_char_count(compressed) + if self._uses_headroom() and plan.candidate_indexes: + warnings[-1:] = [ + "headroom and builtin providers unavailable; used heuristic fallback" + ] + if compressed_chars > request.max_context_chars: + warnings.append("context still exceeds max_context_chars") + return CompactionResult( + messages=compressed, + report=CompactionReport( + provider=self._fallback_provider(), + original_chars=original_chars, + compressed_chars=compressed_chars, + changed=compressed != request.messages, + omitted_messages=omitted, + protected_messages=len( + [item for item in plan.decisions if item.action == "protect"] + ), + compression_ratio=( + compressed_chars / original_chars if original_chars else 1.0 + ), + transforms_applied=["heuristic_summary"] + if compressed != request.messages + else [], + policy=plan.summary, + warnings=warnings, + ), + ) + + def compress_tool_result(self, result: object) -> tuple[object, CompactionReport]: + """Compact a single tool result mapping if it exceeds the configured size.""" + + original_text = self._raw_payload_text(result) + original_chars = len(original_text) + if original_chars <= self.config.max_tool_result_chars: + return result, CompactionReport( + provider=self.config.provider, + original_chars=original_chars, + compressed_chars=original_chars, + changed=False, + ) + + warnings: list[str] = [] + if self._uses_headroom(): + headroom = self._compress_tool_result_with_headroom( + result=result, + original_text=original_text, + original_chars=original_chars, + ) + if headroom is not None: + return headroom + warnings.append("headroom provider unavailable; used builtin fallback") + + builtin = self._compress_tool_result_with_builtin( + result=result, + original_text=original_text, + original_chars=original_chars, + warnings=warnings, + ) + if builtin is not None: + return builtin + + summary = self._summary(original_text, index=0) + compressed: dict[str, object] = { + "harness_compressed": True, + "summary": summary, + "original_chars": original_chars, + } + if isinstance(result, dict) and "error" in result: + compressed["error"] = result["error"] + if isinstance(result, dict) and "status" in result: + compressed["status"] = result["status"] + report = CompactionReport( + provider=self._fallback_provider(), + original_chars=original_chars, + compressed_chars=len(self._raw_payload_text(compressed)), + changed=True, + transforms_applied=["tool_result_summary"], + policy={"mode": "single_tool_result"}, + warnings=warnings + + ( + ["builtin provider unavailable; used heuristic fallback"] + if self._uses_headroom() + else [] + ), + ) + return compressed, report + + def receipt_summary(self, tool_name: str, result: dict[str, object]) -> str: + """Build a concise receipt summary for a tool result.""" + + status = result.get("status") or ("error" if "error" in result else "success") + body = stringify_json_value(result, max_chars=1200) + return f"{tool_name} status={status}; {body}" + + def _summary(self, text: str, *, index: int) -> str: + return "\n".join( + [ + "[Compressed tool context]", + f"message_index: {index}", + f"summary: {summarize_text(text, max_chars=self.config.summary_chars)}", + ] + ) + + def _oldest_removable_index( + self, messages: list[ConversationMessage] + ) -> int | None: + for index, message in enumerate(messages): + if message.role not in {"system", "developer", "user", "assistant"}: + return index + return None + + def _compress_messages_with_headroom( + self, request: CompressionRequest, plan: CompressionPlan + ) -> CompactionResult | None: + if not plan.candidate_indexes: + return None + candidates = [request.messages[index] for index in plan.candidate_indexes] + metadata = self._headroom_metadata( + request.metadata, + mode="model_context", + token_budget=max(1, request.max_context_chars // 4), + ) + result = self._headroom_provider().compress( + CompressionRequest( + messages=candidates, + max_context_chars=request.max_context_chars, + protected_message_count=0, + metadata=metadata, + ) + ) + if result is None or len(result.messages) != len(candidates): + return None + + compressed = list(request.messages) + for index, message in zip(plan.candidate_indexes, result.messages): + compressed[index] = message + + omitted = 0 + while ( + self._messages_char_count(compressed) > request.max_context_chars + and len(compressed) > request.protected_message_count + ): + removable = self._oldest_removable_index(compressed) + if removable is None: + break + compressed.pop(removable) + omitted += 1 + + original_chars = self._messages_char_count(request.messages) + compressed_chars = self._messages_char_count(compressed) + warnings = list(result.report.warnings) + if compressed_chars > request.max_context_chars: + warnings.append("context still exceeds max_context_chars") + transforms = list(result.report.transforms_applied) + if "headroom_candidate_compression" not in transforms: + transforms.append("headroom_candidate_compression") + return CompactionResult( + messages=compressed, + report=result.report.model_copy( + update={ + "original_chars": original_chars, + "compressed_chars": compressed_chars, + "changed": compressed != request.messages, + "omitted_messages": omitted, + "protected_messages": len( + [item for item in plan.decisions if item.action == "protect"] + ), + "compression_ratio": ( + compressed_chars / original_chars if original_chars else 1.0 + ), + "transforms_applied": transforms, + "policy": plan.summary, + "warnings": warnings, + } + ), + ) + + def _compress_messages_with_builtin( + self, + request: CompressionRequest, + plan: CompressionPlan, + warnings: list[str], + ) -> CompactionResult | None: + if not plan.candidate_indexes: + return None + candidates = [request.messages[index] for index in plan.candidate_indexes] + result = self.builtin.compress( + CompressionRequest( + messages=candidates, + max_context_chars=request.max_context_chars, + protected_message_count=0, + metadata=request.metadata, + ) + ) + if result is None or len(result.messages) != len(candidates): + return None + compressed = list(request.messages) + for index, message in zip(plan.candidate_indexes, result.messages): + compressed[index] = message + original_chars = self._messages_char_count(request.messages) + compressed_chars = self._messages_char_count(compressed) + return CompactionResult( + messages=compressed, + report=result.report.model_copy( + update={ + "original_chars": original_chars, + "compressed_chars": compressed_chars, + "changed": compressed != request.messages, + "protected_messages": len( + [item for item in plan.decisions if item.action == "protect"] + ), + "compression_ratio": ( + compressed_chars / original_chars if original_chars else 1.0 + ), + "policy": plan.summary, + "warnings": list(warnings), + } + ), + ) + + def _compress_tool_result_with_headroom( + self, + *, + result: object, + original_text: str, + original_chars: int, + ) -> tuple[dict[str, object], CompactionReport] | None: + metadata = self._headroom_metadata( + {}, + mode="single_tool_result", + token_budget=max(1, self.config.max_tool_result_chars // 4), + ) + headroom_result = self._headroom_provider().compress( + CompressionRequest( + messages=[ + ConversationMessage( + role="tool", + content=original_text, + metadata=coerce_json_object(result), + ) + ], + max_context_chars=self.config.max_tool_result_chars, + protected_message_count=0, + metadata=metadata, + ) + ) + if headroom_result is None or not headroom_result.messages: + return None + summary = headroom_result.messages[0].content + if not summary or len(summary) >= original_chars: + return None + compressed: dict[str, object] = { + "harness_compressed": True, + "provider": "headroom", + "summary": summary, + "original_chars": original_chars, + } + if isinstance(result, dict) and "error" in result: + compressed["error"] = result["error"] + if isinstance(result, dict) and "status" in result: + compressed["status"] = result["status"] + compressed_chars = len(self._raw_payload_text(compressed)) + transforms = list(headroom_result.report.transforms_applied) + if "headroom_tool_result_compression" not in transforms: + transforms.append("headroom_tool_result_compression") + report = headroom_result.report.model_copy( + update={ + "original_chars": original_chars, + "compressed_chars": compressed_chars, + "changed": True, + "compression_ratio": compressed_chars / original_chars, + "transforms_applied": transforms, + "policy": {"mode": "single_tool_result", "provider": "headroom"}, + } + ) + return compressed, report + + def _compress_tool_result_with_builtin( + self, + *, + result: object, + original_text: str, + original_chars: int, + warnings: list[str], + ) -> tuple[dict[str, object], CompactionReport] | None: + builtin_result = self.builtin.compress( + CompressionRequest( + messages=[ + ConversationMessage( + role="tool", + content=original_text, + metadata=coerce_json_object(result), + ) + ], + max_context_chars=self.config.max_tool_result_chars, + protected_message_count=0, + metadata={}, + ) + ) + if builtin_result is None or not builtin_result.messages: + return None + summary = builtin_result.messages[0].content + compressed: dict[str, object] = { + "harness_compressed": True, + "provider": "builtin", + "summary": summary, + "original_chars": original_chars, + } + if isinstance(result, dict) and "error" in result: + compressed["error"] = result["error"] + if isinstance(result, dict) and "status" in result: + compressed["status"] = result["status"] + compressed_chars = len(self._raw_payload_text(compressed)) + report = builtin_result.report.model_copy( + update={ + "original_chars": original_chars, + "compressed_chars": compressed_chars, + "changed": True, + "compression_ratio": compressed_chars / original_chars, + "policy": {"mode": "single_tool_result", "provider": "builtin"}, + "warnings": list(warnings), + } + ) + return compressed, report + + def _messages_char_count(self, messages: list[ConversationMessage]) -> int: + return sum(len(message.content) for message in messages) + + def _uses_headroom(self) -> bool: + return self._provider_name() in { + "headroom", + "managed_compressor", + "context_compressor", + } + + def _uses_builtin_or_default(self) -> bool: + return self._provider_name() in {"", "auto", "builtin", "built_in", "managed"} + + def _fallback_provider(self) -> str: + return "heuristic" + + def _provider_name(self) -> str: + return self.config.provider.strip().lower() + + def _headroom_provider(self) -> HeadroomCompressionProvider: + if self._headroom is None: + self._headroom = HeadroomCompressionProvider() + return self._headroom + + def _headroom_metadata( + self, + metadata: JsonObject, + *, + mode: str, + token_budget: int, + ) -> JsonObject: + merged = dict(metadata) + merged.setdefault("compression_config", {"mode": mode}) + merged.setdefault("token_budget", token_budget) + return merged + + def dict_to_message( + self, role: str, payload: dict[str, object] + ) -> ConversationMessage: + """Convert a mapping into a message projection.""" + + return ConversationMessage( + role=role, + content=stringify_json_value(payload), + metadata=coerce_json_object(payload), + ) + + def _raw_payload_text(self, payload: object) -> str: + try: + return redact_text(json.dumps(payload, ensure_ascii=False, default=str)) + except TypeError: + return redact_text(str(payload)) + + +ToolResultCompressorConfig = ToolResultCompactorConfig +ContextCompressionPolicy = ContextCompactionPolicy +ToolResultCompressor = ToolResultCompactor + +__all__ = [ + "ContextCompactionPolicy", + "ContextCompressionPolicy", + "ToolResultCompactor", + "ToolResultCompactorConfig", + "ToolResultCompressor", + "ToolResultCompressorConfig", +] diff --git a/veadk/extensions/harness/modules/tool_result_compactor/headroom_provider.py b/veadk/extensions/harness/modules/tool_result_compactor/headroom_provider.py new file mode 100644 index 00000000..eded7eec --- /dev/null +++ b/veadk/extensions/harness/modules/tool_result_compactor/headroom_provider.py @@ -0,0 +1,194 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Headroom compression adapter for the atomic Harness SDK.""" + +from __future__ import annotations + +from collections.abc import Mapping, Sequence +import importlib +from typing import Protocol, cast + +from veadk.extensions.harness.schemas import ( + CompactionReport, + CompressionRequest, + CompactionResult, + ConversationMessage, + JsonObject, +) +from veadk.extensions.harness.utils import coerce_json_object, stringify_json_value + + +class _HeadroomCompressFn(Protocol): + def __call__( + self, + messages: list[JsonObject], + *, + model: str, + optimize: bool, + ) -> object: ... + + +class HeadroomCompressionProvider: + """Adapter for local in-process Headroom compression.""" + + name = "headroom" + + def __init__( + self, + *, + auto_install: bool | None = None, + ) -> None: + # Kept for source compatibility; Headroom is now code-only and never + # installed or started by the provider at runtime. + self.auto_install = auto_install + + def compress(self, request: CompressionRequest) -> CompactionResult | None: + """Compress messages with Headroom when a provider is available.""" + + return self._compress_via_sdk(request) + + def _compress_via_sdk(self, request: CompressionRequest) -> CompactionResult | None: + compress = self._load_compress() + if compress is None: + return None + try: + result = compress( + self._message_payloads(request.messages), + model=str(request.metadata.get("model") or "gpt-4o"), + optimize=True, + ) + except Exception: + return None + compressed = self._messages_from_value(getattr(result, "messages", None)) + if compressed is None: + return None + return self._build_result( + original_messages=request.messages, + compressed_messages=compressed, + metrics={ + "tokens_before": getattr(result, "tokens_before", 0), + "tokens_after": getattr(result, "tokens_after", 0), + "tokens_saved": getattr(result, "tokens_saved", 0), + "compression_ratio": getattr(result, "compression_ratio", 0.0), + "transforms_applied": getattr(result, "transforms_applied", []), + }, + ) + + def _load_compress(self) -> _HeadroomCompressFn | None: + return self._import_compress() + + def _import_compress(self) -> _HeadroomCompressFn | None: + try: + module = importlib.import_module("headroom.compress") + except Exception: + return None + compress = getattr(module, "compress", None) + if not callable(compress): + return None + return cast(_HeadroomCompressFn, compress) + + def _message_payloads( + self, messages: list[ConversationMessage] + ) -> list[JsonObject]: + return [message.model_dump(mode="json") for message in messages] + + def _messages_from_value(self, value: object) -> list[ConversationMessage] | None: + if not isinstance(value, Sequence) or isinstance( + value, (str, bytes, bytearray) + ): + return None + messages: list[ConversationMessage] = [] + for item in value: + message = self._message_from_value(item) + if message is None: + return None + messages.append(message) + return messages + + def _message_from_value(self, value: object) -> ConversationMessage | None: + if isinstance(value, ConversationMessage): + return value + if not isinstance(value, Mapping): + return None + role = value.get("role") + content = value.get("content") + if not isinstance(role, str): + return None + metadata = value.get("metadata") + name = value.get("name") + return ConversationMessage( + role=role, + content=content + if isinstance(content, str) + else stringify_json_value(content), + name=name if isinstance(name, str) else "", + metadata=coerce_json_object(metadata), + ) + + def _build_result( + self, + *, + original_messages: list[ConversationMessage], + compressed_messages: list[ConversationMessage], + metrics: Mapping[str, object], + ) -> CompactionResult: + original_chars = self._messages_char_count(original_messages) + compressed_chars = self._messages_char_count(compressed_messages) + tokens_before = self._int_metric(metrics.get("tokens_before")) + tokens_after = self._int_metric(metrics.get("tokens_after")) + tokens_saved = self._int_metric(metrics.get("tokens_saved")) + if tokens_saved == 0 and tokens_before > tokens_after: + tokens_saved = tokens_before - tokens_after + ratio = self._float_metric(metrics.get("compression_ratio")) + if ratio == 0.0 and original_chars: + ratio = compressed_chars / original_chars + return CompactionResult( + messages=compressed_messages, + report=CompactionReport( + provider=self.name, + original_chars=original_chars, + compressed_chars=compressed_chars, + changed=compressed_messages != original_messages, + tokens_before=tokens_before, + tokens_after=tokens_after, + tokens_saved=tokens_saved, + compression_ratio=ratio, + transforms_applied=self._string_list_metric( + metrics.get("transforms_applied") + ), + ), + ) + + def _messages_char_count(self, messages: list[ConversationMessage]) -> int: + return sum(len(message.content) for message in messages) + + def _int_metric(self, value: object) -> int: + try: + return int(value or 0) + except (TypeError, ValueError): + return 0 + + def _float_metric(self, value: object) -> float: + try: + return float(value or 0.0) + except (TypeError, ValueError): + return 0.0 + + def _string_list_metric(self, value: object) -> list[str]: + if not isinstance(value, Sequence) or isinstance( + value, (str, bytes, bytearray) + ): + return [] + return [str(item) for item in value] diff --git a/veadk/extensions/harness/plugins/__init__.py b/veadk/extensions/harness/plugins/__init__.py new file mode 100644 index 00000000..f274795b --- /dev/null +++ b/veadk/extensions/harness/plugins/__init__.py @@ -0,0 +1,35 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Harness plugin entry points for VeADK.""" + +from veadk.extensions.harness.plugins.entrypoints import ( + HarnessCompressPlugin, + HarnessContextPlugin, + HarnessHallucinationPlugin, + HarnessInvocationContextPlugin, + HarnessLongRunControlPlugin, + HarnessResponseVerificationPlugin, + build_harness_plugins, +) + +__all__ = [ + "HarnessCompressPlugin", + "HarnessContextPlugin", + "HarnessHallucinationPlugin", + "HarnessInvocationContextPlugin", + "HarnessLongRunControlPlugin", + "HarnessResponseVerificationPlugin", + "build_harness_plugins", +] diff --git a/veadk/extensions/harness/plugins/_callback_utils.py b/veadk/extensions/harness/plugins/_callback_utils.py new file mode 100644 index 00000000..5a364da4 --- /dev/null +++ b/veadk/extensions/harness/plugins/_callback_utils.py @@ -0,0 +1,35 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Backward-compatible import for plugin callback helpers.""" + +from veadk.extensions.harness.plugins._shared.callback_utils import ( + looks_like_error_result, + message_from_text, + run_context_from_callback, + run_context_from_invocation, + run_context_from_tool, + tool_name, + user_text_from_callback, +) + +__all__ = [ + "looks_like_error_result", + "message_from_text", + "run_context_from_callback", + "run_context_from_invocation", + "run_context_from_tool", + "tool_name", + "user_text_from_callback", +] diff --git a/veadk/extensions/harness/plugins/_shared/__init__.py b/veadk/extensions/harness/plugins/_shared/__init__.py new file mode 100644 index 00000000..9f32f13a --- /dev/null +++ b/veadk/extensions/harness/plugins/_shared/__init__.py @@ -0,0 +1,15 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Shared helpers for Harness plugins.""" diff --git a/veadk/extensions/harness/plugins/_shared/callback_utils.py b/veadk/extensions/harness/plugins/_shared/callback_utils.py new file mode 100644 index 00000000..fa44140d --- /dev/null +++ b/veadk/extensions/harness/plugins/_shared/callback_utils.py @@ -0,0 +1,100 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Shared helpers for Harness Runner plugin callback objects.""" + +from __future__ import annotations + +from collections.abc import Mapping +from typing import TYPE_CHECKING + +from veadk.extensions.harness.schemas import ( + ConversationMessage, + HarnessInvocationRef, + TaskContract, +) +from veadk.extensions.harness.utils import stringify_json_value + +if TYPE_CHECKING: + from google.adk.agents.callback_context import CallbackContext + from google.adk.agents.invocation_context import InvocationContext + from google.adk.tools.base_tool import BaseTool + from google.adk.tools.tool_context import ToolContext + + +def run_context_from_callback( + callback_context: "CallbackContext", + *, + profile: str, +) -> HarnessInvocationRef: + session = callback_context.session + goal = user_text_from_callback(callback_context) + return HarnessInvocationRef( + app_name=getattr(session, "app_name", ""), + user_id=getattr(callback_context, "user_id", ""), + session_id=getattr(session, "id", ""), + invocation_id=getattr(callback_context, "invocation_id", ""), + profile=profile, + task=TaskContract(goal=goal) if goal else None, + ) + + +def run_context_from_invocation( + invocation_context: "InvocationContext", + *, + profile: str, +) -> HarnessInvocationRef: + session = invocation_context.session + return HarnessInvocationRef( + app_name=getattr(session, "app_name", ""), + user_id=getattr(session, "user_id", ""), + session_id=getattr(session, "id", ""), + invocation_id=getattr(invocation_context, "invocation_id", ""), + profile=profile, + ) + + +def run_context_from_tool( + tool_context: "ToolContext", + *, + profile: str, +) -> HarnessInvocationRef: + session = tool_context.session + return HarnessInvocationRef( + app_name=getattr(session, "app_name", ""), + user_id=getattr(tool_context, "user_id", ""), + session_id=getattr(session, "id", ""), + invocation_id=getattr(tool_context, "invocation_id", ""), + profile=profile, + ) + + +def user_text_from_callback(callback_context: "CallbackContext") -> str: + user_content = getattr(callback_context, "user_content", None) + return stringify_json_value(user_content.model_dump()) if user_content else "" + + +def message_from_text(role: str, text: str) -> ConversationMessage: + return ConversationMessage(role=role, content=text) + + +def tool_name(tool: "BaseTool") -> str: + return str(getattr(tool, "name", tool.__class__.__name__)) + + +def looks_like_error_result(result: Mapping[str, object]) -> bool: + if "error" in result or "exception" in result: + return True + status = str(result.get("status", "")).lower() + return status in {"error", "failed", "failure"} diff --git a/veadk/extensions/harness/plugins/builder/__init__.py b/veadk/extensions/harness/plugins/builder/__init__.py new file mode 100644 index 00000000..e5fe151e --- /dev/null +++ b/veadk/extensions/harness/plugins/builder/__init__.py @@ -0,0 +1,19 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Plugin bundle builder exports.""" + +from veadk.extensions.harness.plugins.builder.factory import build_harness_plugins + +__all__ = ["build_harness_plugins"] diff --git a/veadk/extensions/harness/plugins/builder/factory.py b/veadk/extensions/harness/plugins/builder/factory.py new file mode 100644 index 00000000..62850b62 --- /dev/null +++ b/veadk/extensions/harness/plugins/builder/factory.py @@ -0,0 +1,135 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Harness plugin bundle assembly.""" + +from __future__ import annotations + +from collections.abc import Iterable + +from google.adk.plugins import BasePlugin + +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifier, + FinalResponseVerifierConfig, +) +from veadk.extensions.harness.modules.invocation_context import ( + HarnessInvocationContextBuilder, + HarnessInvocationContextConfig, +) +from veadk.extensions.harness.modules.tool_result_compactor import ( + ToolResultCompactor, + ToolResultCompactorConfig, +) +from veadk.extensions.harness.plugins.compactor import HarnessCompressPlugin +from veadk.extensions.harness.plugins.invocation_context import ( + HarnessInvocationContextPlugin, +) +from veadk.extensions.harness.plugins.long_run_control import ( + HarnessLongRunControlPlugin, +) +from veadk.extensions.harness.plugins.response_verification import ( + HarnessResponseVerificationPlugin, +) +from veadk.extensions.harness.stores import HarnessStoreProtocol, InMemoryHarnessStore + +ComponentName = str + + +def build_harness_plugins( + *, + components: Iterable[ComponentName] | str | None = None, + profile: str = "default", + store: HarnessStoreProtocol | None = None, + context_config: HarnessInvocationContextConfig | None = None, + compaction_config: ToolResultCompactorConfig | None = None, + compression_config: ToolResultCompactorConfig | None = None, + verifier_config: FinalResponseVerifierConfig | None = None, +) -> list[BasePlugin]: + """Build a shared-store Harness plugin bundle.""" + + selected = _normalize_components(components) + shared_store = store or InMemoryHarnessStore() + compactor_config = compaction_config or compression_config + plugins: list[BasePlugin] = [] + if "context_engine" in selected: + plugins.append( + HarnessInvocationContextPlugin( + context_builder=HarnessInvocationContextBuilder(context_config), + store=shared_store, + profile=profile, + ) + ) + if "compressor" in selected: + plugins.append( + HarnessCompressPlugin( + compactor=ToolResultCompactor(compactor_config), + store=shared_store, + profile=profile, + ) + ) + if "hallucination" in selected: + plugins.append( + HarnessResponseVerificationPlugin( + verifier=FinalResponseVerifier(verifier_config), + store=shared_store, + profile=profile, + ) + ) + if "long_run_control" in selected: + plugins.append( + HarnessLongRunControlPlugin( + store=shared_store, + profile=profile, + ) + ) + return plugins + + +def _normalize_components(components: Iterable[ComponentName] | str | None) -> set[str]: + if components is None: + raw = ["invocation_context", "compactor", "response_verification"] + elif isinstance(components, str): + raw = [item.strip() for item in components.split(",")] + else: + raw = [str(item).strip() for item in components] + aliases = { + "context": "context_engine", + "context_engine": "context_engine", + "harness_context_plugin": "context_engine", + "invocation_context": "context_engine", + "harness_invocation_context_builder": "context_engine", + "compress": "compressor", + "compression": "compressor", + "compressor": "compressor", + "compact": "compressor", + "compaction": "compressor", + "compactor": "compressor", + "tool_compactor": "compressor", + "tool_compressor": "compressor", + "harness_compress_plugin": "compressor", + "hallucination": "hallucination", + "verifier": "hallucination", + "result_verifier": "hallucination", + "response_verification": "hallucination", + "final_response_verifier": "hallucination", + "harness_hallucination_plugin": "hallucination", + "harness_response_verification_plugin": "hallucination", + "long_run": "long_run_control", + "long_run_control": "long_run_control", + "long_running": "long_run_control", + "run_control": "long_run_control", + "harness_long_run_control_plugin": "long_run_control", + } + return {aliases[item] for item in raw if item in aliases} diff --git a/veadk/extensions/harness/plugins/compactor/__init__.py b/veadk/extensions/harness/plugins/compactor/__init__.py new file mode 100644 index 00000000..719bcb59 --- /dev/null +++ b/veadk/extensions/harness/plugins/compactor/__init__.py @@ -0,0 +1,19 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Compactor plugin exports.""" + +from veadk.extensions.harness.plugins.compactor.plugin import HarnessCompressPlugin + +__all__ = ["HarnessCompressPlugin"] diff --git a/veadk/extensions/harness/plugins/compactor/plugin.py b/veadk/extensions/harness/plugins/compactor/plugin.py new file mode 100644 index 00000000..0244936f --- /dev/null +++ b/veadk/extensions/harness/plugins/compactor/plugin.py @@ -0,0 +1,145 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Tool-result compaction plugin for VeADK Runner.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from google.adk.models import LlmRequest, LlmResponse +from google.adk.plugins import BasePlugin + +from veadk.extensions.harness.modules.tool_result_compactor import ToolResultCompactor +from veadk.extensions.harness.plugins._shared.callback_utils import ( + run_context_from_callback, + run_context_from_tool, + tool_name, +) +from veadk.extensions.harness.plugins.content_adapter import contents_to_messages +from veadk.extensions.harness.schemas import ( + CompactionReport, + CompressionRequest, + HarnessEvent, +) +from veadk.extensions.harness.stores import HarnessStoreProtocol, InMemoryHarnessStore +from veadk.extensions.harness.utils import coerce_json_object + +if TYPE_CHECKING: + from google.adk.agents.callback_context import CallbackContext + from google.adk.tools.base_tool import BaseTool + from google.adk.tools.tool_context import ToolContext + + +class HarnessCompressPlugin(BasePlugin): + """Compacts oversized tool results and historical tool context.""" + + def __init__( + self, + *, + compactor: ToolResultCompactor | None = None, + compressor: ToolResultCompactor | None = None, + store: HarnessStoreProtocol | None = None, + profile: str = "default", + ) -> None: + super().__init__(name="harness_compress_plugin") + self.compactor = compactor or compressor or ToolResultCompactor() + self.compressor = self.compactor + self.store = store or InMemoryHarnessStore() + self.profile = profile + self.compaction_reports: list[CompactionReport] = [] + + async def before_model_callback( + self, + *, + callback_context: "CallbackContext", + llm_request: LlmRequest, + ) -> LlmResponse | None: + tool_reports = self._compact_function_responses(llm_request) + messages = contents_to_messages(llm_request.contents) + self.compaction_reports.extend(tool_reports) + if not messages: + return None + result = self.compactor.compress_messages( + CompressionRequest( + messages=messages, + max_context_chars=self.compactor.config.max_context_chars, + ) + ) + if result.report.changed or tool_reports: + self.store.append_event( + HarnessEvent( + event_type="compressor.model_context", + run_context=run_context_from_callback( + callback_context, + profile=self.profile, + ), + payload={ + "context_report": result.report.model_dump(mode="json"), + "tool_reports": [ + report.model_dump(mode="json") for report in tool_reports + ], + }, + ) + ) + if result.report.changed: + self.compaction_reports.append(result.report) + return None + + async def after_tool_callback( + self, + *, + tool: "BaseTool", + tool_args: dict[str, object], + tool_context: "ToolContext", + result: dict[str, object], + ) -> dict[str, object] | None: + compressed, report = self.compactor.compress_tool_result(result) + if not report.changed: + return None + self.compaction_reports.append(report) + self.store.append_event( + HarnessEvent( + event_type="compressor.tool_result", + run_context=run_context_from_tool(tool_context, profile=self.profile), + payload={ + "tool": tool_name(tool), + "tool_args": coerce_json_object(tool_args), + "report": report.model_dump(mode="json"), + }, + ) + ) + return compressed if isinstance(compressed, dict) else {"result": compressed} + + def reset_diagnostics(self) -> None: + self.compaction_reports.clear() + + def _compact_function_responses( + self, llm_request: LlmRequest + ) -> list[CompactionReport]: + reports: list[CompactionReport] = [] + for content in llm_request.contents: + for part in content.parts or []: + function_response = part.function_response + if function_response is None: + continue + response = function_response.response + if not isinstance(response, dict): + continue + compressed, report = self.compactor.compress_tool_result(response) + if not report.changed: + continue + function_response.response = compressed + reports.append(report) + return reports diff --git a/veadk/extensions/harness/plugins/content_adapter.py b/veadk/extensions/harness/plugins/content_adapter.py new file mode 100644 index 00000000..28711949 --- /dev/null +++ b/veadk/extensions/harness/plugins/content_adapter.py @@ -0,0 +1,136 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Adapters between Google ADK content objects and Harness schemas.""" + +from __future__ import annotations + +from collections.abc import Iterable +import re + +from google.adk.models import LlmRequest +from google.genai import types + +from veadk.extensions.harness.schemas import ConversationMessage +from veadk.extensions.harness.utils import stringify_json_value + +_HARNESS_BLOCK_PATTERNS = { + "Harness Context": re.compile( + r"\n*\[Harness Context\].*?\[/Harness Context\]\n*", + flags=re.DOTALL, + ), + "Harness Long Run Control": re.compile( + r"\n*\[Harness Long Run Control\].*?\[/Harness Long Run Control\]\n*", + flags=re.DOTALL, + ), +} + + +def content_to_text(content: types.Content | None) -> str: + """Extract text-like content from a Google GenAI Content object.""" + + if content is None or not content.parts: + return "" + values: list[str] = [] + for part in content.parts: + if part.text: + values.append(part.text) + elif part.function_response is not None: + values.append(stringify_json_value(part.function_response.model_dump())) + elif part.function_call is not None: + values.append(stringify_json_value(part.function_call.model_dump())) + elif part.executable_code is not None: + values.append(part.executable_code.code or "") + elif part.code_execution_result is not None: + values.append(stringify_json_value(part.code_execution_result.model_dump())) + return "\n".join(value for value in values if value) + + +def contents_to_messages( + contents: Iterable[types.Content], +) -> list[ConversationMessage]: + """Project ADK contents into protocol-neutral Harness messages.""" + + messages = [] + for content in contents: + text = content_to_text(content) + if not text: + continue + messages.append( + ConversationMessage(role=content.role or "unknown", content=text) + ) + return messages + + +def append_system_instruction(llm_request: LlmRequest, instruction: str) -> None: + """Append Harness context to the model system instruction.""" + + existing = llm_request.config.system_instruction + if not existing: + llm_request.config.system_instruction = instruction + return + if isinstance(existing, str): + llm_request.config.system_instruction = _append_latest_harness_context( + existing, instruction + ) + return + existing_text = _system_instruction_to_text(existing) + if existing_text: + llm_request.config.system_instruction = _append_latest_harness_context( + existing_text, instruction + ) + else: + llm_request.config.system_instruction = instruction + + +def response_text(content: types.Content | None) -> str: + """Extract final response text.""" + + return content_to_text(content) + + +def text_response(text: str) -> types.Content: + """Build a model response content from text.""" + + return types.Content(role="model", parts=[types.Part(text=text)]) + + +def _system_instruction_to_text(value: object) -> str: + if isinstance(value, types.Content): + return content_to_text(value) + if isinstance(value, types.Part): + return value.text or stringify_json_value(value.model_dump()) + if isinstance(value, list): + values = [_system_instruction_to_text(item) for item in value] + return "\n".join(item for item in values if item) + return stringify_json_value(value) + + +def _append_latest_harness_context(existing: str, instruction: str) -> str: + cleaned = _remove_previous_block(existing, instruction).strip() + if not cleaned: + return instruction + return f"{cleaned}\n\n{instruction}" + + +def _remove_previous_block(existing: str, instruction: str) -> str: + tag = _instruction_tag(instruction) + return _HARNESS_BLOCK_PATTERNS[tag].sub("\n\n", existing) + + +def _instruction_tag(instruction: str) -> str: + for tag in _HARNESS_BLOCK_PATTERNS: + if f"[{tag}]" in instruction: + return tag + return "Harness Context" diff --git a/veadk/extensions/harness/plugins/entrypoints.py b/veadk/extensions/harness/plugins/entrypoints.py new file mode 100644 index 00000000..d385c0ba --- /dev/null +++ b/veadk/extensions/harness/plugins/entrypoints.py @@ -0,0 +1,40 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Public Harness plugin entry points.""" + +from veadk.extensions.harness.plugins.builder import build_harness_plugins +from veadk.extensions.harness.plugins.compactor import HarnessCompressPlugin +from veadk.extensions.harness.plugins.invocation_context import ( + HarnessInvocationContextPlugin, +) +from veadk.extensions.harness.plugins.long_run_control import ( + HarnessLongRunControlPlugin, +) +from veadk.extensions.harness.plugins.response_verification import ( + HarnessResponseVerificationPlugin, +) + +HarnessContextPlugin = HarnessInvocationContextPlugin +HarnessHallucinationPlugin = HarnessResponseVerificationPlugin + +__all__ = [ + "HarnessCompressPlugin", + "HarnessContextPlugin", + "HarnessHallucinationPlugin", + "HarnessInvocationContextPlugin", + "HarnessLongRunControlPlugin", + "HarnessResponseVerificationPlugin", + "build_harness_plugins", +] diff --git a/veadk/extensions/harness/plugins/invocation_context/__init__.py b/veadk/extensions/harness/plugins/invocation_context/__init__.py new file mode 100644 index 00000000..7c45865d --- /dev/null +++ b/veadk/extensions/harness/plugins/invocation_context/__init__.py @@ -0,0 +1,23 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Invocation context plugin exports.""" + +from veadk.extensions.harness.plugins.invocation_context.plugin import ( + HarnessInvocationContextPlugin, +) + +HarnessContextPlugin = HarnessInvocationContextPlugin + +__all__ = ["HarnessContextPlugin", "HarnessInvocationContextPlugin"] diff --git a/veadk/extensions/harness/plugins/invocation_context/plugin.py b/veadk/extensions/harness/plugins/invocation_context/plugin.py new file mode 100644 index 00000000..af8862dd --- /dev/null +++ b/veadk/extensions/harness/plugins/invocation_context/plugin.py @@ -0,0 +1,120 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Invocation-context plugin for VeADK Runner.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from google.adk.models import LlmRequest, LlmResponse +from google.adk.plugins import BasePlugin + +from veadk.extensions.harness.modules.invocation_context import ( + HarnessInvocationContextBuilder, +) +from veadk.extensions.harness.plugins._shared.callback_utils import ( + message_from_text, + run_context_from_callback, + run_context_from_invocation, + user_text_from_callback, +) +from veadk.extensions.harness.plugins.content_adapter import ( + append_system_instruction, + contents_to_messages, +) +from veadk.extensions.harness.schemas import HarnessEvent +from veadk.extensions.harness.stores import HarnessStoreProtocol, InMemoryHarnessStore +from veadk.extensions.harness.utils import stringify_json_value + +if TYPE_CHECKING: + from google.adk.agents.callback_context import CallbackContext + from google.adk.agents.invocation_context import InvocationContext + from google.genai import types + + +class HarnessInvocationContextPlugin(BasePlugin): + """Injects task context and guardrails before model calls.""" + + def __init__( + self, + *, + context_builder: HarnessInvocationContextBuilder | None = None, + context_engine: HarnessInvocationContextBuilder | None = None, + store: HarnessStoreProtocol | None = None, + profile: str = "default", + ) -> None: + super().__init__(name="harness_invocation_context_plugin") + self.context_builder = ( + context_builder or context_engine or HarnessInvocationContextBuilder() + ) + self.context_engine = self.context_builder + self.store = store or InMemoryHarnessStore() + self.profile = profile + + async def before_model_callback( + self, + *, + callback_context: "CallbackContext", + llm_request: LlmRequest, + ) -> LlmResponse | None: + run_context = run_context_from_callback( + callback_context, + profile=self.profile, + ) + user_text = user_text_from_callback(callback_context) + history = contents_to_messages(llm_request.contents) + receipts = self.store.load_receipts( + run_id=run_context.invocation_id, + session_id=run_context.session_id, + limit=8, + ) + bundle = self.context_builder.prepare_context( + run_context, + user_input=user_text, + history=history, + receipts=receipts, + has_tools=bool(llm_request.tools_dict), + ) + if bundle.header: + append_system_instruction(llm_request, bundle.header) + self.store.append_event( + HarnessEvent( + event_type="invocation_context.injected", + run_context=run_context, + payload={ + "context_chars": bundle.context_chars, + "history_messages": len(history), + "receipt_count": len(receipts), + }, + ) + ) + return None + + async def on_user_message_callback( + self, + *, + invocation_context: "InvocationContext", + user_message: "types.Content", + ) -> "types.Content | None": + text = stringify_json_value(user_message.model_dump(), max_chars=4000) + run_context = run_context_from_invocation( + invocation_context, + profile=self.profile, + ) + self.store.append_message( + run_context.session_id, + message_from_text("user", text), + ) + return None diff --git a/veadk/extensions/harness/plugins/long_run_control/__init__.py b/veadk/extensions/harness/plugins/long_run_control/__init__.py new file mode 100644 index 00000000..72f1055f --- /dev/null +++ b/veadk/extensions/harness/plugins/long_run_control/__init__.py @@ -0,0 +1,21 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Long-run control plugin exports.""" + +from veadk.extensions.harness.plugins.long_run_control.plugin import ( + HarnessLongRunControlPlugin, +) + +__all__ = ["HarnessLongRunControlPlugin"] diff --git a/veadk/extensions/harness/plugins/long_run_control/plugin.py b/veadk/extensions/harness/plugins/long_run_control/plugin.py new file mode 100644 index 00000000..7ebb1166 --- /dev/null +++ b/veadk/extensions/harness/plugins/long_run_control/plugin.py @@ -0,0 +1,97 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Long-run control plugin for VeADK Runner.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from google.adk.models import LlmRequest, LlmResponse +from google.adk.plugins import BasePlugin + +from veadk.extensions.harness.plugins._shared.callback_utils import ( + run_context_from_callback, +) +from veadk.extensions.harness.plugins.content_adapter import append_system_instruction +from veadk.extensions.harness.schemas import HarnessEvent +from veadk.extensions.harness.stores import HarnessStoreProtocol, InMemoryHarnessStore + +if TYPE_CHECKING: + from google.adk.agents.callback_context import CallbackContext + + +class HarnessLongRunControlPlugin(BasePlugin): + """Steers long tool chains toward a final answer near the run budget.""" + + def __init__( + self, + *, + store: HarnessStoreProtocol | None = None, + profile: str = "default", + trigger_after_model_calls: int = 8, + ) -> None: + super().__init__(name="harness_long_run_control_plugin") + self.store = store or InMemoryHarnessStore() + self.profile = profile + self.trigger_after_model_calls = max(1, trigger_after_model_calls) + self._model_call_counts: dict[tuple[str, str], int] = {} + + async def before_model_callback( + self, + *, + callback_context: "CallbackContext", + llm_request: LlmRequest, + ) -> LlmResponse | None: + run_context = run_context_from_callback( + callback_context, + profile=self.profile, + ) + key = (run_context.session_id, run_context.invocation_id) + model_calls = self._model_call_counts.get(key, 0) + 1 + self._model_call_counts[key] = model_calls + if model_calls < self.trigger_after_model_calls: + return None + + append_system_instruction( + llm_request, + _long_run_control_instruction(model_calls=model_calls), + ) + self.store.append_event( + HarnessEvent( + event_type="long_run_control.guidance_injected", + run_context=run_context, + payload={ + "model_calls": model_calls, + "trigger_after_model_calls": self.trigger_after_model_calls, + }, + ) + ) + return None + + +def _long_run_control_instruction(*, model_calls: int) -> str: + return ( + "[Harness Long Run Control]\n" + f"model_calls_so_far: {model_calls}\n" + "objective: finish the current run within the remaining budget.\n" + "guidance:\n" + "- If the task has enough evidence, a complete answer, or generated " + "artifacts, stop calling tools and return the final response now.\n" + "- If files or artifacts were produced, include their filenames, paths, " + "or URIs and a concise summary.\n" + "- Call another tool only when it is strictly required to create the " + "missing final result; avoid repeating searches or code runs.\n" + "[/Harness Long Run Control]" + ) diff --git a/veadk/extensions/harness/plugins/response_verification/__init__.py b/veadk/extensions/harness/plugins/response_verification/__init__.py new file mode 100644 index 00000000..da3b7781 --- /dev/null +++ b/veadk/extensions/harness/plugins/response_verification/__init__.py @@ -0,0 +1,23 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Response verification plugin exports.""" + +from veadk.extensions.harness.plugins.response_verification.plugin import ( + HarnessResponseVerificationPlugin, +) + +HarnessHallucinationPlugin = HarnessResponseVerificationPlugin + +__all__ = ["HarnessHallucinationPlugin", "HarnessResponseVerificationPlugin"] diff --git a/veadk/extensions/harness/plugins/response_verification/plugin.py b/veadk/extensions/harness/plugins/response_verification/plugin.py new file mode 100644 index 00000000..d92e6bfc --- /dev/null +++ b/veadk/extensions/harness/plugins/response_verification/plugin.py @@ -0,0 +1,156 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Final-response verification plugin for VeADK Runner.""" + +from __future__ import annotations + +from typing import TYPE_CHECKING + +from google.adk.models import LlmResponse +from google.adk.plugins import BasePlugin + +from veadk.extensions.harness.modules.final_response_verifier import ( + FinalResponseVerifier, +) +from veadk.extensions.harness.plugins._shared.callback_utils import ( + looks_like_error_result, + run_context_from_callback, + run_context_from_invocation, + run_context_from_tool, + tool_name, +) +from veadk.extensions.harness.plugins.content_adapter import ( + response_text, + text_response, +) +from veadk.extensions.harness.schemas import EvidenceRef, HarnessEvent, ToolReceipt +from veadk.extensions.harness.stores import HarnessStoreProtocol, InMemoryHarnessStore +from veadk.extensions.harness.utils import ( + coerce_json_object, + stringify_json_value, + summarize_text, +) + +if TYPE_CHECKING: + from google.adk.agents.callback_context import CallbackContext + from google.adk.agents.invocation_context import InvocationContext + from google.adk.events.event import Event + from google.adk.tools.base_tool import BaseTool + from google.adk.tools.tool_context import ToolContext + + +class HarnessResponseVerificationPlugin(BasePlugin): + """Records receipts and suppresses unsupported final claims.""" + + def __init__( + self, + *, + verifier: FinalResponseVerifier | None = None, + store: HarnessStoreProtocol | None = None, + profile: str = "default", + ) -> None: + super().__init__(name="harness_response_verification_plugin") + self.verifier = verifier or FinalResponseVerifier() + self.store = store or InMemoryHarnessStore() + self.profile = profile + + async def after_tool_callback( + self, + *, + tool: "BaseTool", + tool_args: dict[str, object], + tool_context: "ToolContext", + result: object, + ) -> dict[str, object] | None: + run_context = run_context_from_tool(tool_context, profile=self.profile) + name = tool_name(tool) + result_object = result if isinstance(result, dict) else {"result": result} + summary = summarize_text(stringify_json_value(result_object), max_chars=1200) + status = "error" if looks_like_error_result(result_object) else "success" + receipt = ToolReceipt( + name=name, + status=status, + summary=summary, + run_id=run_context.invocation_id, + session_id=run_context.session_id, + evidence=[EvidenceRef(source=name, content=summary)], + metadata={"tool_args": coerce_json_object(tool_args)}, + ) + self.store.append_receipt(receipt) + return None + + async def after_model_callback( + self, + *, + callback_context: "CallbackContext", + llm_response: LlmResponse, + ) -> LlmResponse | None: + text = response_text(llm_response.content) + if not text: + return None + run_context = run_context_from_callback( + callback_context, + profile=self.profile, + ) + receipts = self.store.load_receipts( + run_id=run_context.invocation_id, + session_id=run_context.session_id, + limit=20, + ) + report = self.verifier.verify_text(text, receipts=receipts) + intervention = self.verifier.decide(report) + self.store.append_event( + HarnessEvent( + event_type="verifier.report", + run_context=run_context, + payload={ + "intervention": intervention.model_dump(mode="json"), + "receipt_count": len(receipts), + }, + ) + ) + if intervention.action == "block": + return LlmResponse( + content=text_response( + "I cannot verify that result from the available tool evidence. " + "Please rerun the required tool step or provide supporting evidence." + ), + custom_metadata={ + "harness_verification": report.model_dump(mode="json") + }, + ) + metadata = dict(llm_response.custom_metadata or {}) + metadata["harness_verification"] = report.model_dump(mode="json") + llm_response.custom_metadata = metadata + return None + + async def on_event_callback( + self, + *, + invocation_context: "InvocationContext", + event: "Event", + ) -> "Event | None": + run_context = run_context_from_invocation( + invocation_context, + profile=self.profile, + ) + self.store.append_event( + HarnessEvent( + event_type="runner.event", + run_context=run_context, + payload={"author": str(getattr(event, "author", ""))}, + ) + ) + return None diff --git a/veadk/extensions/harness/schemas.py b/veadk/extensions/harness/schemas.py new file mode 100644 index 00000000..6d507ee7 --- /dev/null +++ b/veadk/extensions/harness/schemas.py @@ -0,0 +1,211 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Public Pydantic models shared by Harness modules and plugins.""" + +from __future__ import annotations + +from typing import Literal, TypeAlias + +from pydantic import BaseModel, ConfigDict, Field + +JsonScalar: TypeAlias = str | int | float | bool | None +JsonValue: TypeAlias = JsonScalar | list[object] | dict[str, object] +JsonObject: TypeAlias = dict[str, JsonValue] + + +class HarnessBaseModel(BaseModel): + """Strict base model for public SDK schemas.""" + + model_config = ConfigDict( + extra="forbid", + populate_by_name=True, + validate_assignment=True, + ) + + +class TaskContract(HarnessBaseModel): + """Stable task target used to keep multi-turn runs anchored.""" + + goal: str = "" + acceptance_criteria: list[str] = Field(default_factory=list) + metadata: JsonObject = Field(default_factory=dict) + + +class HarnessInvocationRef(HarnessBaseModel): + """Run identity and profile propagated through Harness modules.""" + + app_name: str = "" + user_id: str = "" + session_id: str + invocation_id: str + profile: str = "default" + task: TaskContract | None = None + metadata: JsonObject = Field(default_factory=dict) + + +class ConversationMessage(HarnessBaseModel): + """Protocol-neutral message projection used by atomic modules.""" + + role: str + content: str + name: str = "" + metadata: JsonObject = Field(default_factory=dict) + + +class InvocationContextBlock(HarnessBaseModel): + """Context header and accounting generated before a model call.""" + + header: str + messages: list[ConversationMessage] = Field(default_factory=list) + original_chars: int = 0 + context_chars: int = 0 + injected: bool = False + warnings: list[str] = Field(default_factory=list) + + +class CompressionDecision(HarnessBaseModel): + """Policy decision for one message.""" + + index: int + action: Literal["protect", "skip", "compress"] + reason: str + role: str + chars: int + + +class CompressionPlan(HarnessBaseModel): + """Role and recency aware compression plan.""" + + decisions: list[CompressionDecision] = Field(default_factory=list) + candidate_indexes: list[int] = Field(default_factory=list) + summary: JsonObject = Field(default_factory=dict) + + +class CompressionRequest(HarnessBaseModel): + """Messages and limits passed to a compressor.""" + + messages: list[ConversationMessage] + max_context_chars: int = 24000 + protected_message_count: int = 2 + metadata: JsonObject = Field(default_factory=dict) + + +class CompactionReport(HarnessBaseModel): + """Compaction accounting and policy trace.""" + + provider: str + original_chars: int + compressed_chars: int + changed: bool = False + omitted_messages: int = 0 + protected_messages: int = 0 + tokens_before: int = 0 + tokens_after: int = 0 + tokens_saved: int = 0 + compression_ratio: float = 0.0 + transforms_applied: list[str] = Field(default_factory=list) + policy: JsonObject = Field(default_factory=dict) + warnings: list[str] = Field(default_factory=list) + + +class CompactionResult(HarnessBaseModel): + """Compacted messages and report.""" + + messages: list[ConversationMessage] + report: CompactionReport + + +class EvidenceRef(HarnessBaseModel): + """Evidence extracted from tool outputs or other trusted context.""" + + source: str + content: str + score: float = 1.0 + metadata: JsonObject = Field(default_factory=dict) + + +class ToolReceipt(HarnessBaseModel): + """Structured record of a tool or capability result.""" + + name: str + status: Literal["success", "error", "unknown"] = "unknown" + summary: str = "" + run_id: str = "" + session_id: str = "" + evidence: list[EvidenceRef] = Field(default_factory=list) + metadata: JsonObject = Field(default_factory=dict) + + +class VerificationReport(HarnessBaseModel): + """Answer grounding result.""" + + status: Literal["pass", "warn", "fail"] = "pass" + reasons: list[str] = Field(default_factory=list) + supported_claims: list[str] = Field(default_factory=list) + unsupported_claims: list[str] = Field(default_factory=list) + evidence: list[EvidenceRef] = Field(default_factory=list) + repaired: bool = False + + +class VerificationDecision(HarnessBaseModel): + """Action a plugin can take after validation.""" + + action: Literal["allow", "observe", "repair", "block"] = "allow" + reason: str = "" + instruction: str = "" + report: VerificationReport | None = None + + +class HarnessEvent(HarnessBaseModel): + """Generic event stored for receipts, reports, and diagnostics.""" + + event_type: str + run_context: HarnessInvocationRef | None = None + payload: JsonObject = Field(default_factory=dict) + + +HarnessRunContext = HarnessInvocationRef +ContextBundle = InvocationContextBlock +CompressionReport = CompactionReport +CompressionResult = CompactionResult +CapabilityReceipt = ToolReceipt +HarnessIntervention = VerificationDecision + +__all__ = [ + "CapabilityReceipt", + "CompactionReport", + "CompactionResult", + "CompressionDecision", + "CompressionPlan", + "CompressionReport", + "CompressionRequest", + "CompressionResult", + "ContextBundle", + "ConversationMessage", + "EvidenceRef", + "HarnessBaseModel", + "HarnessEvent", + "HarnessIntervention", + "HarnessInvocationRef", + "HarnessRunContext", + "InvocationContextBlock", + "JsonObject", + "JsonScalar", + "JsonValue", + "TaskContract", + "ToolReceipt", + "VerificationDecision", + "VerificationReport", +] diff --git a/veadk/extensions/harness/stores/__init__.py b/veadk/extensions/harness/stores/__init__.py new file mode 100644 index 00000000..ea0c4ea6 --- /dev/null +++ b/veadk/extensions/harness/stores/__init__.py @@ -0,0 +1,21 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Harness store implementations.""" + +from veadk.extensions.harness.stores.jsonl import JsonlHarnessStore +from veadk.extensions.harness.stores.memory import InMemoryHarnessStore +from veadk.extensions.harness.stores.protocols import HarnessStoreProtocol + +__all__ = ["HarnessStoreProtocol", "InMemoryHarnessStore", "JsonlHarnessStore"] diff --git a/veadk/extensions/harness/stores/jsonl.py b/veadk/extensions/harness/stores/jsonl.py new file mode 100644 index 00000000..1df1979a --- /dev/null +++ b/veadk/extensions/harness/stores/jsonl.py @@ -0,0 +1,93 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""JSONL Harness store.""" + +from __future__ import annotations + +import json +from pathlib import Path + +from veadk.extensions.harness.schemas import ( + ToolReceipt, + ConversationMessage, + HarnessEvent, +) + + +class JsonlHarnessStore: + """Append-only local JSONL store with lightweight reads.""" + + def __init__(self, root: str | Path) -> None: + self.root = Path(root) + self.root.mkdir(parents=True, exist_ok=True) + + def append_event(self, event: HarnessEvent) -> None: + self._append("events.jsonl", event.model_dump(mode="json")) + + def append_receipt(self, receipt: ToolReceipt) -> None: + self._append("receipts.jsonl", receipt.model_dump(mode="json")) + + def append_message(self, session_id: str, message: ConversationMessage) -> None: + payload = message.model_dump(mode="json") + payload["session_id"] = session_id + self._append("messages.jsonl", payload) + + def load_messages( + self, session_id: str, limit: int | None = None + ) -> list[ConversationMessage]: + messages = [] + for payload in self._read("messages.jsonl"): + if payload.get("session_id") != session_id: + continue + payload.pop("session_id", None) + messages.append(ConversationMessage.model_validate(payload)) + return messages[-limit:] if limit else messages + + def load_receipts( + self, + *, + run_id: str = "", + session_id: str = "", + limit: int | None = None, + ) -> list[ToolReceipt]: + receipts = [] + for payload in self._read("receipts.jsonl"): + receipt = ToolReceipt.model_validate(payload) + if run_id and receipt.run_id != run_id: + continue + if session_id and receipt.session_id != session_id: + continue + receipts.append(receipt) + return receipts[-limit:] if limit else receipts + + def _append(self, filename: str, payload: dict[str, object]) -> None: + path = self.root / filename + with path.open("a", encoding="utf-8") as handle: + handle.write(json.dumps(payload, ensure_ascii=False, sort_keys=True) + "\n") + + def _read(self, filename: str) -> list[dict[str, object]]: + path = self.root / filename + if not path.is_file(): + return [] + rows: list[dict[str, object]] = [] + with path.open("r", encoding="utf-8") as handle: + for line in handle: + line = line.strip() + if not line: + continue + value = json.loads(line) + if isinstance(value, dict): + rows.append(value) + return rows diff --git a/veadk/extensions/harness/stores/memory.py b/veadk/extensions/harness/stores/memory.py new file mode 100644 index 00000000..41110ba3 --- /dev/null +++ b/veadk/extensions/harness/stores/memory.py @@ -0,0 +1,64 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""In-memory Harness store for tests and local development.""" + +from __future__ import annotations + +from collections import defaultdict + +from veadk.extensions.harness.schemas import ( + ToolReceipt, + ConversationMessage, + HarnessEvent, +) + + +class InMemoryHarnessStore: + """Simple process-local store.""" + + def __init__(self) -> None: + self.events: list[HarnessEvent] = [] + self.receipts: list[ToolReceipt] = [] + self.messages: dict[str, list[ConversationMessage]] = defaultdict(list) + + def append_event(self, event: HarnessEvent) -> None: + self.events.append(event) + + def append_receipt(self, receipt: ToolReceipt) -> None: + self.receipts.append(receipt) + + def append_message(self, session_id: str, message: ConversationMessage) -> None: + self.messages[session_id].append(message) + + def load_messages( + self, session_id: str, limit: int | None = None + ) -> list[ConversationMessage]: + messages = list(self.messages.get(session_id, [])) + return messages[-limit:] if limit else messages + + def load_receipts( + self, + *, + run_id: str = "", + session_id: str = "", + limit: int | None = None, + ) -> list[ToolReceipt]: + receipts = [ + receipt + for receipt in self.receipts + if (not run_id or receipt.run_id == run_id) + and (not session_id or receipt.session_id == session_id) + ] + return receipts[-limit:] if limit else receipts diff --git a/veadk/extensions/harness/stores/protocols.py b/veadk/extensions/harness/stores/protocols.py new file mode 100644 index 00000000..6ee61153 --- /dev/null +++ b/veadk/extensions/harness/stores/protocols.py @@ -0,0 +1,52 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Store protocols for Harness events and receipts.""" + +from __future__ import annotations + +from typing import Protocol + +from veadk.extensions.harness.schemas import ( + ToolReceipt, + ConversationMessage, + HarnessEvent, +) + + +class HarnessStoreProtocol(Protocol): + """Minimal persistence protocol used by atomic modules and plugins.""" + + def append_event(self, event: HarnessEvent) -> None: + """Persist a Harness event.""" + + def append_receipt(self, receipt: ToolReceipt) -> None: + """Persist a capability receipt.""" + + def append_message(self, session_id: str, message: ConversationMessage) -> None: + """Persist a message projection for later context engineering.""" + + def load_messages( + self, session_id: str, limit: int | None = None + ) -> list[ConversationMessage]: + """Load recent messages for a session.""" + + def load_receipts( + self, + *, + run_id: str = "", + session_id: str = "", + limit: int | None = None, + ) -> list[ToolReceipt]: + """Load receipts filtered by run or session.""" diff --git a/veadk/extensions/harness/utils.py b/veadk/extensions/harness/utils.py new file mode 100644 index 00000000..f68ad8f2 --- /dev/null +++ b/veadk/extensions/harness/utils.py @@ -0,0 +1,85 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Small text helpers used by Harness modules.""" + +from __future__ import annotations + +import re +from collections.abc import Mapping, Sequence + +from veadk.extensions.harness.schemas import JsonObject, JsonValue + +_SECRET_PATTERNS = ( + re.compile(r"(?i)(authorization\s*[:=]\s*bearer\s+)[^\s,;]+"), + re.compile(r"(?i)((?:api[_-]?key|secret|token|password)\s*[:=]\s*)[^\s,;]+"), + re.compile(r"(?i)((?:access[_-]?key|secret[_-]?key)\s*[:=]\s*)[^\s,;]+"), +) + + +def summarize_text(text: str, *, max_chars: int = 900) -> str: + """Return a compact single-string summary without model calls.""" + + normalized = " ".join(text.split()) + if len(normalized) <= max_chars: + return normalized + head = normalized[: max_chars // 2].rstrip() + tail = normalized[-max_chars // 3 :].lstrip() + omitted = len(normalized) - len(head) - len(tail) + return f"{head} ... [omitted {omitted} chars] ... {tail}" + + +def redact_text(text: str) -> str: + """Mask common credential shapes in traces and receipts.""" + + redacted = text + for pattern in _SECRET_PATTERNS: + redacted = pattern.sub(r"\1[REDACTED]", redacted) + return redacted + + +def stringify_json_value(value: object, *, max_chars: int = 4000) -> str: + """Render dynamic values for receipts without leaking huge payloads.""" + + if isinstance(value, str): + return summarize_text(redact_text(value), max_chars=max_chars) + if isinstance(value, Mapping): + parts = [] + for key, item in value.items(): + parts.append(f"{key}: {stringify_json_value(item, max_chars=400)}") + return summarize_text(redact_text("; ".join(parts)), max_chars=max_chars) + if isinstance(value, Sequence) and not isinstance(value, (bytes, bytearray)): + parts = [stringify_json_value(item, max_chars=300) for item in value[:20]] + return summarize_text(redact_text("; ".join(parts)), max_chars=max_chars) + return summarize_text(redact_text(str(value)), max_chars=max_chars) + + +def coerce_json_value(value: object) -> JsonValue: + """Convert a Python object into the SDK's JSON-safe value type.""" + + if value is None or isinstance(value, (str, int, float, bool)): + return value + if isinstance(value, Mapping): + return {str(key): coerce_json_value(item) for key, item in value.items()} + if isinstance(value, Sequence) and not isinstance(value, (bytes, bytearray, str)): + return [coerce_json_value(item) for item in value] + return str(value) + + +def coerce_json_object(value: object) -> JsonObject: + """Convert a mapping-like object into a JSON object.""" + + if not isinstance(value, Mapping): + return {} + return {str(key): coerce_json_value(item) for key, item in value.items()} diff --git a/veadk/harness.py b/veadk/harness.py new file mode 100644 index 00000000..63157588 --- /dev/null +++ b/veadk/harness.py @@ -0,0 +1,36 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +"""Agent Harness plugin bridge for VeADK users.""" + +from __future__ import annotations + +from collections.abc import Iterable + +from google.adk.plugins import BasePlugin + + +def build_harness_plugins( + *, + components: Iterable[str] | str | None = None, + profile: str = "default", +) -> list[BasePlugin]: + """Build Harness plugins from the bundled VeADK extension.""" + + from veadk.extensions.harness.plugins import build_harness_plugins as _build + + return _build(components=components, profile=profile) + + +__all__ = ["build_harness_plugins"]