Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -412,6 +412,8 @@ The agent-facing surface — tools, prompts, wire schemas, and error envelope
| `domscribe.annotation.get` | Retrieve annotation by ID |
| `domscribe.annotation.list` | List annotations with status/filter options |
| `domscribe.annotation.search` | Full-text search across annotation content |
| `domscribe.verify.baseline` | Capture a pre-edit style/geometry snapshot of a rendered element |
| `domscribe.verify.afterEdit` | Compare the element against the baseline — deterministic verdict + per-property deltas |
| `domscribe.status` | Relay daemon health, manifest stats, queue counts |

See the [`@domscribe/mcp` README](./packages/domscribe-mcp/README.md) for detailed tool schemas, response formats, and prompt definitions.
Expand Down
137 changes: 137 additions & 0 deletions docs/rfcs/0002-post-edit-verify-mcp-tool.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,137 @@
# RFC 0002 — Post-Edit Verification MCP Tools

- **Status:** Implemented (v1: delta-based verification)
- **Packages:** `@domscribe/core`, `@domscribe/verify`, `@domscribe/relay`, `@domscribe/overlay`
- **Depends on:** RFC 0001 (component-style capture)

> This document reconstructs the RFC 0002 spec that earlier commits cite
> (annotation schema v3, `verifyHistory`, `VerifyResult`) and records the
> design as implemented. The original draft was removed in a repo cleanup;
> the normative constraints below match the code.

## Problem

Coding agents editing UI code cannot reliably tell whether their edit took
effect. The universal failure modes:

1. **Silent no-ops** — the agent edits the wrong file, a non-applying style
path (specificity, conditional class), or HMR fails, and the agent
declares success anyway.
2. **Vision-blind regressions** — agents that verify by screenshot rely on
a vision model to compare images, and vision models are demonstrably
unreliable at exactly the deltas that matter for styling work (small
offsets, near-identical shades, spacing changes).

Every mainstream agent stack verifies by "screenshot + eyeball" as of
mid-2026. None offers a deterministic, element-scoped comparison in the dev
inner loop.

## Design

Two MCP tools on the relay close the loop deterministically:

1. **`domscribe.verify.baseline`** — called *before* the edit. Captures a
snapshot of the target element from the live page over the existing
relay↔browser WS channel: the RFC 0001 computed-style allowlist
(≤32 properties) plus the element's bounding rect. Returns an opaque
`baselineId`. Baselines are in-memory and session-scoped (max 100,
oldest evicted) — they describe the live page *now*, so persisting them
would only produce stale comparisons.

2. **`domscribe.verify.afterEdit`** — called *after* the edit and HMR.
Re-captures the same element, computes deltas against the baseline, and
returns a `VerifyResult` (core schema, annotation schema v3):
- `verdict`: `match` | `partial` | `no_change` | `regression`
- `componentStylesDelta`: per-property `{ property, before, after }`
- `boundingRectDelta`: per-field `{ field, before, after }` (0.5 px
epsilon for sub-pixel jitter)
- `notes`: deterministic explanation
When an `annotationId` is supplied, the result is appended to that
annotation's `context.verifyHistory` (append-only).

### Deltas first, pixels later

The v1 verdict is computed from **style and geometry deltas only** — not
pixel diffs. Grounds for this ordering:

- Computed-style deltas are exact, cheap (no screenshot pipeline), immune
to anti-aliasing noise, and directly actionable in code
(`padding: 8px → 12px` tells the agent what to fix; a red pixel overlay
does not).
- The research consensus (Design2Code's low-level metrics, UI2Code^N's
finding that CLIP-similarity rewards *degrade* refinement, the
VLM-blindness benchmarks) is that element-level structured deltas are the
reliable regression oracle, while holistic visual judgment belongs to
the calling agent.
- The pixel path stays open: `VerifyResultSchema.pixelDiffRatio` and
`screenshotRef` are reserved, and `@domscribe/verify` already ships the
pixelmatch comparator used by the falsifier harness. A future revision
can add element-scoped screenshot capture without changing the contract.

### Verdict semantics (deterministic, intent-agnostic)

The tool measures; the agent judges intent. The caller may declare
`expectedChanges: [{ property, value? }]` — the style changes the edit was
meant to make (values compare as `getComputedStyle` strings).

| Situation | Verdict |
| --- | --- |
| No style and no geometry delta | `no_change` |
| No expectations declared, something changed | `match` + caveat note (change detection only) |
| All expectations met, nothing unexpected | `match` |
| All expectations met, extra properties changed | `partial` (unexpected properties listed) |
| Some expectations met (incl. wrong-value changes) | `partial` |
| No expectation met, other properties changed | `regression` |

Geometry deltas never downgrade a verdict on their own — rect movement is
usually a consequence of an intended style change (padding grows the box).
They are always reported.

### Degradation without `captureStyles`

Style capture is gated on the runtime's `captureStyles` flag (default off
in v0.x per RFC 0001). Without it, verification falls back to geometry-only
change detection: `expectedChanges` cannot be evaluated and the result
notes say so. The baseline response reports `hasComponentStyles` so agents
can prompt the user to enable the flag when full deltas matter.

## Wire surface

- `POST /api/v1/verify/baseline` `{ entryId }` →
`{ captured, baselineId?, browserConnected, hasComponentStyles?, hasBoundingRect?, error? }`
- `POST /api/v1/verify/check` `{ baselineId, expectedChanges?, annotationId? }` →
`{ verified, browserConnected, result?: VerifyResult, error? }`

The WS `context:response` payload gains an optional
`elementInfo.boundingRect` (serialized `DOMRect`), captured by the overlay's
relay service.

## Delta engine

`@domscribe/verify` exports the pure functions the relay uses (no DOM, no
I/O — unit-testable in isolation):

- `diffStyleMaps(before, after): StylePropertyDelta[]`
- `diffBoundingRects(before, after, epsilon?): BoundingRectDelta[]`
- `resolveVerdict({ styleDeltas, rectDeltas, expectedChanges? })`

## Falsifier gate

RFC 0002's success criterion is a ≥60% retry-resolution rate on the styling
falsifier corpus: after a failed first attempt, an agent given the
`VerifyResult` deltas should resolve the task on retry at least 60% of the
time. Measuring this requires the agent-driving falsifier mode
(`--mode=agent`), which is tracked separately — see
`docs/sprints/3071-rfc-0001-baseline.md` for the harness gap analysis.

## Agent workflow

```
1. domscribe.query.bySource / domscribe.resolve → entryId
2. domscribe.verify.baseline { entryId } → baselineId
3. edit source, wait for HMR
4. domscribe.verify.afterEdit { baselineId, expectedChanges }
5. verdict = no_change? → the edit did not land; fix and repeat 3–4
verdict = partial/regression? → consult deltas; fix and repeat 3–4
verdict = match? → done (agent confirms intent visually if it can)
```
4 changes: 4 additions & 0 deletions packages/domscribe-core/src/lib/constants/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,10 @@ export const API_PATHS = {
MANIFEST_RESOLVE_BATCH: '/manifest/resolve/batch',
MANIFEST_RESOLVE_BY_SOURCE: '/manifest/resolve-by-source',

// Verify endpoints (RFC 0002)
VERIFY_BASELINE: '/verify/baseline',
VERIFY_CHECK: '/verify/check',

// System endpoints
STATUS: `/status`,
HEALTH: `/health`,
Expand Down
9 changes: 9 additions & 0 deletions packages/domscribe-mcp/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,15 @@ Annotations are created when a developer clicks an element in the Domscribe over
| `domscribe.annotation.list` | List annotations with status and filter options |
| `domscribe.annotation.search` | Full-text search across annotation content |

### Post-Edit Verification

Deterministic verification that a UI edit actually took effect (RFC 0002). Capture a baseline before editing, edit, then verify — the verdict and per-property deltas are exact measurements, not vision-model judgments.

| Tool | Description |
| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `domscribe.verify.baseline` | Capture a pre-edit snapshot (computed styles + geometry) of a rendered element; returns a `baselineId` |
| `domscribe.verify.afterEdit` | Re-capture and compare against the baseline; returns a verdict (`match`/`partial`/`no_change`/`regression`) with style deltas |

### System

| Tool | Description |
Expand Down
35 changes: 35 additions & 0 deletions packages/domscribe-overlay/src/services/relay-service.ts
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,9 @@ export class RelayService {
),
),
innerText: elementInfo.element?.innerText?.slice(0, 500),
boundingRect: this.serializeBoundingRect(
elementInfo.element,
),
}
: undefined,
};
Expand Down Expand Up @@ -188,6 +191,38 @@ export class RelayService {
return true;
}

/**
* Serialize an element's bounding rect into a plain JSON-safe object.
* DOMRect is not JSON-serializable directly (its fields are getters).
*/
private serializeBoundingRect(element: HTMLElement | undefined):
| {
x: number;
y: number;
width: number;
height: number;
top: number;
right: number;
bottom: number;
left: number;
}
| undefined {
if (!element || typeof element.getBoundingClientRect !== 'function') {
return undefined;
}
const rect = element.getBoundingClientRect();
return {
x: rect.x,
y: rect.y,
width: rect.width,
height: rect.height,
top: rect.top,
right: rect.right,
bottom: rect.bottom,
left: rect.left,
};
}

/**
* Refresh annotations from server
*/
Expand Down
1 change: 1 addition & 0 deletions packages/domscribe-relay/package.json
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
"@clack/prompts": "^1.1.0",
"@domscribe/core": "workspace:*",
"@domscribe/manifest": "workspace:*",
"@domscribe/verify": "workspace:*",
"@fastify/cors": "^10.0.0",
"@fastify/websocket": "^11.0.0",
"@modelcontextprotocol/sdk": "^1.0.0",
Expand Down
43 changes: 43 additions & 0 deletions packages/domscribe-relay/src/client/relay-http-client.ts
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,11 @@ import {
ShutdownResponseSchema,
StatusResponse,
StatusResponseSchema,
ExpectedChangeInput,
VerifyBaselineResponse,
VerifyBaselineResponseSchema,
VerifyCheckResponse,
VerifyCheckResponseSchema,
} from '../schema.js';
import {
RelayErrorResponse,
Expand Down Expand Up @@ -354,6 +359,44 @@ export class RelayHttpClient {
return QueryBySourceResponseSchema.parse(await response.json());
}

async verifyBaseline(params: {
entryId: ManifestEntryId;
}): Promise<VerifyBaselineResponse> {
const apiPath = `${API_PATHS.BASE.replace(':version', 'v1')}${API_PATHS.VERIFY_BASELINE}`;
const url = new URL(apiPath, this.baseUrl);
const response = await fetch(url.toString(), {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(params),
});
if (!response.ok) {
throw await this.parseError(response);
}
return VerifyBaselineResponseSchema.parse(await response.json());
}

async verifyCheck(params: {
baselineId: string;
expectedChanges?: ExpectedChangeInput[];
annotationId?: string;
}): Promise<VerifyCheckResponse> {
const apiPath = `${API_PATHS.BASE.replace(':version', 'v1')}${API_PATHS.VERIFY_CHECK}`;
const url = new URL(apiPath, this.baseUrl);
const response = await fetch(url.toString(), {
method: 'POST',
headers: {
'Content-Type': 'application/json',
},
body: JSON.stringify(params),
});
if (!response.ok) {
throw await this.parseError(response);
}
return VerifyCheckResponseSchema.parse(await response.json());
}

async getManifestStats(): Promise<ManifestStatsResponse> {
const apiPath = `${API_PATHS.BASE.replace(':version', 'v1')}${API_PATHS.MANIFEST_STATS}`;
const url = new URL(apiPath, this.baseUrl);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,8 @@ export function createMockRelayClient(
deleteAnnotation: vi.fn(),
patchAnnotation: vi.fn(),
queryBySource: vi.fn(),
verifyBaseline: vi.fn(),
verifyCheck: vi.fn(),
getStatus: vi.fn(),
getHealth: vi.fn(),
shutdown: vi.fn(),
Expand Down
10 changes: 8 additions & 2 deletions packages/domscribe-relay/src/mcp/mcp-adapter.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ function getServer(adapter: McpAdapter) {

describe('McpAdapter', () => {
describe('active mode', () => {
it('should register all 12 tools', () => {
it('should register all 14 tools', () => {
// Act
const adapter = new McpAdapter({
mode: 'active',
Expand All @@ -60,7 +60,7 @@ describe('McpAdapter', () => {

// Assert
const server = getServer(adapter);
expect(server.registeredTools.size).toBe(12);
expect(server.registeredTools.size).toBe(14);
expect(server.registeredTools.has('domscribe.resolve')).toBe(true);
expect(server.registeredTools.has('domscribe.resolve.batch')).toBe(true);
expect(server.registeredTools.has('domscribe.manifest.stats')).toBe(true);
Expand All @@ -83,6 +83,12 @@ describe('McpAdapter', () => {
);
expect(server.registeredTools.has('domscribe.status')).toBe(true);
expect(server.registeredTools.has('domscribe.query.bySource')).toBe(true);
expect(server.registeredTools.has('domscribe.verify.baseline')).toBe(
true,
);
expect(server.registeredTools.has('domscribe.verify.afterEdit')).toBe(
true,
);
});

it('should register all 4 prompts', () => {
Expand Down
4 changes: 4 additions & 0 deletions packages/domscribe-relay/src/mcp/mcp-adapter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ import { AnnotationsRespondTool } from './tools/annotation-respond.tool.js';
import { AnnotationsSearchTool } from './tools/annotation-search.tool.js';
import { StatusTool } from './tools/status.tool.js';
import { QueryBySourceTool } from './tools/query-by-source.tool.js';
import { VerifyBaselineTool } from './tools/verify-baseline.tool.js';
import { VerifyAfterEditTool } from './tools/verify-after-edit.tool.js';

// Prompt classes
import { ProcessNextPrompt } from './prompts/process-next.prompt.js';
Expand Down Expand Up @@ -113,6 +115,8 @@ export class McpAdapter {
new AnnotationsSearchTool(relayHttpClient),
new StatusTool(relayHttpClient),
new QueryBySourceTool(relayHttpClient),
new VerifyBaselineTool(relayHttpClient),
new VerifyAfterEditTool(relayHttpClient),
];

for (const tool of tools) {
Expand Down
2 changes: 2 additions & 0 deletions packages/domscribe-relay/src/mcp/tools/tool.defs.spec.ts
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,8 @@ describe('tool.defs', () => {
expect(MCP_TOOLS.ANNOTATION_PROCESS).toBe('domscribe.annotation.process');
expect(MCP_TOOLS.ANNOTATION_RESPOND).toBe('domscribe.annotation.respond');
expect(MCP_TOOLS.ANNOTATION_SEARCH).toBe('domscribe.annotation.search');
expect(MCP_TOOLS.VERIFY_BASELINE).toBe('domscribe.verify.baseline');
expect(MCP_TOOLS.VERIFY_AFTER_EDIT).toBe('domscribe.verify.afterEdit');
expect(MCP_TOOLS.STATUS).toBe('domscribe.status');
});
});
Expand Down
3 changes: 3 additions & 0 deletions packages/domscribe-relay/src/mcp/tools/tool.defs.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,9 @@ export const MCP_TOOLS = {
ANNOTATION_SEARCH: 'domscribe.annotation.search',
// Query tools
QUERY_BY_SOURCE: 'domscribe.query.bySource',
// Verify tools (RFC 0002)
VERIFY_BASELINE: 'domscribe.verify.baseline',
VERIFY_AFTER_EDIT: 'domscribe.verify.afterEdit',
// System tools
STATUS: 'domscribe.status',
} as const;
Expand Down
Loading
Loading