diff --git a/README.md b/README.md index 240ab94..f87f3ab 100644 --- a/README.md +++ b/README.md @@ -412,6 +412,8 @@ The agent-facing surface — tools, prompts, wire schemas, and error envelope | `domscribe.annotation.get` | Retrieve annotation by ID | | `domscribe.annotation.list` | List annotations with status/filter options | | `domscribe.annotation.search` | Full-text search across annotation content | +| `domscribe.verify.baseline` | Capture a pre-edit style/geometry snapshot of a rendered element | +| `domscribe.verify.afterEdit` | Compare the element against the baseline — deterministic verdict + per-property deltas | | `domscribe.status` | Relay daemon health, manifest stats, queue counts | See the [`@domscribe/mcp` README](./packages/domscribe-mcp/README.md) for detailed tool schemas, response formats, and prompt definitions. diff --git a/docs/rfcs/0002-post-edit-verify-mcp-tool.md b/docs/rfcs/0002-post-edit-verify-mcp-tool.md new file mode 100644 index 0000000..3f8d684 --- /dev/null +++ b/docs/rfcs/0002-post-edit-verify-mcp-tool.md @@ -0,0 +1,137 @@ +# RFC 0002 — Post-Edit Verification MCP Tools + +- **Status:** Implemented (v1: delta-based verification) +- **Packages:** `@domscribe/core`, `@domscribe/verify`, `@domscribe/relay`, `@domscribe/overlay` +- **Depends on:** RFC 0001 (component-style capture) + +> This document reconstructs the RFC 0002 spec that earlier commits cite +> (annotation schema v3, `verifyHistory`, `VerifyResult`) and records the +> design as implemented. The original draft was removed in a repo cleanup; +> the normative constraints below match the code. + +## Problem + +Coding agents editing UI code cannot reliably tell whether their edit took +effect. The universal failure modes: + +1. **Silent no-ops** — the agent edits the wrong file, a non-applying style + path (specificity, conditional class), or HMR fails, and the agent + declares success anyway. +2. **Vision-blind regressions** — agents that verify by screenshot rely on + a vision model to compare images, and vision models are demonstrably + unreliable at exactly the deltas that matter for styling work (small + offsets, near-identical shades, spacing changes). + +Every mainstream agent stack verifies by "screenshot + eyeball" as of +mid-2026. None offers a deterministic, element-scoped comparison in the dev +inner loop. + +## Design + +Two MCP tools on the relay close the loop deterministically: + +1. **`domscribe.verify.baseline`** — called *before* the edit. Captures a + snapshot of the target element from the live page over the existing + relay↔browser WS channel: the RFC 0001 computed-style allowlist + (≤32 properties) plus the element's bounding rect. Returns an opaque + `baselineId`. Baselines are in-memory and session-scoped (max 100, + oldest evicted) — they describe the live page *now*, so persisting them + would only produce stale comparisons. + +2. **`domscribe.verify.afterEdit`** — called *after* the edit and HMR. + Re-captures the same element, computes deltas against the baseline, and + returns a `VerifyResult` (core schema, annotation schema v3): + - `verdict`: `match` | `partial` | `no_change` | `regression` + - `componentStylesDelta`: per-property `{ property, before, after }` + - `boundingRectDelta`: per-field `{ field, before, after }` (0.5 px + epsilon for sub-pixel jitter) + - `notes`: deterministic explanation + When an `annotationId` is supplied, the result is appended to that + annotation's `context.verifyHistory` (append-only). + +### Deltas first, pixels later + +The v1 verdict is computed from **style and geometry deltas only** — not +pixel diffs. Grounds for this ordering: + +- Computed-style deltas are exact, cheap (no screenshot pipeline), immune + to anti-aliasing noise, and directly actionable in code + (`padding: 8px → 12px` tells the agent what to fix; a red pixel overlay + does not). +- The research consensus (Design2Code's low-level metrics, UI2Code^N's + finding that CLIP-similarity rewards *degrade* refinement, the + VLM-blindness benchmarks) is that element-level structured deltas are the + reliable regression oracle, while holistic visual judgment belongs to + the calling agent. +- The pixel path stays open: `VerifyResultSchema.pixelDiffRatio` and + `screenshotRef` are reserved, and `@domscribe/verify` already ships the + pixelmatch comparator used by the falsifier harness. A future revision + can add element-scoped screenshot capture without changing the contract. + +### Verdict semantics (deterministic, intent-agnostic) + +The tool measures; the agent judges intent. The caller may declare +`expectedChanges: [{ property, value? }]` — the style changes the edit was +meant to make (values compare as `getComputedStyle` strings). + +| Situation | Verdict | +| --- | --- | +| No style and no geometry delta | `no_change` | +| No expectations declared, something changed | `match` + caveat note (change detection only) | +| All expectations met, nothing unexpected | `match` | +| All expectations met, extra properties changed | `partial` (unexpected properties listed) | +| Some expectations met (incl. wrong-value changes) | `partial` | +| No expectation met, other properties changed | `regression` | + +Geometry deltas never downgrade a verdict on their own — rect movement is +usually a consequence of an intended style change (padding grows the box). +They are always reported. + +### Degradation without `captureStyles` + +Style capture is gated on the runtime's `captureStyles` flag (default off +in v0.x per RFC 0001). Without it, verification falls back to geometry-only +change detection: `expectedChanges` cannot be evaluated and the result +notes say so. The baseline response reports `hasComponentStyles` so agents +can prompt the user to enable the flag when full deltas matter. + +## Wire surface + +- `POST /api/v1/verify/baseline` `{ entryId }` → + `{ captured, baselineId?, browserConnected, hasComponentStyles?, hasBoundingRect?, error? }` +- `POST /api/v1/verify/check` `{ baselineId, expectedChanges?, annotationId? }` → + `{ verified, browserConnected, result?: VerifyResult, error? }` + +The WS `context:response` payload gains an optional +`elementInfo.boundingRect` (serialized `DOMRect`), captured by the overlay's +relay service. + +## Delta engine + +`@domscribe/verify` exports the pure functions the relay uses (no DOM, no +I/O — unit-testable in isolation): + +- `diffStyleMaps(before, after): StylePropertyDelta[]` +- `diffBoundingRects(before, after, epsilon?): BoundingRectDelta[]` +- `resolveVerdict({ styleDeltas, rectDeltas, expectedChanges? })` + +## Falsifier gate + +RFC 0002's success criterion is a ≥60% retry-resolution rate on the styling +falsifier corpus: after a failed first attempt, an agent given the +`VerifyResult` deltas should resolve the task on retry at least 60% of the +time. Measuring this requires the agent-driving falsifier mode +(`--mode=agent`), which is tracked separately — see +`docs/sprints/3071-rfc-0001-baseline.md` for the harness gap analysis. + +## Agent workflow + +``` +1. domscribe.query.bySource / domscribe.resolve → entryId +2. domscribe.verify.baseline { entryId } → baselineId +3. edit source, wait for HMR +4. domscribe.verify.afterEdit { baselineId, expectedChanges } +5. verdict = no_change? → the edit did not land; fix and repeat 3–4 + verdict = partial/regression? → consult deltas; fix and repeat 3–4 + verdict = match? → done (agent confirms intent visually if it can) +``` diff --git a/packages/domscribe-core/src/lib/constants/index.ts b/packages/domscribe-core/src/lib/constants/index.ts index 1535b1c..b5193ad 100644 --- a/packages/domscribe-core/src/lib/constants/index.ts +++ b/packages/domscribe-core/src/lib/constants/index.ts @@ -26,6 +26,10 @@ export const API_PATHS = { MANIFEST_RESOLVE_BATCH: '/manifest/resolve/batch', MANIFEST_RESOLVE_BY_SOURCE: '/manifest/resolve-by-source', + // Verify endpoints (RFC 0002) + VERIFY_BASELINE: '/verify/baseline', + VERIFY_CHECK: '/verify/check', + // System endpoints STATUS: `/status`, HEALTH: `/health`, diff --git a/packages/domscribe-mcp/README.md b/packages/domscribe-mcp/README.md index 900a2a4..4f16a46 100644 --- a/packages/domscribe-mcp/README.md +++ b/packages/domscribe-mcp/README.md @@ -68,6 +68,15 @@ Annotations are created when a developer clicks an element in the Domscribe over | `domscribe.annotation.list` | List annotations with status and filter options | | `domscribe.annotation.search` | Full-text search across annotation content | +### Post-Edit Verification + +Deterministic verification that a UI edit actually took effect (RFC 0002). Capture a baseline before editing, edit, then verify — the verdict and per-property deltas are exact measurements, not vision-model judgments. + +| Tool | Description | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------ | +| `domscribe.verify.baseline` | Capture a pre-edit snapshot (computed styles + geometry) of a rendered element; returns a `baselineId` | +| `domscribe.verify.afterEdit` | Re-capture and compare against the baseline; returns a verdict (`match`/`partial`/`no_change`/`regression`) with style deltas | + ### System | Tool | Description | diff --git a/packages/domscribe-overlay/src/services/relay-service.ts b/packages/domscribe-overlay/src/services/relay-service.ts index b5a2fc7..2882bf0 100644 --- a/packages/domscribe-overlay/src/services/relay-service.ts +++ b/packages/domscribe-overlay/src/services/relay-service.ts @@ -159,6 +159,9 @@ export class RelayService { ), ), innerText: elementInfo.element?.innerText?.slice(0, 500), + boundingRect: this.serializeBoundingRect( + elementInfo.element, + ), } : undefined, }; @@ -188,6 +191,38 @@ export class RelayService { return true; } + /** + * Serialize an element's bounding rect into a plain JSON-safe object. + * DOMRect is not JSON-serializable directly (its fields are getters). + */ + private serializeBoundingRect(element: HTMLElement | undefined): + | { + x: number; + y: number; + width: number; + height: number; + top: number; + right: number; + bottom: number; + left: number; + } + | undefined { + if (!element || typeof element.getBoundingClientRect !== 'function') { + return undefined; + } + const rect = element.getBoundingClientRect(); + return { + x: rect.x, + y: rect.y, + width: rect.width, + height: rect.height, + top: rect.top, + right: rect.right, + bottom: rect.bottom, + left: rect.left, + }; + } + /** * Refresh annotations from server */ diff --git a/packages/domscribe-relay/package.json b/packages/domscribe-relay/package.json index b48995f..8da2d72 100644 --- a/packages/domscribe-relay/package.json +++ b/packages/domscribe-relay/package.json @@ -24,6 +24,7 @@ "@clack/prompts": "^1.1.0", "@domscribe/core": "workspace:*", "@domscribe/manifest": "workspace:*", + "@domscribe/verify": "workspace:*", "@fastify/cors": "^10.0.0", "@fastify/websocket": "^11.0.0", "@modelcontextprotocol/sdk": "^1.0.0", diff --git a/packages/domscribe-relay/src/client/relay-http-client.ts b/packages/domscribe-relay/src/client/relay-http-client.ts index 0295862..a021874 100644 --- a/packages/domscribe-relay/src/client/relay-http-client.ts +++ b/packages/domscribe-relay/src/client/relay-http-client.ts @@ -51,6 +51,11 @@ import { ShutdownResponseSchema, StatusResponse, StatusResponseSchema, + ExpectedChangeInput, + VerifyBaselineResponse, + VerifyBaselineResponseSchema, + VerifyCheckResponse, + VerifyCheckResponseSchema, } from '../schema.js'; import { RelayErrorResponse, @@ -354,6 +359,44 @@ export class RelayHttpClient { return QueryBySourceResponseSchema.parse(await response.json()); } + async verifyBaseline(params: { + entryId: ManifestEntryId; + }): Promise { + const apiPath = `${API_PATHS.BASE.replace(':version', 'v1')}${API_PATHS.VERIFY_BASELINE}`; + const url = new URL(apiPath, this.baseUrl); + const response = await fetch(url.toString(), { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + }, + body: JSON.stringify(params), + }); + if (!response.ok) { + throw await this.parseError(response); + } + return VerifyBaselineResponseSchema.parse(await response.json()); + } + + async verifyCheck(params: { + baselineId: string; + expectedChanges?: ExpectedChangeInput[]; + annotationId?: string; + }): Promise { + const apiPath = `${API_PATHS.BASE.replace(':version', 'v1')}${API_PATHS.VERIFY_CHECK}`; + const url = new URL(apiPath, this.baseUrl); + const response = await fetch(url.toString(), { + method: 'POST', + headers: { + 'Content-Type': 'application/json', + }, + body: JSON.stringify(params), + }); + if (!response.ok) { + throw await this.parseError(response); + } + return VerifyCheckResponseSchema.parse(await response.json()); + } + async getManifestStats(): Promise { const apiPath = `${API_PATHS.BASE.replace(':version', 'v1')}${API_PATHS.MANIFEST_STATS}`; const url = new URL(apiPath, this.baseUrl); diff --git a/packages/domscribe-relay/src/mcp/__test-utils__/mock-relay-client.ts b/packages/domscribe-relay/src/mcp/__test-utils__/mock-relay-client.ts index 1a9a719..c1dd92a 100644 --- a/packages/domscribe-relay/src/mcp/__test-utils__/mock-relay-client.ts +++ b/packages/domscribe-relay/src/mcp/__test-utils__/mock-relay-client.ts @@ -24,6 +24,8 @@ export function createMockRelayClient( deleteAnnotation: vi.fn(), patchAnnotation: vi.fn(), queryBySource: vi.fn(), + verifyBaseline: vi.fn(), + verifyCheck: vi.fn(), getStatus: vi.fn(), getHealth: vi.fn(), shutdown: vi.fn(), diff --git a/packages/domscribe-relay/src/mcp/mcp-adapter.spec.ts b/packages/domscribe-relay/src/mcp/mcp-adapter.spec.ts index 8eec9f0..9496128 100644 --- a/packages/domscribe-relay/src/mcp/mcp-adapter.spec.ts +++ b/packages/domscribe-relay/src/mcp/mcp-adapter.spec.ts @@ -50,7 +50,7 @@ function getServer(adapter: McpAdapter) { describe('McpAdapter', () => { describe('active mode', () => { - it('should register all 12 tools', () => { + it('should register all 14 tools', () => { // Act const adapter = new McpAdapter({ mode: 'active', @@ -60,7 +60,7 @@ describe('McpAdapter', () => { // Assert const server = getServer(adapter); - expect(server.registeredTools.size).toBe(12); + expect(server.registeredTools.size).toBe(14); expect(server.registeredTools.has('domscribe.resolve')).toBe(true); expect(server.registeredTools.has('domscribe.resolve.batch')).toBe(true); expect(server.registeredTools.has('domscribe.manifest.stats')).toBe(true); @@ -83,6 +83,12 @@ describe('McpAdapter', () => { ); expect(server.registeredTools.has('domscribe.status')).toBe(true); expect(server.registeredTools.has('domscribe.query.bySource')).toBe(true); + expect(server.registeredTools.has('domscribe.verify.baseline')).toBe( + true, + ); + expect(server.registeredTools.has('domscribe.verify.afterEdit')).toBe( + true, + ); }); it('should register all 4 prompts', () => { diff --git a/packages/domscribe-relay/src/mcp/mcp-adapter.ts b/packages/domscribe-relay/src/mcp/mcp-adapter.ts index 1f751d2..2f4b23a 100644 --- a/packages/domscribe-relay/src/mcp/mcp-adapter.ts +++ b/packages/domscribe-relay/src/mcp/mcp-adapter.ts @@ -25,6 +25,8 @@ import { AnnotationsRespondTool } from './tools/annotation-respond.tool.js'; import { AnnotationsSearchTool } from './tools/annotation-search.tool.js'; import { StatusTool } from './tools/status.tool.js'; import { QueryBySourceTool } from './tools/query-by-source.tool.js'; +import { VerifyBaselineTool } from './tools/verify-baseline.tool.js'; +import { VerifyAfterEditTool } from './tools/verify-after-edit.tool.js'; // Prompt classes import { ProcessNextPrompt } from './prompts/process-next.prompt.js'; @@ -113,6 +115,8 @@ export class McpAdapter { new AnnotationsSearchTool(relayHttpClient), new StatusTool(relayHttpClient), new QueryBySourceTool(relayHttpClient), + new VerifyBaselineTool(relayHttpClient), + new VerifyAfterEditTool(relayHttpClient), ]; for (const tool of tools) { diff --git a/packages/domscribe-relay/src/mcp/tools/tool.defs.spec.ts b/packages/domscribe-relay/src/mcp/tools/tool.defs.spec.ts index ec61e0d..4c32d10 100644 --- a/packages/domscribe-relay/src/mcp/tools/tool.defs.spec.ts +++ b/packages/domscribe-relay/src/mcp/tools/tool.defs.spec.ts @@ -17,6 +17,8 @@ describe('tool.defs', () => { expect(MCP_TOOLS.ANNOTATION_PROCESS).toBe('domscribe.annotation.process'); expect(MCP_TOOLS.ANNOTATION_RESPOND).toBe('domscribe.annotation.respond'); expect(MCP_TOOLS.ANNOTATION_SEARCH).toBe('domscribe.annotation.search'); + expect(MCP_TOOLS.VERIFY_BASELINE).toBe('domscribe.verify.baseline'); + expect(MCP_TOOLS.VERIFY_AFTER_EDIT).toBe('domscribe.verify.afterEdit'); expect(MCP_TOOLS.STATUS).toBe('domscribe.status'); }); }); diff --git a/packages/domscribe-relay/src/mcp/tools/tool.defs.ts b/packages/domscribe-relay/src/mcp/tools/tool.defs.ts index c1524b8..e55bd63 100644 --- a/packages/domscribe-relay/src/mcp/tools/tool.defs.ts +++ b/packages/domscribe-relay/src/mcp/tools/tool.defs.ts @@ -25,6 +25,9 @@ export const MCP_TOOLS = { ANNOTATION_SEARCH: 'domscribe.annotation.search', // Query tools QUERY_BY_SOURCE: 'domscribe.query.bySource', + // Verify tools (RFC 0002) + VERIFY_BASELINE: 'domscribe.verify.baseline', + VERIFY_AFTER_EDIT: 'domscribe.verify.afterEdit', // System tools STATUS: 'domscribe.status', } as const; diff --git a/packages/domscribe-relay/src/mcp/tools/verify-after-edit.tool.spec.ts b/packages/domscribe-relay/src/mcp/tools/verify-after-edit.tool.spec.ts new file mode 100644 index 0000000..fd9b944 --- /dev/null +++ b/packages/domscribe-relay/src/mcp/tools/verify-after-edit.tool.spec.ts @@ -0,0 +1,162 @@ +import type { CallToolResult } from '@modelcontextprotocol/sdk/types.js'; +import { VerifyAfterEditTool } from './verify-after-edit.tool.js'; +import { + createMockRelayClient, + getResultText, +} from '../__test-utils__/mock-relay-client.js'; +import { MCP_TOOLS } from './tool.defs.js'; + +describe('VerifyAfterEditTool', () => { + it('should expose the correct tool name', () => { + // Arrange + const tool = new VerifyAfterEditTool(createMockRelayClient()); + + // Assert + expect(tool.name).toBe(MCP_TOOLS.VERIFY_AFTER_EDIT); + }); + + describe('toolCallback', () => { + it('should return the verify result on a match verdict without a hint', async () => { + // Arrange + const verifyResult = { + verdict: 'match', + timestamp: '2026-07-27T00:00:00.000Z', + componentStylesDelta: [ + { property: 'padding', before: '8px', after: '12px' }, + ], + boundingRectDelta: [], + }; + const mockClient = createMockRelayClient({ + verifyCheck: vi.fn().mockResolvedValue({ + verified: true, + browserConnected: true, + result: verifyResult, + }), + }); + const tool = new VerifyAfterEditTool(mockClient); + + // Act + const result: CallToolResult = await tool.toolCallback({ + baselineId: 'vb_123', + expectedChanges: [{ property: 'padding', value: '12px' }], + }); + + // Assert + expect(mockClient.verifyCheck).toHaveBeenCalledWith({ + baselineId: 'vb_123', + expectedChanges: [{ property: 'padding', value: '12px' }], + }); + expect(result.structuredContent).toEqual({ + verified: true, + browserConnected: true, + result: verifyResult, + error: undefined, + hint: undefined, + }); + expect(JSON.parse(getResultText(result))).toEqual( + result.structuredContent, + ); + }); + + it('should hint that the edit did not land on a no_change verdict', async () => { + // Arrange + const mockClient = createMockRelayClient({ + verifyCheck: vi.fn().mockResolvedValue({ + verified: true, + browserConnected: true, + result: { + verdict: 'no_change', + timestamp: '2026-07-27T00:00:00.000Z', + }, + }), + }); + const tool = new VerifyAfterEditTool(mockClient); + + // Act + const result = await tool.toolCallback({ baselineId: 'vb_123' }); + + // Assert + const output = result.structuredContent as { hint?: string }; + expect(output.hint).toContain('did not change'); + expect(output.hint).toContain('hot-reloaded'); + }); + + it('should hint to retry with the same baseline on a partial verdict', async () => { + // Arrange + const mockClient = createMockRelayClient({ + verifyCheck: vi.fn().mockResolvedValue({ + verified: true, + browserConnected: true, + result: { + verdict: 'partial', + timestamp: '2026-07-27T00:00:00.000Z', + }, + }), + }); + const tool = new VerifyAfterEditTool(mockClient); + + // Act + const result = await tool.toolCallback({ baselineId: 'vb_123' }); + + // Assert + const output = result.structuredContent as { hint?: string }; + expect(output.hint).toContain('re-verify with the same baselineId'); + }); + + it('should hint about the wrong element on a regression verdict', async () => { + // Arrange + const mockClient = createMockRelayClient({ + verifyCheck: vi.fn().mockResolvedValue({ + verified: true, + browserConnected: true, + result: { + verdict: 'regression', + timestamp: '2026-07-27T00:00:00.000Z', + }, + }), + }); + const tool = new VerifyAfterEditTool(mockClient); + + // Act + const result = await tool.toolCallback({ baselineId: 'vb_123' }); + + // Assert + const output = result.structuredContent as { hint?: string }; + expect(output.hint).toContain('not how you intended'); + }); + + it('should hint about the browser when no client is connected', async () => { + // Arrange + const mockClient = createMockRelayClient({ + verifyCheck: vi.fn().mockResolvedValue({ + verified: false, + browserConnected: false, + error: 'No browser is connected', + }), + }); + const tool = new VerifyAfterEditTool(mockClient); + + // Act + const result = await tool.toolCallback({ baselineId: 'vb_123' }); + + // Assert + const output = result.structuredContent as { hint?: string }; + expect(output.hint).toContain('open the page in their browser'); + }); + + it('should return a structured MCP error when the client throws', async () => { + // Arrange + const mockClient = createMockRelayClient({ + verifyCheck: vi.fn().mockRejectedValue(new Error('relay down')), + }); + const tool = new VerifyAfterEditTool(mockClient); + + // Act + const result = await tool.toolCallback({ baselineId: 'vb_123' }); + + // Assert + expect(result.isError).toBe(true); + expect(getResultText(result)).toContain('relay down'); + }); + }); +}); diff --git a/packages/domscribe-relay/src/mcp/tools/verify-after-edit.tool.ts b/packages/domscribe-relay/src/mcp/tools/verify-after-edit.tool.ts new file mode 100644 index 0000000..028b3c1 --- /dev/null +++ b/packages/domscribe-relay/src/mcp/tools/verify-after-edit.tool.ts @@ -0,0 +1,127 @@ +import { z } from 'zod'; +import { VerifyResultSchema } from '@domscribe/core'; +import { + McpToolDefinition, + McpToolOutputSchema, + MCP_TOOLS, + mcpErrorResult, +} from './tool.defs.js'; +import { RelayHttpClient } from '../../client/relay-http-client.js'; + +const VerifyAfterEditToolInputSchema = z.object({ + baselineId: z + .string() + .describe('Baseline handle returned by domscribe.verify.baseline'), + expectedChanges: z + .array( + z.object({ + property: z + .string() + .describe('CSS property you intended to change (e.g. "padding")'), + value: z + .string() + .optional() + .describe( + 'Expected resolved value as getComputedStyle reports it (e.g. "12px", "rgb(255, 0, 0)"). Omit to only assert the property changed.', + ), + }), + ) + .optional() + .describe( + 'Declare what your edit was meant to change. With expectations the verdict distinguishes match/partial/regression; without them it only distinguishes changed vs no_change.', + ), + annotationId: z + .string() + .optional() + .describe( + 'When processing an annotation, pass its ID to append this verify result to the annotation’s verifyHistory', + ), +}); + +type VerifyAfterEditToolInput = z.infer; + +const VerifyAfterEditToolOutputSchema = McpToolOutputSchema.extend({ + verified: z.boolean(), + browserConnected: z.boolean().optional(), + result: VerifyResultSchema.optional(), + hint: z + .string() + .optional() + .describe('Actionable guidance based on the verdict'), +}); + +type VerifyAfterEditToolOutput = z.infer< + typeof VerifyAfterEditToolOutputSchema +>; + +export class VerifyAfterEditTool implements McpToolDefinition< + typeof VerifyAfterEditToolInputSchema, + typeof VerifyAfterEditToolOutputSchema +> { + name = MCP_TOOLS.VERIFY_AFTER_EDIT; + description = + 'Verify that a UI edit actually took effect, AFTER editing source and letting HMR apply it. ' + + 'Re-captures the element snapshotted by domscribe.verify.baseline and returns a deterministic verdict with per-property deltas: ' + + '`no_change` means your edit did not reach the rendered element (wrong file, wrong selector, or HMR miss) — do not claim success on a no_change verdict. ' + + '`match`/`partial`/`regression` are computed from your declared expectedChanges. ' + + 'The deltas are exact measurements, not judgments — you decide whether an unexpected delta is acceptable. ' + + 'Style values compare as getComputedStyle strings (colors as rgb(...), lengths resolved to px). ' + + 'Call within a few seconds of the edit so the page state has not drifted.'; + inputSchema = VerifyAfterEditToolInputSchema; + outputSchema = VerifyAfterEditToolOutputSchema; + + constructor(private readonly relayHttpClient: RelayHttpClient) {} + + private buildHint(result: { + verified: boolean; + browserConnected: boolean; + result?: { verdict: string }; + }): string | undefined { + if (!result.verified && !result.browserConnected) { + return 'No browser is connected. Ask the user to open the page in their browser, then retry.'; + } + if (!result.verified) { + return undefined; + } + switch (result.result?.verdict) { + case 'no_change': + return ( + 'The rendered element did not change. Check that you edited the file the manifest points at, ' + + 'that the style path you changed actually applies to this element (specificity, conditional classes), ' + + 'and that the dev server hot-reloaded. Re-verify after fixing.' + ); + case 'partial': + return 'Some of the intended changes applied. Consult the deltas, adjust the edit, and re-verify with the same baselineId.'; + case 'regression': + return 'The element changed, but not how you intended. Your edit likely hit the wrong element or property — review the deltas before retrying.'; + default: + return undefined; + } + } + + async toolCallback(input: VerifyAfterEditToolInput) { + try { + const result = await this.relayHttpClient.verifyCheck(input); + + const output: VerifyAfterEditToolOutput = { + verified: result.verified, + browserConnected: result.browserConnected, + result: result.result, + error: result.error, + hint: this.buildHint(result), + }; + + return { + structuredContent: output, + content: [ + { + type: 'text' as const, + text: JSON.stringify(output, null, 2), + }, + ], + }; + } catch (error: unknown) { + return mcpErrorResult(error); + } + } +} diff --git a/packages/domscribe-relay/src/mcp/tools/verify-baseline.tool.spec.ts b/packages/domscribe-relay/src/mcp/tools/verify-baseline.tool.spec.ts new file mode 100644 index 0000000..c33f028 --- /dev/null +++ b/packages/domscribe-relay/src/mcp/tools/verify-baseline.tool.spec.ts @@ -0,0 +1,132 @@ +import type { CallToolResult } from '@modelcontextprotocol/sdk/types.js'; +import { VerifyBaselineTool } from './verify-baseline.tool.js'; +import { + createMockRelayClient, + getResultText, +} from '../__test-utils__/mock-relay-client.js'; +import { MCP_TOOLS } from './tool.defs.js'; + +describe('VerifyBaselineTool', () => { + it('should expose the correct tool name', () => { + // Arrange + const tool = new VerifyBaselineTool(createMockRelayClient()); + + // Assert + expect(tool.name).toBe(MCP_TOOLS.VERIFY_BASELINE); + }); + + describe('toolCallback', () => { + it('should return the baselineId for a successful capture', async () => { + // Arrange + const mockClient = createMockRelayClient({ + verifyBaseline: vi.fn().mockResolvedValue({ + captured: true, + baselineId: 'vb_123', + entryId: 'aB3dEf7h', + capturedAt: '2026-07-27T00:00:00.000Z', + browserConnected: true, + hasComponentStyles: true, + hasBoundingRect: true, + }), + }); + const tool = new VerifyBaselineTool(mockClient); + + // Act + const result: CallToolResult = await tool.toolCallback({ + entryId: 'aB3dEf7h', + }); + + // Assert + expect(mockClient.verifyBaseline).toHaveBeenCalledWith({ + entryId: 'aB3dEf7h', + }); + expect(result.structuredContent).toEqual({ + captured: true, + baselineId: 'vb_123', + capturedAt: '2026-07-27T00:00:00.000Z', + browserConnected: true, + hasComponentStyles: true, + hasBoundingRect: true, + error: undefined, + hint: undefined, + }); + expect(JSON.parse(getResultText(result))).toEqual( + result.structuredContent, + ); + }); + + it('should hint about the browser when no client is connected', async () => { + // Arrange + const mockClient = createMockRelayClient({ + verifyBaseline: vi.fn().mockResolvedValue({ + captured: false, + browserConnected: false, + error: 'No browser is connected', + }), + }); + const tool = new VerifyBaselineTool(mockClient); + + // Act + const result = await tool.toolCallback({ entryId: 'aB3dEf7h' }); + + // Assert + const output = result.structuredContent as { hint?: string }; + expect(output.hint).toContain('open the page in their browser'); + }); + + it('should hint about navigation when the element is not rendered', async () => { + // Arrange + const mockClient = createMockRelayClient({ + verifyBaseline: vi.fn().mockResolvedValue({ + captured: false, + browserConnected: true, + error: 'Element is not currently rendered', + }), + }); + const tool = new VerifyBaselineTool(mockClient); + + // Act + const result = await tool.toolCallback({ entryId: 'aB3dEf7h' }); + + // Assert + const output = result.structuredContent as { hint?: string }; + expect(output.hint).toContain('not currently rendered'); + }); + + it('should hint about captureStyles when the baseline has geometry only', async () => { + // Arrange + const mockClient = createMockRelayClient({ + verifyBaseline: vi.fn().mockResolvedValue({ + captured: true, + baselineId: 'vb_456', + browserConnected: true, + hasComponentStyles: false, + hasBoundingRect: true, + }), + }); + const tool = new VerifyBaselineTool(mockClient); + + // Act + const result = await tool.toolCallback({ entryId: 'aB3dEf7h' }); + + // Assert + const output = result.structuredContent as { hint?: string }; + expect(output.hint).toContain('captureStyles'); + }); + + it('should return a structured MCP error when the client throws', async () => { + // Arrange + const mockClient = createMockRelayClient({ + verifyBaseline: vi.fn().mockRejectedValue(new Error('relay down')), + }); + const tool = new VerifyBaselineTool(mockClient); + + // Act + const result = await tool.toolCallback({ entryId: 'aB3dEf7h' }); + + // Assert + expect(result.isError).toBe(true); + expect(getResultText(result)).toContain('relay down'); + }); + }); +}); diff --git a/packages/domscribe-relay/src/mcp/tools/verify-baseline.tool.ts b/packages/domscribe-relay/src/mcp/tools/verify-baseline.tool.ts new file mode 100644 index 0000000..88a0244 --- /dev/null +++ b/packages/domscribe-relay/src/mcp/tools/verify-baseline.tool.ts @@ -0,0 +1,103 @@ +import { z } from 'zod'; +import { + McpToolDefinition, + McpToolOutputSchema, + MCP_TOOLS, + mcpErrorResult, +} from './tool.defs.js'; +import { RelayHttpClient } from '../../client/relay-http-client.js'; + +const VerifyBaselineToolInputSchema = z.object({ + entryId: z + .string() + .describe( + 'Manifest entry ID (data-ds attribute value) of the element you are about to edit. ' + + 'Get it from domscribe.resolve, domscribe.query.bySource, or an annotation.', + ), +}); + +type VerifyBaselineToolInput = z.infer; + +const VerifyBaselineToolOutputSchema = McpToolOutputSchema.extend({ + captured: z.boolean(), + baselineId: z + .string() + .optional() + .describe('Pass this to domscribe.verify.afterEdit once you have edited'), + capturedAt: z.string().optional(), + browserConnected: z.boolean().optional(), + hasComponentStyles: z.boolean().optional(), + hasBoundingRect: z.boolean().optional(), + hint: z + .string() + .optional() + .describe('Actionable guidance based on the result'), +}); + +type VerifyBaselineToolOutput = z.infer; + +export class VerifyBaselineTool implements McpToolDefinition< + typeof VerifyBaselineToolInputSchema, + typeof VerifyBaselineToolOutputSchema +> { + name = MCP_TOOLS.VERIFY_BASELINE; + description = + 'Capture a pre-edit baseline snapshot (computed styles + geometry) of a rendered element, BEFORE editing its source. ' + + 'Call this at the start of any styling or layout change, then edit, then call domscribe.verify.afterEdit with the returned baselineId to get a deterministic verdict on whether your edit took effect. ' + + 'The snapshot is captured from the live page in the user’s browser, so the element must currently be rendered. ' + + 'Baselines are session-scoped — they do not survive a relay restart, and stale baselines are evicted after 100 captures.'; + inputSchema = VerifyBaselineToolInputSchema; + outputSchema = VerifyBaselineToolOutputSchema; + + constructor(private readonly relayHttpClient: RelayHttpClient) {} + + private buildHint(result: { + captured: boolean; + browserConnected: boolean; + hasComponentStyles?: boolean; + }): string | undefined { + if (!result.captured && !result.browserConnected) { + return 'No browser is connected. Ask the user to open the page in their browser, then retry.'; + } + if (!result.captured) { + return 'The element is not currently rendered. Ask the user to navigate to a page that renders it, then retry.'; + } + if (result.hasComponentStyles === false) { + return ( + 'Baseline captured with geometry only — computed styles are unavailable because the runtime is not configured with `captureStyles: true`. ' + + 'Verification will still detect whether the element changed, but without per-property style deltas. ' + + 'For full deltas, ask the user to enable captureStyles and re-capture the baseline.' + ); + } + return undefined; + } + + async toolCallback(input: VerifyBaselineToolInput) { + try { + const result = await this.relayHttpClient.verifyBaseline(input); + + const output: VerifyBaselineToolOutput = { + captured: result.captured, + baselineId: result.baselineId, + capturedAt: result.capturedAt, + browserConnected: result.browserConnected, + hasComponentStyles: result.hasComponentStyles, + hasBoundingRect: result.hasBoundingRect, + error: result.error, + hint: this.buildHint(result), + }; + + return { + structuredContent: output, + content: [ + { + type: 'text' as const, + text: JSON.stringify(output, null, 2), + }, + ], + }; + } catch (error: unknown) { + return mcpErrorResult(error); + } + } +} diff --git a/packages/domscribe-relay/src/schema.ts b/packages/domscribe-relay/src/schema.ts index 33fde0b..eff7e74 100644 --- a/packages/domscribe-relay/src/schema.ts +++ b/packages/domscribe-relay/src/schema.ts @@ -13,6 +13,7 @@ import { AnnotationIdSchema, AnnotationInteractionSchema, AnnotationSchema, + BoundingRectSchema, InteractionModeSchema, ManifestEntryIdSchema, ManifestEntrySchema, @@ -21,6 +22,7 @@ import { SourcePositionSchema, AnnotationStatusSchema, AnnotationSummarySchema, + VerifyResultSchema, } from '@domscribe/core'; /* ============================= @@ -395,12 +397,87 @@ export const WSContextResponseSchema = z.object({ tagName: z.string().optional(), attributes: z.record(z.string(), z.string()).optional(), innerText: z.string().optional(), + boundingRect: BoundingRectSchema.optional(), }) .optional(), error: z.string().optional(), }); export type WSContextResponse = z.infer; +/* ============================= + * Verify (RFC 0002) + * ============================= */ +export const VerifyBaselineRequestSchema = z.object({ + entryId: ManifestEntryIdSchema.describe( + 'Manifest entry ID (data-ds attribute value) of the element about to be edited', + ), +}); +export type VerifyBaselineRequest = z.infer; + +export const VerifyBaselineResponseSchema = z.object({ + captured: z.boolean().describe('Whether a baseline snapshot was stored'), + baselineId: z + .string() + .optional() + .describe('Opaque handle to pass to the verify check call'), + entryId: ManifestEntryIdSchema.optional(), + capturedAt: z.string().optional().describe('ISO 8601 capture timestamp'), + browserConnected: z.boolean().describe('Whether a browser client was connected'), + hasComponentStyles: z + .boolean() + .optional() + .describe( + 'Whether the baseline includes a computed-style snapshot (requires runtime captureStyles: true)', + ), + hasBoundingRect: z + .boolean() + .optional() + .describe('Whether the baseline includes element geometry'), + error: z.string().optional(), +}); +export type VerifyBaselineResponse = z.infer< + typeof VerifyBaselineResponseSchema +>; + +export const ExpectedChangeSchema = z.object({ + property: z + .string() + .describe('CSS property expected to change (e.g. "padding")'), + value: z + .string() + .optional() + .describe( + 'Expected resolved value as reported by getComputedStyle (e.g. "12px", "rgb(255, 0, 0)"). Omit to only assert that the property changed.', + ), +}); +export type ExpectedChangeInput = z.infer; + +export const VerifyCheckRequestSchema = z.object({ + baselineId: z + .string() + .describe('Baseline handle returned by the baseline capture call'), + expectedChanges: z + .array(ExpectedChangeSchema) + .optional() + .describe('Declared intent — the style changes the edit was meant to make'), + annotationId: AnnotationIdSchema.optional().describe( + 'When set, the verify result is appended to this annotation’s verifyHistory', + ), +}); +export type VerifyCheckRequest = z.infer; + +export const VerifyCheckResponseSchema = z.object({ + verified: z + .boolean() + .describe('Whether a post-edit snapshot was captured and compared'), + browserConnected: z.boolean(), + result: VerifyResultSchema.optional().describe( + 'Structured verify result (verdict + deltas) when verification ran', + ), + error: z.string().optional(), +}); +export type VerifyCheckResponse = z.infer; + /* ============================= * Query By Source * ============================= */ diff --git a/packages/domscribe-relay/src/server/http-server.ts b/packages/domscribe-relay/src/server/http-server.ts index e4637b3..04bd91c 100644 --- a/packages/domscribe-relay/src/server/http-server.ts +++ b/packages/domscribe-relay/src/server/http-server.ts @@ -15,7 +15,11 @@ import { } from '@domscribe/core'; import path from 'path'; import { ManifestReader } from '@domscribe/manifest'; -import { AnnotationService, FileAnnotationStorage } from './services/index.js'; +import { + AnnotationService, + FileAnnotationStorage, + VerifyService, +} from './services/index.js'; import { registerManifestHandlers, registerAnnotationHandlers, @@ -24,7 +28,11 @@ import { registerShutdownHandler, } from './handlers/index.js'; import { createWSServer, type WSServer } from './ws-server.js'; -import { QueryBySourceRoute } from './routes/index.js'; +import { + QueryBySourceRoute, + VerifyBaselineRoute, + VerifyCheckRoute, +} from './routes/index.js'; import { registerRoute } from './routes/route.interface.js'; import { serializerCompiler, @@ -149,6 +157,10 @@ export async function createRelayServer( // Register routes that depend on WebSocket server registerRoute(QueryBySourceRoute, { app, manifestReader, wsServer: ws }); + const verifyService = new VerifyService(ws, annotationService); + registerRoute(VerifyBaselineRoute, { app, verifyService }); + registerRoute(VerifyCheckRoute, { app, verifyService }); + // Error handler app.setErrorHandler((error: FastifyError, _request, reply) => { const statusCode = error.statusCode ?? HTTP_STATUS.INTERNAL_SERVER_ERROR; diff --git a/packages/domscribe-relay/src/server/routes/index.ts b/packages/domscribe-relay/src/server/routes/index.ts index a1a601b..3fe7ef7 100644 --- a/packages/domscribe-relay/src/server/routes/index.ts +++ b/packages/domscribe-relay/src/server/routes/index.ts @@ -20,3 +20,7 @@ export { ManifestResolveRoute } from './v1/manifest-resolve.route.js'; export { ManifestStatsRoute } from './v1/manifest-stats.route.js'; export { ManifestQueryRoute } from './v1/manifest-query.route.js'; export { QueryBySourceRoute } from './v1/query-by-source.route.js'; + +/** Verify routes (RFC 0002) */ +export { VerifyBaselineRoute } from './v1/verify-baseline.route.js'; +export { VerifyCheckRoute } from './v1/verify-check.route.js'; diff --git a/packages/domscribe-relay/src/server/routes/v1/verify-baseline.route.spec.ts b/packages/domscribe-relay/src/server/routes/v1/verify-baseline.route.spec.ts new file mode 100644 index 0000000..85ec6fa --- /dev/null +++ b/packages/domscribe-relay/src/server/routes/v1/verify-baseline.route.spec.ts @@ -0,0 +1,199 @@ +/** + * Integration tests for POST /api/v1/verify/baseline + * + * Uses a real VerifyService and AnnotationService (in-memory storage). + * WSServer is mocked since the test harness doesn't wire WebSocket. + */ +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; +import Fastify, { type FastifyInstance, type FastifyError } from 'fastify'; +import cors from '@fastify/cors'; +import { + serializerCompiler, + validatorCompiler, +} from 'fastify-type-provider-zod'; +import { HTTP_STATUS, DomscribeErrorCode } from '@domscribe/core'; +import { VerifyBaselineRoute } from './verify-baseline.route.js'; +import { registerRoute } from '../route.interface.js'; +import { + AnnotationService, + InMemoryAnnotationStorage, + VerifyService, +} from '../../services/index.js'; +import type { WSServer } from '../../ws-server.js'; +import type { WSContextResponse } from '../../../schema.js'; + +function createMockWSServer(overrides?: Partial): WSServer { + return { + broadcast: vi.fn(), + getClientCount: vi.fn().mockReturnValue(0), + requestContext: vi.fn().mockResolvedValue(null), + close: vi.fn(), + ...overrides, + }; +} + +function createWSResponse( + overrides: Partial = {}, +): WSContextResponse { + return { + requestId: 'req-1', + success: true, + rendered: true, + context: { + componentStyles: { + computed: { padding: '8px', color: 'rgb(0, 0, 0)' }, + }, + }, + elementInfo: { + tagName: 'button', + boundingRect: { + x: 100, + y: 200, + width: 320, + height: 48, + top: 200, + right: 420, + bottom: 248, + left: 100, + }, + }, + ...overrides, + }; +} + +describe('POST /api/v1/verify/baseline', () => { + let app: FastifyInstance; + let mockWsServer: WSServer; + + beforeAll(async () => { + mockWsServer = createMockWSServer(); + const annotationService = new AnnotationService( + new InMemoryAnnotationStorage(), + ); + await annotationService.initialize(); + const verifyService = new VerifyService(mockWsServer, annotationService); + + app = Fastify({ logger: false }); + app.setValidatorCompiler(validatorCompiler); + app.setSerializerCompiler(serializerCompiler); + await app.register(cors, { origin: true }); + + app.setErrorHandler((error: FastifyError, _request, reply) => { + const statusCode = error.statusCode ?? HTTP_STATUS.INTERNAL_SERVER_ERROR; + reply.status(statusCode).send({ + error: error.message, + code: DomscribeErrorCode.DS_INTERNAL_ERROR, + statusCode, + }); + }); + + registerRoute(VerifyBaselineRoute, { app, verifyService }); + + await app.ready(); + }); + + afterAll(() => { + app.close(); + }); + + it('should report captured:false when no browser is connected', async () => { + // Arrange + vi.mocked(mockWsServer.getClientCount).mockReturnValue(0); + + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/baseline', + payload: { entryId: 'aB3dEf7h' }, + }); + + // Assert + expect(response.statusCode).toBe(200); + const body = response.json(); + expect(body.captured).toBe(false); + expect(body.browserConnected).toBe(false); + expect(body.error).toContain('No browser is connected'); + }); + + it('should capture a baseline with styles and geometry', async () => { + // Arrange + vi.mocked(mockWsServer.getClientCount).mockReturnValue(1); + vi.mocked(mockWsServer.requestContext).mockResolvedValue( + createWSResponse(), + ); + + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/baseline', + payload: { entryId: 'aB3dEf7h' }, + }); + + // Assert + expect(response.statusCode).toBe(200); + const body = response.json(); + expect(body.captured).toBe(true); + expect(body.baselineId).toMatch(/^vb_/); + expect(body.entryId).toBe('aB3dEf7h'); + expect(body.browserConnected).toBe(true); + expect(body.hasComponentStyles).toBe(true); + expect(body.hasBoundingRect).toBe(true); + expect(mockWsServer.requestContext).toHaveBeenCalledWith('aB3dEf7h'); + }); + + it('should capture a geometry-only baseline when styles are unavailable', async () => { + // Arrange + vi.mocked(mockWsServer.getClientCount).mockReturnValue(1); + vi.mocked(mockWsServer.requestContext).mockResolvedValue( + createWSResponse({ context: {} }), + ); + + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/baseline', + payload: { entryId: 'aB3dEf7h' }, + }); + + // Assert + expect(response.statusCode).toBe(200); + const body = response.json(); + expect(body.captured).toBe(true); + expect(body.hasComponentStyles).toBe(false); + expect(body.hasBoundingRect).toBe(true); + }); + + it('should report captured:false when the element is not rendered', async () => { + // Arrange + vi.mocked(mockWsServer.getClientCount).mockReturnValue(1); + vi.mocked(mockWsServer.requestContext).mockResolvedValue( + createWSResponse({ success: false, rendered: false, context: undefined }), + ); + + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/baseline', + payload: { entryId: 'aB3dEf7h' }, + }); + + // Assert + expect(response.statusCode).toBe(200); + const body = response.json(); + expect(body.captured).toBe(false); + expect(body.browserConnected).toBe(true); + expect(body.error).toContain('not currently rendered'); + }); + + it('should return 400 for an invalid entry ID', async () => { + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/baseline', + payload: { entryId: 'not a valid id!' }, + }); + + // Assert + expect(response.statusCode).toBe(400); + }); +}); diff --git a/packages/domscribe-relay/src/server/routes/v1/verify-baseline.route.ts b/packages/domscribe-relay/src/server/routes/v1/verify-baseline.route.ts new file mode 100644 index 0000000..ef08aa8 --- /dev/null +++ b/packages/domscribe-relay/src/server/routes/v1/verify-baseline.route.ts @@ -0,0 +1,89 @@ +import { + API_PATHS, + DomscribeError, + DomscribeErrorCode, + HTTP_STATUS, +} from '@domscribe/core'; +import { + FastifyInstance, + FastifyReply, + FastifyRequest, + HTTPMethods, +} from 'fastify'; +import type { ZodTypeProvider } from 'fastify-type-provider-zod'; +import path from 'path'; +import { + VerifyBaselineRequest, + VerifyBaselineRequestSchema, + VerifyBaselineResponse, + VerifyBaselineResponseSchema, +} from '../../../schema.js'; +import { RelayErrorResponse, RelayErrorResponseSchema } from '../../types.js'; +import { ApiVersion, RelayRoute } from '../route.interface.js'; +import type { VerifyService } from '../../services/verify-service.js'; + +export class VerifyBaselineRoute implements RelayRoute { + apiPath = API_PATHS.VERIFY_BASELINE; + method: HTTPMethods = 'POST'; + version: ApiVersion = 'v1'; + + constructor(private readonly verifyService: VerifyService) {} + + static register({ + app, + verifyService, + }: { + app: FastifyInstance; + verifyService: VerifyService; + }): void { + const route = new VerifyBaselineRoute(verifyService); + const { apiPath, version, method, handler } = route; + const url = path.posix.join( + API_PATHS.BASE.replace(':version', version), + apiPath, + ); + + app.withTypeProvider().route<{ + Body: VerifyBaselineRequest; + Reply: VerifyBaselineResponse | RelayErrorResponse; + }>({ + url, + method, + handler: handler.bind(route), + schema: { + body: VerifyBaselineRequestSchema, + response: { + 200: VerifyBaselineResponseSchema, + 500: RelayErrorResponseSchema, + }, + }, + }); + } + + async handler( + request: FastifyRequest<{ Body: VerifyBaselineRequest }>, + reply: FastifyReply<{ + Reply: VerifyBaselineResponse | RelayErrorResponse; + }>, + ) { + try { + const result = await this.verifyService.captureBaseline( + request.body.entryId, + ); + return reply.status(HTTP_STATUS.OK).send(result); + } catch (error: unknown) { + if (error instanceof DomscribeError) { + return reply.status(HTTP_STATUS.INTERNAL_SERVER_ERROR).send({ + ...error.toProblemDetails(), + error: error.message, + }); + } + const errorMessage = + error instanceof Error ? error.message : 'Unknown error'; + return reply.status(HTTP_STATUS.INTERNAL_SERVER_ERROR).send({ + error: errorMessage, + code: DomscribeErrorCode.DS_INTERNAL_ERROR, + }); + } + } +} diff --git a/packages/domscribe-relay/src/server/routes/v1/verify-check.route.spec.ts b/packages/domscribe-relay/src/server/routes/v1/verify-check.route.spec.ts new file mode 100644 index 0000000..d56e17f --- /dev/null +++ b/packages/domscribe-relay/src/server/routes/v1/verify-check.route.spec.ts @@ -0,0 +1,326 @@ +/** + * Integration tests for POST /api/v1/verify/check + * + * Uses a real VerifyService and AnnotationService (in-memory storage); + * baselines are created through the real baseline route. WSServer is + * mocked since the test harness doesn't wire WebSocket. + */ +import { describe, it, expect, beforeAll, afterAll, vi } from 'vitest'; +import Fastify, { type FastifyInstance, type FastifyError } from 'fastify'; +import cors from '@fastify/cors'; +import { + serializerCompiler, + validatorCompiler, +} from 'fastify-type-provider-zod'; +import { + HTTP_STATUS, + DomscribeErrorCode, + InteractionModeEnum, + InteractionTypeEnum, +} from '@domscribe/core'; +import { VerifyBaselineRoute } from './verify-baseline.route.js'; +import { VerifyCheckRoute } from './verify-check.route.js'; +import { registerRoute } from '../route.interface.js'; +import { + AnnotationService, + InMemoryAnnotationStorage, + VerifyService, +} from '../../services/index.js'; +import type { WSServer } from '../../ws-server.js'; +import type { WSContextResponse } from '../../../schema.js'; + +function createMockWSServer(overrides?: Partial): WSServer { + return { + broadcast: vi.fn(), + getClientCount: vi.fn().mockReturnValue(1), + requestContext: vi.fn().mockResolvedValue(null), + close: vi.fn(), + ...overrides, + }; +} + +function createWSResponse( + computed: Record | undefined, + rectOverrides: Partial<{ + x: number; + y: number; + width: number; + height: number; + top: number; + right: number; + bottom: number; + left: number; + }> = {}, +): WSContextResponse { + return { + requestId: 'req-1', + success: true, + rendered: true, + context: computed ? { componentStyles: { computed } } : {}, + elementInfo: { + tagName: 'button', + boundingRect: { + x: 100, + y: 200, + width: 320, + height: 48, + top: 200, + right: 420, + bottom: 248, + left: 100, + ...rectOverrides, + }, + }, + }; +} + +const BASELINE_STYLES = { padding: '8px', color: 'rgb(0, 0, 0)' }; + +describe('POST /api/v1/verify/check', () => { + let app: FastifyInstance; + let mockWsServer: WSServer; + let annotationService: AnnotationService; + + async function captureBaseline(): Promise { + vi.mocked(mockWsServer.requestContext).mockResolvedValueOnce( + createWSResponse(BASELINE_STYLES), + ); + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/baseline', + payload: { entryId: 'aB3dEf7h' }, + }); + return response.json().baselineId; + } + + beforeAll(async () => { + mockWsServer = createMockWSServer(); + annotationService = new AnnotationService(new InMemoryAnnotationStorage()); + await annotationService.initialize(); + const verifyService = new VerifyService(mockWsServer, annotationService); + + app = Fastify({ logger: false }); + app.setValidatorCompiler(validatorCompiler); + app.setSerializerCompiler(serializerCompiler); + await app.register(cors, { origin: true }); + + app.setErrorHandler((error: FastifyError, _request, reply) => { + const statusCode = error.statusCode ?? HTTP_STATUS.INTERNAL_SERVER_ERROR; + reply.status(statusCode).send({ + error: error.message, + code: DomscribeErrorCode.DS_INTERNAL_ERROR, + statusCode, + }); + }); + + registerRoute(VerifyBaselineRoute, { app, verifyService }); + registerRoute(VerifyCheckRoute, { app, verifyService }); + + await app.ready(); + }); + + afterAll(() => { + app.close(); + }); + + it('should reject an unknown baselineId', async () => { + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/check', + payload: { baselineId: 'vb_does-not-exist' }, + }); + + // Assert + expect(response.statusCode).toBe(200); + const body = response.json(); + expect(body.verified).toBe(false); + expect(body.error).toContain('Unknown baselineId'); + }); + + it('should return no_change when the element did not change', async () => { + // Arrange + const baselineId = await captureBaseline(); + vi.mocked(mockWsServer.requestContext).mockResolvedValueOnce( + createWSResponse(BASELINE_STYLES), + ); + + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/check', + payload: { baselineId }, + }); + + // Assert + expect(response.statusCode).toBe(200); + const body = response.json(); + expect(body.verified).toBe(true); + expect(body.result.verdict).toBe('no_change'); + expect(body.result.componentStylesDelta).toEqual([]); + }); + + it('should return match with deltas when expectations are met', async () => { + // Arrange + const baselineId = await captureBaseline(); + vi.mocked(mockWsServer.requestContext).mockResolvedValueOnce( + createWSResponse( + { padding: '12px', color: 'rgb(0, 0, 0)' }, + { width: 328, right: 428 }, + ), + ); + + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/check', + payload: { + baselineId, + expectedChanges: [{ property: 'padding', value: '12px' }], + }, + }); + + // Assert + expect(response.statusCode).toBe(200); + const body = response.json(); + expect(body.verified).toBe(true); + expect(body.result.verdict).toBe('match'); + expect(body.result.componentStylesDelta).toEqual([ + { property: 'padding', before: '8px', after: '12px' }, + ]); + expect(body.result.boundingRectDelta).toEqual([ + { field: 'width', before: 320, after: 328 }, + { field: 'right', before: 420, after: 428 }, + ]); + }); + + it('should return regression when an unexpected property changed instead', async () => { + // Arrange + const baselineId = await captureBaseline(); + vi.mocked(mockWsServer.requestContext).mockResolvedValueOnce( + createWSResponse({ padding: '8px', color: 'rgb(255, 0, 0)' }), + ); + + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/check', + payload: { + baselineId, + expectedChanges: [{ property: 'padding', value: '12px' }], + }, + }); + + // Assert + expect(response.statusCode).toBe(200); + const body = response.json(); + expect(body.result.verdict).toBe('regression'); + }); + + it('should fall back to geometry-only verification when styles are unavailable', async () => { + // Arrange — baseline without styles + vi.mocked(mockWsServer.requestContext).mockResolvedValueOnce( + createWSResponse(undefined), + ); + const baselineResponse = await app.inject({ + method: 'POST', + url: '/api/v1/verify/baseline', + payload: { entryId: 'aB3dEf7h' }, + }); + const baselineId = baselineResponse.json().baselineId; + + vi.mocked(mockWsServer.requestContext).mockResolvedValueOnce( + createWSResponse(undefined, { height: 60, bottom: 260 }), + ); + + // Act — expectations are declared but cannot be evaluated + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/check', + payload: { + baselineId, + expectedChanges: [{ property: 'padding', value: '12px' }], + }, + }); + + // Assert + expect(response.statusCode).toBe(200); + const body = response.json(); + expect(body.verified).toBe(true); + expect(body.result.verdict).toBe('match'); + expect(body.result.componentStylesDelta).toBeUndefined(); + expect(body.result.boundingRectDelta).toEqual([ + { field: 'height', before: 48, after: 60 }, + { field: 'bottom', before: 248, after: 260 }, + ]); + expect(body.result.notes).toContain('geometry only'); + }); + + it('should report verified:false when the browser disconnected before the check', async () => { + // Arrange + const baselineId = await captureBaseline(); + vi.mocked(mockWsServer.getClientCount).mockReturnValueOnce(0); + + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/check', + payload: { baselineId }, + }); + + // Assert + expect(response.statusCode).toBe(200); + const body = response.json(); + expect(body.verified).toBe(false); + expect(body.browserConnected).toBe(false); + }); + + it('should append the verify result to an annotation verifyHistory', async () => { + // Arrange + const annotation = await annotationService.create({ + mode: InteractionModeEnum.ELEMENT_CLICK, + interaction: { type: InteractionTypeEnum.ELEMENT_ANNOTATION }, + context: { + pageUrl: 'http://localhost:3000/', + pageTitle: 'Fixture', + viewport: { width: 1280, height: 720 }, + userAgent: 'vitest', + }, + }); + const baselineId = await captureBaseline(); + vi.mocked(mockWsServer.requestContext).mockResolvedValueOnce( + createWSResponse({ padding: '12px', color: 'rgb(0, 0, 0)' }), + ); + + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/check', + payload: { + baselineId, + expectedChanges: [{ property: 'padding' }], + annotationId: annotation.metadata.id, + }, + }); + + // Assert + expect(response.statusCode).toBe(200); + expect(response.json().result.verdict).toBe('match'); + + const updated = await annotationService.get(annotation.metadata.id); + expect(updated?.context.verifyHistory).toHaveLength(1); + expect(updated?.context.verifyHistory?.[0].verdict).toBe('match'); + }); + + it('should return 400 when baselineId is missing', async () => { + // Act + const response = await app.inject({ + method: 'POST', + url: '/api/v1/verify/check', + payload: {}, + }); + + // Assert + expect(response.statusCode).toBe(400); + }); +}); diff --git a/packages/domscribe-relay/src/server/routes/v1/verify-check.route.ts b/packages/domscribe-relay/src/server/routes/v1/verify-check.route.ts new file mode 100644 index 0000000..0043c95 --- /dev/null +++ b/packages/domscribe-relay/src/server/routes/v1/verify-check.route.ts @@ -0,0 +1,92 @@ +import { + API_PATHS, + DomscribeError, + DomscribeErrorCode, + HTTP_STATUS, +} from '@domscribe/core'; +import { + FastifyInstance, + FastifyReply, + FastifyRequest, + HTTPMethods, +} from 'fastify'; +import type { ZodTypeProvider } from 'fastify-type-provider-zod'; +import path from 'path'; +import { + VerifyCheckRequest, + VerifyCheckRequestSchema, + VerifyCheckResponse, + VerifyCheckResponseSchema, +} from '../../../schema.js'; +import { RelayErrorResponse, RelayErrorResponseSchema } from '../../types.js'; +import { ApiVersion, RelayRoute } from '../route.interface.js'; +import type { VerifyService } from '../../services/verify-service.js'; + +export class VerifyCheckRoute implements RelayRoute { + apiPath = API_PATHS.VERIFY_CHECK; + method: HTTPMethods = 'POST'; + version: ApiVersion = 'v1'; + + constructor(private readonly verifyService: VerifyService) {} + + static register({ + app, + verifyService, + }: { + app: FastifyInstance; + verifyService: VerifyService; + }): void { + const route = new VerifyCheckRoute(verifyService); + const { apiPath, version, method, handler } = route; + const url = path.posix.join( + API_PATHS.BASE.replace(':version', version), + apiPath, + ); + + app.withTypeProvider().route<{ + Body: VerifyCheckRequest; + Reply: VerifyCheckResponse | RelayErrorResponse; + }>({ + url, + method, + handler: handler.bind(route), + schema: { + body: VerifyCheckRequestSchema, + response: { + 200: VerifyCheckResponseSchema, + 500: RelayErrorResponseSchema, + }, + }, + }); + } + + async handler( + request: FastifyRequest<{ Body: VerifyCheckRequest }>, + reply: FastifyReply<{ + Reply: VerifyCheckResponse | RelayErrorResponse; + }>, + ) { + try { + const { baselineId, expectedChanges, annotationId } = request.body; + const result = await this.verifyService.check({ + baselineId, + expectedChanges, + annotationId, + }); + return reply.status(HTTP_STATUS.OK).send(result); + } catch (error: unknown) { + if (error instanceof DomscribeError) { + return reply.status(HTTP_STATUS.INTERNAL_SERVER_ERROR).send({ + ...error.toProblemDetails(), + error: error.message, + }); + } + const errorMessage = + error instanceof Error ? error.message : 'Unknown error'; + return reply.status(HTTP_STATUS.INTERNAL_SERVER_ERROR).send({ + error: errorMessage, + code: DomscribeErrorCode.DS_INTERNAL_ERROR, + }); + } + } +} diff --git a/packages/domscribe-relay/src/server/services/index.ts b/packages/domscribe-relay/src/server/services/index.ts index 7771143..a00d921 100644 --- a/packages/domscribe-relay/src/server/services/index.ts +++ b/packages/domscribe-relay/src/server/services/index.ts @@ -11,6 +11,7 @@ export type { SearchAnnotationsOptions, SearchAnnotationsResult, } from './annotation-service.js'; +export { VerifyService } from './verify-service.js'; export type { AnnotationStorageProvider } from './storage/index.js'; export { FileAnnotationStorage, diff --git a/packages/domscribe-relay/src/server/services/verify-service.ts b/packages/domscribe-relay/src/server/services/verify-service.ts new file mode 100644 index 0000000..bb1bdb8 --- /dev/null +++ b/packages/domscribe-relay/src/server/services/verify-service.ts @@ -0,0 +1,220 @@ +/** + * Post-edit verification service (RFC 0002). + * + * Captures element snapshots (computed-style allowlist + bounding rect) + * over the existing WS request/response channel, stores them as in-memory + * baselines, and compares a post-edit re-capture against a baseline using + * the deterministic delta functions from `@domscribe/verify`. + * + * Baselines are session-scoped by design: they describe the live page in + * the developer's browser right now, so persisting them across relay + * restarts would only produce stale comparisons. + * + * @module @domscribe/relay/server/services/verify-service + */ +import crypto from 'crypto'; +import type { + BoundingRect, + ManifestEntryId, + VerifyResult, +} from '@domscribe/core'; +import { + diffBoundingRects, + diffStyleMaps, + resolveVerdict, + type ExpectedChange, +} from '@domscribe/verify'; +import type { WSServer } from '../ws-server.js'; +import type { AnnotationService } from './annotation-service.js'; +import type { + VerifyBaselineResponse, + VerifyCheckResponse, +} from '../../schema.js'; + +interface ElementSnapshot { + computed?: Record; + boundingRect?: BoundingRect; +} + +interface VerifyBaseline extends ElementSnapshot { + id: string; + entryId: ManifestEntryId; + capturedAt: string; +} + +/** Cap stored baselines; oldest entries are evicted first. */ +const MAX_BASELINES = 100; + +export class VerifyService { + private readonly baselines = new Map(); + + constructor( + private readonly wsServer: WSServer, + private readonly annotationService: AnnotationService, + ) {} + + /** + * Capture a pre-edit baseline snapshot for a manifest entry. + */ + async captureBaseline( + entryId: ManifestEntryId, + ): Promise { + const capture = await this.captureSnapshot(entryId); + if ('error' in capture) { + return { + captured: false, + browserConnected: capture.browserConnected, + error: capture.error, + }; + } + + const baseline: VerifyBaseline = { + id: `vb_${crypto.randomUUID()}`, + entryId, + capturedAt: new Date().toISOString(), + ...capture.snapshot, + }; + this.baselines.set(baseline.id, baseline); + this.evictOldest(); + + return { + captured: true, + baselineId: baseline.id, + entryId, + capturedAt: baseline.capturedAt, + browserConnected: true, + hasComponentStyles: baseline.computed !== undefined, + hasBoundingRect: baseline.boundingRect !== undefined, + }; + } + + /** + * Re-capture the element and compare against a stored baseline. + */ + async check(params: { + baselineId: string; + expectedChanges?: ExpectedChange[]; + annotationId?: string; + }): Promise { + const { baselineId, expectedChanges, annotationId } = params; + + const baseline = this.baselines.get(baselineId); + if (!baseline) { + return { + verified: false, + browserConnected: this.wsServer.getClientCount() > 0, + error: `Unknown baselineId "${baselineId}" — capture a baseline before verifying. Baselines do not survive relay restarts.`, + }; + } + + const capture = await this.captureSnapshot(baseline.entryId); + if ('error' in capture) { + return { + verified: false, + browserConnected: capture.browserConnected, + error: capture.error, + }; + } + + const result = this.compare(baseline, capture.snapshot, expectedChanges); + + if (annotationId) { + await this.appendToAnnotation(annotationId, result); + } + + return { verified: true, browserConnected: true, result }; + } + + private compare( + baseline: ElementSnapshot, + current: ElementSnapshot, + expectedChanges?: ExpectedChange[], + ): VerifyResult { + const stylesAvailable = + baseline.computed !== undefined || current.computed !== undefined; + const styleDeltas = stylesAvailable + ? diffStyleMaps(baseline.computed ?? {}, current.computed ?? {}) + : []; + + const rectsAvailable = + baseline.boundingRect !== undefined && current.boundingRect !== undefined; + const rectDeltas = + baseline.boundingRect && current.boundingRect + ? diffBoundingRects(baseline.boundingRect, current.boundingRect) + : []; + + // Without style data, declared expectations cannot be evaluated — + // fall back to change detection and say so. + const { verdict, notes } = resolveVerdict({ + styleDeltas, + rectDeltas, + expectedChanges: stylesAvailable ? expectedChanges : undefined, + }); + + const unavailableNote = stylesAvailable + ? undefined + : 'Style snapshots are unavailable (runtime captureStyles is off) — verdict is based on element geometry only. Enable captureStyles for per-property deltas.'; + + return { + verdict, + timestamp: new Date().toISOString(), + componentStylesDelta: stylesAvailable ? styleDeltas : undefined, + boundingRectDelta: rectsAvailable ? rectDeltas : undefined, + notes: [unavailableNote, notes].filter(Boolean).join(' ') || undefined, + }; + } + + private async captureSnapshot( + entryId: ManifestEntryId, + ): Promise< + | { snapshot: ElementSnapshot } + | { error: string; browserConnected: boolean } + > { + if (this.wsServer.getClientCount() === 0) { + return { + error: + 'No browser is connected — open the page in the browser and retry.', + browserConnected: false, + }; + } + + const response = await this.wsServer.requestContext(entryId); + if (!response || !response.success || response.rendered === false) { + return { + error: + response?.error ?? + 'Element is not currently rendered in the browser — navigate to a page that renders it and retry.', + browserConnected: true, + }; + } + + return { + snapshot: { + computed: response.context?.componentStyles?.computed, + boundingRect: response.elementInfo?.boundingRect, + }, + }; + } + + private async appendToAnnotation( + annotationId: string, + result: VerifyResult, + ): Promise { + const annotation = await this.annotationService.get(annotationId); + if (!annotation) return; + + await this.annotationService.patch(annotationId, { + context: { + verifyHistory: [...(annotation.context.verifyHistory ?? []), result], + }, + }); + } + + private evictOldest(): void { + while (this.baselines.size > MAX_BASELINES) { + const oldestKey = this.baselines.keys().next().value; + if (oldestKey === undefined) return; + this.baselines.delete(oldestKey); + } + } +} diff --git a/packages/domscribe-test-fixtures/fixtures/_templates/nuxt/package.json__tmpl__ b/packages/domscribe-test-fixtures/fixtures/_templates/nuxt/package.json__tmpl__ index 423dbaf..6a37717 100644 --- a/packages/domscribe-test-fixtures/fixtures/_templates/nuxt/package.json__tmpl__ +++ b/packages/domscribe-test-fixtures/fixtures/_templates/nuxt/package.json__tmpl__ @@ -13,6 +13,9 @@ "nuxt": "^<%= frameworkVersion %>.0.0", "vue": "^3.0.0", }, + "overrides": { + "@vitejs/devtools": "^0.3.0" + }, "devDependencies": { "@domscribe/nuxt": "*", "typescript": "^5.5.0" diff --git a/packages/domscribe-test-fixtures/fixtures/nuxt/v3/js/package.json b/packages/domscribe-test-fixtures/fixtures/nuxt/v3/js/package.json index 7c5ae09..a29a8bc 100644 --- a/packages/domscribe-test-fixtures/fixtures/nuxt/v3/js/package.json +++ b/packages/domscribe-test-fixtures/fixtures/nuxt/v3/js/package.json @@ -16,5 +16,8 @@ "devDependencies": { "@domscribe/nuxt": "*", "typescript": "^5.5.0" + }, + "overrides": { + "@vitejs/devtools": "^0.3.0" } } diff --git a/packages/domscribe-test-fixtures/fixtures/nuxt/v3/ts/package.json b/packages/domscribe-test-fixtures/fixtures/nuxt/v3/ts/package.json index 8260b86..c44ec43 100644 --- a/packages/domscribe-test-fixtures/fixtures/nuxt/v3/ts/package.json +++ b/packages/domscribe-test-fixtures/fixtures/nuxt/v3/ts/package.json @@ -16,5 +16,8 @@ "devDependencies": { "@domscribe/nuxt": "*", "typescript": "^5.5.0" + }, + "overrides": { + "@vitejs/devtools": "^0.3.0" } } diff --git a/packages/domscribe-verify/src/index.ts b/packages/domscribe-verify/src/index.ts index 8cfa8be..e57c3e0 100644 --- a/packages/domscribe-verify/src/index.ts +++ b/packages/domscribe-verify/src/index.ts @@ -17,3 +17,15 @@ export { loadPng, } from './lib/comparator.js'; export type { DiffResult } from './lib/comparator.js'; + +export { + RECT_EPSILON_PX, + diffBoundingRects, + diffStyleMaps, + resolveVerdict, +} from './lib/delta.js'; +export type { + ExpectedChange, + VerdictInput, + VerdictResult, +} from './lib/delta.js'; diff --git a/packages/domscribe-verify/src/lib/delta.spec.ts b/packages/domscribe-verify/src/lib/delta.spec.ts new file mode 100644 index 0000000..d285095 --- /dev/null +++ b/packages/domscribe-verify/src/lib/delta.spec.ts @@ -0,0 +1,257 @@ +import { describe, it, expect } from 'vitest'; +import type { BoundingRect } from '@domscribe/core'; +import { + RECT_EPSILON_PX, + diffBoundingRects, + diffStyleMaps, + resolveVerdict, +} from './delta.js'; + +function createRect(overrides: Partial = {}): BoundingRect { + return { + x: 100, + y: 200, + width: 320, + height: 48, + top: 200, + right: 420, + bottom: 248, + left: 100, + ...overrides, + }; +} + +describe('diffStyleMaps', () => { + it('should return no deltas for identical maps', () => { + // Arrange + const styles = { padding: '8px', color: 'rgb(0, 0, 0)' }; + + // Act + const deltas = diffStyleMaps(styles, { ...styles }); + + // Assert + expect(deltas).toEqual([]); + }); + + it('should report a changed value with before and after', () => { + // Arrange + const before = { padding: '8px', color: 'rgb(0, 0, 0)' }; + const after = { padding: '12px', color: 'rgb(0, 0, 0)' }; + + // Act + const deltas = diffStyleMaps(before, after); + + // Assert + expect(deltas).toEqual([ + { property: 'padding', before: '8px', after: '12px' }, + ]); + }); + + it('should report a removed property with after null', () => { + // Arrange + const before = { 'box-shadow': '0 1px 2px rgba(0,0,0,0.2)' }; + const after = {}; + + // Act + const deltas = diffStyleMaps(before, after); + + // Assert + expect(deltas).toEqual([ + { + property: 'box-shadow', + before: '0 1px 2px rgba(0,0,0,0.2)', + after: null, + }, + ]); + }); + + it('should report an added property with before null', () => { + // Arrange + const before = {}; + const after = { gap: '16px' }; + + // Act + const deltas = diffStyleMaps(before, after); + + // Assert + expect(deltas).toEqual([{ property: 'gap', before: null, after: '16px' }]); + }); +}); + +describe('diffBoundingRects', () => { + it('should return no deltas for identical rects', () => { + // Arrange + const rect = createRect(); + + // Act + const deltas = diffBoundingRects(rect, createRect()); + + // Assert + expect(deltas).toEqual([]); + }); + + it('should ignore sub-epsilon jitter', () => { + // Arrange + const before = createRect(); + const after = createRect({ width: 320 + RECT_EPSILON_PX }); + + // Act + const deltas = diffBoundingRects(before, after); + + // Assert + expect(deltas).toEqual([]); + }); + + it('should report fields that moved beyond epsilon', () => { + // Arrange + const before = createRect(); + const after = createRect({ width: 328, right: 428 }); + + // Act + const deltas = diffBoundingRects(before, after); + + // Assert + expect(deltas).toEqual([ + { field: 'width', before: 320, after: 328 }, + { field: 'right', before: 420, after: 428 }, + ]); + }); + + it('should honor a custom epsilon', () => { + // Arrange + const before = createRect(); + const after = createRect({ height: 52 }); + + // Act + const deltas = diffBoundingRects(before, after, 10); + + // Assert + expect(deltas).toEqual([]); + }); +}); + +describe('resolveVerdict', () => { + const paddingDelta = { property: 'padding', before: '8px', after: '12px' }; + const colorDelta = { + property: 'color', + before: 'rgb(0, 0, 0)', + after: 'rgb(255, 0, 0)', + }; + const widthRectDelta = { field: 'width' as const, before: 320, after: 328 }; + + it('should return no_change when nothing changed', () => { + // Act + const result = resolveVerdict({ styleDeltas: [], rectDeltas: [] }); + + // Assert + expect(result.verdict).toBe('no_change'); + expect(result.notes).toContain('did not affect the rendered element'); + }); + + it('should return match with a caveat note when no expectations are declared', () => { + // Act + const result = resolveVerdict({ + styleDeltas: [paddingDelta], + rectDeltas: [], + }); + + // Assert + expect(result.verdict).toBe('match'); + expect(result.notes).toContain('change detection only'); + }); + + it('should return match without notes when all expectations are met exactly', () => { + // Act + const result = resolveVerdict({ + styleDeltas: [paddingDelta], + rectDeltas: [widthRectDelta], + expectedChanges: [{ property: 'padding', value: '12px' }], + }); + + // Assert + expect(result.verdict).toBe('match'); + expect(result.notes).toBeUndefined(); + }); + + it('should treat an expectation without a value as met when the property changed', () => { + // Act + const result = resolveVerdict({ + styleDeltas: [paddingDelta], + rectDeltas: [], + expectedChanges: [{ property: 'padding' }], + }); + + // Assert + expect(result.verdict).toBe('match'); + }); + + it('should return partial when expectations are met but other properties changed', () => { + // Act + const result = resolveVerdict({ + styleDeltas: [paddingDelta, colorDelta], + rectDeltas: [], + expectedChanges: [{ property: 'padding', value: '12px' }], + }); + + // Assert + expect(result.verdict).toBe('partial'); + expect(result.notes).toContain('unexpected properties also changed: color'); + }); + + it('should return partial when only some expectations are met', () => { + // Act + const result = resolveVerdict({ + styleDeltas: [paddingDelta], + rectDeltas: [], + expectedChanges: [ + { property: 'padding', value: '12px' }, + { property: 'color', value: 'rgb(255, 0, 0)' }, + ], + }); + + // Assert + expect(result.verdict).toBe('partial'); + expect(result.notes).toContain('1/2 expected changes applied'); + expect(result.notes).toContain('unchanged: color'); + }); + + it('should return partial when a property changed to the wrong value', () => { + // Act + const result = resolveVerdict({ + styleDeltas: [paddingDelta], + rectDeltas: [], + expectedChanges: [{ property: 'padding', value: '16px' }], + }); + + // Assert + expect(result.verdict).toBe('partial'); + expect(result.notes).toContain('padding changed to "12px"'); + expect(result.notes).toContain('expected "16px"'); + }); + + it('should return regression when nothing expected changed but other styles did', () => { + // Act + const result = resolveVerdict({ + styleDeltas: [colorDelta], + rectDeltas: [], + expectedChanges: [{ property: 'padding', value: '12px' }], + }); + + // Assert + expect(result.verdict).toBe('regression'); + expect(result.notes).toContain('other properties changed: color'); + }); + + it('should return regression when only geometry moved and expectations were unmet', () => { + // Act + const result = resolveVerdict({ + styleDeltas: [], + rectDeltas: [widthRectDelta], + expectedChanges: [{ property: 'padding', value: '12px' }], + }); + + // Assert + expect(result.verdict).toBe('regression'); + expect(result.notes).toContain('only geometry changed: width'); + }); +}); diff --git a/packages/domscribe-verify/src/lib/delta.ts b/packages/domscribe-verify/src/lib/delta.ts new file mode 100644 index 0000000..a8ae2a2 --- /dev/null +++ b/packages/domscribe-verify/src/lib/delta.ts @@ -0,0 +1,236 @@ +/** + * Deterministic delta computation for post-edit verification (RFC 0002). + * + * Pure functions — no DOM, no browser, no I/O. The relay's verify tools + * capture element snapshots (computed styles + bounding rect) before and + * after an agent edit and feed them here; the resulting deltas are the + * structured, deterministic half of the verify contract. Judging whether a + * detected change matches the *intent* is deliberately left to the calling + * agent — these functions only report what changed, precisely. + * + * @module @domscribe/verify/delta + */ +import type { + BoundingRect, + BoundingRectDelta, + StylePropertyDelta, + VerifyVerdict, +} from '@domscribe/core'; + +/** + * A change the agent intended to make, declared at verify time. + * When `value` is present, the property must resolve to exactly that value + * (as reported by `getComputedStyle`) for the expectation to count as met. + */ +export interface ExpectedChange { + property: string; + value?: string; +} + +export interface VerdictInput { + styleDeltas: StylePropertyDelta[]; + rectDeltas: BoundingRectDelta[]; + /** + * Optional declared intent. When absent the verdict degrades to pure + * change detection: `no_change` vs `match`-with-a-caveat-note. + */ + expectedChanges?: ExpectedChange[]; +} + +export interface VerdictResult { + verdict: VerifyVerdict; + notes?: string; +} + +/** + * Sub-pixel geometry jitter tolerated before a rect field counts as changed. + * Browsers report fractional pixels; half a device pixel is noise. + */ +export const RECT_EPSILON_PX = 0.5; + +const RECT_FIELDS = [ + 'x', + 'y', + 'width', + 'height', + 'top', + 'right', + 'bottom', + 'left', +] as const; + +/** + * Diff two style maps (property → resolved value). + * Reports one entry per property whose value differs, with `null` marking + * a property absent on one side. Key order follows the before-map first, + * then after-only properties in their own order — deterministic for tests. + */ +export function diffStyleMaps( + before: Record, + after: Record, +): StylePropertyDelta[] { + const deltas: StylePropertyDelta[] = []; + + for (const [property, beforeValue] of Object.entries(before)) { + const afterValue = after[property] ?? null; + if (afterValue !== beforeValue) { + deltas.push({ property, before: beforeValue, after: afterValue }); + } + } + + for (const [property, afterValue] of Object.entries(after)) { + if (!(property in before)) { + deltas.push({ property, before: null, after: afterValue }); + } + } + + return deltas; +} + +/** + * Diff two bounding rects field-by-field. + * Deltas at or below `epsilon` (default half a pixel) are dropped as + * rendering jitter. + */ +export function diffBoundingRects( + before: BoundingRect, + after: BoundingRect, + epsilon: number = RECT_EPSILON_PX, +): BoundingRectDelta[] { + const deltas: BoundingRectDelta[] = []; + + for (const field of RECT_FIELDS) { + const beforeValue = before[field]; + const afterValue = after[field]; + if (Math.abs(afterValue - beforeValue) > epsilon) { + deltas.push({ field, before: beforeValue, after: afterValue }); + } + } + + return deltas; +} + +interface ExpectationOutcome { + met: ExpectedChange[]; + wrongValue: { expected: ExpectedChange; delta: StylePropertyDelta }[]; + unchanged: ExpectedChange[]; + unexpected: StylePropertyDelta[]; +} + +function classifyExpectations( + styleDeltas: StylePropertyDelta[], + expectedChanges: ExpectedChange[], +): ExpectationOutcome { + const deltaByProperty = new Map(styleDeltas.map((d) => [d.property, d])); + const expectedProperties = new Set(expectedChanges.map((e) => e.property)); + + const met: ExpectedChange[] = []; + const wrongValue: { expected: ExpectedChange; delta: StylePropertyDelta }[] = + []; + const unchanged: ExpectedChange[] = []; + + for (const expected of expectedChanges) { + const delta = deltaByProperty.get(expected.property); + if (!delta) { + unchanged.push(expected); + } else if (expected.value !== undefined && delta.after !== expected.value) { + wrongValue.push({ expected, delta }); + } else { + met.push(expected); + } + } + + const unexpected = styleDeltas.filter( + (d) => !expectedProperties.has(d.property), + ); + + return { met, wrongValue, unchanged, unexpected }; +} + +function formatProperties(items: { property: string }[]): string { + return items.map((i) => i.property).join(', '); +} + +/** + * Resolve a deterministic verdict from the computed deltas. + * + * Rules: + * - No style and no geometry delta → `no_change` (the edit did not reach + * the rendered element — wrong file, wrong selector, or HMR miss). + * - No declared expectations → `match` with a caveat note; the caller must + * inspect the deltas to confirm intent. + * - All expectations met, nothing unexpected → `match`. + * - All expectations met but extra properties changed → `partial`. + * - Some expectations met → `partial`. + * - No expectation met while other properties changed → `regression`. + * + * Geometry deltas never downgrade a verdict on their own: rect movement is + * usually a consequence of an intended style change (padding grows the + * box). They are reported for the agent to judge. + */ +export function resolveVerdict(input: VerdictInput): VerdictResult { + const { styleDeltas, rectDeltas, expectedChanges } = input; + + if (styleDeltas.length === 0 && rectDeltas.length === 0) { + return { + verdict: 'no_change', + notes: + 'No style or geometry delta detected — the edit did not affect the rendered element. ' + + 'Check the file, the selector, and that HMR applied the change.', + }; + } + + if (!expectedChanges || expectedChanges.length === 0) { + return { + verdict: 'match', + notes: + 'Change detected. No expectedChanges were declared, so this verdict reflects ' + + 'change detection only — inspect the deltas to confirm they match the intent.', + }; + } + + const outcome = classifyExpectations(styleDeltas, expectedChanges); + const allMet = + outcome.wrongValue.length === 0 && outcome.unchanged.length === 0; + + if (allMet && outcome.unexpected.length === 0) { + return { verdict: 'match' }; + } + + if (allMet) { + return { + verdict: 'partial', + notes: `All expected changes applied, but unexpected properties also changed: ${formatProperties(outcome.unexpected)}.`, + }; + } + + if (outcome.met.length > 0 || outcome.wrongValue.length > 0) { + const parts: string[] = []; + if (outcome.wrongValue.length > 0) { + parts.push( + outcome.wrongValue + .map( + ({ expected, delta }) => + `${expected.property} changed to "${delta.after}" (expected "${expected.value}")`, + ) + .join('; '), + ); + } + if (outcome.unchanged.length > 0) { + parts.push(`unchanged: ${formatProperties(outcome.unchanged)}`); + } + return { + verdict: 'partial', + notes: `${outcome.met.length}/${expectedChanges.length} expected changes applied. ${parts.join('. ')}.`, + }; + } + + const changed = + styleDeltas.length > 0 + ? `other properties changed: ${formatProperties(styleDeltas)}` + : `only geometry changed: ${rectDeltas.map((d) => d.field).join(', ')}`; + return { + verdict: 'regression', + notes: `None of the expected changes were applied, but ${changed}. The edit likely landed on the wrong element or property.`, + }; +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 88a1b49..986fe01 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -243,6 +243,9 @@ importers: '@domscribe/manifest': specifier: workspace:* version: link:../domscribe-manifest + '@domscribe/verify': + specifier: workspace:* + version: link:../domscribe-verify '@fastify/cors': specifier: ^10.0.0 version: 10.1.0