Skip to content

follow-up: MCP.md tool-call output structural comparison (entirely unaddressed) #2509

Description

@carlos-alm

Context

#2369 added Tier 3 structural output-label diffing for docs/examples/CLI.md (two commands, context and audit --quick, comparing only ## Section header labels against a real re-run). That issue's own text flagged a second, entirely separate piece of scope it deliberately did not address:

MCP.md tool-call comparison: comparable idea for MCP tool responses (key-shape match against real tool output from an actual MCP server call), entirely unaddressed by #2212 — MCP.md's own protection remains JSON-syntax-validity only (#2005).

Problem

docs/examples/MCP.md's only current protection (#2005) is JSON-syntax validity of its documented ```json blocks — nothing verifies that a documented tool-call's response shape (which keys are present) still matches what an actual MCP server call returns today. A tool's response schema could add/remove/rename a top-level key and MCP.md would show no drift.

Fix direction

Comparable idea to #2369's CLI.md work, but needs its own design pass since the mechanics are different:

  • CLI.md's harness runs the CLI directly via execFileSync and diffs plain-text ## headers. MCP.md's equivalent would need to actually stand up (or drive) an MCP server and make a real tool call, then diff the RESPONSE's key shape (e.g. top-level JSON keys, or a normalized recursive key-shape) against the documented JSON block's own keys.
  • Needs picking a small representative subset of MCP tools to start with (mirroring CLI.md Tier 2/3's "curated subset" pattern), not all of them at once.
  • Needs a normalization policy for the JSON case specifically — likely simpler than CLI.md's plain-text case (compare object key sets/shapes, not exact values), but still a real design decision, not something to decide unilaterally in an unrelated PR.

Filed per this repo's scope-discipline convention — out of scope for #2369, which was specifically about CLI.md's two starter commands, not MCP.md.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions