Replace CRUD operations with JSON patch

[jacobsimionato]

Background

Currently, A2UI has an underlying data structure which is basically a set of components and a data model. But our messages are for incrementally updating each of them, just imply defining them.

Problems

• There is no well-defined format to serialize the current state of a surface
• Our CRUD operations are a little irregular. E.g. why updateComponents rather than updateSurface? Why not include initial components in CreateSurface etc (see createSurface that includes initial components and data model #1239). If we add more kinds of data, we need to invent new CRUD operations to represent them.
• In practice, most UIs are created and never updated, so it makes sense for the format to emphasize the surface state, rather than incremental updates. Right now, we have an incremental API makes it simple to represent changes to state (a single message) but complex to represent the current state (list of message which may overwrite each other and thus be redundant).
• There is no way to delete components currently, only replace them.

Recent changes in direction

Since we defined A2UI v0.8 and v0.9, some things have changed about our direction which could motivate this design change.

• There is more emphasis on MCP, which is typically a "generate once" pattern, rather than a "generate and update" pattern. A2UI is kind of awkward for MCP, because they need to include a list of messages, and there is a question about which order they should be in etc (it doesn't matter).
• We are looking at separating transport and inference format - https://docs.google.com/document/d/1mH2DJ_jZZA2vZ9EQlWaU4v8_xCAcmqrox_LlElmJ3ws/edit?tab=t.0#heading=h.c0uts5ftkk58 . LLMs cannot write JSON patch well, and this motivated us to avoid it previously. But, if we are defining a separate LLM format anyway, let's go ahead and use it seeing as it entirely solves the issue of representing incremental data updates.

Proposal

Let's focus on defining a JSON object which represents the state of a surface including data model and components. Then, when creating new surfaces, we just send the data. To update a surface, we send a JSON patch based on the previous data to apply.

Pros and Cons

Pros

• Simplify the transport format
• Simplify the mental model of a Surface snapshot and incremental updates
• Make it easy for us to expand the surface state representation (e.g. add something more than data model and components) without needing to rethink CRUD operations
• Simplify core library implementation. Currently, we have a message processor that understands how to apply our messages to a data model. Instead, we would just have some JSON and a JSON patch library that modifies it.
• Obvious way to represent surface snapshot state to LLMs. Currently, in the most simple a2ui implementations, the LLM would see the history of A2UI messages in its context. This is unnecessarily verbose. If there are many messages updating a surface, it should likely just see a snapshot of surface state. This new structure makes it obvious how to achieve this.
• Ability to represent arbitrarily small updates to state, e.g. update an individual property value

Cons

• Creates more churn - this is relatively big change to the spec. Though, I believe we can make this update in the core library and renderers would be unaffected.
• Agents will really need full access to the accurate existing UI state in order to generate correct patches. They will probably have this anyway, though.
• JSON patch is hard for humans to read
• To incrementally update surfaces, developers will need an inference library to help. They won't be able to wing the JSON patch generation.

Sketch

Surface

{
  "$defs": {
    "Surface": {
      "type": "object",
      "description": "The complete state of an A2UI surface.",
      "properties": {
        "surfaceId": {
          "type": "string",
          "description": "Unique identifier for the surface."
        },
        "catalogId": {
          "type": "string",
          "description": "The URI of the catalog defining the components used."
        },
        "components": {
          "type": "object",
          "description": "A map of ComponentId to component definitions.",
          "additionalProperties": {
            "$ref": "catalog.json#/$defs/anyComponent"
          }
        },
        "dataModel": {
          "type": "object",
          "description": "The initial data model for the surface.",
          "default": {}
        },
        "theme": {
          "$ref": "catalog.json#/$defs/theme"
        },
        "sendDataModel": {
          "type": "boolean",
          "default": false
        }
      },
      "required": ["surfaceId", "catalogId", "components"]
    }
  }
}

Server to client

{
  "oneOf": [
    { "$ref": "#/$defs/CreateSurfaceMessage" },
    { "$ref": "#/$defs/UpdateSurfaceMessage" },
    { "$ref": "#/$defs/DeleteSurfaceMessage" }
  ],
  "$defs": {
    "CreateSurfaceMessage": {
      "type": "object",
      "properties": {
        "version": { "const": "v0.10" },
        "createSurface": {
          "type": "object",
          "properties": {
            "surface": { "$ref": "common_types.json#/$defs/Surface" }
          },
          "required": ["surface"]
        }
      },
      "required": ["createSurface", "version"]
    },
    "UpdateSurfaceMessage": {
      "type": "object",
      "properties": {
        "version": { "const": "v0.10" },
        "updateSurface": {
          "type": "object",
          "properties": {
            "surfaceId": { "type": "string" },
            "patch": {
              "type": "array",
              "items": { "$ref": "#/$defs/JsonPatchOperation" }
            }
          },
          "required": ["surfaceId", "patch"]
        }
      },
      "required": ["updateSurface", "version"]
    },
    "JsonPatchOperation": {
      "type": "object",
      "properties": {
        "op": { "enum": ["add", "remove", "replace", "move", "copy", "test"] },
        "path": { "type": "string", "description": "JSON Pointer to the surface property." },
        "value": { "description": "The value to add or replace." },
        "from": { "type": "string", "description": "Required for move/copy." }
      },
      "required": ["op", "path"]
    }
  }
}

[jacobsimionato]

This is a more radical change which goes against my push to avoid churn, but I think it could simplify things in the long term, and actually be relatively easy to add to core libraries.

Curious your thoughts @gspencergoog @yjbanov @wrenj especially

[jacobsimionato]

Note that this could solve #1239 and #235

[jacobsimionato]

@yjbanov this solves your problem about updating individual property values too I think

[gspencergoog]

So, for the streaming case, how would this work? You'd send an initial empty surface and follow it with a bunch of JSON patch messages that update it to add a new item?

[wrenj]

Overall I like it, its conceptually much simpler. The surface is just a big object and you do mutations to it.

The issue with json patch is it can fail if you don't have the latest model, since insert/replace/delete will fail if the key is or isn't present. So client updates the data model locally and when agent sends an add or replace and it might fail if there is a key conflict.

Consider looking at json-merge patch which is a little less picky about the keys (it just overwrites and moves on): https://zuplo.com/learning-center/what-is-json-merge-patch. It does have its own issues about limited array options (have to replace the entire array instead of manipulating just one element, maybe we can work around that using keys).

[gspencergoog]

Also, if the patches get out of order, they can also fail.

[wrenj]

Another option is firebase syntax

patches are in dot notation:
{"components.header": {"type": "header", "title": "Loading..."}}
{"dataModel.theme": "dark"}

plus some sentinels for array manipulation, deletes, etc (full list https://googleapis.dev/nodejs/firestore/latest/FieldValue.html):

{"components.txt_2": { "$delete": true }}
{"components.root_container.children": { "$arrayRemove": ["txt_2"] }}
{"components.root.children": {"$arrayUnion": ["header"]}}

[wrenj]

You could probably just go with JSON Patch (esp since we already use JSON Pointer) and add operators for missing functionality we want. I think just two things:

{ "op": "removeIfExists", "path": "/user/age" } # remove and don't error if key not found
{ "op": "removeValue", "path": "/user/favoriteFoods", "value": "pasta" } # remove array element by value

According to JSON Patch "op": "add" is already an upsert so that's covered.

Maybe you could also add "op": "increment" or "decrement" although doesn't seem high priority.

[dmandar]

I strongly agree with having a unified createSurface. Also, keeping both component and data updates unified under updateSurface works perfectly for the template and hydration pattern.
We can establish a pattern where createSurface is always the full initial snapshot (hence good for templates). For such canned UIs, the application sends the complete component tree pre-wired with data bindings, but with an empty data model.

So, in my mind, an example createSurface (Template without data):
{ "createSurface": { "surface": { "surfaceId": "weather-1", "components": { "city_text": { "type": "Text", "properties": { "text": "${dataModel.city}" } } }, "dataModel": {} } } }

Then, as the agent runs tools and streams results, it doesn't need to rebuild or touch the component tree. It just streams lightweight updateSurface patches to hydrate the data model. If we use the dot-notation syntax wrenj suggested:
Example updateSurface (Hydration):
{ "updateSurface": { "surfaceId": "weather-1", "patch": { "dataModel.city": "Sunnyvale" } } }

I imagine this would be the typical use case (vs updating components on long lived surfaces) for agentic chat use cases (template + hydration).