Building Latch: Agent to UI Exploration

Every AI dashboard demo follows the same script. Someone types "show me sales by region", a chart appears, the room nods. Then you close the tab and it's gone, and tomorrow you type the same sentence again.

I wanted to build the version where the good answers stay. This is how it went, including the parts where I was wrong.

Wanna try it now? Visit latch.isaacantwi.com

Where it started

The first version of the idea was a paragraph I wrote down before any research. Roughly:

The front end is going to have a set of cards and charts, and the back end is going to know what those primitives are. Whatever the person types is going to use tools that call the REST APIs, and the user's token is passed across so the AI inherits what that person can and cannot see. Front end designers create static shells so the AI does not invent new colour schemes. And the user is allowed to pin a card so it stops being dynamic. Best of both worlds.

So, a dashboard with the AI built in, but optional. Instead of landing on a wall of forty widgets, you ask. "What was my spending trend last quarter?" The model works out the time range, picks a card that fits, fetches the data, and the card appears.

The pin was the last sentence, almost an afterthought. It turned out to be the whole product.

Then I found Google's A2UI

https://developers.googleblog.com/introducing-a2ui-an-open-project-for-agent-driven-interfaces/

Before writing code I spent an hour looking for prior art. It was the most useful hour of the project, because four of my six assumptions did not survive it.

The big one: I thought nobody had packaged the contract between an agent and a set of components it is allowed to render. Google had. A2UI is their open spec for agents that describe UI instead of writing it. The agent sends JSON naming a component and its data, the client holds the catalog, and the agent cannot ask for a component that was not published. That is my "designers create static shells" sentence, written down as a standard by someone else, with renderers for React and Flutter already shipping.

It wasn't alone. CopilotKit has AG-UI, and there are two more generative UI specs beside it. CopilotKit's own guide even has a name for what I was describing: Static Generative UI, the safe and obvious end of a spectrum, with tutorials. ThoughtSpot lets you embed their agent in your own app. And permission-scoped tools were already being worked on by the auth vendors.

I had arrived, independently, at a well-populated corner of the map. Better to know that on day zero than after a launch post.

What survived

Three things, and each one is written down as out of scope by the people building the specs. That is what made them openings rather than wishful thinking.

Persistence. A2UI v1.0 is session-scoped, with no way to save or restore a surface, and the omission reads as deliberate. Vercel's AI SDK scopes its UI state to the conversation. Wren AI, the leading open source generative BI tool, calls what it produces "disposable charts". The whole field had agreed that agent-composed UI is a thing that evaporates.

Authorisation joined to the UI. A2UI defers all of it to the transport layer. Meanwhile the auth vendors were solving per-tool scopes for agents. Two live conversations that had never met. Nobody had connected them into "the composed surface differs by role, and a pinned card degrades on its own when access is revoked."

Determinism. A published critique of Wren AI: LLM planning is nondeterministic, so the same question can plan differently, and there is no native refusal on ambiguity. If the model's whole output is a small typed object, that problem mostly goes away, and the output becomes something you can diff and test.

Those three gave me the sentence the project is built around:

Agent-composed UI today forgets everything on reload, ignores who's asking, and can't be tested.

The idea did not change. The research told me which parts of it were the valuable parts. I called it Latch. A card arrives loose, and latching fixes it in place. In the interface the word is pin, because that is what people say.

One more thing A2UI's basic catalog is missing: chart components. It ships text, images, cards, tabs, buttons, sliders, form fields. Nothing for data. An analytics catalog is a contribution on its own before you add persistence or scoping to it.

What I ended with

Three layers, one for each hole.

User: "what did I spend last quarter?"
        |
        v
  /api/ask
    token -> tool manifest            LAYER 2: only the tools this role may call
        |
        v
  model emits ONLY { tool, args, card, title }
        |
        v
  REST call, same token forwarded
        |
        v
  renders from the catalog            LAYER 1: designer-built shells only
        |
        v
  [ pin ]                             LAYER 3: stores the PLAN, not the pixels,
                                               and re-runs it on every load

Layer one is the catalog: five cards, each declaring the shape of data it accepts and the limits it breaks under. A line chart takes a time series with up to four lines. A donut takes categories, at most six, none negative. A stat tile takes one number with an optional comparison. A table takes rows. Bars take almost anything and are the fallback for everything else. That is data, not code, so both the model and the validator can read it.

Layer two is the manifest: four parameterised tools over the API, not forty question-shaped functions. Time series, breakdown, scalar, list. Each declares the shape it returns, and the shape is the join between the two layers, because it decides which cards are eligible before the model is asked anything. The manifest is generated per token. There are three demo identities over one household ledger: the owner sees everything, the partner sees the joint accounts only, and the accountant sees totals but never transaction rows or merchant names.

Layer three is the pin. A pinned card stores this and nothing else:

{
  "toolId": "metrics.breakdown",
  "args": { "metric": "spend", "dimension": "category", "limit": 6, "range": "last_quarter" },
  "cardId": "DonutCard",
  "props": { "title": "Where the money went" },
  "question": "what did I spend last quarter by category?"
}

No pixels, no data, no snapshot. Loading the board runs every pinned plan again against the token you hold now. So the numbers stay current. The card is still editable by conversation, because the plan is the same object the model produces. And if your access is revoked, the card degrades on its own: pin a merchant breakdown as the owner, switch to the accountant, and that card comes back saying you do not have access to merchant-level detail. It stays pinned. It just cannot answer any more. That last one is the part I have not seen anywhere else.

The principles the code runs on

These are the rules I kept coming back to. Most of them are one line in the codebase somewhere.

Publish what the UI can render and what the API will serve, then give the model a very small set of legal moves between them. Everything else follows from this.

Make illegal choices unrepresentable rather than rejected. The card is a parameter on each tool, and its allowed values are the two or three cards that can render that tool's result. The model cannot name a card that would not fit, because that value does not exist in what it was shown.

Absence, not prohibition. A tool the caller cannot use is missing from the manifest, not marked forbidden. The merchant dimension is stripped from the enums before the model sees them, so it cannot ask for the thing it is not allowed to have.

Validate against the data, not the intent. Ask for a donut over twenty merchants and the renderer rejects it after the data comes back, swaps to ranked bars, and says so on the card. The model never gets to be wrong in a way you see.

Give the model a legal way to say no. Without one, a model told to always call a tool will answer a narrower question and label it as yours. So there is a decline tool, and the prompt lists what the current identity cannot see, so the refusal can be specific.

Store the question, not the picture. The plan is the dashboard's own configuration format, and the model is one way to write one. The six cards you see on first load are plans I typed by hand, identical in shape to anything the model produces.

Keep the model's output small enough to test. The whole output is a tool, its arguments, a card and a title. That is diffable, so it is snapshot-testable, so it runs in CI. Nobody tests their natural language to chart layer because the output is a chart. Here it is four fields. The same property means the model is swappable. If the AI layer needed a frontier model to be correct, the contract would be too loose.

Measure before guessing. See the fourth bug below.

Bugs worth writing down

The interesting part was not the architecture. It was watching where a constrained system still leaks.

The model answered a different question and labelled it as mine. My first prompt said "if the question cannot be answered with the tools you have, choose the closest tool". Asked, as the accountant, which merchants I spend the most at, it produced a breakdown by category and titled it Top Merchants by Spending. It looked like an answer. It was the exact failure the project exists to prevent, and I had written the instruction that caused it. The fix was the decline tool, not a better sentence.

The chat invented a number. The ordinary text chat that sits next to the board was asked a follow-up, "and what was the biggest category?", and answered "groceries, with $1,845" without calling anything. Wrong category, invented figure. Earlier tool results are not in the history the client replays, so a model that skips the lookup is working from nothing. Instructions did not fix it. A guard did: if the reply states a money figure and no tool ran that turn, the answer is thrown away and asked again with the tool call forced.

"Other" was reported as the biggest category. With the guard in, the model called the breakdown with a limit of one and got a single bucket called Other holding the whole month. The schema floor is now three, because a breakdown into one group is not a breakdown, and the parameter description says plainly that Other is a leftover bucket. Both of these were schema problems wearing model-problem clothes.

Twenty-five seconds a question. Early on, questions took 25 to 74 seconds and I assumed the tool schema was too big. A benchmark that prints prompt and completion tokens killed that in a minute: the prompt was fine, the model was thinking at length about which enum value to pick. Qwen's no-think marker does nothing through Ollama's OpenAI-compatible endpoint. A reasoning effort of none does, and it is now the default. Five times faster from one request parameter.

And one from the end. I tried swapping in gpt-oss through Ollama and six of seven questions failed validation, because Ollama's template for that model drops the enum values from tool parameters, so it invented metric names the schema had never offered. The contract was fine. The runtime was quietly hiding it from the model. Back to Qwen.

What it runs on

Next.js, with the charts from Tremor's copy-paste distribution, so the chart source lives in the repo and a designer can change any of it. The data is a seeded household ledger, 708 transactions over twelve months, generated deterministically so the tests never drift. Rent and groceries land mostly on joint accounts, shopping and subscriptions mostly on personal ones, which is why the owner and the partner get different charts for the same question rather than the same chart with smaller numbers.

I built it against Qwen3 14B running locally through Ollama. The live demo runs Qwen3 27B on Groq's free tier, over the same OpenAI-compatible protocol, which is the swap the contract promised. The tests, 34 of them, cover the query engine, the permission boundary and the exact list of tools each role's model is handed, including one that asserts merchant is gone from the accountant's options before the model ever sees them. None of them need a model.

What I am still unsure about

Whether the chat should exist at all. I added it next to the board so the contrast is visible: prose that scrolls away, or a card that stays. It makes the argument well. It also doubles the surface area, and the two worst bugs above both came from it.

Whether pinning survives contact with users. Everything here assumes people want to curate. Some will pin everything and end up with the forty-widget wall they were trying to escape. There is no eviction, no decay, no limit, and I do not know yet whether that needs one.

Whether the specs will absorb this. CopilotKit owns AG-UI, ships quickly, and persistence is an obvious next move for them. That argues for building on their protocol rather than beside it.

Try it

The live board is at latch.isaacantwi.com and the code is at github.com/okraks/latch. With no model configured the board still works: the six seed cards, the role switcher, and no command bar. Point it at anything that speaks the OpenAI protocol to turn the asking on.