Shrinking MCP Answers Bought Nothing. Answering the Question Did.

Copilot chat in VS Code on a SWCraft project: one question about the architecture model, and the agent answers it with read call after read call to the MCP server

In the last post about SWCraft’s MCP server I cut 39 tools to 13, because the surface had been designed for a program and the caller was a language model. I thought the surface was done. Then I watched one agent do one change.

SWCraft is an experimental AUTOSAR architecture-as-code environment I am building, introduced in the opening article of this series. The architecture model lives in version-controlled YAML; deterministic tools validate it and generate ARXML, diagrams and documentation; an LLM agent reads and changes the same model through MCP. The agent in this story is GitHub Copilot on the model its automatic setting picks, at the default reasoning effort. Not the strongest agent available, on purpose: if the design works for that one, it works for the rest.

A word on words, because this post has two things that could both be called a model. The architecture model is the YAML: components, interfaces, connectors. The agent is the language model. When a size matters, I say bench, and there are two of them below.

The number that started it

The task was ordinary: add a brake data logger component with a port on an existing interface, a runnable, a task mapping and a connector. Five steps an architect does in ten minutes.

The agent did it right. It took 27 tool calls and 22 turns of the agent, and exactly one of the 27 was the change. The rest was the agent looking. Ten calls were it learning what a component looks like: one discover of 30 KB, which the host could not hold inline and saved to a file, then five reads of that file, then four more discover calls for the layers it wanted. Nine calls gathered context, whole elements, two of them for an element the agent was about to create. Three calls were it finding a rule by being refused. Two were checks after the commit had already succeeded.

Every call was reasonable on its own. Together they were the agent reading the model to see it.

The surface I had built was CRUD: discover, read, change. It is a good surface for a program. For a language model it is an invitation to chain, because the agent cannot act on what it has not seen, and the only way to see is to read.

Measure first

Before changing anything I built a benchmark: seventeen architect tasks, nine questions (“which ports does nothing connect?”, “does a 40 KB component fit the memory this core reaches?”) and eight changes (connect two components, move a runnable, rename a data element, import a component, and the logger task, verbatim). A runner drives Copilot and Codex headless on a fresh copy of the bench per run, and a checker scores every answer against the model’s state after the run. A rule I added later, nothing else moved, fails a change that also tidied a description nobody asked about. Codex answers questions only: it declines the server’s review question from inside itself, so it cannot commit a change.

Two benches. The small one is the public showcase, 122 elements. The large one is generated: 150 components, 1 200 interfaces, 1 450 connectors around one hub with 550 ports, 3 745 elements in all. Nothing in it is realistic except its proportions, which are a real export’s, and size was the point.

Baseline, small bench, the release build, every task twice: 297 calls, 34 of 34 right. About nine calls a task. The agent was not wrong. It was expensive, and the large bench was not even attempted on that build: I already knew what a 30 KB discover would do on a model thirty times the size.

Step one: the agent knows the schema before its first call

The first idea was the cheapest. The assistant profile SWCraft writes into a project (the file Copilot reads before every conversation) now carries the model’s schema, generated from the same contract the validator uses: every layer, its fields, which are required. discover answers a baseline by default, about 1.5 KB, because the profile already has the layers. And a change set can ask the server to complete it: when a new component needs a type mapping, the server adds the mapping instead of refusing and letting the agent work out what the refusal meant.

Step two: the map

With the schema in the prompt, the remaining reads were for context: what exists, what is it called, what does it connect to. So I gave the agent a map. A read with no address answers every layer’s element count and the names in each layer, and a read can take up to sixteen addresses at once, so the context for a change is one call instead of nine.

Steps one and two were measured together, as one prototype, on the small bench: 297 calls became 112, 32 of 34 right. The logger task, verbatim as task two, went from about 24 calls to 4 or 5. The two failures were one task, both of its runs: a new profile rule said “read only what the change depends on”, the agent took it literally, and removed a whole sequence where the task asked for one step of it. A sentence in the prompt had bought calls and cost a correct answer, the unsafe change that passes validation again, this time invited by my own rule. I fixed the sentence and the task passed twice; the lesson stayed. A rule that shortens the agent’s path has to say what the shorter path must still look at.

Then the map met the large bench, and that is where the first design assumption broke. The arithmetic alone says so: every name of every layer is 128 KB there, which the server refuses outright, and the names of 1 200 interfaces and 1 450 connectors alone are far more than the 8 KB the host accepts in one answer before it saves it to a file. So the map was bounded before the large bench ever ran: every layer is counted, and only a layer of at most fifty elements is listed by name. On the large bench the map is under 4 KB and says, honestly, “1 453 connectors”, with no names.

A map works when the model is small enough that seeing it is cheap. On a large model there is nothing small to look at, and an agent that still reads to see falls back to walking the layers. The prototype rebuilt as a release build, with the bounded map, showed what that walk costs. Small bench: 146 calls, 33 of 34, the same sequence task failing once more. Large bench: 187 calls, 31 of 34. The change tasks held, 16 of 16, because the schema and the batched read carried them. The questions fell apart. “Which ports does nothing connect?” needs every connector end; the agent made reads of up to 69 KB and gave up, both runs. Codex, which runs its calls from a script, looped read 1 243 times on the same question and timed out. The server bounded every answer. It did not bound their number.

Two more things that build taught. The same design, measured twice with nothing tuned, cost 112 calls once and 146 calls the next time. The spread of one design is itself a figure, and it is larger than I had assumed. And 21 of the 34 large-bench runs carried at least one answer the host would have saved to a file.

Step three: a budget that bought nothing

The obvious lever was to make the server respect the host’s limit: cap every answer at 8 KB on the wire, page the rest, split a batch into the entries that fit plus hints for the rest. I built it, and the benchmark said 123 calls against the prototype’s 112, the same 32 of 34. Then I measured it where the limit actually lives, in VS Code, on the showcase: eight runs, with the cap and without. Every read answer in all eight runs was already under 8 KB, at both budgets. The one answer that spilled was the agent choosing a full discover of a layer, which no budget on reads would touch.

The budget was not wrong. It was aimed at the wrong thing. The agent’s path varied more by its own mood (one task took 7 turns in one run and 13 in the next) than by any byte limit I set. What decided the number of calls was not how much the server sent back. It was how many questions the agent had to ask to get an answer it could use. The budget was not adopted, and on hosts that save answers at 20 KB rather than 8 it would have split batches for nothing.

Step four: the server answers the question

So the question became the unit of design. For each benchmark task: what is the one call whose answer is the answer?

  • “Which ports does nothing connect?” is not a read of the connectors. It is a computed value, unconnected, on the component: the server walks the connectors and delegations and returns the ports with their direction and interface. One call on either bench.
  • “Which tasks run on this core?” The task knows its partition and the partition knows its core; the agent kept comparing the task’s partition with the core’s name and answering “none”. A computed core on the task, filterable, and the question is one call.
  • “How many components are rated B?” A count form. 152 bytes.
  • “Remove this component.” The agent had been reading the dependents, then removing each. remove with with_dependents lets the server remove the connectors, the composition members with their delegations, the task entries and the sequence steps that name it, and list every row it added in the review. Two calls on either bench, where it had been between four and fourteen.
  • “Is there something called brake temp?” A find tool: up to sixteen names at once, exact first, then ignoring case, then by whole words or their beginnings, then the nearest spellings, five candidates each with the address to send back unchanged. And a change set that names an element that does not exist is refused with the nearest candidates, so a typo costs one short answer instead of a search.

From this build on, each task ran once, not twice, to keep inside the month’s Copilot budget. Small bench: 57 calls, 17 of 17 right. Large bench, measured for the first time on an equal footing: 55 calls, 17 of 17. The eight change tasks cost 45 calls on the small bench and 41 on the large. For the first time the number of calls did not depend on the size of the model. Codex, which had answered five of nine questions on the large bench the build before, answered eight.

Step five: the answers the questions had left out

Three tools still answered with the model’s size rather than the question’s: impact analysis (36 KB on the large bench when asked about an interface the hub uses), the resource report (29 KB, whatever was asked), and the change set’s reply after a remove, which carried the removed component whole (9.8 KB). Each got a form the question chooses: impact as a count first, then a summary; the resource report for one core, one memory, or one fits question; a removed element’s before summarised to its identity, its required fields, its references and the counts of what it held, which halved the reply.

Two tools left the server. The port-compatibility and connector-consistency checks had been the agent’s “check again after the commit”, and the staging already does both when a change is staged. A tool that lets the agent re-verify what the server has verified is a call the agent will make.

Small bench: 45 calls, 17 of 17. Large bench: 72 calls, 16 of 17. The extra calls on the large bench are three change runs that read more before writing, and with one run per task that says which way a figure moved, not by how much. The wrong answer is the most instructive failure of the whole road: “does a 40 KB component fit the memory this core reaches?” The agent asked for the core’s resources, read a headroom of about 28 KB, and answered “yes”. The server has a fits form that decides exactly that threshold. I added a sentence to the profile telling the agent to use it and re-ran the task: the same two calls, the same answer, word for word but the order of a clause. On the small bench the agent asked fits and was right. Given a number and a question about a number, the weak agent compares them itself and gets it wrong, and no prompt fixes that reliably. The answer has to carry the verdict. That is the next change, and it is not in this release.

Task by task, the two benches on this build:

The same seventeen tasks on the showcase and on a bench thirty times its size: calls per task after the answers were bounded by the question

The questions cost the same at thirty times the size, one or two calls each. What still grows with the model is three changes that read before they write, the import of a component above all: 5 calls on the showcase, 17 on the large bench, eleven of them reads.

Step six: the questions nobody had asked

The benchmark had no requirement questions, and the requirements are where the data is least formal. Seven tasks: which elements satisfy a requirement, which requirements nothing satisfies, which links need confirming after a requirement changed version, which requirements talk about a topic, which elements could realise one, link a port, confirm every link at a new version.

Measured on the step-two build, before any requirement form, once per task: about 7 calls a task on the small bench with two of seven right, four on a re-run; about 9 calls on the large bench with five right, then four. Codex got one of five at best. Every failure was the same shape. The agent read element-level links and missed the one on a runnable; it re-derived “unallocated” by hand and got it wrong on every Codex run; it compared versions by eye and lost one of two stale links; asked for candidates, every run reasoned from the text and none reached the deterministic trace search the CLI already had. And the re-versioning of a link, which takes a remove and a create per site, failed on two refusals that were each exact and together did not say the one accepted shape.

So the server got three computed values on the requirements layer (who realises it, with the member sites; whether anything does; which links are stale against the version), a text search and a for this requirement search in find that reaches the same ranking the CLI uses, and an update of a requirement’s version that confirms its links in one operation.

After: 13 calls for the seven tasks on the small bench and 16 on the large, all seven right on both, and Codex five of five on both. About two calls a task. The question that had cost sixteen calls and a wrong answer on the large bench costs one and is right. The seventeen architect tasks, re-measured on the small bench with the new profile, cost 55 calls and were all right: ten more than the step before, with nothing tuned, which is the one-run spread again.

The road, in one table

Calls per architect task, Copilot, and the runs answered right. The first four rows are two runs per task; the last three are one. The budget row is the one that was not kept.

BuildSmall bench, 122 elementsLarge bench, 3 745 elements
The release8.7, 34 of 34not run
Schema in the profile, completion, the map, batched reads3.3, 32 of 34 (34 after the profile fix)the map refused at 128 KB
+ an 8 KB host budget, not kept3.6, 32 of 34—
The same design as a release build, the map bounded4.3, 33 of 345.5, 31 of 34
+ find, nearest names, remove with dependents, computed values, count3.4, 17 of 173.2, 17 of 17
+ answers bounded by the question, two tools withdrawn2.6, 17 of 174.2, 16 of 17
+ the requirement forms3.2, 17 of 17not re-taken

Seven requirement tasks: about 7 calls and two to four of seven right, then about 2 and all seven, on the small bench; about 9 calls and four or five right, then about 2 and all seven, on the large.

And the same seventeen tasks one by one, on the showcase, before the first step and after the last:

The same seventeen architect tasks before and after the redesign: calls per task on the showcase

The changes are where the calls went: the logger task from about 24 to 6, the import from 20 to 7, the remove from about 14 to 3. The questions that were already one call stayed one call, and the two that had cost four and six, the unconnected ports and the components in a partition, are one call each. One question got dearer, the ports of a component, from 2 to 4: one run, and a figure to watch rather than explain.

The same forms on two real models

The large bench is synthetic. On the day of the release I called every question form the profile names, once each, on two real AUTOSAR exports, through the same in-memory client a host talks to. No agent ran on them: the models are a customer’s, and nothing of their content leaves the machine, so this is answer sizes only.

Export AExport B
Components / interfaces / connectors148 / 1 199 / 1 44876 / 489 / 1 375
Largest component, by ports550428
Heaviest interface, by ports naming it41597
Ports no connector reaches107285

The large bench had been generated to export A’s proportions. Export B is the other shape: the weight sits in one interface, not one component.

Every form the profile’s recipes name stayed small on both: the baseline discover 2.4 KB, the map under 2 KB, every count under 200 bytes, find of four names 3.7 KB, the impact count about 200 bytes and its bounded page under 600, the batched summary of the sixteen largest components under 5 KB, the resource report under 400 bytes. The forms that grow with the model crossed 8 KB on both exports: the impact summary at 47 to 79 KB, and refused outright for export B’s heaviest interface; the unconnected ports of a whole layer at 18 and 56 KB, where the same read with a limit of twenty is 3.8 KB; find with sixteen names at 15 to 17 KB. And one question has no one-call form yet: “which of this component’s ports are unconnected” is answered over the layer’s rows, not the element, so on a real export it is a paged walk. That is a finding, and it is on the list.

What I would tell myself at the start

The agent’s cost is the number of questions it must ask, not the size of what it gets back. The 8 KB budget was an afternoon well spent, because it proved the opposite of what I expected.

A map is a small-bench design. Anything that lists the model stops working at the size where you needed it most. Count, and let the server search.

Every answer is bounded by the question, never by the model. This is the sentence the whole road reduces to. A question about one port’s connections should cost the same on 122 elements and on 3 745. If it does not, the server is making the agent do the server’s work.

A computed value beats a rule in the prompt. Every time I told the agent how to derive something (follow partition to core, compare the headroom, read the dependents first), the weak agent did it wrong often enough to matter. Every time the server derived it, the agent was right.

Design for the weakest agent, and measure every step on the large bench. The small bench flattered the map. The large bench is where the design is decided.

One run is an observation, not a rate. The same design measured 112 calls and 146 calls on two measurements. Any step that moves a figure by less than that spread has not been shown to move it.

Not everything is solved. The large bench still costs more calls than the small one once answers are bounded (about 4 a task against 2.6), one fit question is answered wrong by an agent that compares numbers itself, a member question about one element has no one-call form, and a Codex run that scripts its own loop can still make a thousand calls the server cannot refuse one by one. Those are the next measurements.

The surface now has twelve tools: one added, two withdrawn. I had treated “thirteen” as a result, not a rule: find was a new question, and a new question may be a new tool, with everything adding a tool does to the agent. The width of an answer is a parameter. The number of calls is a measurement. I will keep taking it.