Notes ·
The tool result is the big number
Code mode comparisons count tool definitions. In my runs the definitions took about 2 percent of the window, and one tool call took all of it. What a call returns decides whether a task runs at all.
Two numbers, one of them measured
An MCP tool fills an agent’s context in two ways. The definition is the card that says which tools exist. The result is what comes back from a call. Both stay in the window and get paid for in every later turn.
Almost every published number about code mode measures the first. Code mode is the pattern where the agent writes a small program that calls its tools in a sandbox, and only what the program returns reaches the model. The best-known figures for it are a worked example and a count of definition bytes. Neither counts what a task consumed. I did not find a code mode benchmark that measures the result size as its own term, so I set up codemode-lab to measure it.
The card is small
I took three public MCP servers. Together they expose 8 tools, and the whole definition surface is 11 268 bytes. That is about 3 000 to 4 700 tokens, 1.5 to 2.3 percent of a 200K window. A real cost, and a small one.
Then I called one of those tools. One call against the documentation of cloudflare/agents returned 683.7 KiB of text, which is 700 150 bytes. For apify/apify-mcp-server it was 728.2 KiB. In tokens that is 92 to 155 percent of the same 200K window, from one call to one tool.
The band is wide because I count bytes, not tokens, and I only trust the ratio where I billed it. One result of 392 601 bytes grew a prompt on claude-sonnet-5 by 163 895 tokens: 2.4 bytes per token. Before that measurement my band ended at a kinder number, and every share of the window I printed was too low.
So the definitions take two percent and the result takes the window. If you optimise the two percent and ignore the rest, you are tuning the wrong term.
What happened when a model had to read it
Five questions, each asked twice with the same model and the same tools: once by an agent that calls the tools directly, once by an agent that writes a program. Five runs per arm. Tokens read off the API response, so billed, not estimated.
On the two large tasks the direct agent could not hold the result. Its client writes any result over 25 000 tokens to a file. An arm that reads that file in parts can reach the data, but every part stays in the prompt of every later turn. On one task that came to 1.7 million prompt tokens in the median run, and it ran out of turns at 40 in three runs of five. An arm with a raised limit got the whole result into its window: 168 548 prompt tokens for one question.
Code mode read up to 1.33 MiB on those tasks and returned under 2 KiB. Its prompts stayed near 5 500 tokens, and it answered correctly in all 20 of its runs. At list price the direct arms cost 12 to 63 times as much, and they were right less often.
So the difference is not a percentage saved. On a large result the question is whether the task finishes.
Where it was close
A comparison with no losing case is marketing, so I wrote a prediction down before each run, and two of the five said code mode should lose.
Where the data fits in the window, the two arms came out close. On one task the prompts were within 1 percent of each other, on another within 3 percent. Code mode wins only when its program returns much less than it reads.
One predicted loss was a clear win. On a task I called for the direct arm, the program counted entries in the sandbox and returned 102 bytes. It used 35 percent fewer prompt tokens and cost 31 percent of the direct arm’s list price. Two of my four directional predictions held on prompt tokens.
The result size also explains how this kind of number gets gamed. One losing task first showed a 17x saving, because the program returned a 300 character excerpt of a document the task asked for in full. Returning the whole document scored 1x. A ratio is only honest when both arms answer the same question.
The number I still do not have
The open question is the break-even point: at what result size, with and without the prompt cache, does code mode start to pay. One study found a break-even for the number of calls. Its tools returned nothing, so it could not find one for size. I have not found one either.
What I have is a rule I now apply before I pick an approach. I check what one call returns, in bytes, against the window. If it fits with room to spare, the approach matters little. If it does not fit, the result has to be processed outside the context, and the only real choice is where.