# codemode-lab

> MCP cost tools audit the tool definitions. That is the small number. What one tool call gives back is the number that decides whether your task runs at all, and code mode benchmarks do not measure it as its own term.

- **Role:** Author & Maintainer
- **Year:** 2026
- **Tech:** TypeScript, MCP, Code Mode, Astro, Sandboxing, Vitest, Claude Agent SDK
- **Repository:** https://github.com/ondraulehla/codemode-lab
- **Canonical page:** https://ulehla.dev/projects/codemode-lab

**Problem.** The best-known numbers for MCP "code mode" are illustrations or counts of tool definitions, compared against a baseline nobody would ship. The careful studies measure definitions and the number of calls, not the size of what one call returns.

**Solution.** Five questions, each asked twice: once by an agent calling tools directly, once by an agent writing a program. Every prediction was written down before the run, and two of the five say code mode should lose.

**Outcome.** Five runs per arm on claude-sonnet-5. When the direct arm had to read a large result, it cost 12 to 63 times as much as code mode at list price and answered correctly less often. Where the data fits, the two arms came out close, and one task called as a code mode loss was a clear win. Two of four directional predictions held on prompt tokens.

---

## An agent has a desk, and the desk is small

An agent has a finite memory. Every tool result lands in that memory and stays there.

Picture an assistant at a desk. You ask which of five thick manuals mentions a thing.

Way A photocopies all five manuals and piles the copies on the desk. The desk overflows after
the second manual. The assistant never finishes, and the work they did do is buried.

Way B sends a note to the archive. Go through these five, count the mentions, come back with one
sheet. The desk only ever holds one sheet.

The desk is the context window. The photocopies are tool results. Way B is code mode: the agent
writes a small program that calls its tools, the program runs in a sandbox, and only what it
returns reaches the model.

The headline numbers for way B are striking. Anthropic's write-up shows 150,000 tokens falling to
2,000. Cloudflare reports 99.9 percent. Both are honest about what they are, if you read closely:
the first is a worked example of a hypothetical task, and the second measures the size of a tool
definition blob across 2,500 endpoints. Neither counts the tokens a task consumed.

So I measured it.

## Most numbers measure the catalogue card

Before an agent calls anything it reads a card that lists which tools exist. That card is what
most published cost comparisons measure. It is the **tool definition tax**. It is a real number,
and it is easy to measure.

It is also the small one. Three public MCP servers expose 8 tools between them, and their whole
definition surface is 11,268 bytes. About 3,000 to 4,700 tokens. That is 1.5 to 2.3 percent of a
200K window.

Then I called one of those tools.

| repo                   | text returned | of a 200K window |
| ---------------------- | ------------: | ---------------: |
| apify/apify-mcp-server |     728.2 KiB |       98 to 155% |
| vercel/ai              |     712.8 KiB |       96 to 152% |
| cloudflare/agents      |     683.7 KiB |       92 to 146% |
| nodejs/undici          |     274.4 KiB |        37 to 59% |

The byte counts are exact, measured on 21 September 2026. A KiB is 1,024 bytes. The right-hand
column is not exact: this case study counts bytes and ships no tokenizer, so it estimates tokens at
2.4 to 3.8 bytes per token and prints every such figure as a band.

The high end of the band comes from a billed count. In the sweep further down, one result of
392,601 bytes grew the prompt of claude-sonnet-5 by 163,895 tokens, the median of five runs: 2.4
bytes per token. claude-opus-5 read it the same way. The band used to end at 3.3, and every share of
the window on this page was lower than it is now.

Take the third row. 683.7 KiB is 700,150 bytes, and that is one call to one tool.

**The card and the manual, drawn to one scale.** Byte counts, exact.

| bar | what it is | bytes | share of the manual |
| --- | --- | ---: | ---: |
| the card | DeepWiki, 3 tools | 1,526 B | 0.22% |
| all three cards | 8 tools, three servers | 11,268 B | 1.61% |
| one manual | one call, one tool | 700,150 B | 100% |

One manual is 459 times the card of the server that produced it, and 62 times all three cards together.

A ratio with no denominator is how this kind of claim goes wrong, including for people who are not
trying to mislead anybody. That is why both denominators are named above.

The result size is not unknown. Anthropic's own post describes results passing through the
context, and one benchmark counts how often a result is too long. What I did not find is a code
mode benchmark that measures it as its own term, call by call. The card is the small number.

## The recorded run

The same three calls, made twice against the live DeepWiki server, once, on a laptop. No model ran
in this recording. Arm A shows what a client that keeps every result would have to hold. Arm B is
the real program in a sandbox. The results further down are the run with a model in it.

**The recorded run, as figures.** The same calls, made twice against the live DeepWiki server on 21 September 2026. No model ran. The run took 94 s. Byte counts are exact. Token figures derived from them are estimates, printed as a band.

*Arm A, direct tool calling, simulated. It shows what a client that keeps every result would have to hold. Claude Code does not keep them: it writes a result over 25,000 tokens to a file.*

| call | text into the window | time | state |
| --- | ---: | ---: | --- |
| cloudflare/agents | 700,150 B | 45.3 s | returned |
| e2b-dev/E2B | 412,636 B | 24.2 s | returned |
| nodejs/undici | – | – | not sent |

Arm A put 1,112,786 B into the window after 2 of 3 planned calls. That is an estimated 293,000 to 464,000 tokens, or 146 to 232 percent of a 200,000 token window.

Arm A then stopped: The results already in the window pass 200,000 tokens at the most generous rate this page uses. The remaining calls are not sent. Unsent: nodejs/undici.

*Arm B, code mode. The same results land inside a sandbox.*

| payload pulled into the sandbox | bytes |
| --- | ---: |
| cloudflare/agents | 700,150 B |
| e2b-dev/E2B | 412,636 B |
| nodejs/undici | 280,987 B |

Arm B held 1,393,773 B inside the sandbox and returned 462 B to the model, which is 3,017 times less. Counting the program and the answer together, 1,889 B reached the window, an estimated 497 to 787 tokens, or 0.2 to 0.4 percent of the same window.

What the program printed inside the sandbox:

- cloudflare/agents: 699986 chars, 61 pages, top page 62 rows
- e2b-dev/E2B: 412590 chars, 48 pages, top page 32 rows
- nodejs/undici: 280976 chars, 20 pages, top page 52 rows

The question: DeepWiki returns each repository's documentation as one markdown dump. The dump is a series of pages. Every page starts with a line of the form '# Page: <title>'. Count a table row as any line that starts with the pipe character. For cloudflare/agents, e2b-dev/E2B and nodejs/undici, find the page that holds the most table rows. Give the exact page title for each repo, copied character for character, and copy the first table line of that page as evidence.

## What a real agent did

Five questions, each asked twice. Once by an agent calling the tools directly, once by an agent
writing a program that calls them inside a sandbox. Same model, same question, the same tools
allowed. Each approach ran as two arms, one with the prompt cache and one without, and every arm ran
five times.

I wrote the prediction for each one down **before** the run, and two of the five say code mode
should lose. A comparison with no losing case is marketing.

The harness runs those five tasks through the Claude Agent SDK, on claude-sonnet-5 at low effort,
and reads the tokens off the API response. Those figures are billed, not estimated.

On the two large tasks there is a catch. Claude Code writes any tool result over 25,000 tokens to a
file, and the plain direct arm has no tool that can open one, so it never sees the data. A normal
session could open that file, so a third arm does: it keeps Read and Grep and reads the file in
parts.

**The results.** Five tasks, model claude-sonnet-5, 5 runs per arm, 22 September 2026. "Read, then returned" is what the code mode program took from the tools and what it handed back to the model. The cost is the list price of the billed tokens next to code mode, at prices of 2026-06-24: an equivalent of billed tokens, not a bill. "About the same" means the runs do not separate the two.

| task | called before the run | read, then returned | code mode | direct | direct, reads the file |
| --- | --- | ---: | --- | --- | --- |
| `table-heavy-page` | code mode, held | 1.33 MiB, then 1.4 KiB | right 10 of 10 | never saw the data | right 2 of 5, 63x the cost |
| `one-big-payload` | code mode, held | 383.4 KiB, then 451 B | right 10 of 10 | never saw the data | right 3 of 5, 12x the cost |
| `structure-rank` | direct, did not hold | 4.4 KiB, then 102 B | right 10 of 10 | right 10 of 10, 3x the cost | not run |
| `outline-leaves` | direct, tie | 1.0 KiB, then 1.0 KiB | right 10 of 10 | right 10 of 10, about the same cost | not run |
| `topic-overlap` | no call made | 9.0 KiB, then 9.8 KiB | right 10 of 10 | right 9 of 10, about the same cost | not run |

*The exact figures.* Prompt tokens (input, cache reads and cache writes) counted and billed by the Anthropic API, the median of the uncached arms. No token figure in this table is an estimate. The ratios are code mode over direct at list price: below 1x, code mode cost less.

| task | predicted | direct | code mode | direct answer | code mode answer | list price, uncached | list price, cached | prediction |
| --- | --- | ---: | ---: | --- | --- | ---: | ---: | --- |
| `table-heavy-page` | code mode | 7,105 | 5,715 | never saw the data | correct | 0.67x | 0.76x | prompt tokens held, list price held, cached held |
| `one-big-payload` | code mode | 5,458 | 5,456 | never saw the data | correct | 0.98x | 0.86x | prompt tokens held, list price held, cached held |
| `structure-rank` | direct | 7,769 | 5,039 | correct | correct | 0.31x | 0.30x | prompt tokens not held, list price not held, cached not held |
| `outline-leaves` | direct | 5,297 | 5,269 | correct | correct | 1.04x | 0.72x | prompt tokens not held, list price held, cached not held |
| `topic-overlap` | uncertain | 10,717 | 10,456 | correct | correct | 0.96x | 1.06x | no verdict |

Of the two rows written down in advance as losses for code mode, on `structure-rank` code mode used 2,730 fewer, and on `outline-leaves` code mode used 28 fewer prompt tokens than direct.

| task | direct arm that reaches the data | direct | code mode | direct right | code mode right | direct over code mode, list price |
| --- | --- | ---: | ---: | ---: | ---: | ---: |
| `one-big-payload` | A-files, reads the spilled file | 163,814 | 5,531 | 3 of 5 | 5 of 5 | 12x (6x to 21x) |
| `one-big-payload` | A-raw, holds the whole result | 168,548 | 5,456 | 4 of 5 | 5 of 5 | 28x (26x to 30x) |
| `table-heavy-page` | A-files, reads the spilled file | 1,683,743 | 5,347 | 2 of 5 | 5 of 5 | 63x (28x to 79x) |

In 35 of 55 runs, an arm set to run with the prompt cache off read it all the same. That is why these tables count the whole prompt and not input tokens alone.

"Never saw the data" means Claude Code wrote the result to a file and the arm had no tool to open it.

What the harness says about its own limits, in its own words:

- Every token figure here is what the Anthropic API counted, read from modelUsage on the final result message.
- Per-turn figures are read from each API response and cover the input side only.
- Both arms declare the same tools: the direct arms have every other tool of the server removed.
- Claude Code writes an MCP result over 25,000 tokens to a file. A-uncached and A-cached have no tool to read it; A-files does; A-raw raises the limit.
- The Node worker used by arm B is an isolation boundary for measurement, not a security boundary.
- No dollar figure here is billed. Costs derived from these tokens use list prices and say so.

**Code mode wins when its program returns much less than it reads.** On the two large tasks it read
up to 1.33 MiB and returned under 2 KiB. On outline-leaves and topic-overlap it returned about as
much as it read, and the two arms came out even.

**Where the data fits, the two arms are close.** On outline-leaves the two prompts are within 1
percent of each other. On topic-overlap they are within 3 percent, and code mode answered correctly
in all ten of its runs, the direct arm in nine.

**One predicted loss was a clear win.** structure-rank was called for direct. Code mode used 35
percent fewer prompt tokens and cost 31 percent of the direct arm's list price. The direct arm read
four outlines into its window and wrote about three times as many output tokens, 854 against 293.
The program counted the entries in the sandbox and returned 102 bytes. So two of four directional
predictions held on prompt tokens, and the miss on structure-rank is not noise.

**Reading the file works, and it costs.** The third arm reads the file in parts, and every part
stays in the prompt of every later turn. On table-heavy-page that came to 1.7 million prompt tokens
in the median run, and it ran out of turns, at 40, in three runs of five. A fourth arm raises the
limit, so the whole result lands in its window: 168,548 prompt tokens for one question, about 28
times the cost of code mode. Code mode answered both tasks correctly in all 20 of its runs, with
prompts near 5,500 tokens.

The first sweep found the direct arm wrong on topic-overlap, with every result in its window. On
claude-sonnet-5 it was right in nine runs of ten. A smaller check on claude-opus-5, with the same
harness and two runs per arm, found it wrong in all four runs and code mode right in all four. So
that finding holds for one model and not for the other.

## How I know I am not fooling myself

The first measurement was wrong four ways. Reading the transcripts is what found all four. Each one
would have produced a number, and every one of those numbers would have been wrong.

1. **The model already knew the answer.** The question asked whether some repositories document a
   WebSocket API. The model could answer correctly having called no tool at all. The benchmark
   measured recall, not data access.
2. **The answer was inside the question.** The strings the grader looked for appeared in the
   question itself. Restating the question passed the grade.
3. **The two arms had different tools.** The direct arm loaded three tools. Code mode got one. The
   first fix let both arms call only the same tool, but the direct arm still carried the two other
   definitions in its context. Now they are removed.
4. **A permission setting bypassed a ban.** One tool was banned so the run would not spend a
   provider's money. A permission setting silently overrode the ban, and the tool was called anyway.

Every replacement question is now graded on a string the documentation generator invented, not on
anything the repository says. Each string is checked to return zero hits inside its own repository.
A test fails the build if a grading string ever appears in its own question again.

A second review, a day later, found nine more faults. The ones a reader of this page should know:

- **The number I published as the tool tax was not the tool definitions.** It was every input token
  over a tool-less baseline across every turn, tool results included. The definitions are now
  counted on the first request only, where they are the only difference.
- **The cached code mode arm counted the uncached arm's calls too.** One counter was reset once per
  task instead of once per arm.
- **The typed surface told the model the wrong return type.** It promised an object and the sandbox
  delivered a string. A test now type-checks every reference program against the surface the model
  reads.
- **The daily check that re-measures every claim went green while it failed.** A pipe swallowed its
  exit code, and it stayed green through two stale claims.
- **Only one arm got the server's instructions.** Claude Code hands a server's instructions to the
  direct arm, about 3 KB for DeepWiki, and nothing handed them to code mode. Every code mode request
  started lighter for a reason that had nothing to do with code mode. Both arms get them now.
- **The page said "no answer" where the direct arm had answered wrongly.** The results table now tells the two apart.

The sweep itself found three more:

- **An arm set to run without the prompt cache used it anyway.** It happened in 35 of 55 runs. The
  harness now flags such a run and keeps it, and the summary counts the whole prompt, cache reads
  included. A run like that costs less at list price than a truly uncached one, so the uncached price
  ratios are the noisier ones.
- **Raising the output limit did not put the result in the window.** Claude Code has a second limit
  that writes a large result to disk, and only the tool itself can raise it. The arm with the raised
  limit now calls the live server through a small proxy that declares that limit.
- **An arm that ran out of turns lost its whole record.** The SDK throws after its last message, and
  the harness dropped the run with its tokens. It now keeps the run and counts it as not completed.

**A ratio is only honest when both arms answer the same question.** One of my losing tasks reported
a 17x saving. The program was returning a 300 character excerpt of a document the task had asked
for in full. Returning the whole document scored 1x. The second number is the true one, and the
first is how a volume comparison gets gamed.

**One number needs one definition.** I recorded the definition size as wire bytes for one server
and as JSON length for another, and the daily liveness probe then reported a 75 percent drift that
had not happened. It was comparing a handshake reply against a tool list. The repo now measures
`JSON.stringify({ tools })` everywhere, whatever transport the server speaks.

**One label needs one unit.** "kB" meant 1,024 bytes in one place and 1,000 in another, so one
payload appeared at two sizes, and once a KiB figure was written as bytes. Sizes are now printed in
KiB and MiB.

### The caveat, not buried

The first sweep ran every arm with no file tools, and a normal session keeps them. That caveat now
has an answer. The arm that reads the file is that session. On the two large tasks it cost 12 and 63
times as much as code mode and was right less often.

The narrow claim stands. A payload that does not fit has to be processed outside the context window.
Code mode is one way to do that, and a file with a search tool is another. On these two tasks the
file was the expensive way.

What remains is the model. The check on claude-opus-5 agrees on cost. Code mode used 36 percent
fewer prompt tokens on structure-rank there too, and outline-leaves was a tie again. On
one-big-payload, the only large task it ran the two arms that reach the data on, both answered
correctly and cost 9 and 27 times as much as code mode. So the narrow loss on the first sweep came
from the harness it ran on, which has been fixed since, not from the model. Where the two models
differ is accuracy, on topic-overlap. Both ran at low effort.

## What others found

- **Anthropic's own documentation** reports a case where the pattern loses: on τ²-bench, scores did
  not change and cost rose about 8 percent.
- **A study of 425 runs on GitHub tasks** put code mode at 84 percent success, 0.101 dollars a task,
  and the slowest of five approaches. A plain command line tool reached 86 percent at 0.054 dollars.
- **A study of 14 models** found that programmatic tool calling costs more tokens below about 26
  parallel calls. Its tools returned nothing, so it measured the number of calls, not the size of a
  result.
- **A test with 50 runs a task** cut input tokens by 78.5 percent, mostly tool definitions, and
  raised output tokens by 120 percent.

The sources, and what each one measured, are in the repository's
[prior art notes](https://github.com/ondraulehla/codemode-lab/blob/main/docs/prior-art.md).

## What comes next

One open question: where is the break-even point? At what result size, with and without caching,
does code mode start to pay? One study found a break-even for the number of calls. I found none for
the size of a result, and that is the number this project is built to measure. The harness now has a
synthetic server that returns a result of a size it is given, so the next run can look for it.