Notes · 4
Writing
Notes from building and measuring AI tooling. Each one starts from a number I measured.
MCP tools cost 1.8× more than the estimate everybody usedI counted what the tool definitions of every server in the MCP registry cost in Claude. The common o200k estimate was low by 1.8×, and the gap only showed up once I asked for the exact number.The tool result is the big numberCode mode comparisons count tool definitions. In my runs the definitions took about 2 percent of the window, and one tool call took all of it. What a call returns decides whether a task runs at all.Measuring agents like flightsA demo that works once tells you nothing. What pass rate, flakiness and cost per success taught me about building AI agents – and why none of it works without a clean room per run.Thirteen tools through a sandbox, and the race that hid themA scanner can warn you about an untrusted MCP server. It cannot stop the code once it runs. What I learned from moving the server into a throwaway VM, and from the one bug that only a real cloud runtime could show me.