Code mode: make your agents 2x faster and 2x cheaper on long tasks

Everyone I talk to has an opinion about code mode right now. Mine was lit by Prime Intellect's Prime Agent, a coding harness built around a persistent REPL, and by Alex Zhang's speculative programmatic tool calling, which (among other things) can run tools while the model is still streaming code. Cloudflare has one. Anthropic has one. We've had a few of our own, from MCP code execution to Pydantic Monty. Coding agents are moving the same way: OpenCode 2 now sends MCP tools through Code Mode by default, and Pi ships a codemode tool. The pitch usually goes like this: stop making the model call your tools one at a time, let it write a little program that calls them instead, and everything gets cheaper, faster and better. I believe that. But almost every number I'd seen came from agents with hundreds of tools or huge payloads, and many agents in the wild are boring: a handful of CRUD tools and a job to do. So I built a benchmark out of exactly that. Five small Pydantic AI apps, single-item tools only, and three kinds of work: everyday requests, long multi-step jobs, and five tasks I could dial from 5 items to 200. Eight models, code mode off and on, 5,276 graded runs on an internal benchmark. The short version: the longer the task, the bigger the win. And it gets big. #TL;DR On long tasks, code mode roughly halved cost and wall time. Across all eight models: cost -58%, wall time -51%, success +16.0 points. Without code mode 67 of 288 runs failed; with it, 21. It scales. For gpt-6.1-sol, every 100 extra items cost 30 to 79 seconds and $0.04 to $0.07 per run without code mode. With it: 4 to 9 seconds on four of the five tasks, and under a cent on three. At 200 items, code mode was 47% to 75% cheaper. Correctness falls off without it. From 5 items to 200, success without code mode dropped from 92-100% to 62-79%. With code mode it was 92-100% at 200. It isn't about round trips. Models already batch: one response carried 200 tool calls. At 200 items, averaged over the eight models, model requests fell by between 1% and 22% per task; deepseek-v4.1-flash on its own fell much further. What code mode cuts is what goes into each request. Small jobs pay a tax. At 5 items code mode cost more on three of the five tasks (+21% to +38%). On the 50 everyday tasks it was 16% cheaper overall, but success was flat (+1.6 points, noise). It's sandboxed, and it leaves a trail. In Pydantic AI Harness the model's code runs in Monty, which starts with no access to the host and, by default, stops any snippet after 30 seconds of execution time or at a 256 MiB heap. With Pydantic Logfire tracing the agent, every tool call that code makes shows up as its own span. How much faster and cheaper each extra 100 items are with code mode on, for gpt-6.1-sol: Task Faster Cheaper cart 7x 8x calendar 9x 25x+ chat 5x 2.4x kanban 20x 11x inventory 2.2x 2.2x My read: models are much better at writing a loop than at being one, and the gap grows with the length of the task. #The loop, and why it hurts In a normal tool-calling agent, the model picks a tool, emits some JSON, stops, waits for the harness to run it, reads the result, and picks again. The model sits in the middle of every step, and the costs compound: tool definitions crowd the context, every intermediate result flows through the model, and, the usual story goes, every tool call is another model turn that resends the whole growing context. Code mode flips it. The tools become functions inside a sandbox, the model writes one program that loops and branches over them, and only the result comes back. In Pydantic AI Harness that sandbox is Monty, our Python interpreter written in Rust, and the model sees a single run_code tool. Here's one of the 50 tasks, verbatim: "can you get rid of everything in my cart that costs more than $50 a piece? it's getting out of hand". The cart has 122 lines, a list tool that returns at most 50 per page, and a remove tool that takes one id. 22 lines need to go. Without code mode that's three pages of listing and 22 removes, every one of them chosen by the model. With code mode, it's this: import asyncio items, offset = [], 0 while offset is not None: page = await list_items(offset=offset, limit=50) items += page["items"] offset = page["next_offset"] pricey = [i for i in items if i["unit_price"] > 50] await asyncio.gather(*(remove_item(item_id=i["id"]) for i in pricey)) len(pricey) The model never reads the 100 cheap lines. It doesn't have to remember which ids it already removed. The comparison > 50 is done by Python, not by a language model squinting at a float. #Code mode with guardrails That program is code a model wrote, and something has to run it. What that something lets the code touch is the biggest design decision in code mode. Prime Agent makes the open choice. Its model's only tool is a persistent IPython kernel, and the rest of the harness, sub-agents included, lives in that kernel as functions. An IPython kernel is a full Python process: unless something around it says otherwise, the code can import any library, read and write files and open network connections. For a coding assistant on your own machine, that's what you want. It's not something I'd put in a web server and let every user's prompt drive. Pydantic AI Harness makes the opposite choice. CodeMode runs the model's program in Monty, our Python interpreter written in Rust, which starts from nothing and adds only what you allow: No host access by default. Sandboxed code starts with no access to the host's files, environment or clock, and there is no network. If the agent needs a directory, an environment variable or the time, you grant it per request. This benchmark granted the built-in OSAccess handler: an in-memory filesystem and environment, and the host clock. Your tools are the way out. The program reaches the world through the functions you register, plus any OS access you grant it, and each tool call goes through Pydantic AI to your real tool. It can do what those allow, and nothing else. Hard limits. By default each run_code snippet gets 30 seconds of execution time and a 256 MiB heap (details), which is what stops a runaway loop. Fast to start. Each Monty session runs in a local worker process, and a new sandbox takes under a millisecond from a running pool, so the guardrails don't add a container start to every snippet. So you get code mode with guardrails. Every code mode run in this benchmark ran in Monty. #The code is the graph Samuel wrote a whole post about this, The code is the graph, and if you build agents you should read it. His argument is that graph libraries rebuild control flow badly: "if is a branch. for is a loop. A function call is an edge", and asyncio.gather is a fan-out and a join. Or in his words about a five-line example: "Fan out, join, branch. That is a graph, and it's also a Python function." The program above is exactly that. Here's the graph it encodes: I wrote that program for the figure, but the models write the same shape on their own in the benchmark, at run time, for a task they've never seen (in one cart run, gpt-6.1-sol spread it over three run_code calls). Nobody drew that graph in advance. That's the bit that makes me think code mode is where agents are going: the model isn't just calling your tools more cheaply, it's designing the workflow on the fly. But this is a blog post with a benchmark in it, so let's find out whether any of that is true. #The benchmark The benchmark is trivial on purpose. Five tiny Pydantic AI apps: a shopping cart, a calendar, a chatroom, a kanban board and an inventory system. Each one has only single-item CRUD tools (list, get, create, update, delete), with paginated lists. No shell, no filesystem, no bulk endpoints. If the agent wants to change 200 things, it has to make 200 calls, or write a loop. No single step here is hard. What's hard is doing an easy step a few hundred times, in the right order, without losing count, and that's the part I wanted to isolate: iteration over a long horizon. The code is the graph. Let your agent write the graph. On top of those apps sit three tiers of work: Everyday: 50 tests, ten per app. Single edits, bulk changes, rules with branches, lookups and sums. Without code mode, a run made about 13 tool calls. Long horizon: 12 multi-step jobs over 60 to 500 records, two per app and two that cross apps. "Alex is leaving the team. go through Alex's open tasks (not done) in order of id: if it's urgent or high priority, give it to whoever has the fewest open points at that moment..." About 68 tool calls per run. Scaling: one task per app, with the same rule at N = 5, 20, 50, 100 and 200 items. "I have a pile of meeting invites I never answered. for each one: accept it if it's on a weekday and ends by 5pm, decline it if it's on a weekend or ends after 5pm." At N = 200, runs without code mode averaged 168 tool calls. Each app is a Pydantic AI agent. With code mode on, it gets Pydantic AI Harness's CodeMode capability with eager execution, which can start running a complete statement before the model finishes writing the run_code call. With code mode off, the same agent calls the same tools directly. Everything was pinned to one commit, and code mode was flipped with a single setting, so the only difference between arms is code mode itself. The prompts are written the way a person actually types, and every run is graded by a deterministic oracle: the app's end state has to match the expected state exactly (money within a cent), and the reply has to contain the right numbers. No LLM judge. Methodology Availability: the benchmark code isn't public, so this note gives the full setup: versions, models, cost, arms, runs, validation and statistics, plus its known warts. System under test: Pydantic AI 2.56.1.dev13 and Pydantic AI Harness 0.56.1.dev13, both built from pydantic/pydantic-ai commit b82f0f258f4d, the same settings in every arm apart from code mode.