Is the cheap model good enough? How to test a new model in your agent

Last week a paper called GVS5H (Gorilla vs 5 humans) hit the top of Hacker News with an interesting claim for those that pay a frontier-model bill. Wrap a small, cheap model in a manager/worker loop, and it closes the gap to the best models on hard coding problems. In the paper, Qwen3.8-Flash-Next goes from 84.2 to 93.0 on the 100 newest hard LiveCodeBench problems with the loop, against 90.4 for Fable 5 as a single call. Not just better, it was also an order of magnitude cheaper at $5.76 per pass for the Qwen loop against $61.11 for Fable 5. I wanted to do a test against this claim. If I swapped Opus 5.5, my daily driver, for Qwen3.8 Flash and put the GVS5H loop around it, would I save money without making the agent worse? This test is ideal material for a quick refresher on how you can use observability, evals, datasets, and experiments to check your own agents against different models to decide if a change to a newer/better/cheaper model is worth it. What this covers This post has seven steps, and each one is the input to the next: Define what good looks like. Without knowing what good looks like, you can’t check that your agent is better or worse when changing the model. Traces. Your agent, instrumented, sending every run to an Arize AX project. Those production traces are your baseline. Evals on those traces. Checks that score each run: did it generate code, does the code work, for loops did the loop make progress. A dataset. A set of real inputs pulled from the traces, mixing runs that passed and runs that failed. Experiments. Each candidate model or setup runs over the dataset, scored by the same evals, as well as scoring latency and cost. Compare. Pass rate, cost and latency side by side, measured against what you defined in step 1. Read the traces. Open the traces behind any difference, because the numbers alone can mislead you. The agent I’m testing My example agent solves competitive programming problems from LiveCodeBench, as I wanted to base my tests on the original GVS5H paper. The agent is one function with two settings: model and mode. Today it runs Opus 5.5 in single mode, calling Anthropic directly with the Anthropic SDK. That’s the current architecture, and the baseline every variant has to beat. There are three modes in this test: single: one model call, problem in, code out. This is the base mode that will run against the original model. manager: the GVS5H loop. A manager call writes a plan, a worker call brainstorms approaches, then the manager hands the worker one task at a time. Every call starts fresh and the only shared memory is a few files (plan.md, notes.md, solution.py). Public tests run after every worker round, and failing tests override the manager if it says “done”. ralph: the classic Ralph loop, added as another mode to compare against. The same prompt every round with a fresh context, with state kept in files on disk. After each round the code runs against the problem’s public tests, and the loop stops when they pass. The variants swap the model for Qwen3.8 Flash called through OpenRouter, giving me four setups to compare: Opus 5.5, single call: the current architecture, as the control. Qwen3.8 Flash, single call. Qwen3.8 Flash + Ralph loop. Qwen3.8 Flash + GVS5H manager loop. Both loops, the manager and Ralph loop, are capped at 10 rounds. Every setup uses the paper’s settings: reasoning effort high and a 128k output cap. The single call for Qwen gives me a baseline, so that I can measure how effective the loops are, and be able to attribute scores to the loops, not Qwen’s abilities. I also added the classic Ralph loop to see which loop approach is the most effective. Step 1: Define what good looks like My goal is to find out if swapping to smaller models using the GVS5H loop is worth it. But what does “worth it” mean? I’m focusing on these things: Cheaper: is the cost per solved problem lower. No noticeable loss in speed: is it as fast, or not noticeably slower to solve each problem. Same or better answers: the results of interactions need to be as correct and helpful with the new model. For testing against LiveCodeBench, this means it solves the same or higher percentage of coding problems. It’s important to define this up front so that you can plan your evals early. You don’t want to ignore latency, for example, then publish an unusable agent that saves you money on tokens, but loses more in sales as users get fed up waiting and bounce. I also picked a goal of completing each task in under 15 seconds. Short enough for a developer to not wander off and get coffee, or disappear into a doom scrolling loop. Step 2: Trace your agent Traces are the raw material for this entire process. Each one is a detailed representation of a real request with its input, its output, every model call, the token counts, the cost and the timing. Evals score them, datasets are built from them. Arize AX tracing is built on OpenInference, a set of OpenTelemetry instrumentors for model SDKs and agent frameworks. The instrumentor captures the model calls for you. Each call to the agent becomes a separate trace in Arize AX. For the single mode, this is fine, there is one call per problem. For the loops, each iteration of the loop would be a single trace using the instrumentor as is. This doesn’t model the real world of the loops, as an entire loop is really one trace. To make this work, I can wrap the loop in a chain span. This way all the spans captured for each run of that loop become children of that span, and I get one trace for the entire loop. For consistency, I wrapped all runs, including the single call, in a chain span. Top tip: put everything an eval might need into that root span’s output. Mine holds the final code, whether any code was produced, the number of rounds, the end-to-end latency, the total cost across every model call, and a round-by-round record of what the agent tried and how the public tests went. Every evaluator later in this post reads that output and has access to all the data it needs. It means my LLM-as-a-judge doesn’t need to process unnecessary tokens from all the spans for each iteration of the loop to find what it needs, it can just check the final output. My production traces My agent is already in production running the current architecture, Opus 5.5 as a single call. Every request is traced to an Arize AX project called gvs5h-screen, and each trace is one request to solve one problem. For this test I’m using 95 of them, all problems from the GVS5H set. These traces are also my baseline for everything I defined in step 1 that isn’t correctness. The 95 Opus runs cost $7.96 in total, with a median latency of 19 seconds. The slowest took nine and a half minutes. Step 3: Add evals to your traces A good rule of thumb of testing is to test one thing in isolation. That way you can attribute changes to that one thing. When testing agent changes as we will be doing in a later step, this rule means you should be testing them against the evals you already run in production so you can attribute the scores to the changes you are making to the agent, rather than having to prove it’s not an eval change that affects the scores. Essentially the evals you run on production traces must be the same evals you run on experiments. In my case, I am testing if the loops with a Qwen model make a difference. For my base scores, I have traces in production from running my agent, and have online evals built to run against every trace as it comes in. You don’t need an evaluator for cost or latency. Arize AX already records both on every trace, and experiments show them for every run, so they’re there to compare without writing any code. I used three kinds of evaluator: Code evaluators for things you can compute Code evaluators are for the deterministic checks that are specific to your agent. Mine is produced_code: did the run end with any code at all? It separates “wrong answer” from “never answered at all”, and that difference is important, as we’ll see in step 7. It’s a few lines of Python that read the root span’s output. Code evaluators run in a sandbox in Arize AX. I wrote one more in the same shape for the manager loop, false_done, which flags runs where the manager says “done” while the public tests are still failing. Remote evaluators for checks that need your infrastructure The most important eval for a coding agent is “does the code actually work”, and that means running untrusted model-written code against hidden tests. You don’t want that running anywhere near anything you care about, and the hidden tests shouldn’t leave your machine either. Remote evaluators handle this. Arize AX sends the run to an HTTPS endpoint you own, and your endpoint returns a label, a score and an explanation. Mine is a small FastAPI service running in a Podman container with no network access, a read-only filesystem and no Linux capabilities, exposed through a Cloudflare tunnel. This pattern is great for local runs for testing, and you can use hosted sandboxes in the cloud for production evals. The Arize AX remote evaluator cookbook has an example of this in action. I created two remote evaluators. correct uses the GVS5H team’s fixed LiveCodeBench grader. correct_original uses the upstream grader, which has the bug the paper found: a broken mock of sys.stdin.buffer.readline that always returns the first line of input, which the paper says moved scores by up to 12.6 points. Running both lets me test the eval itself. I checked it against known answers. All 97 known-good solutions shipped with GVS5H pass it, and 43 of the 47 cases the paper says flip between the two graders flip for me too. This gave me confidence in my evals. LLM judges for things no test can check Tests tell you whether the final code works. They can’t tell you whether a loop made real progress between rounds or jus