A VP of Engineering rolls out an AI coding agent to a few hundred developers. A quarter later, finance asks why the AI line doubled. The first instinct is to look at the model: a cheaper one, a smaller one, a different vendor.
That instinct skips the part of the system that may be spending the money.
AI coding agent costs are set by the model and the harness around it. In September 2026, Arena ran seven models through Claude Code, Codex CLI, and the open-source Pi harness, and found harness choice moved cost far more than success. On SWE-bench Lite, the same models cost about 2x as much in Claude Code as in Pi, while success rates stayed within about 2%. Arena calls that gap the harness tax.
What is the harness tax?
A model on its own is not an agent. The harness is the runtime that wraps it: it runs the loop, provides the tools, manages context, persists memory, and enforces control. We covered those five jobs in our guide to the agent harness.
Every one of those jobs consumes tokens. System instructions, tool definitions, command output, and the growing conversation all ride along on each model call.
The harness tax is what that machinery costs when it is not doing work your task needs. Developers rarely see it as a line item. Claude in Claude Code feels like one product, and so does GPT in Codex. The bill does not separate them.
How much does the harness change the cost of the same model?
Arena's team evaluated 21 model and harness pairs on 30 tasks each from SWE-bench Lite and Terminal-Bench 2.0, three attempts per task (Arena, September 2026).
The clearest single case is Claude Fable 5 on SWE-bench Lite:
| Harness | Success rate | Average cost per attempt |
|---|---|---|
| Claude Code | 97.8% | $1.33 |
| Pi | 96.7% | $0.67 |
One point of success, twice the cost. A team choosing between those two setups on success rate alone would see almost no difference and pay double on one of them.
Across all shared models, Claude Code cost about 2.0x Pi and 1.6x Codex on SWE-bench Lite, and about 1.5x Pi on Terminal-Bench 2.0. The harness moved success rates by roughly 2% on the first benchmark and 5% on the second.
Where does the extra cost come from?
Mostly from what the agent carries before it starts working.
Arena examined initial context as one of the drivers behind the cost gap, and the design difference is easy to see. Claude Code ships with detailed system instructions and a broad set of built-in tools, and every one of them is described to the model before work begins.
Pi takes the opposite approach. It calls itself a minimal harness, and in Arena's configuration the model had four tools: read, write, edit, and bash. Plan mode and subagents are left out of the default and added through extensions when a team wants them.
Context compounds. Within a session it does not reset between turns, so whatever the harness loads at the start is paid for again on every call. As we wrote in our Claude Code guide, an endless session drifts as the window fills. The same mechanism that degrades quality also raises the bill.
None of that makes Claude Code's context wasteful. Its permission modes, hooks, deny rules, and managed settings are the controls that make shell access safe to hand an agent. The question is whether your task uses them.
Should the model and the harness be separate decisions?
Arena's results say yes. You would expect each lab's model to do best in its own lab's harness. Across six Anthropic and OpenAI models and both benchmarks, a different harness recorded the highest observed success rate in nine of twelve comparisons.
This is not a verdict on any lab or harness. HatchWorks AI builds on Anthropic, OpenAI, Google, and an assortment of other models, and the pairing that wins changes by task. What the data supports is narrower and more useful: hold the model constant, change the harness, and measure. Pi makes that easy because it runs models from many providers in the same harness.
What should you measure before choosing a coding agent setup?
Model leaderboards compare Model A with Model B. Production compares stacks. Two teams on the same model can land on very different economics because their tools, context, and loops differ.
The unit worth measuring is closer to model × harness × tools × context × workflow. For each pairing, track:
- Task success: did it complete the work correctly?
- Cost per successful task: not per token, per outcome
- Latency: how long it took
- Retries: how often it had to recover
- Human intervention: how much steering it needed
- Context growth: what accumulated as the task ran
Cost per outcome is the line finance will ask about. Our AI agent cost optimization guide makes the same point for agents in general: most cost problems start at the architecture stage, before a line of production code ships.
When is the extra harness worth paying for?
When it does work your task needs. Arena's sample is two open benchmarks, 30 tasks each, and real repositories add requirements a benchmark does not test. Read the 2x as a reason to measure, not as a forecast of your bill.
| If your work looks like this | Lean toward |
|---|---|
| Well-scoped fixes, small diffs, a strong model | A lean harness and tight context |
| Shell access near production systems | A harness with deny rules, hooks, and audit |
| Many teams on one budget | A harness you can configure centrally and meter |
| Long tasks across a large, unfamiliar codebase | Richer context management, then measure it |
If more scaffolding moves your results, keep it. If results barely move, find out what you are paying for.
How HatchWorks AI treats the harness as part of the build
In AI-Driven Development, the configuration around the agent is part of the deliverable. Shared context, repeatable workflows, and quality gates with automated guardrails are built into the engagement rather than left to each developer's settings.
That is where the harness tax gets managed: what context every task starts with, which tools are loaded for which work, and which controls are enforced rather than requested.
Will better models need less harness?
Possibly, for everyday work. Arena's conclusion is that as models improve, coding agents may need less of today's scaffolding, while harder problems at the edge of model capability still benefit from structure.
Pi is a live test of that idea: give the model files, a shell, and a small set of tools, be deliberate about context, and see how much more it needs.
Before your next model decision, three questions:
- Do you know your cost per successful task, by harness?
- Have you ever run the same model in a second harness on your own repository?
- Which of your harness features would you miss if they were gone tomorrow?
If you cannot answer the first two, the model is not the next thing to change.
FAQ
What is the harness tax in AI coding agents?
It is the extra cost a coding agent's harness adds on top of the model's own work, mostly from instructions, tool definitions, and context loaded on every call. Arena uses the term for its finding that the same model cost up to about 2x more in one harness than another with similar success rates.
Does the harness affect the cost of AI coding agents more than the model?
In Arena's September 2026 study, harness choice moved cost by 1.5x to 2x across shared models, while moving success rates by only 2% to 5%. On those benchmarks, the harness was the larger cost lever. Results on your own codebase may differ.
Is a minimal harness like Pi enough for an enterprise team?
For well-scoped tasks with a strong model, a minimal harness can match larger ones at lower cost. Teams that need centralized permissions, audit, and guardrails around shell access may still want a fuller harness, and should measure what those features cost.
