The Agent Harness: What to Build Around the Model and What to Rip Out

Give two teams the same model and the same task, and you can get a working app from one and a pile of half finished features from the other. The model didn’t change. What changed is everything around it: what it sees, which tools it can reach, where its commands run, and when it has to stop and ask. That surrounding code is the harness, and it has quietly become the part of an agent people actually engineer.

A name for the code around the model

Anthropic’s April 2 post on agent harness design gives the cleanest definition I’ve seen. The harness is the software scaffolding around a model, meaning the loop, the tools, context management and guardrails, and harness design is deciding what goes into that scaffolding and what you can take back out as models get better. OpenAI describes its own version almost the same way. In an August 19 post on building on the Codex harness, they call the harness the execution system around the model, one that manages conversation state, streams execution, uses tools, enforces sandbox and approval policies, and carries work across turns. They also open sourced it, so you can read that layer instead of guessing at it.

The practical consequence is that you never really test a model on its own. Anthropic’s January guide to evals for AI agents says it outright: evaluating an agent means evaluating the harness and the model working together. Their example is a little humbling. Opus 4.5 first scored 42% on CORE-Bench. After a researcher dug in, fixed grading bugs and ambiguous tasks, and switched to a less constrained scaffold, it scored 95%. Part of that jump was the grader rather than the harness, but the lesson holds either way. OpenAI’s post has a harness only version of the story, where adding retained reasoning and context compaction took a model’s ARC-AGI-3 score from 13.3% to 38.3% while using about a sixth of the output tokens. When an agent looks dumb, the harness deserves a hard look before the model does.

Work that outlives a context window

Things get interesting when a job takes longer than one context window, because every fresh session starts with no memory of the last one. Anthropic’s November post on harnesses for long-running agents compares it to a project staffed by engineers working in shifts where nobody remembers the previous shift. The Claude Agent SDK already had compaction, and that still wasn’t enough. Given only a high level prompt, the agent either tried to build the whole app at once and ran out of context halfway through a feature, or a later session looked around, saw a lot of code, and declared the job done.

Their fix borrowed habits from human teams. A first session with a different prompt writes a feature list as a JSON file (over 200 features for their claude.ai clone), all marked as failing, plus an init.sh script to start the dev server, a progress notes file and an initial git commit. Every later session reads the notes and the git log, runs a quick end to end check to catch anything the last shift broke, picks one failing feature, tests it the way a user would, and finishes with a commit and a progress update. They landed on JSON for the feature list because the model was less likely to wrongly change or overwrite a JSON file than a Markdown one.

The harness in that post protected the feature list with strongly worded instructions. My suggestion is to back rules like that with code wherever you can, because a prompt is a request and a check is a guarantee. Here’s a small check an orchestrator could run after each session, comparing the file from the last commit with the new one and rejecting anything other than a flipped passes flag:

import json, sys

def without_passes(feature):
    return {k: v for k, v in feature.items() if k != "passes"}

def check(before_path, after_path):
    with open(before_path) as f:
        before = json.load(f)
    with open(after_path) as f:
        after = json.load(f)
    if len(after) != len(before):
        return [f"feature count changed from {len(before)} to {len(after)}"]
    return [f"feature {i} was edited, only 'passes' may change"
            for i, (old, new) in enumerate(zip(before, after))
            if without_passes(old) != without_passes(new)]

if __name__ == "__main__":
    problems = check(sys.argv[1], sys.argv[2])
    for problem in problems:
        print(problem, file=sys.stderr)
    sys.exit(1 if problems else 0)

If it exits nonzero, revert the commit and hand the error text to the next session so the agent knows why its work was thrown away.

Don’t let the builder grade its own homework

The next failure Anthropic ran into was self evaluation. In their March post on harness design for long-running apps, they note that agents asked to judge their own work tend to praise it confidently even when it’s mediocre. So they split the roles into a planner that turns a short prompt into a full product spec, a generator that builds it, and an evaluator that clicks through the running app with Playwright and fails the work if any grading criterion drops below its threshold. Even the evaluator needed tuning. Out of the box it would spot a real bug and then talk itself into approving the work anyway.

It worked, and it wasn’t cheap. From one retro game maker prompt, a solo agent ran for 20 minutes and cost $9, and the core game didn’t work. The full harness ran for 6 hours and cost $200, and the game was playable. A heavier harness buys reliability with time and tokens, so every piece had better earn its keep.

Every piece is a bet against the model

This is the idea I’d pin above my desk. Anthropic keeps repeating that every component in a harness encodes an assumption about what the model can’t do on its own, and those assumptions go stale. Their best example is what they call context anxiety. Sonnet 4.5 would start wrapping up early as it sensed its context limit getting close, so the harness got full context resets with a structured handoff between agents. On Opus 4.5 that behavior was largely gone, the resets turned into dead weight, and the app building harness dropped them in favor of one continuous session with compaction. When Opus 4.6 arrived, they removed the sprint structure too, and moved the evaluator to a single pass at the end. They found it was only worth its cost on tasks sitting past what the model could reliably do solo.

How they got there matters too. Cutting the harness down all at once lost performance and made it hard to tell which parts had been load bearing. What worked was removing one component at a time and checking the effect, which is just good debugging, but easy to forget when a new model lands.

Boundaries belong in the harness, not the prompt

The harness is also where security actually lives, because the model only ever emits requests. OpenAI’s Codex security docs split this into two layers that work together. The sandbox mode sets what Codex can technically do when it runs model generated commands, such as where it can write and whether it can reach the network, and locally it’s enforced with OS level mechanisms. The approval policy decides when Codex has to stop and ask first. With workspace-write and on-request approvals, it can read files, make edits and run commands in the working directory on its own, but it asks before editing anything outside the workspace or running a command that needs the network.

Claude Code takes a complementary route with hooks, shell commands that run at fixed points in the agent’s lifecycle so that certain things always happen instead of relying on the model to remember them. A PreToolUse hook that exits with code 2 blocks the tool call, and whatever it wrote to stderr goes back to Claude as feedback so it can change course. The April 2 post adds a design hint that fits here. A bash tool hands the harness nothing but a command string, while a dedicated tool gives it typed arguments it can gate, log or show to the user, so actions that are hard to reverse make good candidates for their own tools.

Credentials deserve the strictest treatment. Anthropic’s April 8 post on Managed Agents admits that their first design ran the model’s generated code in the same container as the credentials, which meant a prompt injection only had to talk Claude into reading its own environment. The fix was structural. Git tokens get wired into the repo remote when the sandbox is set up, so push and pull work without the agent ever holding a token, and MCP calls go through a proxy that fetches OAuth tokens from a vault outside the sandbox. They also split the agent into a harness, a sandbox and an append only session log, so a crashed harness can be replaced and pick up from the log, and a dead container shows up as nothing worse than a failed tool call.

The cost lever hiding in plain sight

The Messages API is stateless, so the harness resends instructions, tool definitions and history every turn, and prompt caching is what keeps that affordable. The April 2 post notes cached tokens cost 10% of base input tokens, and gives rules that are easy to break by accident. Put stable content like the system prompt and tools first, append updates as messages instead of editing the prompt, don’t switch models mid session because caches are per model, and avoid adding or removing tools because they sit in the cached prefix. A harness that tweaks its system prompt every turn keeps throwing away its own cache.

Where I’d start

If you’re building your own harness, start with the smallest thing that could work and read the transcripts on realistic tasks before you add anything. Put your guardrails in code the model can’t argue with, like a sandbox, approvals for anything irreversible and checks on the files it shouldn’t touch, and keep credentials out of reach of anything the model runs. Then, every time a new model ships, go back through the harness one piece at a time and ask whether each part is still doing a job. The model gets most of the attention, but the harness is the half you control, and it’s usually where the next big improvement is hiding.