What's Actually Inside an AI Agent

“Agent” has become one of those words that means whatever the person selling it wants it to mean. It sounds like there must be some deep new architecture underneath, and the surprise when you go looking is how little there is. Once you can see the handful of moving parts, every agent product starts to look familiar, and you get much better at guessing where one will break.

The definition that actually helps

Anthropic’s Building effective agents post from December 2024 draws a line I still find useful. A workflow is a system where the model and its tools follow code paths you wrote ahead of time. An agent is a system where the model directs its own process and decides which tools to use and when. By the time they published their context engineering guide in September 2025, they had boiled it down even further: an agent is an LLM autonomously using tools in a loop.

That sentence is basically the whole architecture: a model that can ask for things, tools that do things, a loop that keeps going until the job is done, and a context window holding everything the model knows about the job so far. Let’s go through those four pieces.

The model never runs anything

When a model “uses a tool,” it isn’t executing code. The Claude Platform docs on how tool use works describe it as a contract: you tell the model what operations exist and what their inputs look like, and the model decides when to call them. What comes back from the API is a structured request, a tool_use block with a tool name and a JSON object of arguments. Your application runs the actual operation and sends the output back in a tool_result block on the next request.

A tool definition is just a name, a description and a JSON Schema for the input:

tools = [{
    "name": "read_file",
    "description": "Read a UTF-8 text file. Path must be absolute.",
    "input_schema": {
        "type": "object",
        "properties": {"path": {"type": "string"}},
        "required": ["path"],
    },
}]

The docs point out that the model never sees your implementation, only the schema you provided and the result you returned. So the description is doing the job that documentation, type hints and code review would normally do for a human caller. In the same December 2024 post, Anthropic says that while building their SWE-bench coding agent they spent more time optimizing tools than the overall prompt. One fix was making a file tool require absolute paths, because the model kept making mistakes with relative paths after it had moved out of the root directory.

The loop is a while loop

Because the model can’t run your code, every tool call is a round trip driven by a loop in your application. With the Anthropic Messages API, the signal is the stop_reason field. If it’s "tool_use", the response contains one or more tool calls for you to run. Anything else ("end_turn", "max_tokens", "stop_sequence" or "refusal") means the model has finished or stopped for a reason you should handle.

Here’s roughly what that looks like with the Python SDK. It isn’t production code, but it’s the real shape:

import anthropic

client = anthropic.Anthropic()
MODEL = "your-model-id"  # any current Claude model

def run_agent(task, tools, handlers, max_turns=10):
    messages = [{"role": "user", "content": task}]
    for _ in range(max_turns):
        response = client.messages.create(
            model=MODEL, max_tokens=4096, tools=tools, messages=messages
        )
        if response.stop_reason != "tool_use":
            return response
        messages.append({"role": "assistant", "content": response.content})
        results = []
        for block in response.content:
            if block.type != "tool_use":
                continue
            try:
                output = handlers[block.name](**block.input)
                results.append({"type": "tool_result",
                                "tool_use_id": block.id,
                                "content": str(output)})
            except Exception as err:
                results.append({"type": "tool_result",
                                "tool_use_id": block.id,
                                "content": f"Error: {err}",
                                "is_error": True})
        messages.append({"role": "user", "content": results})
    raise RuntimeError("Agent hit the turn limit")

A few details in there are easy to miss. The whole message history goes back to the model on every call, because the Messages API is stateless and doesn’t remember anything between requests. The tool results have to come right after the assistant turn that asked for them, and the handle tool calls docs say the tool_result blocks must come first in that user message, before any text. And when a tool fails, you don’t crash the loop. You send the error back with is_error set so the model can see what went wrong and try something else.

That max_turns guard matters too. Anthropic’s post points out that it’s common to add stopping conditions, such as a maximum number of iterations, to keep control of an agent, and the idea isn’t specific to Claude. The OpenAI Agents SDK describes exactly the same loop in its running agents docs at v0.20.0: call the model, stop if the output is a final answer, run the tool calls and go around again if it isn’t, and raise MaxTurnsExceeded if the run goes past max_turns. In that release the default limit is 10.

The SDK defines “final output” as text of the expected type with no tool calls, and adds handoffs so one agent can delegate to another, but underneath it’s the same loop you’d write by hand.

Context is the agent’s working memory

The context window drains faster than you’d expect, because every tool result gets appended to a history that’s resent each turn. Anthropic’s context engineering guide names the problem context rot: as the number of tokens in the window grows, the model gets worse at accurately recalling what’s in it. They tie it to how transformers work, where every token attends to every other token, so n tokens means n² pairwise relationships competing for attention. Bigger windows help, but they don’t make the problem go away.

The guide describes three techniques for long tasks. Compaction summarizes a conversation that’s close to the limit and starts a fresh window with the summary. In Claude Code that summary keeps things like architectural decisions and unresolved bugs, and the agent carries on with it plus the five most recently accessed files. Structured note taking has the agent write notes to storage outside the window (think a NOTES.md file) and read them back later. Sub-agents hand a focused job to a separate agent with a clean window, which might burn tens of thousands of tokens exploring and then return a condensed summary, often 1,000 to 2,000 tokens.

The cheapest win is in the tools themselves. In their guide to writing tools for agents, Anthropic says Claude Code limits tool responses to 25,000 tokens by default, and recommends pagination, filtering or truncation with sensible defaults for anything that could return a lot. A tool that dumps an entire table into the context isn’t just expensive, it crowds out everything the agent needs to pay attention to for the rest of the run.

MCP is how the tools get plugged in

Writing every integration by hand gets old, and that’s the problem the Model Context Protocol is meant to solve. MCP gives applications a standard way to discover and call tools that live somewhere else. The latest revision of the spec, dated 2026-07-28 and tagged at the end of July, keeps the same basic shape. A host application runs one MCP client per server, clients talk to servers using JSON-RPC 2.0, and a server can be a local subprocess over stdio or a remote service over Streamable HTTP. The client sends tools/list to find out what’s available and tools/call to run something, and the results slot straight into the loop above.

Under the hood that revision is a big one. It makes the protocol stateless by removing the initialize handshake, so every request carries its own protocol version and client capabilities. If you maintain an MCP server or client, read the changelog carefully. If you only use MCP servers, the mental model above hasn’t changed.

One design principle deserves more attention than it gets: servers shouldn’t be able to read the whole conversation, which stays with the host. That’s what lets you plug a third party server into an agent without handing it everything you’ve said.

Where it goes wrong: security and cost

Every tool you give an agent is a capability that can be triggered by text the model reads, and a lot of what an agent reads (web pages, emails, file contents, tool results) comes from places you don’t control. The MCP spec is blunt about it. Tools represent arbitrary code execution, hosts must get explicit user consent before invoking any tool, and descriptions of tool behavior such as annotations should be treated as untrusted unless they come from a trusted server.

The spec’s Tools page also says clients should show tool inputs to the user before calling a server, to avoid malicious or accidental data exfiltration, and should implement timeouts and log tool usage for audits. On the framework side, the OpenAI Agents SDK lets you mark a tool with needs_approval so the run pauses until a person approves or rejects the call. My rule of thumb is simple: anything that sends, deletes, pays or publishes gets an approval step, and anything that only reads gets scoped as narrowly as you can manage.

Cost is the other trap. Anthropic’s post is upfront that agents trade latency and cost for better task performance, and that autonomy brings higher costs and the potential for compounding errors, so they recommend extensive testing in sandboxed environments with guardrails. And since the full history is resent every turn, a 30 turn run isn’t 30 small requests, it’s 30 requests that each carry everything before them. The turn limit in your loop is a security control and a budget control at the same time.

Where I’d start

Anthropic’s advice is to start with the LLM APIs directly, since many of these patterns take only a few lines of code, and if you do use a framework, make sure you understand what’s underneath it. I agree. Write the loop above against a toy tool or two, print the message history on every turn, and watch how quickly it grows. Then break a tool on purpose and see how the model reacts to the error.

Every agent product is a model, a set of tool schemas, a loop with a stopping condition and some strategy for keeping the context under control. The good ones put serious work into their tool descriptions, their context budget and their approval prompts. The fragile ones skipped at least one of those, and now you know where to look.