Why Your AI Prompt Cache Keeps Missing

A document assistant can send the same handbook to a model all afternoon and still pay to process it repeatedly. The handbook has not changed. The questions are arriving seconds apart. Somebody added a cache setting. Everything sounds right until you inspect what the application actually sends.

At the beginning of every request sits a helpful little line: the current timestamp.

That is the kind of mistake I would look for before changing models or shortening the document. Prompt caching depends on how a request is assembled, and application code has plenty of ways to turn stable material into a moving target.

Start with what is being reused

This discussion uses Anthropic’s explicit prompt caching interface and its default five minute cache pricing. Anthropic’s prompt caching announcement describes reuse of frequently supplied context, with cache writes carrying a premium and reads receiving a discount. That makes a long document followed by several questions a promising workload, provided the requests actually share reusable input.

The useful distinction is between processing the shared input and generating an answer. A cached handbook still participates in the new request. The model generates a fresh response to the new question. An application that needs identical answers served instantly would need its own answer cache, with separate rules about freshness and authorization.

For document analysis, think about the stable beginning of the conversation. Put the durable instructions and document before the changing question, and place the cache breakpoint at the end of the shared material. Anthropic’s January 2025 cookbook example demonstrates this arrangement using a text block with cache_control followed by a question.

A breakpoint is not a command to search the whole request for reusable paragraphs. If a value changes earlier in the input, the shared prefix ending at that breakpoint has changed too. The position of the variable material matters as much as the position of the document.

Build one stable document block

Suppose the application answers questions about an internal support handbook. The following is a request construction example for a Python application with an initialized Anthropic client. handbook contains the full authorized document, and question contains the current question. Use a substantial document for the experiment because cache eligibility has a minimum prompt length; a tiny placeholder is not a useful cache test.

response = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=300,
    messages=[{
        "role": "user",
        "content": [
            {
                "type": "text",
                "text": "Support handbook:\n" + handbook,
                "cache_control": {"type": "ephemeral"},
            },
            {"type": "text", "text": question},
        ],
    }],
)
print(response.usage.model_dump())

The model identifier and request shape were already present in the historical cookbook. This example deliberately keeps the question outside the marked document block. It also leaves timestamps out because this task does not require one. If the answer depends on today’s date, include that changing context with the question after the breakpoint.

I would load the handbook once from a versioned source and reuse the resulting string. Fetching and converting an HTML page on every request introduces unrelated opportunities for changes. A generated footer, shuffled navigation, or a different whitespace conversion can alter the material even when the policy itself is identical.

The same reasoning applies to prompts assembled from several files. Pick a deterministic file order and preserve it. Do not trust the accidental order of asynchronous fetch completion. This is ordinary reproducibility work that happens to affect an AI bill.

Watch the accounting before the stopwatch

A quick response does not prove a cache hit. Network conditions, output length, and service load can all affect the total duration. The response usage is the better starting point.

The versioned SDK usage definition exposes cache_creation_input_tokens and cache_read_input_tokens alongside input_tokens and output_tokens. Record all four. Repeated cache creation without corresponding reads is a strong reason to inspect the request construction and spacing between calls.

Send the initial request and let it finish before sending the next one. Keep the model, document, and request structure fixed while changing only the question. That gives you a simple reuse experiment without introducing simultaneous cold requests. Then deliberately insert a different timestamp before the document and compare the usage. Finally, move that timestamp after the breakpoint and repeat.

The prompt caching documentation specifies a default five minute lifetime, refreshed when cached content is used. Run the initial comparison within that window, then repeat after a gap longer than five minutes to observe expiration. Longer cache duration is a separate option with different pricing, outside this example.

Those are proposed experiments, not measurements from a deployment. Save the actual counters from your environment. If neither creation nor reads appear, check eligibility and the breakpoint before interpreting the result as an expiration problem. A cache cannot miss an entry it never created.

For troubleshooting, I would record a digest of the stable text, its source version, and the model identifier. Comparing digests makes an unexpected document change visible without copying confidential handbook contents into ordinary logs. The digest is application telemetry, not a cache key supplied to Anthropic.

Calculate savings for the work you repeat

For the original short lived pricing, the announcement specifies a write at 1.25 times the normal input price and a read at 0.1 times that price. Treat those as rates for the shared portion, not discounts on the entire request.

Consider a shared prefix whose ordinary input charge would be one cost unit per request. Ten requests without reuse cost ten units for that prefix. One creation followed by nine reads costs 1.25 plus 0.9, or 2.15 units. That is a 78.5 percent reduction for this portion of the input. The arithmetic is illustrative; it is not a claim about measured application spending.

Now consider ten requests that each create a new entry and never reuse it. The same prefix costs 12.5 units. Enabling caching has increased that portion of the bill. Questions and generated answers still add their own charges in both cases.

This is why I would evaluate reuse at the level of the document and actual traffic pattern. A handbook queried repeatedly during a support shift has a different opportunity from a unique uploaded file that receives one question. A great aggregate request count can conceal very little reuse per document.

Keep the application rules intact

Moving content around for caching should preserve the meaning and authorization of the request. Retrieve only documents that the current user may access. Keep the document’s identity and version attached to the application session, and make a deliberate choice when the source changes. Cheaper reuse is not a reason to keep answering from an obsolete policy.

Also measure answer quality after restructuring prompts. Even if the same words remain somewhere in the request, changing their placement can change model behavior. A few representative questions with known answers are a useful check before applying the new layout broadly.

My first optimization would be an observable, deterministic request builder. Once the shared input stops changing accidentally, the usage counters can tell you whether caching fits the workload. Until then, a cache flag can give you the appearance of an optimization while faithfully charging for fresh work.