A Million Tokens of Context, and What It Actually Buys You

A month ago the biggest context window you could actually use was 200,000 tokens. Now Google is letting a small group of developers push a million tokens into Gemini 1.5 Pro, and Anthropic’s new Claude 3 models ship with 200K and a note that they can accept more than a million. The obvious question for a lot of teams is this: if the whole manual, the whole repo or the whole call transcript fits in one prompt, do we still need the search layer we spent last quarter building?

The short answer is sometimes. It depends on what the vendors actually measured, what every request will cost you, and what research says about how models use a long prompt.

What got announced

Google announced Gemini 1.5 on February 15. The Google blog post describes 1.5 Pro as a mid-size multimodal model with a standard 128,000 token context window, plus a private preview through AI Studio and Vertex AI where a limited group can try up to 1 million tokens. Google puts that million in concrete terms: about an hour of video, 11 hours of audio, a codebase of more than 30,000 lines, or more than 700,000 words. They also say they’ve tested up to 10 million tokens in research. The developer blog post pitches uploading large PDFs, code repositories and long videos straight into AI Studio as prompts.

Anthropic followed on March 4 with the Claude 3 family: Haiku, Sonnet and Opus. Opus and Sonnet are in the API now, and Haiku is listed as coming soon. All three launch with a 200K window, and Anthropic says all three can accept inputs over 1 million tokens, which it may open up to select customers.

What the vendors say about recall

Both companies lead with the same benchmark, needle in a haystack. You hide one small fact somewhere in a huge block of unrelated text, ask the model for it, and repeat at different lengths and depths.

Google says 1.5 Pro found the needle 99% of the time in inputs up to 1 million tokens. Anthropic says Claude 3 Opus passed 99% accuracy on its version of the test, which it made harder by drawing from 30 different needle and question pairs and using a varied crowdsourced set of documents. Anthropic also mentions that Opus sometimes pointed out that the needle sentence looked like a human had inserted it into the text.

Those are strong results, and I don’t want to wave them away. They show that a model can find a single, distinctive fact in a very long input. But that’s a narrower claim than “the model will use everything you give it”, and Google’s own Gemini 1.5 technical report, posted to arXiv on March 8, is refreshingly direct about the gap. When the team hid 100 needles in one haystack and asked for all of them, 1.5 Pro recalled around 70% up to 128K tokens and more than 60% up to 1 million. That’s better than GPT-4 Turbo managed within its own 128K limit in the same test, but it’s a long way from 99%. The report also notes that the prompting method and the kind of needle affect the scores, and it argues for harder evaluations that make a model combine facts scattered across the context.

Real work looks more like the 100 needle version. Nobody asks a contract review tool to find one planted sentence. They ask which clauses conflict.

Fitting in the window is not the same as using it

The best-known research on this is Lost in the Middle by Nelson F. Liu and colleagues at Stanford, UC Berkeley and Samaya AI. They gave models a question plus a set of documents, only one of which held the answer, and moved that document around. Accuracy followed a U shape: best when the answer was at the very beginning or end of the input, and noticeably worse when it sat in the middle. The paper found this even in models built for long context, and found that extended-context versions of a model often performed about the same as their shorter siblings on the same inputs. In one setup, GPT-3.5-Turbo did worse with the answer buried in the middle of the documents than it did with no documents at all.

Two caveats matter. The models in that paper (GPT-3.5-Turbo, Claude 1.3, MPT-30B-Instruct and LongChat-13B) are last year’s generation, and the contexts were far shorter than a million tokens. Gemini 1.5 and Claude 3 may well do better. But until someone shows otherwise on your kind of data, assume the position of information in a prompt can change whether the model uses it.

The same paper has a result I think every tech lead should remember. In an open-domain question answering setup, going from 20 retrieved documents to 50 improved accuracy by only about 1.5% for GPT-3.5-Turbo and about 1% for Claude 1.3, while making the input much longer, which the authors point out means more latency and cost. More context stopped paying for itself well before the retriever ran out of relevant material.

Anthropic’s own long context prompting study from last September points the same way, just from the fix side. Testing recall on documents around 70K to 95K tokens, it found that asking Claude to pull relevant quotes into a scratchpad before answering improved accuracy, that examples drawn from the same document helped, and that instructions belong at the end of the prompt. If position and prompting didn’t matter, none of those tricks would move the numbers.

What it costs to send the whole thing

On the Claude 3 announcement, Opus is $15 per million input tokens and $75 per million output tokens, Sonnet is $3 and $15, and Haiku is $0.25 and $1.25. Input is billed per request, so if your app sends the same 180,000 token document along with every question, you pay for those 180,000 tokens every single time. On Opus that’s about $2.70 of input before the model writes a word. A hundred questions against one document is about $270 in input alone. On Sonnet the same hundred questions cost about $54, and on Haiku, once it’s available, about $4.50.

Gemini 1.5 Pro’s million-token preview is free during testing, so there’s no bill to quote yet. Google has said it plans pricing tiers that start at the standard 128K window and scale up to 1 million tokens, which suggests the price will depend on how much of the window you use. Google also tells early testers to expect longer latency with the experimental 1M window and says it’s still working on speed. For anything a user waits on, that matters as much as the price.

There’s a security cost too. “Upload the whole repository” is a great demo and a questionable default. A repo often includes things that shouldn’t leave your network, like config files with credentials, customer data in test fixtures, or internal hostnames. Sending everything means sending everything, so decide what goes in on purpose.

Where long context really shines

None of this makes long context a gimmick. It’s a real change when the whole thing is the point. Reviewing a long contract or spec where clauses refer to each other is a good fit, because chopping it into chunks can separate a definition from the clause that depends on it. Getting oriented in an unfamiliar codebase is another, since questions like “where does this request get authorized” cross many files, and Google’s developer post pitches exactly that for onboarding. Long meeting or support call transcripts, where the important moment might be an offhand remark 40 minutes in, are a natural fit as well, and so is a one-off analysis where you’d spend more time building retrieval than you’d ever spend on tokens.

The pattern I’d look for is low query volume, a bounded set of material, and questions that need connections across the whole thing rather than one fact from it.

When I’d still retrieve first

Retrieval still earns its place when the material is bigger than any window, like a full knowledge base or years of tickets, or when it changes often enough that you’d rather re-index a page than resend a corpus. It also wins when you’re answering thousands of questions a day and per-request input cost dominates, when latency is part of the product, and when different users may only see certain documents so you need to control exactly what reaches the model. Sending a small, relevant set of passages also makes the answer easier to check, because you know what the model was looking at.

The two also combine well. Use retrieval to pick the few documents that matter, then let the long window take them whole instead of trimming them into tiny chunks.

How I’d approach it this month

Before changing architecture, I’d run a small test with your own documents rather than trusting any haystack chart. Plant several facts, not one, at different positions, include a question that needs two of them combined, and compare full-context answers against your current retrieval pipeline on accuracy, latency and input tokens per request. Put the question after the document, and try asking the model to quote the passages it’s relying on before answering, since that’s what Anthropic’s own testing supports.

Big context windows are a genuine new tool, and for whole-document work they can replace a lot of fiddly chunking. Just treat “it fits” as the start of the evaluation, not the end of it.