Reasoning Models Without the Hype: What Thinking Costs You

Six months ago “reasoning model” meant a couple of preview models from OpenAI. Now OpenAI, DeepSeek, Anthropic and Google all ship something that thinks before it answers, and the word is doing a lot of marketing work. If you’re deciding what to wire into a product, you don’t need a theory of machine cognition. You need to know what the API actually does differently, what it bills you for, and whether your task gets better.

I’m going to stick to what the vendors themselves say in their docs, repos and announcements. Anything more about their internals is guesswork.

What the vendors mean by thinking

The common idea is simple. Before writing the answer you see, the model generates extra tokens that work through the problem. Vendors differ on whether you can read those tokens, how you control how many get spent, and whether thinking is a separate model or a switch on an ordinary one.

OpenAI: hidden reasoning tokens and an effort setting

OpenAI’s reasoning guide describes reasoning tokens that sit alongside the usual input and output tokens, which the model uses to break down the prompt and consider several approaches before responding. The important practical detail is that those tokens are not visible through the API. You get a count of them in the usage object, and that’s it.

The full o1 model reached the API in December 2024, and o3-mini followed on January 31, 2025. Both take a reasoning_effort parameter on Chat Completions. The docstring in the openai-python SDK as of early March marks it as o1 and o3-mini only, lists low, medium and high as the supported values, and says lower effort can mean faster responses and fewer tokens spent on reasoning. The same file describes max_completion_tokens as a cap covering both visible output and reasoning tokens, which matters more than it sounds.

DeepSeek-R1: the thinking is in the open, and so are the weights

DeepSeek published R1 on January 20, 2025, and the release is a different animal because you can download it. The DeepSeek-R1 repository describes two models. DeepSeek-R1-Zero was trained with large scale reinforcement learning without supervised fine tuning as a first step, and the README is frank that it suffered from endless repetition, poor readability and language mixing. DeepSeek-R1 added cold start data before the reinforcement learning stages to fix that. Both are listed at 671B total parameters with 37B activated and a 128K context length, which is not something you run on a laptop. The repo also ships six smaller dense models, from 1.5B to 70B, fine tuned from Qwen and Llama bases on samples generated by R1.

The license is the part your legal team will ask about. The code and weights are under the MIT License, and the README explicitly allows commercial use, modifications and distillation into other models. The distilled models carry their base models’ terms too: the Qwen based ones come from Apache 2.0 licensed models and the Llama based ones from Llama 3.1 and 3.3 licenses.

Unlike OpenAI’s models, R1’s reasoning comes back as text in its output, wrapped in <think> tags. The README even warns that the model sometimes emits an empty think block and skips reasoning, and recommends forcing each response to start with <think> followed by a newline.

Claude 3.7 Sonnet: one model with a thinking budget

Anthropic released Claude 3.7 Sonnet on February 24, 2025, and the announcement calls it a hybrid reasoning model. It’s one model that can answer normally or use extended thinking, and the thinking is shown to you rather than hidden. Through the API you set a budget, telling Claude to think for no more than N tokens up to its output limit, which Anthropic frames as trading speed and cost for answer quality. Pricing is the same in both modes, $3 per million input tokens and $15 per million output tokens, with thinking tokens counted as output.

In code that’s a thinking object with type set to enabled and a budget_tokens value. Anthropic’s extended thinking cookbook notebook spells out the rules: the minimum budget is 1,024 tokens, it suggests starting at the minimum and working up, thinking isn’t compatible with changing temperature, top_p or top_k or with prefilling the response, and thinking tokens count toward the context window, rate limits and the output bill. Responses come back with thinking content blocks, and occasionally a redacted_thinking block when safety systems flag part of the reasoning. That block is encrypted, and the API decrypts it when you send it back so Claude can carry on.

Gemini 2.0 Flash Thinking

Google’s entry is still labeled experimental. Its February 5 developer post describes Gemini 2.0 Flash Thinking Experimental as the Flash variant that reasons before answering. Google’s own cookbook quickstart for it uses the model ID gemini-2.0-flash-thinking-exp-01-21, says the model shows its thoughts, and notes that the flag separating thoughts from the answer only works on the v1alpha API. It’s worth trying, but I wouldn’t put an experimental model on a production path yet.

The bill and the wait

This is where reasoning models surprise people. Reasoning tokens are billed as output tokens even though, with OpenAI, you never see them. OpenAI’s guide says a model can produce anywhere from a few hundred to tens of thousands of reasoning tokens depending on the problem, and recommends reserving at least 25,000 tokens for reasoning plus output when you start experimenting. If you set the token cap too low, the model can burn through it while still thinking, and you pay for input and reasoning without getting a visible answer back.

Those tokens also take time to generate, so latency goes up along with cost. OpenAI’s reasoning best practices guide is blunt that its GPT models are faster and tend to cost less, and it calls the reasoning models the planners and the GPT models the workhorses. The same guide notes that for o1 and o3-mini, reasoning from earlier turns is ignored in follow up requests, so a long conversation doesn’t carry the thinking forward. Each turn pays for its own.

With Claude 3.7 Sonnet, thinking shows up inside output_tokens in the usage numbers. With OpenAI, look at completion_tokens_details.reasoning_tokens. Either way, log it, because averages on a dashboard hide the one request that thought for 20,000 tokens.

When a reasoning model is worth it

The vendors’ own guidance points at the same kind of work. OpenAI’s best practices guide highlights ambiguous tasks, pulling the relevant details out of large piles of documents, multi step planning, code review and grading other models’ outputs. Anthropic says extended thinking helps on math, physics, instruction following and coding. DeepSeek’s comparisons focus on math, code and reasoning tasks.

My reading: reasoning helps when there are several steps and one wrong step ruins the answer. Planning a refactor across files, working out whether a policy applies to an edge case, or solving a math problem with a checkable answer all fit.

It usually doesn’t help when the answer is one step away. Pulling an invoice number out of an email, classifying a support ticket, rewriting a sentence, and chatting in an interface where a person is watching a spinner are all cases where a regular model is cheaper, faster and probably just as right. OpenAI’s own example architecture uses GPT-4o models for the triage work and hands only the final policy decision to o3-mini, which is a sensible split.

Prompting them differently

OpenAI’s best practices say these models do best with straightforward prompts. Keep instructions simple and direct, skip “think step by step” because the model already reasons internally, use delimiters like XML tags or section titles to separate parts of the input, and try zero shot before adding examples. Since o1-2024-12-17, they take developer messages instead of system messages, and they avoid Markdown in responses unless your developer message starts with the line “Formatting re-enabled”.

DeepSeek’s README goes further for R1: avoid a system prompt entirely and put all instructions in the user message, and set temperature between 0.5 and 0.7 with 0.6 recommended. Anthropic, by contrast, says prompting Claude 3.7 Sonnet works about the same in both modes, although you lose control of temperature once thinking is on. So the old chain of thought scaffolding is mostly something to delete, not something to port.

Test it on your own work

Benchmarks tell you about benchmarks. The honest way to decide is to take 20 to 50 real tasks from your product where you know what a good answer looks like, and run them through the regular model you use now and a reasoning model at its lowest setting. Record whether each answer is correct, how long the request took, and how many tokens it used including reasoning.

import time
from openai import OpenAI

client = OpenAI()
start = time.perf_counter()
resp = client.chat.completions.create(
    model="o3-mini",
    reasoning_effort="low",
    max_completion_tokens=25000,
    messages=[{"role": "user", "content": task_prompt}],
)
elapsed = time.perf_counter() - start
details = resp.usage.completion_tokens_details
print(elapsed, resp.usage.completion_tokens, details.reasoning_tokens)

The Claude version swaps in model="claude-3-7-sonnet-20250219", a max_tokens larger than your budget, and thinking={"type": "enabled", "budget_tokens": 1024}, then reads resp.usage.output_tokens.

If the lowest setting already wins, stay there. If it doesn’t, raise it one step and check whether accuracy moves enough to justify the extra latency and tokens. If the reasoning model never wins, you learned that cheaply. If it wins only on a slice of tasks, route that slice and leave the rest alone.

The short version is that “thinking” is a feature you pay for per token and per second. Treat it like any other expensive dependency: turn it on where your own tests show it earns its keep, cap it, and watch the usage numbers.