Getting JSON You Can Trust Out of an LLM

The first time you wire a language model into a real app, the fun part is the prompt. The annoying part is the line right after it, where you call json.loads() on whatever came back and hope. Sometimes the model wraps the object in a Markdown code fence. Sometimes it adds a friendly sentence before the brace. Sometimes it invents a field name that’s close to yours but not quite, and your code falls over at 2am.

Two weeks ago OpenAI shipped a feature aimed squarely at that problem, so it feels like a good moment to walk through the whole progression, from asking nicely to actual guarantees, and to be clear about what those guarantees don’t cover.

Asking nicely

The oldest technique is to describe the JSON you want in the prompt and parse the reply. It works surprisingly often, and “often” is the problem. Nothing in the API is holding the model to your format, so every failure mode above is on the table, and the usual workaround is a retry loop that sends the request again when parsing fails. That costs tokens and latency, and it still gives you no promise about field names or types.

Google’s own Gemini docs were refreshingly blunt about this as of this month: you can ask Gemini for JSON in the prompt, but Google can’t guarantee you’ll get JSON and nothing but JSON.

JSON mode: valid, but not necessarily yours

OpenAI announced JSON mode at DevDay in November 2023. You set response_format to {"type": "json_object"} and the model’s output is constrained to syntactically valid JSON. That kills the stray prose and code fences in one go.

What it doesn’t do is look at your schema. It will give you valid JSON, and that JSON can have the wrong keys, missing keys or a string where you wanted a number. The SDK docstring for response_format also carries two warnings worth knowing. You still have to tell the model to produce JSON in a system or user message, because without that instruction it may emit whitespace until it hits the token limit, which looks like a hung request. And if finish_reason comes back as length, the content may be cut off partway through, so “valid JSON” only holds when generation actually finished.

Gemini has a similar switch. The Gemini API added a response_mime_type option back in April that you can set to application/json.

Function calling and tools

Function calling came at the problem from another angle. You describe a function with a JSON Schema for its parameters, and the model replies with a function name and an arguments object instead of prose. You can also use it purely to get structured data out, by defining a function whose only job is to receive the extraction.

It helped, but the guarantee was still soft. The openai-python SDK’s own docstring for the arguments field says the model doesn’t always produce valid JSON and may hallucinate parameters your schema never defined, and tells you to validate the arguments in your code before calling your function. That is about as clear a warning as a vendor writes.

Anthropic took the same approach with Claude. Tool use became generally available on May 30 across the Claude 3 model family on the Anthropic API, Amazon Bedrock and Google Cloud’s Vertex AI, and the launch included forced tool use so you can make Claude use a particular tool. Anthropic’s tool use docs say outright that tools don’t have to be real functions and that you can use one whenever you want JSON that follows a schema. You define the tool with an input_schema, set tool_choice to {"type": "tool", "name": "record_summary"} or similar, and read the input off the tool_use block in the response.

This is a solid pattern, but there is no strict mode here. The same docs have a troubleshooting note for when Claude’s attempted tool use is invalid, with missing required parameters as the example. So with Claude as of today, treat the schema as strong guidance and validate what comes back.

Structured Outputs: the schema becomes a constraint

On August 6, OpenAI introduced Structured Outputs, and it goes a big step past JSON mode: output is promised to match a schema you supply, not just to parse. It comes in two forms. For function calling you add strict: true to the function definition, and that works with every model that supports tools, back to gpt-4-0613 and gpt-3.5-turbo-0613. For plain responses there is a new response_format type, json_schema, which currently works with gpt-4o-2024-08-06 and gpt-4o-mini-2024-07-18.

The interesting bit is how. OpenAI says it converts your JSON Schema into a context-free grammar and, after each token, masks out every token that would break the grammar. The model literally can’t pick a token that would take the output outside your schema. OpenAI also trained the new model to handle complex schemas better, but the announcement is honest that training alone got them to 93% on their benchmark, and the constrained decoding is what gets them to 100%.

That machinery has consequences you’ll notice. The first request with a new schema is slower while the grammar is built; OpenAI says typical schemas take under 10 seconds to process, and complex ones up to a minute, after which it’s cached. Only a subset of JSON Schema is supported. Strict mode isn’t compatible with parallel function calls, so OpenAI recommends setting parallel_tool_calls to false. And the Python SDK’s helper, which converts a Pydantic model into a schema for you, rewrites every object so that additionalProperties is false and every property is listed as required, because that’s the strict form the API expects. If you wanted an optional field, you model it as something that can be null rather than something that can be absent.

The announcement is equally clear about the edges. The schema guarantee holds only when the response isn’t a refusal and generation wasn’t cut short, as indicated by finish_reason. That brings us to the two cases your code needs to handle.

Refusals and truncation

A model that refuses an unsafe request can’t also fill in your schema, so Structured Outputs adds a refusal field to the message. When it’s set, you get the model’s refusal text there instead of content that matches your schema. You do have to check for it.

Truncation is the other one. If the model hits max_tokens before closing the last brace, no amount of constrained decoding makes that half an object valid. In the openai-python helpers, the new client.beta.chat.completions.parse() method raises LengthFinishReasonError when finish_reason is length and ContentFilterFinishReasonError when it’s content_filter, rather than handing you something broken. With Claude, the equivalent signal is a stop_reason of max_tokens, and Anthropic’s docs suggest retrying with a higher max_tokens if the truncated response contains an incomplete tool use block.

Here’s roughly what I’d write with version 1.41.0 of the Python SDK, handling all three outcomes before trusting the data.

from typing import Literal
from pydantic import BaseModel
from openai import OpenAI, LengthFinishReasonError

class Ticket(BaseModel):
    category: Literal["billing", "bug", "account", "other"]
    priority: int
    summary: str

client = OpenAI()

def triage(text: str) -> Ticket | None:
    try:
        completion = client.beta.chat.completions.parse(
            model="gpt-4o-2024-08-06",
            messages=[
                {"role": "system", "content": "Classify the support ticket. Priority is 1 (low) to 3 (urgent)."},
                {"role": "user", "content": text},
            ],
            response_format=Ticket,
            max_tokens=300,
        )
    except LengthFinishReasonError:
        return None  # cut off: retry with more room or a shorter input
    message = completion.choices[0].message
    if message.refusal:
        return None  # log it and send the ticket to a person
    ticket = message.parsed
    if ticket.priority not in (1, 2, 3):
        return None  # matches the schema, still wrong
    return ticket

Schema valid is not the same as correct

That last check is the whole point of this post. The schema says priority is an integer, and 7 is an integer. The schema says category is one of four strings, and “billing” is one of them, even when the customer was actually reporting a bug. Structured Outputs guarantees shape, not truth. OpenAI’s announcement says so directly: the model can still make mistakes in the values, like getting a step of a math problem wrong.

So validation in your own code doesn’t go away. It moves. You stop writing defensive parsing and start writing the checks that encode what your application means by valid: ranges, cross-field rules, IDs that must exist in your database, dates that can’t be in the future. Many of those rules don’t belong in the model-facing schema anyway, since strict mode only supports a subset of JSON Schema.

For the actual validation, use real tools. If you’re working with raw schemas, the Python jsonschema library’s validate function checks an instance against a schema. One gotcha from its docs: the format keyword, the one that says a string is an email or an IPv4 address, isn’t enforced by default. You have to plug in a format checker. If you’d rather work with Python types, Pydantic’s model_validate_json parses and validates in one step and raises a ValidationError with details you can log.

Where each provider stands right now

If you’re on OpenAI, use Structured Outputs with a pinned model that supports it, handle refusals and length errors explicitly, and keep your business validation. If you’re on Claude, forced tool use with a clear input_schema is the way to get structured data, and you should validate every response because nothing in the API enforces the schema. On Gemini, response_mime_type gets you JSON output, and schema controlled generation through response_schema is limited to Gemini 1.5 Pro for now; Google’s JSON mode quickstart notes that Flash only takes a text description of the JSON you want.

The takeaway is simple. Constrained decoding means you can finally stop worrying about whether you’ll get parseable JSON, and that’s a genuinely good change. What you get back is still a model’s guess dressed up in the right shape, so check the values like you’d check any other untrusted input before it touches your database.