Getting Reliable Structured Outputs from LLMs: Validation, Retries and Constrained Decoding

In this article, you will learn how to get reliable structured outputs from large language models by validating them against a contract and the source text, retrying only when it helps, and constraining generation.

Topics we will cover include:

• How to define a strict Pydantic contract and check every validated object against the source it came from.
• How to build a bounded retry loop that regenerates only on shape errors and keeps the SDK’s hidden transport retries under control.
• How to use constrained decoding with OpenAI’s Responses API without dropping validation, and how to test each failure path with a mocked client.

Structured output is a standard feature now. OpenAI’s Responses API takes a JSON schema or a Pydantic model and promises a response that fits it, vLLM does the same on a local server with xgrammar behind it, and the Python SDKs hand you a parsed object instead of a string. Most tutorials stop at the moment the JSON parses. That’s the wrong place to stop, and here’s the whole problem in four lines of invoice:

Ask a model for the invoice id, the currency, the total in cents and the line it took the total from, and you might get this back:

It’s valid JSON, every key is present, the types are right, EUR is in the allowed set, and the quoted line really is in the invoice. The total is 12,500 cents, which is 125 euros, for an invoice that says 1,250. No parser catches that and no schema catches it, because 12500 is a perfectly good integer. The only thing that catches it is a comparison with the source text, and your application has to make that comparison itself.

The three terms at the end of the title are the three layers that do the catching. Validation means your code checks the returned object against a contract you own, then against the invoice it came from. Retries means a second attempt is allowed only when another attempt can fix what went wrong, with a hard stop at three. Constrained decoding means the generator can only emit tokens that keep the output inside your schema, so a whole class of shape failures never reaches validation. The examples below were run on Python 3.10.12 and Pydantic 2.14.1; the code extracts fields and nothing more, and never writes to a real system.

Define the Contract Before Calling the Model

The contract has four fields, and your application owns it, not the prompt:

Every key is required, so a response that leaves one out fails. Three fields are nullable without a default, which is different from optional: the model may say “the invoice doesn’t state this” by sending null, but it may not stay silent and it may not invent a number to avoid a null. Two currencies is a tutorial scope, not a claim about invoices. Cents are stored as an integer so that 1,250.00 never becomes 1249.9999.

The two configuration options do most of the work. Strict mode switches off Pydantic’s default coercion, which would otherwise accept the string "125000" for an integer field and convert it silently; in strict mode that string fails with the error type int_type, and so does the boolean True, which the default mode would turn into 1. extra="forbid" turns an unexpected key into an extra_forbidden error instead of ignoring it. Feed Invoice four bad inputs and you get four distinct error types back:

You can use dataclasses for application-owned data, but model output needs validation before it becomes a trusted record, and Invoice is the gate it passes through. One thing the gate can’t hold: Invoice.model_json_schema() exports four properties, all required, additionalProperties false, and not a word about the invoice. The rule that the cents have to match the quoted line can’t be written in JSON Schema, so a decoder that enforces this schema perfectly will still produce the 12,500-cent total from the opening. A valid object is the entry ticket to the next check, not the result.

Validate the Object, Then Check It Against the Source

accept() runs four steps and ends in one of four states: accepted, needs_review, rejected or shape_error.

Step one is the parser and the contract, and it’s the only step that can return shape_error, which matters because a shape error is the only outcome the next section is allowed to retry. Step two checks that total_quote is in the invoice character for character. Step three checks the id you asked about and that the currency the model picked is actually printed in the text. Step four catches the opening’s mistake: the digits in “Total due: EUR 1,250.00” with the separators stripped are 125000; the model sent 12500, so the object is rejected with the reason “total_cents 12500 does not match the quoted line”. For a real invoice format you’d write a check that understands your decimals; the principle holds either way: the number has to be derivable from the evidence or it isn’t accepted.

needs_review is where a null goes when the invoice genuinely didn’t state an amount or a currency; under this policy a missing fact goes to a person, not to a guessed zero. rejected is terminal, because another call to the model can’t change what the invoice says. Expected values in the test fixtures are written by hand from the source text, never copied from a model’s output.

One boundary before the loop. Everything here starts from text that already exists. When you’re extracting text from scanned invoices, inspect the transcription before you ask the model to populate the schema, because a wrong digit upstream passes every check downstream as long as the object is internally consistent. What happens to a shape_error from accept() is the next question, and the answer is a policy rather than a loop that runs until something parses.

Retry Only When Another Attempt Can Help

extract_with_retries() has three exits, and only one of them is a retry. A shape_error gets another attempt with the errors fed back. accepted, needs_review and rejected are terminal, because the source won’t change. A refusal or a configuration error stops at once, because repeating the request won’t help.

generate is whatever calls your model; it gets the invoice, the schema and the feedback list, and returns text. The schema and the source never change between attempts. The feedback is capped at five reasons, enough to tell the model that vendor isn’t a field or that total_cents has to be an integer, and not enough to turn the prompt into a transcript of everything that went wrong. Use the field-level validation errors to identify the failed check and send only that; each error carries a loc, a type and a msg, and the message is usually all the model needs. Every candidate is validated from scratch, because a repair call can change the amount while it fixes the key.

The budget of three is only real if you also own the transport retries. The OpenAI Python SDK retries connection errors, timeouts, 409, 429 and 5xx responses twice by default with exponential backoff, so three application attempts can be nine HTTP requests without your code noticing. Set max_retries=0 on the client for a budget you can demonstrate, add a per-call timeout (the default is ten minutes, far too long for an extraction) and keep an overall deadline in the loop; the SDK retry configuration is documented per client and per request. Three attempts is this tutorial’s policy, not a universal optimum; what matters is counting the hidden work and stopping on the terminal cases, which the table below sorts.

Table 1. Match the failure to the next step.

A loop like this fixes shape failures. It can’t fix a model that was never constrained, and that’s the third layer.

Constrain Generation Without Dropping Validation

Asking for JSON in the prompt and restricting what the generator can emit are different things. A prompt is a request, and the model can still answer with a code fence, a stray sentence or an extra key. A schema constraint runs inside decoding: at each step the generator may only choose from the tokens that keep the output valid. Adrian Tam’s walkthrough of decoding strategies puts it in one line: “The idea is to mask out tokens that would make the output invalid”; this example uses a supported schema constraint rather than implementing the masking. On that path the malformed, extra-key and wrong-type cases from the fixtures stop occurring, because the decoder can’t produce them.

The call is the one OpenAI’s structured-output guide documents for the Python SDK as of October 2026: client.responses.parse(...) with text_format=Invoice, which accepts the Pydantic model directly and returns output_parsed as an instance of it. The guide’s examples name gpt-6-astra; pin whichever model you run and record it. I wrote this adapter from the documentation and exercised it with a mocked client in the fixtures below, not against a paid API, and I’d rather say that than imply a live result.

Two lines do the work the decoder can’t. The status check runs before anything reads the object: a response that isn’t completed is routed, not used. A completed response with no parsed output is a refusal, not an empty invoice. Then the object goes through accept() unchanged, because the decoder enforced the shape and only the shape; the constraint guarantees that total_cents is an integer, not that it’s 125000 rather than 12500. Keep the contract plain too, since not every Pydantic feature survives the trip into a provider’s schema subset.

Where each failure leaves the pipeline. The dashed arrow is the only retry, and it only runs for a shape error, at most three attempts in all. Refusals, incomplete responses and configuration errors leave at the completion check; a wrong amount leaves at the source checks. Illustrative application design, not measured provider behavior.

Run Both Paths Through the Same Failure Cases

The fixture suite turns the sentences above into fourteen test cases, from a missing required key and a code fence around the JSON to a fabricated quote, a refusal and exhaustion with no fourth call. Four of them, with the mocked client, look like this:

Read what the mock proves and what it doesn’t. It proves the routing: a wrong amount is rejected on the first attempt and never retried, exhaustion returns after exactly three calls, an incomplete response never reaches the source checks, and a refusal stops after one. It proves nothing about how often a real model produces the 12,500-cent total, and nothing about cost or latency; there are no percentages in this article because a success rate from canned responses would describe the fixtures, not a model. When you run the suite against a real adapter, record the model name, the SDK version, the outcome class and the attempt count per case, mask the source text in the logs, and report parsed, structurally valid and correct-against-the-source as three separate columns, with needs_review in the third as not yet correct.

Let Only Accepted Results Reach the Next Step

The pipeline consumes three outcomes and treats them differently. accepted goes downstream. needs_review goes to a person with the candidate and the reasons attached. rejected, whether from a contradiction, a refusal or an exhausted budget, goes to an error path that records what happened and does nothing else; a retry never repeats a write, and a repair attempt never acquires permissions the first attempt didn’t have.

Prefer constrained generation when the provider supports your contract, because it removes the shape failures and the retries they’d cause; fall back to the prompt-only loop with the same accept() when it doesn’t. In both cases I’d keep the typed contract and the source check even though the generator already enforces structure, because structure is the one thing it can enforce and correctness is the thing you need. The deployment test is a single question you can ask of a pipeline you already run: does every failure path stop before the downstream action? If the evidence is missing, the response is incomplete or the three attempts are spent, the result has to be an explicit review or error outcome, not a plausible object that happened to parse.