LLM Structured Output Validation and Retry Patterns for Backends

LLM structured output should never be trusted just because you asked for JSON. Validate every response against a schema, retry a limited number of times with the validation error included, and have a defined fallback when retries fail. Provider features that constrain output to a schema reduce failures, but they do not remove the need for validation in your own code.

Why Can You Not Trust Structured Output Blindly?

Language models generate text token by token. Even with clear instructions, a response can include extra commentary, truncated JSON, wrong field names, values of the wrong type, or plausible but invalid data. Output can also be cut off when it reaches the maximum token limit, leaving an incomplete object.

Many providers now offer modes that constrain output to a JSON schema, which makes syntactically valid output much more likely. These features vary by provider and model, and they typically guarantee structure rather than meaning. A value can match the schema and still be wrong, for example a date in the future where only past dates make sense.

Treat model output like any other untrusted input. Your backend should parse it, validate it, and reject it when it does not fit, exactly as you would with data from an external client.

How Should You Validate the Response?

Use two levels of validation. First, check structure: the output must parse as JSON and match a schema with the right fields, types, and allowed values. Libraries for schema validation in your language make this straightforward, and you can define the schema once and use it both to instruct the model and to check the result.

Second, check semantics and business rules. Verify that identifiers exist, numbers are within plausible ranges, and enumerated values are valid. If the output will trigger an action, such as sending an email or running a query, add stricter checks and consider human approval for high-impact operations.

def parse_ticket(raw: str) -> Ticket:
    data = json.loads(raw)          # may raise JSONDecodeError
    ticket = Ticket.model_validate(data)  # may raise ValidationError
    if ticket.priority not in ALLOWED_PRIORITIES:
        raise ValueError("invalid priority")
    return ticket

What Are Good Retry Patterns?

When validation fails, a retry often succeeds, but unbounded retries waste money and time. Use a small, fixed number of attempts and make each one more informative than the last.

• Repair with feedback: send the original request again along with the invalid output and the specific validation error, asking for a corrected response.
• Lower variability: reduce temperature on retries to make output more predictable, where your task allows it.
• Cap attempts and time: stop after two or three tries and respect the request deadline.
• Distinguish failure types: retry on validation errors, but handle rate limits and timeouts with backoff, and do not retry on safety refusals.
• Log failures: record the invalid output and error, with sensitive data removed, to find prompt problems.

Retries multiply cost and latency, so measure the failure rate. A high rate usually signals a prompt or schema problem, such as an overly complex schema, ambiguous field descriptions, or insufficient examples, and fixing it is cheaper than retrying.

What Should Happen When Retries Fail?

Define a fallback before you need it. Depending on the feature, that might be returning a clear error to the caller, using a simpler default result, routing the item to a queue for manual review, or trying a different model. Silent failures are the worst outcome, because bad data can flow downstream unnoticed.

Keep operations idempotent so a retried or reprocessed request does not repeat side effects. Store the raw model response alongside the parsed result when you need to audit behavior. Also keep schemas simple. Flat objects with clear field names and short enumerations are easier for models to produce reliably than deeply nested structures.

Frequently Asked Questions

Does JSON mode guarantee valid output?

It improves syntactic validity, but behavior varies by provider. You still need schema and business validation in your code.

How many retries are reasonable?

Typically one to three. Beyond that, success rates drop and costs rise, so fall back instead.

Should I include the validation error in the retry prompt?

Yes. A specific error message helps the model correct the exact problem and usually improves retry success.

How do I handle truncated JSON?

Check the stop reason, increase the output limit if appropriate, or shorten the requested output. Do not try to guess missing parts.

Summary

Validate structure and meaning, retry a few times with feedback, and define a fallback for failures. Keep schemas simple, log invalid outputs, and monitor failure rates. Check your provider's documentation for schema enforcement features.