LLM Structured Output Validation: JSON Schema vs Retry Loops

Quick answer: never trust a model to return valid structured data just because you asked. Use schema-constrained output modes when your provider offers them, validate every response against a schema in your own code, and add a bounded retry that feeds the validation error back to the model. Fail clearly or fall back after a small number of attempts.

Why Does Structured Output Fail?

Language models generate text token by token, and a prompt that says return JSON is a request, not a guarantee. Even strong models occasionally add commentary around the JSON, wrap it in code fences, use a wrong type, invent a value outside your allowed list, or omit a required field. At scale, a failure rate of one percent is thousands of broken requests.

Truncation is another common cause. If the response hits the maximum token limit in the middle of an object, you receive invalid JSON that no amount of prompting will fix. Set generous limits for structured outputs, and check the finish reason, which tells you whether the model stopped naturally or was cut off.

Failures also differ in severity. Malformed syntax is easy to detect. A well-formed object with a plausible but wrong value passes parsing and breaks downstream logic later, which is harder to catch. That is why validation must check meaning where it can, not only syntax.

Constrained Output or Validation After the Fact?

Many providers offer modes that constrain generation to a schema, so the output is syntactically valid by construction. These reduce parsing failures sharply and are worth using where available. Their support for schema features such as nested objects, optional fields, and unions varies, so check the documentation for what your provider and model version accept.

Constrained decoding does not remove the need for validation. It guarantees shape, not truth. A field can be valid and still wrong, a number can be out of the range your business allows, and a value can contradict another field. Validate with your own schema library plus business rules, whichever mode you use.

• Use enums for closed sets of values, and include an explicit unknown option so the model need not guess.
• Keep schemas small and flat. Deeply nested or very large schemas increase errors.
• Add field descriptions, since they guide the model as much as the prompt does.
• Reject unknown fields, or strip them deliberately, so extra data never slips through.

How Do You Design a Safe Retry Loop?

When validation fails, a retry that includes the specific error often succeeds, because the model can correct itself. Send back the invalid output and the validation message, and ask for a corrected version. Limit attempts to two or three, since repeated failures usually mean the prompt or schema is the problem, not bad luck.

def extract(text, max_attempts=3):
    messages = build_messages(text)
    for attempt in range(max_attempts):
        raw = call_model(messages)
        try:
            return Schema.model_validate_json(raw)
        except ValidationError as err:
            messages += [assistant(raw), user(f"Invalid output: {err}. Return corrected JSON only.")]
    raise ExtractionError("validation failed after retries")

Decide in advance what happens after the last failure. Options include returning a clear error, routing to a human review queue, or using a default value for non-critical fields. Log every failure with the raw output, the schema version, and the prompt version, because these logs reveal patterns you can fix at the source.

Retries cost money and latency, so track the retry rate as a metric. A rising rate is an early warning that a model or prompt change has degraded output quality.

Frequently Asked Questions

Is JSON mode enough to skip validation?

No. It improves syntax, but values can still be wrong, out of range, or inconsistent. Always validate in your application.

How many retries should I allow?

Two or three is typical. More rarely helps and increases cost and latency. Investigate prompt or schema issues instead.

Should I strip code fences from the response?

You can as a defensive step, but prefer a mode that returns pure JSON. Parse strictly and treat anything unexpected as a failure.

Why does the model invent enum values?

The model may prefer a natural-sounding value over your list. Use constrained modes, clear descriptions, and an explicit unknown option.

Summary

Combine schema-constrained generation with your own validation, keep schemas simple, and use a bounded retry that returns the error to the model. Decide the failure path ahead of time and monitor retry rates. Consult your provider documentation for supported schema features.