Production AI systems need outputs that downstream code can consume. Structured output is how you get that reliability.
The three approaches
1. JSON mode + schema
- OpenAI
response_format: { type: "json_schema", schema, strict: true }— the model is constrained to emit JSON matching the schema. Strict mode is enforced server-side. - Gemini
responseMimeType + responseSchema— similar. - Anthropic: no native JSON mode, but tool-use achieves it (define a tool whose input is your output schema).
- Open-source:
outlines,instructor,guidanceuse grammar-constrained decoding for the same effect.
2. Tool / function calling
- The canonical mechanism for side-effectful structured outputs.
- LLM emits a tool call:
{ name, arguments: {...schema...} }. - Your code executes the tool and returns the result; LLM continues.
- Modern APIs support parallel tool calls — multiple in one turn.
3. Pydantic / Zod + retry
- Define a Pydantic / Zod model.
- Prompt: "Return JSON matching this schema: ...".
- Parse; on failure, retry with the validation error appended.
- Less reliable than native JSON mode but works on any model.
Reliability tactics
- Strict mode wherever supported. It eliminates most reliability issues at zero cost.
- Schema design discipline. Avoid optional + required mix; avoid recursive schemas; prefer flat over nested when possible.
- One tool call per concept. Resist mega-tools with 20 fields; smaller tools called in parallel are more reliable.
- Examples in the system prompt for ambiguous fields (e.g., date formats).
- Deterministic decoding — temperature 0 for extraction; not for creative generation.
Common bugs
- Asking for a nested array of objects with optional fields → models occasionally drop fields. Flatten.
- Asking for an enum value via free text → model invents synonyms. Use schema enum constraint.
- Multi-valued tool calls in older models → retry with parallel-tool support.
Real production stack
- OpenAI structured output (strict JSON schema) for clean extraction.
- Anthropic tool use for agent tools and structured generation.
- Pydantic for schema definitions; share between server validation and LLM prompt.
- Instructor (Python) or Vercel AI SDK (TS) wrap most of this with retry logic.