Every team building with LLMs has a graveyard, and it is full of regex. Yours truly once maintained a 200-line parsing function whose job was to turn "the model will probably return JSON" into "the database will definitely get rows." It had a comment that just said SORRY.
The graveyard exists because for two years the industry's reliability strategy was typing "please respond in JSON" into a prompt and then acting surprised. If you have searched "llm structured outputs json mode extraction" at 1 a.m. while a pipeline burned, you already know the prayer-based approach. This post is about what replaced it.
The short answer: structured outputs constrain an LLM's decoding so the response must match a JSON schema you define, turning extraction from a probabilistic hope into a contractual guarantee. JSON mode guarantees valid JSON; strict structured outputs guarantee your JSON. That distinction is the difference between a demo and a pipeline.
The parsing problem nobody admits
Ask a room of engineers who shipped LLM features in 2023 about their parse failure rate and watch the eye contact disappear. A typical honest number for a plain-prompt extraction pipeline: 70 to 90 percent of responses parse cleanly on a good day, with failures clustering exactly where you cannot afford them, long documents and weird edge cases.
The failure modes were a genre unto themselves. The model adds a friendly preamble ("Sure! Here is the JSON you requested:") and your parser eats it. It wraps output in markdown fences. It invents a field. It helpfully converts your date format. Each one is rare; multiplied across thousands of nightly runs, rare becomes Tuesday.
What LLM structured outputs and JSON mode do for extraction
Strip the jargon and the mechanism is simple: instead of letting the model pick any token it likes, decoding only allows tokens that keep the output valid against your schema. When the schema says the next thing must be a number, prose is not a possibility. The model is not trying harder; the wrong answers are structurally unavailable, like a multiple-choice exam instead of an essay.
Two tiers matter in practice. JSON mode guarantees syntactically valid JSON: braces balanced, types legal, but the shape is up to the model. Structured outputs, meaning strict schema adherence, guarantee the shape too: your fields, your types, your enums. For LLM data extraction into a database, you want the second one. JSON mode alone still lets the model hand you perfectly valid JSON about something else entirely.
Why this changed what LLMs are for
Before schema constraints, LLMs were interactive toys with infrastructure dreams. After, they became pipeline components: you can put one between a document store and a database and sleep at night. That is the entire trick, and it is boring in the way plumbing is boring, which is to say, civilization depends on it.
The reliability math tells the story. Illustrative numbers from a typical document-extraction migration: plain prompting parsed 72 percent cleanly, "please use JSON" got to 88, JSON mode reached 96, and strict schema outputs hit 99.5 percent, with the residual handled by validation rather than prayer. That jump is what makes reliable LLM pipelines possible, and it is why the AI agents running in production lean on structured tool calls for everything they do.
Designing schemas that survive production
A schema is a user interface for a model. Four rules keep yours out of the graveyard.
Keep schemas shallow
Deeply nested objects multiply the model's chances to get confused. Flat structures with clear names extract better. If your schema looks like a family tree, split the extraction into two calls.
Enums over free text
Every free-text field is an invitation to creativity. If the value should be one of six things, make it an enum of six things. Classification tasks improve the most from this: the model stops composing poetry and starts checking boxes.
Nullable fields for missing data
Documents in the wild lack fields. If the schema demands a value, the model will invent one, and an invented invoice total is a special kind of bug. Mark optional fields nullable so "not present" is a legal answer instead of a hallucination prompt.
Field descriptions are prompt surface
The JSON schema LLM providers accept usually supports a description per field. Use it. "Net amount after discounts, before tax, as printed on the final line" beats "amount" by a wide margin, and the guidance lives in the contract rather than floating in a prompt string.
Where structured outputs still break
Schema-constrained decoding fixes syntax, not judgment. Long documents still strain context. Ambiguous fields still get ambiguous answers, just well-formatted ones. Domain edge cases, the invoice with two totals in two currencies, still need a policy.
So production systems wrap the guarantee in a workflow: validate values against business rules, flag low-confidence or out-of-range results to a human review queue, and log everything. The teams behind our LLM document processing playbook treat structured output as the floor, not the ceiling. And if your documents are too varied for one schema, that is a retrieval problem; fine-tuning vs RAG covers that fork in the road.
A pattern worth stealing: have the model emit a short confidence note per record, explaining ambiguous calls in one line ("two totals present; used the one labeled Balance Due"). It costs a few tokens and turns your review queue from forensic archaeology into skimming. Low-confidence records go to humans, high-confidence records go to the database, and the boundary is a business decision you tune weekly.
A worked example: invoice extraction
Say you are extracting invoices into an ERP. The schema: vendor name (string), invoice date (string, ISO format, description says "the date labeled Invoice Date, not Due Date"), total (nullable number, described as "final amount due after tax"), currency (enum of the five you actually support), and line items (an array of flat objects, capped at fifty).
The call returns rows that parse every time. Your validation layer then checks totals against purchase orders and routes anything off by more than two percent to a human. What ships to the database is clean; what ships to the review queue is interesting. Parse failures, the old 4 a.m. pager genre, effectively stop happening.
That is the whole pitch: the model got more constrained in exactly the way you needed. Limit the output, keep the judgment, and spend your engineering time on the checks that matter. Before you build, an AI readiness audit will tell you whether your data is clean enough for any of this to help. The schema is easy. The data is the adventure.