{"id":541,"date":"2026-08-03T13:04:03","date_gmt":"2026-08-03T13:04:03","guid":{"rendered":"https:\/\/abrarqasim.com\/blog\/openai-structured-outputs-json-schema-what-strict-mode-wont-do\/"},"modified":"2026-08-03T13:04:03","modified_gmt":"2026-08-03T13:04:03","slug":"openai-structured-outputs-json-schema-what-strict-mode-wont-do","status":"publish","type":"post","link":"https:\/\/abrarqasim.com\/blog\/openai-structured-outputs-json-schema-what-strict-mode-wont-do\/","title":{"rendered":"OpenAI Structured Outputs JSON Schema: What Strict Mode Won’t Do"},"content":{"rendered":"

Short version for the impatient: strict: true<\/code> guarantees the shape<\/em> of your JSON. It guarantees nothing about whether the values inside it are right. If you already knew that, skip ahead to the bit about optional fields, because that’s where I actually lost a day.<\/p>\n

Here’s the setup. I had an extraction endpoint pulling line items out of scanned supplier invoices. Schema locked down, strict: true<\/code>, Pydantic model on the other end. It had been quiet for weeks. Then finance flagged an invoice where the tax amount came back as 0.0<\/code> on a document that clearly showed 5% VAT.<\/p>\n

My first instinct was that the API had regressed. It hadn’t. The response was perfectly schema-valid. It was just wrong. Constrained decoding had done exactly what it promised and nothing more, and I’d quietly started treating “parses cleanly” as “is correct” somewhere around week three.<\/p>\n

Strict mode is a grammar, not a fact checker<\/h2>\n

The mechanism is worth understanding, because it explains precisely which class of bugs this feature kills and which it leaves alone.<\/p>\n

OpenAI compiles your JSON Schema into a context-free grammar, then masks the sampler at every step so only tokens valid under that grammar can be emitted. Their announcement post on Structured Outputs<\/a> walks through this: once the model has produced {\"val<\/code>, an opening brace is no longer a legal next token, so its probability gets zeroed. That’s also why the first request against a brand new schema is slow. The grammar has to be preprocessed and cached, and OpenAI says complex schemas can take up to a minute the first time.<\/p>\n

So the guarantee is real, and it’s a token-level guarantee. Braces balance. Enums stay inside their allowed values. Required keys show up. What no grammar can check is whether 5.0<\/code> should have been 12.50<\/code>. OpenAI is direct about this in their own limitations section: structured outputs “doesn’t prevent all kinds of model mistakes,” and they specifically call out wrong values inside a well-formed object.<\/p>\n

I find it useful to think of it as the difference between a type system and a test suite. Nobody ships a TypeScript codebase and declares it correct because it compiles. Same energy here. I made a similar mistake in the other direction once, stuffing so much context in that the model started confidently misreading it, which I wrote up in the anti-hallucination prompt that backfired on me<\/a>.<\/p>\n

The before and after is genuinely dramatic<\/h2>\n

Credit where it’s due. The old pattern was miserable. This is roughly what my invoice extractor looked like in 2023, on JSON mode, which guaranteed valid JSON and absolutely nothing about the schema:<\/p>\n

import json\n\nSYSTEM = """Return ONLY valid JSON matching this shape:\n{"vendor": str, "total": float, "tax": float, "currency": "AED"|"USD"}\nDo not include markdown fences. Do not explain."""\n\ndef extract(text, attempts=4):\n    for i in range(attempts):\n        r = client.chat.completions.create(\n            model="gpt-4-0613",\n            response_format={"type": "json_object"},\n            messages=[\n                {"role": "system", "content": SYSTEM},\n                {"role": "user", "content": text},\n            ],\n        )\n        raw = r.choices[0].message.content\n        try:\n            data = json.loads(raw)\n        except json.JSONDecodeError:\n            continue\n        # JSON mode gave us valid JSON. It did not give us OUR JSON.\n        if not {"vendor", "total", "tax", "currency"} <= data.keys():\n            continue\n        if data["currency"] not in ("AED", "USD"):\n            continue\n        return data\n    raise RuntimeError(f"gave up after {attempts} attempts")\n<\/code><\/pre>\n
Four attempts. Manual key checking. An enum I had to police by hand. And a retry loop that burned tokens every time the model decided \"amount_total\"<\/code> was a nicer key name than \"total\"<\/code>.<\/p>\n
The current version:<\/p>\n
from typing import Literal\nfrom pydantic import BaseModel\n\nclass Invoice(BaseModel):\n    vendor: str\n    total: float\n    tax: float\n    currency: Literal["AED", "USD"]\n\ncompletion = client.chat.completions.parse(\n    model="gpt-4.1",\n    messages=[\n        {"role": "system", "content": "Extract the invoice fields."},\n        {"role": "user", "content": text},\n    ],\n    response_format=Invoice,\n)\n\ninvoice = completion.choices[0].message.parsed\n<\/code><\/pre>\n
The retry loop is gone. The key checking is gone. The enum policing is gone. That’s a real improvement and I’m not going to pretend otherwise. The SDK converts the Pydantic model to a schema, sets strict: true<\/code>, and deserializes the response back into a typed object.<\/p>\n
What survived, unchanged, is the possibility that tax<\/code> is 0.0<\/code> when it should be 12.50<\/code>.<\/p>\n
Every field is required, and that’s where I lost the day<\/h2>\n
This is the rule that produced my actual bug, and it’s the one I’d tattoo on a junior dev if they let me.<\/p>\n
In strict mode, every property you declare must appear in required<\/code>. Not “should.” Must. You also need additionalProperties: false<\/code> on every object in the schema, no exceptions. Microsoft’s Azure OpenAI structured outputs docs<\/a> spell out both rules, and they mirror OpenAI’s own subset exactly.<\/p>\n
So how do you express a field that genuinely might not exist on the document? You don’t make it optional. You union it with null:<\/p>\n
{\n  "type": "object",\n  "properties": {\n    "vendor":   { "type": "string" },\n    "total":    { "type": "number" },\n    "tax":      { "type": ["number", "null"] },\n    "po_number":{ "type": ["string", "null"] }\n  },\n  "required": ["vendor", "total", "tax", "po_number"],\n  "additionalProperties": false\n}\n<\/code><\/pre>\n
Every key is required. Two of them are allowed to be null. Fine so far.<\/p>\n
Now read that schema the way the model reads it. tax<\/code> is a number-or-null and it is mandatory. The model must emit something<\/em> in that slot on every single document. When the tax line is smudged, or written in a layout the model hasn’t seen, or expressed as “VAT included,” the grammar offers exactly two escape hatches: a number, or null. There is no “I couldn’t tell.” So it picks one. And a plausible number often beats null on the model’s own priors, because most invoices do have a tax amount.<\/p>\n
That’s my 0.0<\/code>. Not a hallucination in the dramatic sense. A forced choice under a constraint I built.<\/p>\n
What fixed it was making “unknown” a first-class value instead of an absence:<\/p>\n
from typing import Literal, Union\nfrom pydantic import BaseModel\n\nclass Known(BaseModel):\n    status: Literal["found"]\n    value: float\n    source_text: str   # verbatim span it read the number from\n\nclass Unknown(BaseModel):\n    status: Literal["not_found"]\n    reason: str\n\nclass Invoice(BaseModel):\n    vendor: str\n    total: float\n    tax: Union[Known, Unknown]\n<\/code><\/pre>\n
Two changes doing the work. not_found<\/code> is now a legal, cheap answer, so the model stops guessing to satisfy the grammar. And source_text<\/code> forces it to quote the span it read the number from, which means I can grep the OCR text for that string and reject the row when it isn’t there. Schema compliance I get for free. Groundedness I have to design in.<\/p>\n
One caveat: the root object can’t be an anyOf<\/code>, so this union pattern works on nested fields, not at the top level.<\/p>\n
The keywords your schema is quietly ignoring<\/h2>\n
Here’s the part that stings if you came from ordinary JSON Schema validation, where pattern<\/code> and minimum<\/code> actually mean something. In strict mode a large chunk of the vocabulary is accepted and then ignored.<\/p>\n
Per the Azure docs, the ignored list includes minLength<\/code>, maxLength<\/code>, pattern<\/code> and format<\/code> on strings; minimum<\/code>, maximum<\/code> and multipleOf<\/code> on numbers; minItems<\/code>, maxItems<\/code> and uniqueItems<\/code> on arrays. Objects lose patternProperties<\/code>, propertyNames<\/code>, minProperties<\/code> and maxProperties<\/code>. There’s also a ceiling of 100 object properties total and five levels of nesting. Recursion via $ref<\/code> and shared definitions in $defs<\/code> are both supported, which surprised me the first time I tried it.<\/p>\n
Which means this schema is lying to you:<\/p>\n
{\n  "email":  { "type": "string", "format": "email" },\n  "iban":   { "type": "string", "pattern": "^AE[0-9]{21}$" },\n  "rating": { "type": "integer", "minimum": 1, "maximum": 5 }\n}\n<\/code><\/pre>\n
You’ll get a string in email<\/code>. It may not contain an @<\/code>. You’ll get an integer in rating<\/code>. It may be 47<\/code>.<\/p>\n
Two things help. Push what you can into enum<\/code>, which is enforced, so rating<\/code> becomes \"type\": \"integer\", \"enum\": [1,2,3,4,5]<\/code>. Then validate the rest after the fact, in your own code, where the constraint actually runs:<\/p>\n
from pydantic import BaseModel, EmailStr, Field, field_validator\n\nclass Contact(BaseModel):\n    email: EmailStr\n    iban: str = Field(pattern=r"^AE[0-9]{21}$")\n\n    @field_validator("iban")\n    @classmethod\n    def check_mod97(cls, v: str) -> str:\n        # the model can produce a well-formed IBAN that isn't a real one\n        if not iban_mod97_valid(v):\n            raise ValueError("checksum failed")\n        return v\n<\/code><\/pre>\n
Note that Pydantic will happily send pattern<\/code> up in the generated schema, where it does nothing, and then enforce it locally on the way back, where it does. That asymmetry caught me out for a while.<\/p>\n
Three ways you still get something that isn’t your schema<\/h2>\n
The grammar holds right up until it doesn’t, and there are exactly three documented exits.<\/p>\n
The model can refuse. Safety refusals bypass the schema entirely and come back on a separate refusal<\/code> field on the message. If you’re reading .parsed<\/code> without checking .refusal<\/code> first, you’ll get a None<\/code> and a confusing traceback three functions later.<\/p>\n
The generation can hit max_tokens<\/code> mid-object. You get truncated, invalid JSON. Always check finish_reason == \"stop\"<\/code> before you trust the payload, especially on schemas with unbounded arrays, since maxItems<\/code> isn’t enforced and nothing stops the model from emitting four hundred line items.<\/p>\n
And parallel tool calls break it. Structured outputs aren’t compatible with parallel function calling. Set parallel_tool_calls: false<\/code> when you’re using strict: true<\/code> on tools.<\/p>\n
msg = completion.choices[0].message\n\nif msg.refusal:\n    raise ModelRefused(msg.refusal)\nif completion.choices[0].finish_reason != "stop":\n    raise Truncated(completion.choices[0].finish_reason)\n\ninvoice = msg.parsed\n<\/code><\/pre>\n
Nine lines. I’ve now added them to every extraction path I own, including the ones in the document pipelines I build for clients<\/a>.<\/p>\n
What I’d do this week<\/h2>\n
Pick your highest-volume structured call and open its schema. Find every field typed as [\"something\", \"null\"]<\/code> and ask what the model does when it can’t tell. If null is the only alternative to a value, you’re pushing it toward a guess. Add an explicit not_found<\/code> branch and a source_text<\/code> field, then sample fifty recent responses and check how often that quoted span actually appears in your input. My hit rate was 91%. I’d assumed 100%, which was the whole problem.<\/p>\n
The parsing problem is solved. I’d started acting like the extraction problem was solved too, and those aren’t the same thing. Same lesson I keep relearning every time a framework makes something easy, most recently when AI SDK 7 let me delete my own agent loop<\/a> and I briefly forgot I still had to think about what the loop was doing.<\/p>\n
If you want the full list of what the subset does and doesn’t accept, OpenAI’s structured outputs guide<\/a> is the reference, and it’s worth twenty minutes before you design a schema rather than after.<\/p>\n","protected":false},"excerpt":{"rendered":"
Strict mode guarantees your JSON matches the schema, not that the values are right. The optional-field trap that cost me a day, and the checks I added after.<\/p>\n","protected":false},"author":2,"featured_media":540,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"rank_math_title":"","rank_math_description":"Strict mode guarantees your JSON matches the schema, not that the values are right. The optional-field trap that cost me a day, and the checks I added after.","rank_math_focus_keyword":"openai structured outputs json schema","rank_math_canonical_url":"","rank_math_robots":"","footnotes":""},"categories":[4,45],"tags":[613,612,609,610,106,611,608],"class_list":["post-541","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-programming","tag-constrained-decoding","tag-data-extraction","tag-json-schema","tag-llm-api","tag-openai","tag-pydantic","tag-structured-outputs"],"_links":{"self":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts\/541","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/comments?post=541"}],"version-history":[{"count":0,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/posts\/541\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/media\/540"}],"wp:attachment":[{"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/media?parent=541"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/categories?post=541"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/abrarqasim.com\/blog\/wp-json\/wp\/v2\/tags?post=541"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}