How to Get Structured JSON Output from Any LLM (Reliably)

JSON mode, function calling, schema constraints, and prompt engineering — the complete toolkit for reliable structured output across GPT-4o, Claude, and Gemini.

Author

AICredits Team

Published

8 Apr 2026

Reading time

7 min read

Why structured output is hard

LLMs are trained to produce human-readable text, not machine-parseable data. Even when you ask for JSON, the model might wrap it in markdown fences, add a preamble, include trailing commas, or hallucinate fields. In production, any of these cause your JSON parser to throw and break your application.

The good news: every major model now has a mechanism for enforcing structured output.

Technique 1: JSON mode

The most reliable approach for OpenAI-compatible APIs. Set response_format to {"type": "json_object"} — the model is constrained to produce only parseable JSON.

import json
from openai import OpenAI
 
client = OpenAI(
    base_url="https://api.aicredits.in/v1",
    api_key="sk-your-aicredits-key",
)
 
response = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    response_format={"type": "json_object"},
    messages=[
        {
            "role": "system",
            "content": (
                "Extract these fields and return valid JSON only: "
                '{"name": string, "email": string, "issue": string}'
            ),
        },
        {
            "role": "user",
            "content": "Hi, I'm Priya ([email protected]) and my invoice shows the wrong amount.",
        },
    ],
)
 
data = json.loads(response.choices[0].message.content)
print(data)
# {'name': 'Priya', 'email': '[email protected]', 'issue': 'Invoice shows wrong amount'}

Technique 2: Function calling / tool use

The strictest approach — define a JSON Schema and the model will fill it exactly. Use this for complex nested schemas with required fields and enum constraints.

tools = [{
    "type": "function",
    "function": {
        "name": "extract_ticket",
        "description": "Extract structured data from a support ticket",
        "parameters": {
            "type": "object",
            "properties": {
                "name":     {"type": "string"},
                "email":    {"type": "string", "format": "email"},
                "issue":    {"type": "string"},
                "priority": {"type": "string", "enum": ["low", "medium", "high"]},
            },
            "required": ["name", "email", "issue", "priority"],
        },
    },
}]
 
response = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    tools=tools,
    tool_choice={"type": "function", "function": {"name": "extract_ticket"}},
    messages=[{"role": "user", "content": "I'm Rahul ([email protected]), payment keeps failing, urgent!"}],
)
 
import json
args = json.loads(response.choices[0].message.tool_calls[0].function.arguments)
print(args)
# {'name': 'Rahul', 'email': '[email protected]', 'issue': 'Payment keeps failing', 'priority': 'high'}

Technique 3: Pydantic validation with retry

Parse the JSON, then validate it with Pydantic. On failure, re-prompt with the error message:

from pydantic import BaseModel, ValidationError
 
class SupportTicket(BaseModel):
    name: str
    email: str
    issue: str
    priority: str  # "low" | "medium" | "high"
 
def extract_ticket(user_message: str) -> SupportTicket:
    for attempt in range(2):
        response = client.chat.completions.create(
            model="openai/gpt-4o-mini",
            response_format={"type": "json_object"},
            messages=[
                {"role": "system", "content": "Return JSON: {name, email, issue, priority (low/medium/high)}."},
                {"role": "user",   "content": user_message},
            ],
        )
        try:
            data = json.loads(response.choices[0].message.content)
            return SupportTicket(**data)
        except (json.JSONDecodeError, ValidationError) as e:
            if attempt == 1:
                raise
            # Re-prompt with error on second attempt
            user_message = f"{user_message}\n\nPrevious output failed: {e}. Fix and return valid JSON."
 
ticket = extract_ticket("I'm Rahul, [email protected], payment failed, urgent!")
print(ticket)

Technique 4: Schema-first prompt engineering

For models without JSON mode, include the target schema directly and use a completion cue:

# End the user message with the opening brace to trigger JSON completion
prompt = (
    "Extract: name, email, issue from this message.\n"
    "Return only valid JSON matching: {\"name\": string, \"email\": string, \"issue\": string}\n\n"
    f"Message: {user_message}\n\nJSON: {{"
)

Choosing the right technique

| Scenario | Best technique | |----------|---------------| | Simple flat schema | JSON mode | | Complex schema with enums | Function calling | | Model without JSON mode | Schema-first prompt | | Production with validation | JSON mode + Pydantic |

AICredits

Build with any AI model. Pay in INR.

One OpenAI-compatible API for every major model, with a prepaid rupee wallet. Top up via UPI — no international credit card, no forex surprises.

Top up via UPI, prepaid INR wallet, no international card needed

Chain-of-Thought, Few-Shot or Plain Prompt? Pick by Cost and Quality

6 min read

Write Cheaper Prompts: Six Habits That Cut Your LLM Token Bill

6 min read

Build Your First AI App in a Weekend (India Edition)

8 min read

Continue in Docs

Need implementation commands and endpoint details? Go to quickstart or API reference.