Draw the schema boundary before relying on recovery

Downstream code breaks on the first malformed response. A support classifier feeding a queue, an extractor writing to a database, or a triage step calling another API cannot pause to interpret prose around a JSON block. Teams usually respond by tightening the prompt, then by adding retries with the validation error appended. Both help, but they leave the central question unanswered: which component guarantees the shape, and what happens when the model still misses it.

AIVAX answers with two paths. Send response_schema when the gateway should own validation and repair. Send response_format when the provider should enforce the schema natively, with gateway healing available when explicitly enabled or when the account default applies. The names look similar; the ownership differs, and that difference decides how failures surface, what retries cost, and where application validation still belongs.

How AIVAX draws the boundary

response_schema runs through AIVAX's own pipeline: the gateway stores the schema, instructs the model in plain text to follow it, then extracts, validates, and repairs the answer itself. Nothing about the shape is delegated to a provider parameter, which is why this path works with any model, including ones without native structured-output support.

After generation, AIVAX tries to extract JSON from the full text, from brace- and bracket-repaired variants such as a missing opening or closing delimiter, and from JSON code blocks in the answer. The common generation slips — prose around the object, fences, a dropped brace — usually resolve here, silently, without spending another model call. When extraction yields a candidate, the gateway validates it against the schema. A schema mismatch appends the validation errors, with path, message, and keyword, and asks the model to generate the JSON again. Unparseable output triggers a retry with a plain-format correction message. The loop continues until valid JSON arrives or the attempt limit is reached; past the limit, the request fails instead of returning a near-miss. The guarantee is one-sided by design: an invalid object never reaches the caller, but success is not promised — a pathological request fails loudly rather than delivering a plausible-looking miss. During streaming under healing, only the validated JSON chunks reach the caller.

response_format with type: "json_schema" takes the other route. AIVAX keeps the schema internally and forwards it as a native provider response_format carrying the name response_schema and the model's strict flag. Gateway healing joins only when response_format.json_schema.healing_options sets max_attempts, or when the account's automatic JSON Healing default applies (enabled by default). Other response_format types, such as plain JSON-object mode, pass through to the provider body without gateway validation. Healing applies to the schema-constrained final answer, including answers that follow tool results, not to tool-call arguments mid-turn. Tools, MCP sources, reasoning, and multi-step agentic work run in the earlier turns; the model then produces the final object under the same gateway-owned validation. Because the schema travels as instruction text rather than a provider capability flag, this composition holds for any model — the gateway never needs the provider to support tools and structured output together.

json_only: true controls the envelope rather than the guarantee. With streaming off, the HTTP body carries only the final JSON as application/json. With streaming on, AIVAX emits the complete JSON as one SSE data event followed by [DONE].

What providers guarantee, and what they leave to you

Provider dialects differ enough that one vendor's strict mode says little about another's. OpenAI distinguishes JSON mode from Structured Outputs: JSON mode constrains output to valid JSON but does not match it to a schema, requires an explicit JSON instruction, and keeps edge cases the application must handle. Structured Outputs bind the response to the supplied schema under strict: true, within a rigid dialect that demands additionalProperties: false and explicit required entries.

Google's Gemini structured-output documentation promises syntactically correct JSON following the schema, then directs applications to validate values in their own code before use. That phrasing marks the practical boundary across vendors: even a sampler-level guarantee covers shape, not business correctness. A value can satisfy the schema type while violating permissions, identifiers, ranges, or account rules.

This is why AIVAX documents healing as reliability improvement rather than a guarantee. Retries repair syntax slips and missing keys; they cannot make an underspecified schema sufficient or a too-small model careful. In practice, recent models — including the smaller ones — rarely produce malformed JSON on straightforward schemas, so healing mostly sits idle and the misses that remain are schema mismatches worth reading, not noise worth retrying away.

Choose the mode by ownership

Use response_schema when the gateway should own the outcome: the model lacks native structured support, the final JSON follows tool use, or the consumer cannot tolerate malformed output. Accept that each retry is another generation with its own latency and token spend.

Use response_format with type: "json_schema" and no healing options when the integrated model supports native enforcement and you want the provider's mechanism to bind the output. Healing stays available per request through healing_options, and the account default keeps it on unless deliberately disabled.

{
    "model": "@openai/gpt-4o",
    "messages": [{ "role": "user", "content": "Return a short status object." }],
    "response_format": {
        "type": "json_schema",
        "json_schema": {
            "schema": {
                "type": "object",
                "properties": {
                    "status": { "type": "string" },
                    "message": { "type": "string" }
                },
                "required": ["status", "message"]
            },
            "healing_options": { "max_attempts": 5 }
        }
    }
}

max_attempts accepts values from 1 to 10. Raise it only after the schema and prompt earn it; repeated limit hits point at an overly rigid schema, missing required fields, vague instructions, noisy tool results, or a model too small for the task.

Make retries converge instead of repeating

Schema design decides whether a retry can succeed. Mark every field the application must receive as required, give arrays explicit items, and prefer enum, format, and length or pattern constraints over prose descriptions when the accepted values are known. For extraction, state which fields may be null when absent and which must never be invented. For classification, pair the enum label with a short reason field so reviewers can audit the decision. For tool-backed answers, let the model call tools first and include source fields when the consumer needs provenance.

Watch the failure pattern before touching the limit. Syntax failures that heal on the first retry need no change. Repeated schema mismatches on the same path indicate a constraint the model cannot satisfy from the prompt; relax the schema, sharpen the field description, or split the request. Failures scattered across paths suggest the model is guessing, which a higher limit converts into a slower guess.

Validate shape in the gateway, rules in the application

AIVAX validates shape: types, required fields, enums, patterns, numeric bounds, and array constraints within its documented JSON Schema subset. Business rules stay with the application. Re-check permissions, identifier validity, date ranges, and account-specific constraints after parsing, including for responses the gateway healed successfully. A healed object arrives well-formed; whether it may execute a write or call an external API remains a policy decision outside the schema.

The practical setup is therefore three layers: a strict schema for shape, bounded healing for recoverable misses, and application checks for everything the schema cannot express. Keep the attempt budget low enough that a pathological request fails loudly, and log the validation errors from healing feedback alongside the accepted result so evaluations can distinguish first-try success from repaired success.

The full parameter reference, schema feature list, and json_only behavior live in the Structured Responses documentation.