All posts

Reasoning Is a Protocol, Not Just a Model Setting

Reasoning APIs now carry summaries, signatures, encrypted state and provider-specific stream events. A production gateway must preserve that protocol, not flatten it into text.

Aug 25, 2026 By AIVAX Labs

A reasoning model does more than return a longer answer. Modern APIs can emit readable summaries, opaque state, signatures, usage details and several kinds of streaming events before the final text arrives. Some of that data is useful to a person. Some of it must be preserved unchanged so the provider can continue a tool-using or multi-turn interaction.

That makes reasoning an integration protocol, not merely a model setting.

Applications that treat every response as a string will miss this distinction. The first turn may look correct while a later tool result loses its reasoning state, a provider switch breaks the stream parser, or an observability system records a summary but discards the opaque item required for continuity. A production AI gateway therefore has two jobs: expose a stable application contract and preserve the provider semantics that cannot safely be normalized away.

The visible answer is only one output channel

Reasoning-capable APIs increasingly separate at least three concerns:

The final answer is user-facing text or structured output.
A reasoning summary is a provider-generated explanation suitable for display or inspection when the model and account support it.
Reasoning state is provider-specific data used to maintain continuity, especially around tools and stateless multi-turn requests.

Those channels are not interchangeable. A summary is not necessarily the model's raw chain of thought, and opaque state is not content an application should render. Flattening all three into one text field loses provenance and makes it difficult to apply different storage, display and security policies.

The provider contracts make this separation explicit. OpenAI's Responses API represents reasoning as output items and can return summary parts or encrypted reasoning content. Anthropic returns thinking blocks with signatures and may return encrypted redacted_thinking blocks. Google's Gemini documentation describes thought signatures as encrypted representations of internal reasoning that must be preserved for stateless continuity.

The shared concept is reasoning continuity. The wire formats are different.

Providers stream different event grammars

A streaming integration cannot assume that reasoning arrives as ordinary text tokens.

OpenAI Responses uses typed server-sent events such as response.reasoning_summary_text.delta, response.reasoning_summary_text.done and response.reasoning_text.delta. The event names distinguish incremental content from completed parts and distinguish summaries from reasoning text.

Anthropic's Messages API organizes a stream around content blocks. Thinking can arrive through thinking_delta events, while a separate signature_delta carries the signature associated with the block. Text, tool input and thinking are different delta types even though they share the same connection.

Google's Interactions API uses step-oriented events. A stream can contain step.start, step.delta and step.stop for model output, function calls and thought steps. Its documentation also warns clients to skip unknown event types gracefully because the protocol can gain new event and delta types over time.

These are not cosmetic naming differences. Each protocol makes different promises about ordering, item identity, completion and replay. A parser that extracts only a text property may appear to work for a simple answer but fail when reasoning and tool calls interleave.

Control, display and continuity are separate decisions

A useful gateway contract separates three questions that application code often mixes together.

How much reasoning should the model use?

Providers expose different controls. OpenAI supports reasoning effort levels. Anthropic supports effort and, for compatible models, adaptive or budget-based thinking. Gemini exposes thinking levels for supported models. These settings affect latency, token use and response quality, but their accepted values and model compatibility vary.

A portable application-level field such as reasoning_effort is useful only if the gateway maps it deliberately and rejects or documents unsupported combinations. Silently forwarding every option to every provider turns portability into guesswork.

What reasoning information may the user see?

Readable reasoning output should be treated as a presentation and policy choice, not as ordinary answer text. Providers may return a summary, omit readable thinking, or redact part of it. The application may choose to show a summary in a developer console while hiding it from an end-user chat.

This distinction also matters for observability. A reasoning summary can help diagnose why an agent selected a tool or abandoned a plan, but it should not be mistaken for a complete execution trace. Tool calls, tool results, timings, errors, usage and the final outcome remain separate evidence.

What state must survive the next turn?

This is the least visible and most operationally important question. Anthropic instructs clients to pass thinking and redacted-thinking blocks back unchanged in tool and multi-turn workflows. Gemini requires stateless clients to resend thought blocks exactly as received. OpenAI can return encrypted reasoning content for use when the application manages state without server-side storage.

A gateway may normalize readable summaries for the client, but it must retain enough identity and opaque data to reconstruct the provider-native item when the conversation continues. Editing, concatenating or dropping that state can break the protocol even when the displayed answer was correct.

A stable contract needs a lossless path

A practical reasoning gateway should support two representations at once:

a normalized view for common application behavior;
a lossless provider view for state that must round-trip.

The normalized view can expose final text, reasoning summaries, usage and a consistent stream shape. The lossless view should keep item identifiers, formats, signatures and encrypted content with enough structure to replay them only to the compatible provider transport.

AIVAX applies this pattern when translating OpenAI Responses output into its OpenAI-compatible chat-completions surface. Reasoning text and summary events can be emitted through a consistent reasoning delta, while structured reasoning details preserve the distinction between text, summary and encrypted data. The details carry item identity and transport format so a later request can reconstruct a compatible Responses reasoning item instead of sending an ambiguous text blob.

On the request path, an application-level reasoning_effort can be translated into the Responses reasoning configuration, including an automatic summary request. On the response path, the translator handles both incremental and completed reasoning events while avoiding duplicate emission of the same completed part.

The important architectural point is not the name of one field. It is that normalization remains reversible where continuity requires it.

Tool use is the real interoperability test

A single-turn prompt is a weak test of reasoning support. The harder path is:

The model reasons about a task.
It emits one or more tool calls.
The application executes the tools.
The next request returns tool results alongside the prior assistant state.
The model resumes with the correct reasoning context.

This flow tests whether the integration preserved item ordering, tool-call identifiers and opaque reasoning data. It also exposes whether streaming assembly produced a replayable final message rather than only a visually correct sequence of deltas.

Test at least these cases for every reasoning-capable transport:

reasoning followed by a final answer without tools;
reasoning followed by one tool call;
interleaved reasoning and multiple tool calls;
an interrupted stream after partial reasoning;
a response containing only a summary, only opaque state, or redacted state;
a stateless next turn that replays the prior provider data;
a model change between turns;
unknown stream events that should be logged and skipped rather than crash the request.

The acceptance criterion is not “the UI showed some thinking.” It is “the complete interaction remains valid, observable and replayable under the provider's documented rules.”

Reasoning changes cost and latency accounting

Reasoning controls are operational controls because they affect more than answer style.

Google documents separate thought-token usage and states that response pricing includes both output and thinking tokens. Anthropic notes that thinking adds processing time and exposes thinking-token details in usage for supported responses. OpenAI exposes reasoning-token details and recommends preserving reasoning items across requests because doing so can improve continuity and reduce unnecessary regeneration in supported flows.

A gateway should therefore record the provider's usage detail instead of collapsing every token into one undifferentiated output count. Operators need to answer:

Did higher reasoning effort improve the task outcome?
How much latency occurred before visible answer text?
How many tokens were spent on reasoning versus the answer?
Did tool loops cause repeated reasoning work?
Did a transport translation omit usage fields or count them twice?

Reasoning effort should be evaluated against task success, not enabled globally because a model supports it. Classification, extraction and simple routing may benefit from low or minimal effort. Complex coding, planning and tool-heavy work may justify more. The right setting is workload-specific and should be tested with representative conversations.

Do not turn reasoning into an accidental data leak

Reasoning data deserves an explicit retention policy.

A readable summary may contain user data, retrieved context or intermediate hypotheses that should not appear in an end-user interface. Encrypted or signed state should remain opaque; logging it indiscriminately increases storage and exposure without making operations easier. Raw chain-of-thought should not be requested as ordinary answer text, and a summary should not be advertised as a complete or authoritative explanation of model behavior.

A production design should decide separately whether to:

stream summaries to the calling application;
display them to users, developers or neither;
store them with conversation logs;
retain opaque replay state and for how long;
redact sensitive inputs from traces;
allow reasoning data to cross a provider or regional boundary.

The safest default is purpose limitation: retain the minimum data required for continuity and debugging, keep opaque state transport-scoped, and expose readable summaries only where they provide a concrete product or operational benefit.

The gateway is where protocol churn should stop

Reasoning APIs will continue to evolve. Event types, supported effort levels, output visibility and replay requirements already differ across providers and models. Applications should not have to implement every variation directly.

A gateway can absorb that churn if it maintains a precise boundary: normalize the common controls and outputs, preserve non-portable state without flattening it, expose usage that operators can measure, and fail clearly when a requested capability cannot be translated safely.

That is the difference between routing a request and operating a reasoning system. The model setting starts the work. The protocol determines whether the whole interaction survives.