Introducing Agentic Tests

A support agent can give a polished first answer and still fail the conversation. It may skip a required question, misuse a tool two turns later, lose track of an objection or end without the action the user needed. A single-response check sees one frame. The user experiences the sequence.

Agentic Tests evaluates that sequence. AIVAX simulates a goal-oriented user, sends every assistant turn through the AI Gateway under test and asks an independent judge whether the conversation reached its goal, remains recoverable or has persistently moved away from the expected outcome.

This turns a real conversational job—qualify a prospect, resolve a support case, complete onboarding, use a tool correctly—into a repeatable regression check for the gateway configuration that performs it.

Three roles, one conversation

An Agentic Test coordinates three distinct roles:

1. The simulated user pursues the goal and produces the next message from the conversation so far.
2. The AI Gateway responds with its configured model, instructions, tools, skills, knowledge and inference settings.
3. The independent judge evaluates progress against the goal and any judge-only criteria.

The separation matters. The simulated user is not grading the assistant, and the judge is not producing the assistant's answer. Each role sees the information needed for its part of the evaluation.

After an evaluated turn, the judge reports reasoning, a normalized score, a state and trajectory measurements. A run can move through active, at_risk, success or loss as the conversation develops. The test therefore preserves not just the final result, but how the gateway arrived there.

Write the goal as an observable outcome

A useful test starts with the result a user should be able to observe. Consider a plan-recommendation flow:

Identify the customer's team size, recommend the correct plan, explain why it fits, and provide the next signup step.

That is a goal, not a script for the assistant. It leaves room for the simulated user and gateway to have a natural conversation while giving the judge a concrete outcome to evaluate.

Optional validation criteria add requirements for the judge alone:

The recommendation must name the selected plan and connect it to the stated team size. The final response must include a direct signup step.

Keeping those criteria away from the simulated user prevents the test actor from unnaturally steering the exchange toward the checks. The gateway under test does not receive them either.

Start messages can establish a customer objection, a prior assistant response or another specific point in a flow. An external user identifier can also be forwarded when gateway behavior depends on identity from your application.

Focused tests produce the clearest failures. If qualification, pricing explanation and objection handling are independent behaviors, separate them. A failed run then points to a smaller surface that can be inspected and corrected.

A weak turn can still recover

Conversation quality is not always monotonic. An assistant may ask an unhelpful question, then recover when the user clarifies. Treating every low-scoring turn as an immediate failure would hide that distinction.

Agentic Tests tracks both the current score and the cumulative trajectory. Success occurs when the configured baseline is reached or the simulated user declares the goal complete. A loss requires the score and trajectory to stay at or below the loss threshold for the required consecutive evaluations. If neither happens before the turn budget is exhausted, the result is incomplete.

This produces three final outcomes with different meanings:

• success — the conversation reached the configured goal threshold or the simulated user declared completion;
• loss — the conversation established a persistent unsuccessful trajectory;
• incomplete — the turn budget ended without success or persistent loss.

You can choose when judging begins, when the simulated user may end the exchange and how many turns the scenario receives. Early discovery-heavy flows may benefit from delaying the first judged turn; short regression checks can use a tighter budget. These controls change the evaluation boundary, not the model configured on the gateway.

Inspect the evidence behind a run

A persistent test definition can be reused, while every execution creates its own run. Editing the test later does not replace earlier run history.

The run inspector keeps the simulated-user, assistant and judge messages in chronological order. It also retains timestamps, prompt and completion usage, judge opinions and trajectory values, the final evaluation, failure information and total charged cost. A run can be exported as JSON when the record needs offline analysis.

That evidence helps answer more useful questions than “did it pass?”:

• At which turn did the conversation become at risk?
• Did the gateway recover after a weak response?
• Which judge requirement remained unsatisfied?
• Did a tool or retrieval step change the trajectory?
• How much simulated-user, gateway and judge inference did the run consume?

Persistent tests can also run on a five-field cron schedule, with a minimum interval of five minutes. Failure notifications can wait for a configurable number of consecutive failed executions, and a recovery notification can report the first subsequent success. Successful and failed runs are retained for one month; cancelled runs are retained for one day, so results needed beyond those windows should be exported.

Run ephemeral evaluations from a trusted backend

Some applications need the evaluation immediately rather than as a stored dashboard run. The direct generation endpoint executes an ephemeral Agentic Test and streams its progress with Server-Sent Events. It accepts the same core evaluation settings, but it does not create a persistent test or run in the dashboard.

The stream distinguishes user generation, gateway generation, judge analysis, usage updates, retryable errors and the final validation event. Streamed content arrives in chunks, so consumers route messages by event type and concatenate chunks in order.

Direct execution requires a private AIVAX API key and Accept: text/event-stream. Keep that key on a trusted backend—never in browser code or a distributed application bundle.

Start with one conversation that matters

Choose a flow whose outcome is visible and whose regression would matter: a support resolution, a sales qualification, an onboarding milestone, a grounded answer or a tool-mediated action. Write the outcome as the goal, reserve implementation requirements for judge-only validation criteria and give the conversation only the turns it reasonably needs.

Then inspect the trajectory, not only the final label. That is where a conversational test becomes operational evidence rather than another prompt snapshot.

Explore the Testing page or open AIVAX Console to create a focused evaluation for an AI Gateway.