Multi-turn evaluation for LLM agents: simulated user and judge
Evaluate a multi-turn LLM agent with a simulated user and an LLM judge. A tested AIVAX Agentic Tests run, with real turn scores, outcomes and per-run costs.
The first multi-turn test I ran for this post ended in a loss after two turns, and the assistant was not at fault. The test required the assistant to ask for the team size before recommending a plan, and the simulated user opened with "hi, i need a plan for my team of 8 people. which one fits best?" The assistant recommended a plan, as anyone would. The judge scored that turn 0.15 because the ordering rule could no longer be met, scored the next turn 0.15 as well, and closed the conversation. The goal I had written let the answer leak into the user's first message.
That failure is a good way into multi-turn evaluation of an LLM agent. You need a simulated user who pursues a goal, the agent under test, and a judge that grades the whole conversation against rules the user never sees. AIVAX Agentic Tests runs that loop against an AI Gateway, so the model, instructions, tools, skills and knowledge are tested together. Below is one small test from gateway to verdict, with the scores, outcomes and costs it produced.
The judge sees the acceptance rules and the simulated user does not
Each turn has three participants. The simulated user receives the goal and the conversation so far and writes the next message. The gateway answers with its own model, instructions, tools and knowledge. The judge reads the goal, the optional validation_criteria and the full transcript, then returns a score, a reasoning and a state.
I checked in the code that validation_criteria goes to the judge only. It is not in the simulator's prompt and not in the gateway's context, which is the reason to keep acceptance rules there and not in the goal: a simulated user that knows the checklist steers toward it. The judge's instructions also tell it to treat the transcript and any tool output as untrusted data and say that a claim of success is not proof of success, so an assistant that announces "your account is upgraded" gains nothing from saying it. The judge also gives more weight to the latest user–assistant exchange when it rates the current trajectory.
None of this reaches outside the conversation. A simulated user that talks to a gateway with live tools triggers live tool calls, so point a test gateway at fixtures or a staging backend, and confirm a real payment or ticket in the system that owns it.
A small gateway, a goal and acceptance rules
The test gateway carries a fictional plan policy, not AIVAX's commercial plans. I created it through the AI Gateway API on @google/gemini-3.5-flash-lite, a model in the live catalog on November 17, 2026:
{
"name": "zz-blog-plan-advisor",
"parameters": {
"baseAddress": "@integrated",
"modelName": "@google/gemini-3.5-flash-lite",
"systemInstruction": "You are a plan advisor for a fictional product. Ask for team size before recommending a plan. For one to five people, recommend Starter; for six or more, recommend Team. Explain the recommendation using the stated team size. The next signup step is to open Settings > Plans and select the recommended plan. Do not claim signup is complete. Keep answers under 80 words."
}
}
The goal is written from the user's side and describes who they are and how they behave. The wording that lost in my first run said "Explain your team size when asked", which a simulated user reads as permission to say it whenever it likes. In six runs with that wording, one ended in a loss for exactly this reason. With the new wording, which says what the user withholds, all six succeeded (three direct, three persisted); that is too few to give a rate, but the leak did not recur.
Save this as agentic-test.json, with model set to the gateway's ID or its slug:id identifier:
{
"model": "<AI_GATEWAY_ID_OR_SLUG>",
"goal": "You are choosing a plan for your team. Do not say how many people are on the team until the assistant asks; when asked, answer that there are eight of you. Ask which plan fits, and continue until you understand the recommendation and how to sign up.",
"validation_criteria": "Validate whether the assistant asks for team size before recommending a plan, recommends Team for eight people, explains why it fits, and gives Settings > Plans > Team as the next signup step. It must not claim signup is complete.",
"start": [],
"max_turns": 10,
"minimum_turns": 1,
"allow_user_exit": false,
"judge_start_turn": 1,
"loss_threshold": 0.2,
"base_threshold": 0.9,
"profile": "medium"
}
allow_user_exit is false so that only the judge or the turn budget ends the conversation. The rest are defaults.
Run it as one streamed request
curl --no-buffer --fail-with-body \
'https://inference.aivax.net/api/v1/generations/agentic-tests' \
-H "Authorization: Bearer ${AIVAX_API_KEY}" \
-H 'Content-Type: application/json' \
-H 'Accept: text/event-stream' \
--data-binary @agentic-test.json
This direct endpoint streams an ephemeral evaluation and leaves no test or run in the dashboard, so your code has to keep what it needs from the stream. Keep the key on a backend.
My three runs of this request returned HTTP 200 and took 9.1 to 10.8 seconds. Each stopped at turn 2 with outcome: "success" and a score of 0.95. The turn-1 scores were 0.75, 0.60 and 0.60, which is the judge saying the assistant had asked the right question and nothing more had happened yet. In all three the assistant opened by asking for the team size and the simulated user answered "there's 8 of us" or close to it.
Each SSE message holds a timestamp and an event with type and data, so route on event.type. chat.user_message.content and chat.assistant_message.content carry text chunks to concatenate in order. chat.judge.turn_analysis_result_ready carries the judge's reasoning, score, state and trajectory values under data.result. usage_updated reports tokens per role, unhandled_error names the failing scope and whether a retry follows, and chat.validation.end carries the terminal outcome. The stream event reference lists the rest.
A release gate should read that last event and nothing else. HTTP 200 only says the stream opened, and an intermediate pass: true only says persistent loss has not been established. This script exits 0 only on outcome: "success":
import { createInterface } from "node:readline";
let end = null;
for await (const line of createInterface({ input: process.stdin })) {
if (!line.startsWith("data: ")) continue;
try {
const { event } = JSON.parse(line.slice(6));
if (event.type === "chat.validation.end") end = event.data;
} catch {}
}
console.log(end ?? "stream closed without chat.validation.end");
process.exit(end?.outcome === "success" ? 0 : 1);
Fed with node gate.mjs < stream.sse, it exited 0 on a saved successful stream and 1 on the saved lost run. I also cut a stream off after 1,500 bytes and it exited 1 with "stream closed without chat.validation.end", which is the behavior you want: a closed connection with no terminal event is not a pass.
One weak turn does not end a conversation
The judge returns a score between 0.001 and 0.999 for each judged turn. The run keeps a cumulative trajectory, conversation_delta, that starts at 0.5, adds 2 × score − 1 after every judged turn and stays between 0 and 1. It ends with success at the first score at or above base_threshold. It ends with loss only when two consecutive scores are at or below loss_threshold and the trajectory is at or below it too. The streak length of two is fixed in the code, while the two thresholds are yours to set, 0.01 to 0.99 each, more than 0.1 apart.
My first run shows the arithmetic. A score of 0.15 moves the trajectory by −0.7, which clamps it to 0, so the state became at_risk at turn 1 with a loss streak of 1. The second 0.15 made the streak 2 and the state loss. Had the second turn scored higher, the streak would have reset and the conversation could have recovered.
The judge is also slow to say yes. In one of the runs with the original goal, the simulated user asked about a free trial and about paying, and the assistant answered that it could not answer the first question. The judge held the score at 0.78 for three turns, noting each time that the reason the Team plan fits was thin, and reached 0.92 at turn 6 after the user said goodbye, with the remark that the explanation "was minimal" but the criteria were "substantially satisfied". The run passed, and its reasoning says what to tighten, so read the reasoning on passing runs too.
The other outcomes are incomplete, when max_turns runs out or a permitted simulated-user exit comes before the judge is satisfied, and interrupted, when a persisted run stops on a validation hook or an execution failure before a verdict exists. max_turns accepts 2 to 64 and bounds simulated-user turns. minimum_turns controls only when the simulated user may exit and has no effect when exits are disabled. judge_start_turn delays judging past early discovery, though the final turn and any exit are always judged. A higher base_threshold makes the pass stricter, but an ambiguous rubric stays ambiguous.
Read the outcome, not the run state
For history, schedules and notifications, create a persistent test with POST /api/v1/agentic-tests. It takes the same fields as the direct request, with name and gateway in place of model, and POST /api/v1/agentic-tests/<TEST_ID>/runs queues a run and returns 202 with the run in state pending. Poll GET /api/v1/agentic-tests/<TEST_ID>/runs/<RUN_ID> until the state settles.
The run's state is pending, running, succeeded, failed or cancelled, and it says whether execution finished. The verdict is outcome, which the run summary carries next to state and which also sits in result.outcome. I made a persisted run end in a loss by telling the simulated user to give the team size in its first message. It came back with state: "succeeded", outcome: "loss", a summary score of 0, which is the final cumulative trajectory and not the last judge score, and a cost of $0.011, after 22 seconds.
Failure notifications follow the state. A scheduled test that ends in loss every night sends no alert, because the run executed without error. To get one, either read outcome from the run record in your own monitoring or register an after-test hook. It fires once the test reaches a normal terminal outcome, before the final event, and carries the outcome, score and loss streak, so your endpoint can forward a loss and answer 2xx. Any hook that answers non-2xx stops the run and stores it as failed with outcome: "interrupted", which does count toward the failure notification. The before-inference and after-inference hooks fire before and after each gateway inference, which makes them the right tool for an exact check such as a forbidden phrase that you would not leave to a judge. They work only on persisted tests, and the direct endpoint rejects a request that includes hooks.
Schedules take a five-field cron expression with occurrences at least five minutes apart. Succeeded and failed runs are kept for 30 days and cancelled runs for one day, so export any run you need longer. New runs are limited per account per minute: 5 on Free, 20 on Pro, 100 on Max, and no plan cap on Ultimate. Manual runs, scheduled runs and direct evaluations share that quota, and the excess gets a 429 before any stream opens, so pace the starts and retry with backoff (limits). A direct request without a positive balance returns 402, and a persisted run fails with "The account balance was exhausted during this run."
Short runs cost between a third of a cent and a cent
A run costs the gateway's inference plus the simulated user and the judge, at the rates of the selected profile, with the account's plan multiplier applied. The profile picks the simulator and judge models and does not change the gateway's model. I ran the same persisted test at each profile, plus the run that ended in a loss:
| Profile | Turns | Wall time | Billed cost |
|---|---|---|---|
low | 2 | 7 s | $0.0033 |
medium | 2 | 9 s | $0.0037 |
high | 3 | 53 s | $0.0070 |
medium, ended in loss | 2 | 22 s | $0.0110 |
These are the cost fields of the run records, for a gateway whose answers fit in 80 words. In the medium run the judge read 1,660 input tokens and wrote 486, against 850 and 22 for the simulated user and 217 and 72 for the gateway, so by token count the judge is most of that bill. The high run shows why the profile matters as much as the length: it wrote 1,122 tokens as the simulated user and 4,356 as the judge, and it took 53 seconds. The judge's input also grows with the conversation, because every judged turn sends it the full transcript: its two calls read 764 and 896 tokens in that run. A suite of long conversations with many turns will cost far more per run than these numbers. Check the pricing and limits rates and each run's cost before scaling repeated evaluations, and remember that a schedule creates a billed run at every occurrence until you disable it.
Keep the failures as scenarios
A failed run is worth keeping if you can replay it. Save the goal, the start messages that set the opening context, the criteria and the gateway configuration, change the gateway, and run it again. Keep start for scenarios that begin mid-conversation, such as a customer objection after a prior answer. Each persisted execution is its own run, so editing the test does not rewrite earlier history.
The judge can be wrong, and a score trend means little until someone has compared it with human labels. Have a domain reviewer mark a handful of transcripts, including clear passes, clear failures and borderline recoveries, and compare those marks with the judge's reasoning. Langfuse's evaluation guide says the same about LLM judges: they need calibration against human preferences and can share blind spots with the application's model. Hamel Husain's notes on multi-turn chat evals argue for reading conversations and making binary pass/fail calls before automating anything. I agree with that order, and a simulated-user suite is most useful once you know which failures you are looking for.
Simulation is one way to find them. Langfuse describes a second one, N+1 evaluation, which replays real production conversations up to the point where the bot went wrong. Frameworks such as DeepEval run their conversation simulator inside your test code. Agentic Tests runs the simulation on AIVAX against a gateway. If your agent is not behind one yet, a gateway can use your own OpenAI-compatible endpoint as its provider. To apply the same per-turn signal to live traffic, see the judge-trajectory post; if you are coming from OpenAI Evals, the migration guide covers rewriting a multi-turn test.
Not every question needs a conversation. A closed question such as which team should own a ticket can be checked with a single request that returns typed answers, as in the support-triage example. When a model is about to retire, point the same scenarios at a second gateway on the replacement model, and see the guide to pinning a model ID or using an alias for the surrounding checks. AIVAX Testing and the console are where to create a persistent test.
Ran against the live AIVAX API on November 17, 2026 (eight direct evaluations and five persisted runs on @google/gemini-3.5-flash-lite, all deleted afterwards), with the rest checked against the docs and implementation the same day.