From test to monitor: the judge trajectory as a production signal

A gateway can pass every pre-deployment check and still lose conversations in production. The model is the same, the instructions are the same, the tools are the same — but real users do not follow the script. They ask ambiguous questions, change their mind mid-flow, paste unexpected input, and abandon conversations that a simulated user would have patiently continued. The failure shows up days later in a support ticket or a dropped conversion, long after the turn where the conversation actually drifted.

This post assumes the Agentic Tests mechanism: a simulated user pursues a goal through the gateway while an independent judge scores every turn, tracking the trajectory through active, at_risk, success, and loss states. That post covered the pre-deployment side — writing goals, reading trajectories, building regression checks. Here we cover the other half: the same trajectory vocabulary, applied to live traffic, as a monitoring signal.

Tests end at deployment. Conversations do not

A scheduled Agentic Test answers a bounded question: does this gateway configuration still perform a specific job? It runs on a five-field cron schedule, compares the outcome against a goal, and notifies on repeated failure. That is regression coverage, and it works because the input is controlled — the same goal, the same turn budget, the same judge criteria.

Production traffic is uncontrolled by definition. No two real conversations share a goal statement, so there is no per-conversation judge score waiting to be read. What production does offer, and tests cannot, is volume: hundreds or thousands of real trajectories per day, each one a record of how the gateway behaved when nobody wrote the script. The monitoring question is narrower: which of these conversations drifted, when, and in the same way as conversations that drifted before.

That shift — from judging one conversation against a goal to comparing many conversations against each other — is where trajectory structure earns its place.

Behavior has shape, and shape is monitorable

Recent research gives this intuition a precise form. In Automata from Agent Traces: Failure and Next-Step Prediction (arXiv 2608.23670), the authors collapse entire corpora of agent traces into compact finite-state machines of 7–43 states that replay held-out traces at 0.997 fitness or better. Per-state behavioral features predict failure at up to 0.94 held-out AUROC, and an online monitor ranks failing runs above passing ones from a partial trace. The failure analysis (Appendix E.1) reports the substantive claim for operators: failed traces are consistently longer across all datasets yet visit the same or fewer unique states, suggesting that failure shows up as cycling through familiar states rather than exploring new ones. Structure diverges before the outcome does.

You do not need to build that FSM to use the lesson. The operational primitive is simpler: define the small set of behavioral states your gateway can be in — gathering information, calling a tool, waiting on retrieval, summarizing, asking a clarification, closing — and watch how conversations move between them. A healthy conversation progresses through the states and terminates. A drifting one cycles: clarification after clarification, tool call after tool call, retrieval returning nothing useful twice in a row. The cycle is visible in the transcript long before the user abandons it.

The judge vocabulary works on real transcripts

The Agentic Tests judge already speaks this language. Each evaluated turn produces a normalized score, a state, and cumulative trajectory values (turn_delta, conversation_delta, loss_streak), streamed as chat.judge.turn_analysis_result_ready events. A conversation becomes at_risk when the cumulative trajectory falls to or below the loss threshold or a low-scoring streak begins — not on a single weak turn, which the judge explicitly tolerates as recoverable.

Apply that vocabulary to production exports and three review questions become mechanical:

• Where did it become at risk? Export conversations as JSONL, replay the transcript turn by turn, and mark the first turn where the trajectory would have crossed into at_risk. That turn — not the abandonment three turns later — is the incident.
• Did it recover, and how? The judge distinguishes a dip that recovers from a dip that persists. In production review, the same distinction separates gateway problems (no recovery without intervention) from normal conversational noise (user clarifies, flow resumes).
• Did a tool or retrieval step change the trajectory? In the retest, the run inspector retains per-message token usage and judge opinions together; in the production export, the same turn shows the tool or retrieval output that explains the drop. When the trajectory drops on the same turn a tool returns an empty result or retrieval returns nothing relevant, the cause is in the record.

This is retrospective analysis, not live judging: running a judge model over every production turn would multiply inference cost. The point is sampling and pattern-finding: review the conversations users abandoned, find the at-risk turn, and check whether the same turn shape recurs across many failures.

From pattern to action

A recurring at-risk shape is an operational input. Each destination has a trigger:

1. Escalate the live conversation — trigger: a conversation currently in at_risk, detectable cheaply (same tool failing twice, same clarification asked twice, retrieval empty on a high-stakes flow). Route to a human or a fallback flow before the user leaves.
2. Retest the gateway — trigger: the same shape recurs across exported conversations. Turn it into a new Agentic Test: write the failed outcome as a goal, add the missing requirement as judge-only validation criteria, and run it against the current configuration. If it fails, the regression suite just grew by one real failure.
3. Adjust the gateway — trigger: the new test reproduces the failure. Most recurring shapes point at configuration, not the model: instructions that never ask the disambiguating question early, a tool description that invites malformed arguments, retrieval that returns nothing for a common phrasing. Change the configuration, then watch the shape disappear from exports.
4. Fix the boundary system — trigger: the new test passes but production turn_delta still drops on tool or retrieval turns. The trajectory drop means a tool is down, a document is missing, or an API changed. The conversation record is then incident evidence for another team, with timestamps and the exact failing turn attached.

The loop closes on a named metric: conversations entering at_risk per 1,000 conversations, before versus after the change — measured on the next export, not assumed.

What the signal is not

Three boundaries keep this honest. First, the trajectory vocabulary describes drift, not quality: a conversation can stay active throughout and still produce a mediocre answer the user accepts. Task-specific evals still own output quality. Second, retrospective review finds patterns in failures that already happened; it does not prevent the next novel failure mode, which by definition has no prior shape. Scheduled tests and live escalation cover the modes review cannot see yet. Third, the research numbers cited above (0.94 AUROC, 0.997 fitness) come from the paper's twelve datasets and its FSM construction — they describe what trajectory structure can predict, not what any particular gateway review will achieve. Treat them as evidence that the signal exists, then measure it on your own traffic.

The trajectory is the handoff between testing and operations

Pre-deployment tests and production monitoring usually live in different tools with different vocabularies — pass rates on one side, latency and error dashboards on the other. The judge trajectory is one signal that spans both: the same score, state, and cumulative values that gate a release can describe a live conversation's drift, and the same export that documents an incident can become next week's regression test.

Start with the conversations users already abandoned. Find the turn where each one became at risk. If the same shape recurs, write it as a test; if that test starts failing repeatedly, it trips the notification threshold you already configured for scheduled tests — then it is not anecdote, it is the next escalation rule to set. The Agentic Tests documentation describes the judge events, trajectory fields, and run exports this workflow builds on.