Gemini Decision 1 vs Jev vs OpenAI Decisions: 40 tickets, one request body
I sent 40 labeled support tickets to Gemini Decision 1, Jev 1.14, GPT-6 Luna Decisions and Perplexity Decider v1.2 with the same request. Accuracy came out nearly even; option order, calibration and tokens per request did not.
Reversing the order of five team names in a JSON object sent four of my forty support tickets to a different team on GPT-6 Luna Decisions. On Gemini Decision 1 the same reversal changed nothing: not one choice, and not one probability by as much as 0.005.
I did not set out to measure that. I wanted the comparison people have been asking for since Google shipped Gemini Decision 1 and its Decisions API on November 11: how it stacks up against TypeSafe's Jev and against OpenAI's Decisions API on GPT-6 Luna. The comparisons I found either run a single prompt or quote the vendors' own leaderboards. AIVAX, the platform this blog belongs to, serves all of these models behind one endpoint, POST /api/v1/generations/decisions, with one request body, so I could send identical requests to each and read tokens and cost from the same usage field. I added Perplexity Decider v1.2, which lists a fifth of their input price.
The test set is small and deliberately awkward
I wrote 40 support tickets and labeled each with one of five teams (billing, technical, sales, account, other) and whether it needs an answer within the hour. Thirteen are urgent. Sixteen are hard on purpose: sarcasm ("Great, another outage. Third one this month."), urgency stated as its opposite ("Not urgent at all, but the checkout page has been returning an error for every customer since 9am."), a lone "hi", spam, a sales threat with a Friday deadline, and tickets that sit between two teams. Three are in Portuguese or Spanish.
Every request carried the ticket as state and the same two questions:
{
"model": "@google/gemini-decision-1",
"state": "How much would it cost to add 10 more users to our current plan?",
"questions": {
"department": {
"type": "choice",
"instructions": "Which team should handle this support message?",
"criteria": {
"billing": "Charges, refunds, invoices, payment methods, subscription changes and cancellations",
"technical": "Bugs, errors, outages, API or SDK problems, integrations not working",
"sales": "Pricing questions, quotes, plans, discounts, feature questions before buying",
"account": "Login, passwords, 2FA, SSO, permissions, account security, profile changes, account deletion",
"other": "Thanks, feedback, spam, or anything that is not a request"
}
},
"is_urgent": {
"type": "noul",
"instructions": "Does this message need a response within the hour?",
"criteria": {
"true": "Production is down, the customer is blocked now, a security incident, or a deadline within hours",
"false": "Can wait for the normal queue, including messages that only sound emotional"
}
}
}
}
I ran the set three times on each model: once as written, once again unchanged to check repeatability, and once with the five department options in reverse order. That is 480 answered calls; the ones rejected by the rate limit, described further down, were sent again and are not counted. Forty tickets cannot separate models that differ by one or two answers. They can show behavior that repeats across many tickets, which is what the option order and the urgency values did.
Accuracy was a near tie
| Model | Department, original order | Department, reversed | Urgency at 0.5 | Changed by reversal | Median input tokens | Cost per 1,000 | Median time |
|---|---|---|---|---|---|---|---|
@google/gemini-decision-1 | 38/40 | 38/40 | 37/40 | 0 | 296 | $0.030 | 465 ms |
@typesafe/jev-1.14 | 38/40 | 38/40 | 39/40 | 2 | 505 | $0.022 | 458 ms |
@openai/gpt-6-luna-decisions | 36/40 | 38/40 | 36/40 | 4 | 411 | $0.042 | 438 ms |
@perplexity/pplx-decider-v1.2 | 38/40 | 38/40 | 38/40 | 0 | 252 | $0.005 | 484 ms |
No model got every department right, and no ticket fooled all four. Two tickets fooled three. "My API key stopped working after I reset my password. Every call returns 401." is labeled account in my set; Gemini, Luna and Decider said technical, and Jev agreed with me by 0.52 to 0.48. "Is there a way to pay yearly instead of monthly, and would that be cheaper?" is labeled sales; Gemini, Jev and Luna said billing, and only Decider said sales. I am not sure my labels are right. "Subscription changes" sits in the billing description I wrote, and an API error after a password reset belongs to both teams. When the models split on a ticket, I would check the label and the criteria text before blaming the model.
Every model read the sarcastic outage ticket as technical, the lone "hi" as other, and the Portuguese and Spanish tickets correctly. The negated-urgency ticket ("Not urgent at all, but the checkout page...") came back urgent on all four, at 0.58 on Decider and 0.78 to 1.00 on the others.
Repeating the set unchanged moved almost nothing. Gemini and Luna returned identical answers. Jev flipped one ticket that it had split 0.58 to 0.42, and Decider's urgency values moved by at most 0.04. Input tokens were identical in both passes on every model.
Option order moved two of the four models
The OpenAI forum thread for the Decisions API beta has a post from a developer who found that "choice order changes the probabilities", on Luna and on Jev. AIVAX forwards the options in the order it receives them, and reversing the five team names reproduced the effect on my tickets. Luna changed its answer on four tickets and Jev on two, with single probabilities shifting by up to 0.45 and 0.43. On "How much would it cost to add 10 more users to our current plan?", labeled sales, Luna went from billing at 0.53 to sales at 0.61, and Jev from billing at 0.72 to sales at 0.71. Decider kept every choice, but its probabilities moved by up to 0.24, so a threshold tuned on one order would not carry over exactly. Gemini Decision 1 returned the same distribution in both orders, to every digit.
All six changed answers came from tickets where the model reported a confidence below 0.7, and Luna's accuracy happened to go up in the reversed order. That is why the effect is easy to miss. It still means the order of keys in your criteria object changes routing, and nobody reviews key order in a pull request. If you run Jev or Luna, keep the option order fixed in code, never build it from a set or from a database query without an ORDER BY, and treat a reordering as a change that needs the same evaluation as a new model.
Luna was confidently wrong on urgency
Twenty-five of Luna's forty urgency values sat at 0.02 or below or at 0.98 or above. Gemini had nine values that far out, Decider 22 and Jev one. Extreme values are fine when they are right, and most of Luna's were. Its three misses on urgent tickets came back at 0.20, 0.35 and 0.15, though. The last is "My trial ends tomorrow and I still can't figure out if your plan supports multiple workspaces", from a customer who needs an answer before being charged. Decider gave that one 0.78 and Gemini 0.59.
A common way to use a noul is to act automatically outside a middle band and send the band to a person, and TypeSafe's confidence guide suggests 0.7 and 0.3 as the edges. With that band, Jev and Decider made no mistakes outside it on my set, sending five and seven tickets to review. Gemini made one, rating the 401 ticket urgent at 0.83. Luna made two, both urgent tickets it called routine at 0.20 and 0.15. One or two errors is not a difference I would rank on. What makes me keep a person in Luna's loop is the combination: it is the model that commits hardest, and the review band catches the fewest of its answers.
The angriest ticket in the set, "URGENT!!! Refund my last payment, I never authorized it.", was missed by three of the four. I labeled it urgent because an unauthorized charge is a security question and the true criterion mentions security incidents. Gemini gave it 0.34, Decider 0.47 and Luna 0.20; Jev gave it 0.57. None of them read "never authorized" as the security incident my criterion described.
Confidence on choice behaved better. With a 0.7 confidence floor for routing without review, Gemini routed 36 tickets with no error, Decider 33 with no error, Jev 36 with one (the yearly-plan ticket, billing at 0.95 confidence), and Luna 33 with one.
Gemini Decision 1 was also the only one of the four that sent a confidence field on noul answers; the others sent one only on choice. AIVAX returns what each model reports and the decisions page lists confidence as optional, so routing code has to work without it, which the support triage example handles by falling back to the top probability.
Each model counts the same request differently
Gemini Decision 1 and GPT-6 Luna Decisions both list $0.10 per million input tokens in the AIVAX model catalog, and output costs nothing on any of the four. Gemini read my request in a median of 296 tokens and Luna in 411, so the same ticket cost about 40% more on Luna. Jev lists $0.042 per million and counted 505 tokens, which still made it cheaper than both. Decider v1.2 lists $0.02 per million and counted 252 tokens: $0.005 per thousand tickets, about a quarter of Jev's cost and a sixth of Gemini's. Jev also reports about 72 output tokens per call, which do not reach the bill.
My tickets ranged from one word to about thirty, and the token count on each model moved by less than 40 across the whole set. The rest is the two questions and their criteria, which every call pays for again, so the criteria text is what to shorten once volume grows. Google says its own API bills the shared input once per request no matter how many questions it carries, and AIVAX bills the input tokens the model reports in usage.input_tokens.
I checked this on Gemini Decision 1 with three of the tickets, sent as the department question alone, the is_urgent question alone and both together. The first ticket counted 193, 161 and 294 input tokens, so one request with both questions cost $0.0000300 where two requests cost $0.0000361. Grouping saved 17%, 17% and 16% on the three tickets, so I send both questions in one request.
Median latency fell within 50 ms across the four, and each model had runs above 700 ms, so latency did not separate them.
All 480 calls together cost under two cents. The rate limit was the one operational limit I hit. I started the repeat pass right after the first, with four requests in flight, and 117 calls came back 429. The decisions quota is one per-minute counter per account, shared by all models and keys, and every request uses one unit however many questions it carries. On Max it is 200 requests a minute (10 on Free, 30 on Pro, no plan limit on Ultimate). One pass over the set is 160 calls, so the end of the first pass and the start of the second landed in the same minute and went over. For the reversed-order pass I dropped to two requests at a time. A backlog loop should stay under the plan's per-minute limit (200 on Max, so about three requests a second sustained) and retry a 429 with backoff, because an immediate retry meets the same window. One Luna call also returned a 503, which the decisions page puts down to temporary provider unavailability, and it succeeded on the first retry, so the loop should retry that code a bounded number of times as well.
What I would pick, and for what
For a routing step whose options will be edited by more than one person, I would start with Gemini Decision 1. It was the only model indifferent to option order, and its confidence values were usable on both question types. It is also the only one of the four that accepts audio, video and PDFs in state, which matters for a support queue that receives voicemail and attached invoices. For high-volume text classification where cost dominates, Decider v1.2 matched it on accuracy at a sixth of the price and kept its choices when the order changed. I would keep Jev where its thresholds are already tuned, with the option order fixed in code. I would let Luna route departments but not decide urgency without a review band.
Perplexity released Decider v1.3 on November 13, and its launch post reports 93.98% agreement with LLM judges on a million production requests, against 93.68% for v1.2 and 93.35% for Jev 1.14. The AIVAX catalog lists Decider up to v1.2, and @perplexity/pplx-decider:latest resolves to v1.2, so I did not test v1.3. The gap Perplexity reports between v1.2 and Jev is smaller than anything forty tickets could detect.
The thresholds I quoted belong to these exact versions. When v1.3 or a Jev 1.15 reaches the catalog, an alias such as @typesafe/jev:latest will move to it and the urgency values will move with it, which is why I pin canonical names in routing code, as the post on pinning model IDs argues. A labeled set like this one is cheap enough to keep next to the routing code and rerun on every model, option-order or criteria change: all three passes here cost less than two cents. The Complexity Router uses the same classify-then-branch shape to pick a model per request, and it is the kind of step where an order-sensitive classifier would quietly change which model handles a request.
Measured on November 16, 2026 through the live AIVAX API, with usage.cost on a Max account; the grouping check was run and the catalog read on November 17. The tickets and labels are mine, not a public benchmark, and one person labeled them. I did not call the vendors' APIs directly, so results there may differ.