Batch is an admission-control problem, not a queue

Bulk AI work looks like a queue problem. Collect the records, line them up, process each one, export the results. The queue is the easy part. Batch incidents rarely start in the queue itself; they start at the boundary the queue hides: the point where queued work meets a provider with rate limits, a wallet with a balance, a schema the output must satisfy, and an error distribution that is never uniform.

A batch system earns its keep by deciding what gets admitted, in what order, under what budget — and what stops the line when the failures share a cause. The queue stores intent. Admission control protects the run.

The queue is honest storage; the boundary is where runs die

Consider the failure that a plain queue cannot express. A batch job kicks off with ten thousand items, hammers the inference API with unbounded concurrency, and hits a rate limit halfway through. Some items succeeded, some failed transiently, some were never attempted. The queue says "items remaining." It does not say which failures deserve a retry, which ones signal a broken workflow configuration, or whether the account still has balance to finish.

The enterprise rate-limiting analysis describes exactly this discovery pattern: the batch process either crashes, produces partial results nobody knows are partial, or creates a thundering herd that degrades unrelated traffic. The post-mortem always prescribes rate limiting — but the deeper lesson is that LLM APIs limit on several dimensions at once (requests, tokens, concurrency, daily budget), and a queue that admits work without consulting any of them will discover each limit as an incident.

Research sharpens the point for agentic workloads. CONCUR (Chen et al., arXiv:2601.22705) identifies "middle-phase thrashing" in batch inference: long-lived agents accumulate state, KV-cache efficiency collapses well before memory is exhausted, and throughput degrades severely — recomputation consuming roughly half of end-to-end latency in their measurements. Their prescription is agent-level admission control driven by runtime cache signals, reporting up to 4.09x throughput on Qwen3-32B and 1.9x on DeepSeek-V3 across their evaluated workloads. The mechanism differs from an API gateway's problem, but the thesis is the same:

Five admission decisions every batch run needs

A batch design that survives contact with production answers five questions before the first item is processed:

• How much enters at once? Bounded parallelism with a waiting queue, so a burst of ten thousand items becomes a steady flow the provider can absorb. Unbounded fan-out turns every transient slowdown into a pile-up.
• Whose work goes first? When jobs share capacity, scheduling needs a policy: tier priority, fairness across accounts so one tenant's backlog does not starve another, and FIFO within a tier so ordering stays predictable.
• What stops the line? A handful of consecutive failures usually means the workflow is misconfigured — wrong instruction, wrong schema, dead model — not that a thousand items are individually unlucky. Continuing to spend on a broken configuration is the most expensive failure mode a batch operator controls.
• Which failures deserve another attempt? A rate-limit rejection, a provider overload, and a schema violation are three different events. Retrying them identically wastes money on the ones that can never succeed and starves the ones that would.
• Who pays, and is there budget left? Every item consumes tokens against an account balance. A run that cannot check the wallet before admitting work will die mid-queue with partial results and no clean resume point.

How AIVAX Batch answers: bounded admission, validated items, shaped retries

AIVAX Batch structures bulk work as three layers: a reusable workflow (instruction, result schema, model, validation, retry policy), a job (one execution queue under a workflow), and items (individual inputs with their own output, validation, state, confidence, and cost). Each layer exists so that one of the admission decisions has an explicit owner.

Admission starts before any token is spent. Jobs are created paused: import the items, inspect the list, then start deliberately. The processing queue behind them is bounded — 4,096 slots with backpressure on producers, at most 4 concurrent jobs, 16 items per job in flight — and scheduling prefers higher-tier work first, then jobs from accounts with nothing already running, then FIFO. That ordering is a fairness policy written into the scheduler, not documentation advice: one account's backlog cannot permanently crowd out the rest.

The spending gates sit on the admission path, not beside it. Before an item is admitted, the worker checks the account's operating balance and the batch rate limiter; a 402 pauses the job with an insufficient-funds marker instead of burning partial credit, and 429s or provider-overload signals accumulate toward a transient-failure pause rather than hammering a struggling provider. Consecutive errors — execution or validation — count against the workflow's errorStopThreshold (default 5, configurable 1–100) and pause the job when crossed. A broken instruction stops the line after a handful of items instead of after a thousand.

Each admitted item then runs a generate–validate–retry loop with a fixed output contract. The workflow's JSON schema is mandatory, a second validation instruction is optional, and failures carry their shape: Pending, Finished, Refused, ExecutionError, ValidationError, Cancelled. Retries are addressed to the failure shape — errors, execution-error, validation-error, low-confidence — with maxRetries (default 2, up to 10) bounding each attempt, and validation feedback fed back into the next generation so a regenerated output has a reason to differ from the one that failed.

Export closes the loop back into the caller's systems: finished results stream as JSONL, filterable by state and confidence, with pending items excluded so a partial run cannot masquerade as a complete one. Stable record identifiers in the input join exported outputs back to the source system.

Operate the run, not just the queue

The documented best practices read as admission discipline: pilot 5–20 representative items before the full workload, set a low errorStopThreshold on new workflows, retry the narrowest failure mode the evidence supports, retry low-confidence items separately from errors, and export finished before errors when humans review in stages. Each rule exists because someone paid to learn it — usually by spending a full run's budget on a configuration a pilot would have caught.

Batch pricing reinforces the same point from the provider side. Hosted batch APIs commonly discount bulk work up to 50% precisely because deferrable, well-shaped demand is cheaper to serve than spiky realtime traffic. The discount is the market telling you what the infrastructure already knows: a batch that arrives metered, budgeted, and resumable costs less to run than the same tokens fired blindly. Designing for admission is not overhead on top of bulk work. It is what makes bulk work cheap.

The queue was never the hard part. Storing ten thousand intents is trivial. Deciding which ten enter next, under whose budget, with what evidence that the run is still healthy — that is the system. Build the gates first, then fill the queue.