How to run thousands of LLM requests in batch with rate limits and retries
Use bounded concurrency, balance checks, targeted retries, and resumable jobs to process thousands of LLM requests without replaying completed work.
To run thousands of LLM requests in batch, bound concurrency, check quota and balance before processing, and keep a durable outcome for each item. Resume pending work after rate limits clear, retry only the relevant failures, and pause when repeated errors suggest a shared configuration problem. In AIVAX Batch, these are admission-control decisions: a queue records what you want to run; the worker decides what can run next.
How should you prepare a large batch job?
AIVAX Batch separates a reusable workflow from a job and its items. The workflow holds the instruction, model, required result schema, optional validation instruction, and retry policy. A job groups one workload; each item records its own state, output, confidence, validation result, and cost.
Start with a small representative pilot, not the full list. Include inputs likely to fail: missing fields, ambiguous records, and text that should be refused. Inspect the outputs and costs before scaling. A retry policy cannot repair a consistently wrong instruction.
Jobs are created paused. Import and inspect the items, then start deliberately. Include a stable source identifier in each input so exported results can be reconciled with your database. Keep the job and item identifiers rather than reimporting the entire list whenever processing stops.
Check Plans and limits and Pricing before the full run. Batch has an account-level item-processing quota; model and provider limits still matter. Budget for generation, optional model-based validation, and additional attempts, not just the number of imported rows. This is a managed execution workflow, not a promise of discounted provider batch pricing.
How does admission control limit work in flight?
The current worker uses a channel buffer with 4,096 slots, which makes producers wait when that buffer is full. It schedules up to four concurrent jobs, with up to 16 items in flight per job, per worker process. These are implementation details, not a throughput guarantee or a limit on how many records a job can contain. The channel drains into per-job queues, so its capacity is not a bound on the entire waiting workload.
Scheduling prefers higher-tier work, then accounts without an already active job, then queue arrival order. This gives other accounts a preference within the same tier; it is not a guarantee against starvation or a completion-time SLA.
Before model processing, the worker checks operating balance and the batch rate limiter. Bounded concurrency limits simultaneous work, but it does not make every request admissible: daily quota can still run out, and a provider can still reject or overload.
What happens after a 429, an empty balance, or repeated errors?
The worker handles these conditions differently:
- Insufficient balance: a failed operating-balance check or a
402pauses the job with an insufficient-funds event. The affected item remains pending. This prevents further admission; it does not refund work already performed or reserve the whole job's budget. - Rate limits and temporary provider failures: a
429, retryable inference exception, or provider-overload exception leaves the affected item pending. The worker counts these events and pauses the job at a threshold of three transient failures. That counter is separate from ordinary item errors and resets when the job is reactivated; it is not a consecutive-error counter. - Repeated item errors: completed processing flows that return
ExecutionErrororValidationErrorcount towarderrorStopThreshold. Its default is 5, with a supported range of 1–100. A different returned outcome resets that consecutive-error count.
A pause stops new admissions, not work already in flight. With concurrent items, more attempts may complete after a threshold is reached; the threshold is not an exact spending cap.
For a quota-related pause, wait for the applicable window to reset or resolve the quota constraint before resuming. For provider failures, inspect the job event and provider availability. Do not treat the worker's transient-failure handling as a documented exponential-backoff guarantee, and do not repeatedly restart a job while the same limit remains active.
Which failures should you retry?
There are two separate controls: retries inside one item's processing flow, and operator-triggered retries of selected stored outcomes.
Within the flow, maxRetries defaults to 2 and accepts 0–10. It bounds extra attempts after eligible execution or validation failures, not the total number of times an operator can requeue that item. A required result schema is supplied to structured generation; the Batch flow checks the response envelope and can run an optional validation instruction. Validation feedback and the previous completion are passed into regeneration when available. A refusal returns Refused rather than automatically entering that correction loop.
Syntax repair and correctness are different questions. The structured-output healing boundary explains why parseable JSON is not sufficient evidence that a result is correct. Use validation rules that test what your downstream system actually needs.
After inspecting stored outcomes, choose the narrowest retry mode:
execution-error: retry items whose execution failed.validation-error: retry items that failed validation, ideally after understanding the pattern.errors: retry execution errors, validation errors, and refusals. Review refusals before selecting this broader mode.low-confidence: retry successfully finished items with confidence below 0.3. Confidence is model-reported, not a calibrated probability of correctness.
These modes select non-running items, reset them to pending, and clear their previous output and validation details. Export evidence before retrying if you need to retain it. When at least one item is selected, the retry endpoint also starts a job that was not active; it is not a staging-only operation.
How do you export partial results without losing track of unfinished work?
The export endpoint streams JSONL with each item's ID, metadata, input, and output. Filter by state and confidence to separate successful results from errors or review candidates. Pending items are excluded; export is not limited to successful results unless you request that filter.
An export with no pending rows does not prove the job is complete. Compare the job state and item counts with the source manifest. Preserve source identifiers in the input, ingest successful outputs idempotently, and track unfinished records separately. The same idempotency principles used for LLM tool calls apply to downstream writes: replaying a delivery must not create another business operation.
Frequently asked questions
Should I use a provider Batch API or a managed job?
Use a provider's native batch offering when its supported models, execution window, and result contract fit your pipeline. Use a managed job when you need workflow-level validation, per-item outcomes, selective retries, and pause/resume controls together. AIVAX Batch provides the latter; do not assume it inherits a provider's native batch discount or completion guarantee.
What should I do when my batch hits 429 rate limits?
Read the job event, identify the applicable quota or inference limit, and wait until it clears before resuming pending work. AIVAX pauses after its transient-failure threshold; increasing maxRetries is not the remedy for an active quota window. If your own API client receives a 429, pace requests and use bounded backoff rather than an immediate retry loop.
Can I resume without paying for completed items again?
Resume the existing job instead of reimporting the same input list. Normal scheduling selects pending, non-running items from active jobs, so already finished items are not selected again. This is not an exactly-once billing guarantee: new attempts can incur usage, and an interrupted attempt may have performed billable work before its final state was saved.
How do I reprocess only failures?
Use execution-error or validation-error after reviewing the cause. The broader errors mode includes refusals; low-confidence deliberately reprocesses some successful items. Export the existing evidence first, because retry clears selected outputs and can reactivate the job immediately.
When should I pause or finish a job?
Pause when errors share a cause, quota is unavailable, balance is insufficient, or results require investigation. Correct the condition before resuming, and allow for in-flight work to complete. Finish a job when you intend to terminate it rather than continue its pending workload. For new workflows, use a low error threshold and a small pilot before admitting the full list.