Batch
Bulk inference: many requests submitted as one job, processed in the background and collected when they finish.
What batch is for
Batch is for work with no one waiting on the answer. Instead of sending requests one at a time and holding a connection open for each, you hand over a file of requests and collect the results when the job completes.
It suits transcription of a back catalogue of recordings, classification or extraction over a dataset, bulk translation, and transformation jobs where the same instruction is applied to thousands of records. Audio can be submitted this way alongside text, so transcription archives do not need a different mechanism from everything else.
How it works
The shape follows the OpenAI Batch API, which is what the providers offering batch on this platform implement:
- Upload a JSONL file in which each line is one request: the target model and its parameters.
- Create a batch job referencing that file. Each line is processed as an individual request.
- Poll the job until its status is completed, then retrieve the output.
Results come back as two files: one carrying the responses, one carrying the failures, so a single bad line does not have to be untangled from the rest of the job. Where the provider writes these to object storage, the output is named after the input file.
Turnaround
A completion window is set when the job is created. One provider offers a choice of 24, 48 or 72 hours, with 48 hours as the default.
Billing
Batch is charged at the provider's batch rate, which is lower than the same model's synchronous rate. The discounts are the provider's, not ours, and they differ between providers; Model Pond's 2.5% is applied on top of whichever rate applies to the job.
Batch rates are a percentage reduction on the model's listed rate, applied by the provider at billing time.
Provider support
| Provider | Batch support | Documentation |
|---|---|---|
| Scaleway | Yes — JSONL input against Object Storage | Batch processing |
| OVHcloud | Yes — OpenAI-compatible /v1/batches |
Batch mode |
What is recorded
Batch jobs are metered the same way as synchronous requests, and the same narrow record applies: time taken, success and failure counts, input tokens, cached tokens and output tokens, attributed to the API key and target model and recorded against a billing period rather than with a timestamp. Prompt and response content is not recorded. See Gateway for the full list.
What to read next
The parts of the platform people usually read after this page.