Batch Inference API

# Batch Inference API

Send a JSONL file of requests for open models and get a file of results back within 24 hours, or within 72 hours at the same price for work that can wait. Same wire format as OpenAI's Batch API: change the base URL and the key.

## Overview

The Batch Inference API is a Files and Batches API that follows OpenAI's. The OpenAI Python and Node SDKs work unchanged with Gridline's base URL and a Gridline token as the API key:

```
https://api.gridlinegpu.com/openai/v1
```

1. Upload a JSONL file of requests with `purpose=batch`.
2. Create a batch from it, naming the endpoint and the completion window: `24h`, or `72h` at the same price (see [Completion window](https://gridlinegpu.com/docs/batch-api#completion-window)).
3. Poll the batch until it is `completed`, then download its output file and error file.

Endpoints: `/v1/chat/completions` for the chat models and `/v1/embeddings` for `bge-m3`. Create your token in your [dashboard](https://gridlinegpu.com/dashboard/tokens), or with `gridline setup --apply` (see the [quickstart](https://gridlinegpu.com/docs#access)); it carries the `read` and `routing:write` scopes the API needs. Your account needs credits: add them by card at [Get started](https://gridlinegpu.com/get-started). A missing or revoked token is answered `401 invalid_api_key`, and one without a needed scope `403 insufficient_permissions`.

## Models

Each model is pinned to a full Hugging Face commit and is listed at `GET /openai/v1/models` once it has passed qualification. Use the id in `body.model`; OpenAI model names such as `gpt-*` are refused.

| Model id | Source | Licence | Endpoints |
| --- | --- | --- | --- |
| `qwen2.5-0.5b-instruct` | Qwen/Qwen2.5-0.5B-Instruct, bf16 | Apache-2.0 | chat completions |
| `qwen2.5-1.5b-instruct` | Qwen/Qwen2.5-1.5B-Instruct, bf16 | Apache-2.0 | chat completions |
| `qwen2.5-7b-instruct` | Qwen/Qwen2.5-7B-Instruct, bf16 | Apache-2.0 | chat completions |
| `mistral-7b-instruct-v0.3` | mistralai/Mistral-7B-Instruct-v0.3, bf16 | Apache-2.0 | chat completions |
| `llama-3.1-8b-instruct` | meta-llama/Llama-3.1-8B-Instruct (from RedHatAI/Llama-3.1-8B-Instruct), bf16 | Llama 3.1 Community License | chat completions |
| `bge-m3` | BAAI/bge-m3 | MIT | embeddings |

- `bge-m3` returns dense embeddings only, 1,024 dimensions, for inputs of up to 8,192 tokens; its sparse and multi-vector outputs are not offered.
- The chat models take up to 8,192 tokens of context and write at most 1,024 tokens per request. See [Models](https://gridlinegpu.com/docs/models) for the commits.
- Fine-tunes of `gpt-oss-20b` and `qwen3.5-9b` are download-only for now: their `ft:` models cannot be named in a batch yet.
- Models of 70B parameters and larger are not offered.

## Quickstart

With the `openai` Python package, set `base_url` and pass your Gridline token as the key:

```
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.gridlinegpu.com/openai/v1", api_key=os.environ["GRIDLINE_TOKEN"])

batch_input = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_input.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

batch = client.batches.retrieve(batch.id)  # poll until batch.status == "completed"
if batch.output_file_id:
    open("results.jsonl", "wb").write(client.files.content(batch.output_file_id).content)
```

When you poll, give each request a short timeout, for example `OpenAI(base_url=..., api_key=..., timeout=60)`. By default the `openai` package waits up to 600 s for one answer, so a dropped connection stalls a poll for ten minutes. With 60 s it times out and the package retries it.

The same flow with curl:

```
curl https://api.gridlinegpu.com/openai/v1/files \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -F purpose=batch \
  -F [[email protected]](https://gridlinegpu.com/cdn-cgi/l/email-protection)

curl https://api.gridlinegpu.com/openai/v1/batches \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input_file_id": "file-...", "endpoint": "/v1/embeddings", "completion_window": "24h"}'

curl https://api.gridlinegpu.com/openai/v1/batches/batch_... \
  -H "Authorization: Bearer $GRIDLINE_TOKEN"

curl https://api.gridlinegpu.com/openai/v1/files/file-.../content \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" -o results.jsonl
```

A batch your credit cannot cover fails at validation with `insufficient_quota`, before anything runs. Credits are held when it is validated and settled when it ends. A `72h` batch first waits for a top-up: see [Completion window](https://gridlinegpu.com/docs/batch-api#completion-window).

## Completion window

Set `completion_window` to `24h` or `72h`. Both cost the same: a `72h` batch is charged exactly as a `24h` batch on the same model. Choose `72h` for work that can wait.

- `24h`, the standard window: a batch waits up to 12 hours for your account's GPU capacity to become free, then fails with `capacity_unavailable`. Nothing ran and nothing was charged.
- `72h`: your `24h` batches get free GPU capacity before your `72h` ones, and a `72h` batch waits up to 48 hours to start before it fails with `capacity_unavailable`, with nothing run or charged. A `72h` batch that starts has at least 24 hours left before its deadline.
- If your credit cannot cover the most a `72h` batch can cost, it waits up to the same 48 hours for a top-up instead of failing at once, then fails with `insufficient_quota` if it is still short. A `24h` batch fails at once.
- A batch's `expires_at` is 24 or 72 hours after its `created_at`. A batch still running then ends `expired`: the requests that completed are in the output file and are charged, and the rest are in the error file with `batch_expired`. A `72h` batch whose GPU time runs out before its deadline ends the same way.

With curl, send `"completion_window": "72h"`. The OpenAI Python SDK types `completion_window` as `"24h"` only, so a type checker flags `"72h"`, though the SDK sends it unchanged. For a type-clean call, set it through `extra_body`:

```
batch = client.batches.create(
    input_file_id=batch_input.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    extra_body={"completion_window": "72h"},
)
```

The batch then reports `completion_window` `"72h"`.

## Request and result lines

Each input line is one request. `custom_id` is yours and must be unique in the file; `url` must equal the batch's endpoint. One base model per batch; see [Fine-tuned models](https://gridlinegpu.com/docs/batch-api#fine-tuned) for mixing in your `ft:` models.

```
{"custom_id": "q-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen2.5-7b-instruct", "messages": [{"role": "user", "content": "Summarize this ticket."}], "max_tokens": 256}}
{"custom_id": "e-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "bge-m3", "input": ["first passage", "second passage"]}}
```

The output file has one line per completed request, in any order, with the standard response body including `usage`. Requests that fail individually, such as one over the context length, go to the error file with a code: `invalid_request`, `context_length_exceeded`, `batch_expired`, `batch_cancelled`, `server_error` or `structured_output_invalid` (a structured answer that stopped on its own but is not the JSON its `response_format` asked for; not billed). A file with any invalid line fails the whole batch at validation, as with OpenAI.

Batch statuses: `validating`, `in_progress`, `finalizing`, `completed`, `failed`, `expired`, `cancelling` and `cancelled`. A cancelled batch still delivers, and charges for, the requests that completed. A new batch can stay `validating` while your account's other batches and fine-tuning jobs use its GPU capacity, then `in_progress` with no completed requests for a few minutes while Gridline finds and starts a GPU; at busy times it can take longer. You aren't charged for time spent waiting.

Waiting for a GPU is not charged. Check the job in your dashboard or its API response:` gridline.reason`, when available, explains the wait and what to do. A service health check does not guarantee that a GPU can start.

- Account capacity: wait for another job to end. An accepted job retries automatically; do not resubmit it.
- Customer credit: check your current balance before adding more. If you already added enough, wait for the next check. Resubmit only after a failed job.
- Gridline capacity: wait. Adding credit or submitting duplicates will not help.
- Platform daily limit: wait for capacity; the budget resets at 00:00 UTC.

New self-serve accounts normally allow 1 open fine-tune, 2 open batches and $100 per UTC day. Your account may have different limits. An account-limit check returns the effective limit and current use in `error.limit`. Other refusals, including a concurrent create that fills the last slot, may only have an error code and message. A refused job create does not create or charge a job. For an open-job limit, retry after a job ends. Daily limits reset at 00:00 UTC; a deposit does not raise them. Contact [[email protected]](https://gridlinegpu.com/cdn-cgi/l/email-protection) with your job ID to ask for help or a higher limit.

## Fine-tuned models

Set `body.model` to an `ft:` model from one of your [fine-tuning jobs](https://gridlinegpu.com/docs/fine-tuning):

```
{"custom_id": "t-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "ft:qwen2.5-7b-instruct:o3f9a1c2d4e5:7b2c9e41a0d3", "messages": [{"role": "user", "content": "Classify: the parcel never came."}], "max_tokens": 16}}
```

- The model must be yours. An `ft:` id that does not exist, belongs to another account or is no longer active is answered `model_not_found`.
- One base model per batch. A batch may mix that base model with up to 8 different `ft:` models fine-tuned from it, within what the base has been proved to serve: on qwen2.5-0.5b-instruct, qwen2.5-1.5b-instruct, 8 of any rank; on qwen2.5-7b-instruct, mistral-7b-instruct-v0.3, llama-3.1-8b-instruct, 4 adapters of rank 64 or 8 of rank 32 or lower; on qwen3-8b, 4 of any rank. Together their files stage at most 1,008 MiB. A batch over either limit fails validation with `model_not_found` and a message saying which.
- The input file and every `ft:` model in it must belong to the same account.
- A QLoRA adapter is trained against a 4-bit quantized base and served on the bf16 base, so its outputs can differ slightly from what training measured.

## Structured outputs

`/v1/chat/completions` lines accept `response_format` of type `json_object` or `json_schema`.

- Set `max_tokens` on structured requests.
- A structured answer that stops at `max_tokens` (`finish_reason: "length"`) is returned as it is and billed for the tokens generated, as with OpenAI. Its JSON may be truncated or invalid.
- A schema that allows unbounded recursion, such as an optional property that references its own definition (a linked list's `next` or a tree's `children`), can make the model keep generating until `max_tokens`. Bound it in the prompt (for example “exactly two items”) or in the schema (`maxItems`), and set `max_tokens`.

## Limits and retention

- Completion window: 24 or 72 hours (see [Completion window](https://gridlinegpu.com/docs/batch-api#completion-window)).
- Open batches: a new account can have 2 at once. Another create answers `429 limit_exceeded`, with your limit in `error.limit`, until one ends.
- Each input file: at most 200 MB and 50,000 requests.
- Per account: at most 500 files and 20 GB stored at a time. Delete files you no longer need to make room.
- Input files are kept for at least 7 days after upload, and while a batch uses them, for 7 days past that batch's completion window (24 or 72 hours). Output and error files are kept for 30 days after they are written.
- Chat and completions: `n` up to 8, `logprobs` and `top_logprobs` up to 5; `seed` is best-effort and `user` is ignored.

## Not supported

These are refused with `400 invalid_request_error` at validation, never silently ignored: `/v1/responses`, moderations, images and audio endpoints; completion windows other than `24h` and `72h`; `stream`, tools and `tool_choice`, any `guided_*` parameter, `logit_bias`, `service_tier`, `store`, `prediction`, and image, audio or file content parts; several base models in one batch; and more than 8 fine-tuned models in one batch.

## Privacy

Your requests and results are stored under your account and are never logged. Batches run on datacenter hosts only.

## Pricing

Fine-tuning and batch jobs are paid for from your prepaid credits. A batch is charged for the input and output tokens of the results it delivers, at its model's rates (a fine-tuned model's are its base's); embeddings are charged for their input tokens only. Credits are held when a batch is validated and settled when it ends. A 72h batch is charged exactly as a 24h batch on the same model.

What a credit is worth, the credit packs and each model's rates are on the [pricing page](https://gridlinegpu.com/pricing).

---

Source: https://gridlinegpu.com/docs/batch-api