Batch Inference API

Overview

The Batch Inference API is a Files and Batches API that follows OpenAI's. The OpenAI Python and Node SDKs work unchanged with Gridline's base URL and a Gridline token as the API key:

https://api.gridlinegpu.com/openai/v1

1. Upload a JSONL file of requests with purpose=batch.
2. Create a batch from it, naming the endpoint and the completion window: 24h, or 72h at the same price (see Completion window).
3. Poll the batch until it is completed, then download its output file and error file.

Endpoints: /v1/chat/completions for the chat models and /v1/embeddings for bge-m3. Create your token in your dashboard, or with gridline setup --apply (see the quickstart); it carries the read and routing:write scopes the API needs. Your account needs credits: add them by card at Get started. A missing or revoked token is answered 401 invalid_api_key, and one without a needed scope 403 insufficient_permissions.

Models

Each model is pinned to a full Hugging Face commit and is listed at GET /openai/v1/models once it has passed qualification. Use the id in body.model; OpenAI model names such as gpt-* are refused.

• bge-m3 returns dense embeddings only, 1,024 dimensions, for inputs of up to 8,192 tokens; its sparse and multi-vector outputs are not offered.
• The chat models take up to 8,192 tokens of context and write at most 1,024 tokens per request. See Models for the commits.
• Fine-tunes of gpt-oss-20b and qwen3.5-9b are download-only for now: their ft: models cannot be named in a batch yet.
• Models of 70B parameters and larger are not offered.

Quickstart

With the openai Python package, set base_url and pass your Gridline token as the key:

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.gridlinegpu.com/openai/v1", api_key=os.environ["GRIDLINE_TOKEN"])

batch_input = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_input.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

batch = client.batches.retrieve(batch.id)  # poll until batch.status == "completed"
if batch.output_file_id:
    open("results.jsonl", "wb").write(client.files.content(batch.output_file_id).content)

When you poll, give each request a short timeout, for example OpenAI(base_url=..., api_key=..., timeout=60). By default the openai package waits up to 600 s for one answer, so a dropped connection stalls a poll for ten minutes. With 60 s it times out and the package retries it.

The same flow with curl:

curl https://api.gridlinegpu.com/openai/v1/files \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -F purpose=batch \
  -F [email protected]

curl https://api.gridlinegpu.com/openai/v1/batches \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input_file_id": "file-...", "endpoint": "/v1/embeddings", "completion_window": "24h"}'

curl https://api.gridlinegpu.com/openai/v1/batches/batch_... \
  -H "Authorization: Bearer $GRIDLINE_TOKEN"

curl https://api.gridlinegpu.com/openai/v1/files/file-.../content \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" -o results.jsonl

A batch your credit cannot cover fails at validation with insufficient_quota, before anything runs. Credits are held when it is validated and settled when it ends. A 72h batch first waits for a top-up: see Completion window.

Completion window

Set completion_window to 24h or 72h. Both cost the same: a 72h batch is charged exactly as a 24h batch on the same model. Choose 72h for work that can wait.

• 24h, the standard window: a batch waits up to 12 hours for your account's GPU capacity to become free, then fails with capacity_unavailable. Nothing ran and nothing was charged.
• 72h: your 24h batches get free GPU capacity before your 72h ones, and a 72h batch waits up to 48 hours to start before it fails with capacity_unavailable, with nothing run or charged. A 72h batch that starts has at least 24 hours left before its deadline.
• If your credit cannot cover the most a 72h batch can cost, it waits up to the same 48 hours for a top-up instead of failing at once, then fails with insufficient_quota if it is still short. A 24h batch fails at once.
• A batch's expires_at is 24 or 72 hours after its created_at. A batch still running then ends expired: the requests that completed are in the output file and are charged, and the rest are in the error file with batch_expired. A 72h batch whose GPU time runs out before its deadline ends the same way.

With curl, send "completion_window": "72h". The OpenAI Python SDK types completion_window as "24h" only, so a type checker flags "72h", though the SDK sends it unchanged. For a type-clean call, set it through extra_body:

batch = client.batches.create(
    input_file_id=batch_input.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
    extra_body={"completion_window": "72h"},
)

The batch then reports completion_window "72h".

Request and result lines

Each input line is one request. custom_id is yours and must be unique in the file; url must equal the batch's endpoint. One base model per batch; see Fine-tuned models for mixing in your ft: models.

{"custom_id": "q-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen2.5-7b-instruct", "messages": [{"role": "user", "content": "Summarize this ticket."}], "max_tokens": 256}}
{"custom_id": "e-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "bge-m3", "input": ["first passage", "second passage"]}}

The output file has one line per completed request, in any order, with the standard response body including usage. Requests that fail individually, such as one over the context length, go to the error file with a code: invalid_request, context_length_exceeded, batch_expired, batch_cancelled, server_error or structured_output_invalid (a structured answer that stopped on its own but is not the JSON its response_format asked for; not billed). A file with any invalid line fails the whole batch at validation, as with OpenAI.

Batch statuses: validating, in_progress, finalizing, completed, failed, expired, cancelling and cancelled. A cancelled batch still delivers, and charges for, the requests that completed. A new batch can stay validating while your account's other batches and fine-tuning jobs use its GPU capacity, then in_progress with no completed requests for a few minutes while Gridline finds and starts a GPU; at busy times it can take longer. You aren't charged for time spent waiting.

Fine-tuned models

Set body.model to an ft: model from one of your fine-tuning jobs:

{"custom_id": "t-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "ft:qwen2.5-7b-instruct:o3f9a1c2d4e5:7b2c9e41a0d3", "messages": [{"role": "user", "content": "Classify: the parcel never came."}], "max_tokens": 16}}

• The model must be yours. An ft: id that does not exist, belongs to another account or is no longer active is answered model_not_found.
• One base model per batch. A batch may mix that base model with up to 8 different ft: models fine-tuned from it, within what the base has been proved to serve: on qwen2.5-0.5b-instruct, qwen2.5-1.5b-instruct, 8 of any rank; on qwen2.5-7b-instruct, mistral-7b-instruct-v0.3, llama-3.1-8b-instruct, 4 adapters of rank 64 or 8 of rank 32 or lower; on qwen3-8b, 4 of any rank. Together their files stage at most 1,008 MiB. A batch over either limit fails validation with model_not_found and a message saying which.
• The input file and every ft: model in it must belong to the same account.
• A QLoRA adapter is trained against a 4-bit quantized base and served on the bf16 base, so its outputs can differ slightly from what training measured.

Structured outputs

/v1/chat/completions lines accept response_format of type json_object or json_schema.

• Set max_tokens on structured requests.
• A structured answer that stops at max_tokens (finish_reason: "length") is returned as it is and billed for the tokens generated, as with OpenAI. Its JSON may be truncated or invalid.
• A schema that allows unbounded recursion, such as an optional property that references its own definition (a linked list's next or a tree's children), can make the model keep generating until max_tokens. Bound it in the prompt (for example “exactly two items”) or in the schema (maxItems), and set max_tokens.

Limits and retention

• Completion window: 24 or 72 hours (see Completion window).
• Open batches: a new account can have 2 at once. Another create answers 429 limit_exceeded, with your limit in error.limit, until one ends.
• Each input file: at most 200 MB and 50,000 requests.
• Per account: at most 500 files and 20 GB stored at a time. Delete files you no longer need to make room.
• Input files are kept for at least 7 days after upload, and while a batch uses them, for 7 days past that batch's completion window (24 or 72 hours). Output and error files are kept for 30 days after they are written.
• Chat and completions: n up to 8, logprobs and top_logprobs up to 5; seed is best-effort and user is ignored.

Not supported

These are refused with 400 invalid_request_error at validation, never silently ignored: /v1/responses, moderations, images and audio endpoints; completion windows other than 24h and 72h; stream, tools and tool_choice, any guided_* parameter, logit_bias, service_tier, store, prediction, and image, audio or file content parts; several base models in one batch; and more than 8 fine-tuned models in one batch.

Privacy

Your requests and results are stored under your account and are never logged. Batches run on datacenter hosts only.

Pricing

Fine-tuning and batch jobs are paid for from your prepaid credits. A batch is charged for the input and output tokens of the results it delivers, at its model's rates (a fine-tuned model's are its base's); embeddings are charged for their input tokens only. Credits are held when a batch is validated and settled when it ends. A 72h batch is charged exactly as a 24h batch on the same model.

What a credit is worth, the credit packs and each model's rates are on the pricing page.