Batch and fine-tuning API

# Evals

Grade a model's answers on your own test set with OpenAI's Evals API. Each run's model calls are ordinary batches, billed at batch prices.

## Overview

1. Create an eval: how each row is graded (its testing criteria) and the shape of your rows.
2. Start a run on a model, a catalog model or one of your `ft:` models, over a file of rows. Gridline sends the rows to the model as a batch, then grades each answer.
3. Read the run's pass and fail counts, then each graded row.

The routes and objects are OpenAI's, under Gridline's base URL, with a Gridline token as the API key (see the [quickstart](https://gridlinegpu.com/docs#access) for the token). Code written for OpenAI's `client.evals` runs with the base URL changed, within the [differences](https://gridlinegpu.com/docs/evals#differences) below.

```
https://api.gridlinegpu.com/openai/v1
```

| Method | Path | What it does |
| --- | --- | --- |
| `POST` | `/evals` | Create an eval: its testing criteria and the shape of its items. |
| `GET` | `/evals` | List your evals. |
| `GET` | `/evals/{eval_id}` | Retrieve an eval. |
| `POST` | `/evals/{eval_id}` | Update an eval's name or metadata. |
| `DELETE` | `/evals/{eval_id}` | Delete an eval and its ended runs. Refused while a run is open. |
| `POST` | `/evals/{eval_id}/runs` | Start a run, or quote one with dry_run. |
| `GET` | `/evals/{eval_id}/runs` | List an eval's runs. |
| `GET` | `/evals/{eval_id}/runs/{run_id}` | Retrieve a run: its status, counts and usage. |
| `POST` | `/evals/{eval_id}/runs/{run_id}` | Cancel a run. Its batches are cancelled. |
| `DELETE` | `/evals/{eval_id}/runs/{run_id}` | Delete a run and its results file. |
| `GET` | `/evals/{eval_id}/runs/{run_id}/output_items` | A run's graded rows. |
| `GET` | `/evals/{eval_id}/runs/{run_id}/output_items/{output_item_id}` | One graded row. |

## Create an eval

An eval holds up to 8 testing criteria. Templates read the row with `{{item.<field>}}` and the model's answer with `{{sample.output_text}}`. With the OpenAI Python SDK:

```
import os
from openai import OpenAI

client = OpenAI(base_url="https://api.gridlinegpu.com/openai/v1", api_key=os.environ["GRIDLINE_TOKEN"])

support_eval = client.evals.create(
    name="support-answers",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {"question": {"type": "string"}, "ideal": {"type": "string"}},
        },
        "include_sample_schema": True,
    },
    testing_criteria=[
        {
            "type": "string_check",
            "name": "exact",
            "input": "{{sample.output_text}}",
            "reference": "{{item.ideal}}",
            "operation": "eq",
        },
        {
            "type": "text_similarity",
            "name": "close",
            "input": "{{sample.output_text}}",
            "reference": "{{item.ideal}}",
            "evaluation_metric": "fuzzy_match",
            "pass_threshold": 0.8,
        },
        {
            "type": "score_model",
            "name": "judge",
            "model": "qwen2.5-7b-instruct",
            "input": [
                {"role": "system", "content": "Score from 0 to 1 how well the answer matches the ideal answer. Reply with the number only."},
                {"role": "user", "content": "Question: {{item.question}}\nIdeal: {{item.ideal}}\nAnswer: {{sample.output_text}}"},
            ],
            "range": [0, 1],
            "pass_threshold": 0.7,
        },
    ],
)
```

A row passes when every criterion passes. Update an eval's `name` or `metadata` with `POST /evals/{eval_id}`; its criteria are fixed.

## Items

One JSON object per line, each with an `item`. Upload the file with `purpose="batch"`, or send up to 500 rows inline as `file_content`:

```
{"item": {"question": "How do I reset my password?", "ideal": "Use the Forgot password link on the sign-in page."}}
{"item": {"question": "Can I change my plan mid-month?", "ideal": "Yes. The change applies from the next billing day."}}
```

```
items = client.files.create(file=open("support-items.jsonl", "rb"), purpose="batch")
```

A run grades the first 500 usable rows. A row that is not valid JSON, has no `item`, or makes a request the batch API would refuse is skipped and counted. To grade answers you already have, give each row a `sample` (`{"output_text": "…"}`) and run with a `jsonl` data source: nothing is sent to a model.

## Run it

A `completions` run renders each row into chat messages, sends them to the model as one batch, then grades the answers. Ask for the quote first with `dry_run` (a Gridline field: it answers the run's maximum charge and writes nothing):

```
run_request = {
    "name": "base model",
    "data_source": {
        "type": "completions",
        "model": "qwen2.5-7b-instruct",
        "input_messages": {
            "type": "template",
            "template": [
                {"role": "system", "content": "Answer the customer in one sentence."},
                {"role": "user", "content": "{{item.question}}"},
            ],
        },
        "sampling_params": {"temperature": 0, "max_completion_tokens": 128},
        "source": {"type": "file_id", "id": items.id},
    },
}

quote = client.post(
    f"/evals/{support_eval.id}/runs",
    body={**run_request, "dry_run": True},
    cast_to=object,
)
print(quote["quote"]["max_charge_usd"])

run = client.evals.runs.create(support_eval.id, **run_request)
```

Poll the run until its `status` is `completed`, `failed` or `canceled`. It carries OpenAI's `result_counts`, `per_testing_criteria_results` and `per_model_usage`, and a `gridline` object with its batch ids, its results file and its quote:

```
import time

while run.status in ("queued", "in_progress"):
    time.sleep(30)
    run = client.evals.runs.retrieve(run.id, eval_id=support_eval.id)

print(run.result_counts)  # passed, failed, errored, total
for item in client.evals.runs.output_items.list(run.id, eval_id=support_eval.id, status="fail"):
    print(item.datasource_item["question"], item.results)
```

Cancel a run with `POST /evals/{eval_id}/runs/{run_id}` (`client.evals.runs.cancel`); its batches are cancelled. Output items are kept 30 days after the run ends.

## Evaluate a fine-tune

Run the same eval twice over the same items: once on your `ft:` model and once on its base model. Compare the two runs' `result_counts` and `per_testing_criteria_results`.

```
for model in ["ft:qwen2.5-7b-instruct:o3f9a1c2d4e5:7b2c9e41a0d3", "qwen2.5-7b-instruct"]:
    request = {**run_request, "name": model}
    request["data_source"] = {**run_request["data_source"], "model": model}
    client.evals.runs.create(support_eval.id, **request)
```

Use your model's own id from `GET /models` or the succeeded job's `fine_tuned_model`. Hold out rows you did not train on, or the comparison measures memory, not quality.

## Graders

### string_check

Compares the rendered input with the rendered reference. Passes or fails (1 or 0).

| Operation | Passes when |
| --- | --- |
| `eq` | The input equals the reference. |
| `ne` | The input differs from the reference. |
| `like` | The input contains the reference. |
| `ilike` | The input contains the reference, ignoring case. |

### text_similarity

A score from 0 to 1, passing at or above `pass_threshold`. Metrics: `fuzzy_match`, `bleu`, `rouge_1`, `rouge_2`, `rouge_3`, `rouge_4`, `rouge_5`, `rouge_l`. Not supported yet: `gleu`, `meteor`, `cosine`. `fuzzy_match` and `rouge_l` compare at most the first 1000 characters or words of each side. BLEU and ROUGE split text into lower-cased runs of letters and digits, so scores can differ slightly from OpenAI's on punctuation-heavy text.

### score_model

A catalog chat model reads your grading messages for each row and replies with a number within `range` (default 0 to 1); the row passes at or above `pass_threshold`. A reply that is not a number in range errors the row; it is never guessed. Each eval uses at most 2 distinct score models, and a score model cannot be an `ft:` model. Its reply is at most 512 tokens (default 64).

## What it costs

- A run's model calls are ordinary batches in your account: one for the answers, and one per score model for the grading. Each is billed at the published batch prices on [/pricing](https://gridlinegpu.com/pricing), as any batch, and appears in `GET /batches` with its receipt.
- Each batch costs at least its model's per-batch minimum ($0.20 on qwen2.5-7b-instruct). So a small run on one model costs at least one minimum, and two with a score model.
- A `jsonl` run graded only by `string_check` and `text_similarity` sends nothing to a model and costs nothing.
- The quote (`dry_run`, and the run's `gridline.quote`) is an upper bound: every answer at its full `max_completion_tokens`. You pay what the batches actually use.
- A run is refused before anything is created when your credit is below its quote. A batch that fails or is cancelled ends the run and is billed as any batch with that end.

## Limits

- 500 rows per run; 500 rows inline.
- 8 testing criteria per eval, 2 distinct score models.
- Answers up to 4096 tokens (`max_completion_tokens`, default 1024).
- 100 evals per account, 20 runs kept per eval, and 2 runs open at a time per account.
- Output items are kept 30 days after a run ends; the run itself is deleted 90 days after it ends. An eval stays until you delete it.

## Differences from OpenAI

- Data sources are jsonl and completions. The responses data source is refused.
- data_source_config is custom. The logs and stored_completions types are refused: Gridline does not store completions.
- Graders are string_check, text_similarity and score_model. The python and label_model graders are refused.
- A score_model grader runs on a catalog chat model, not an ft: model.
- Upload items with purpose batch (or use a fine-tune file). Files with purpose evals are not accepted yet.
- item_schema is stored and echoed; items are not checked against it.
- report_url is null: there is no dashboard view of a run yet.

## Errors

Refusals use OpenAI's error body; `error.message` says what to change.

| Status | error.code | When |
| --- | --- | --- |
| 400 | `invalid_request` | A field is missing or out of range. error.param names it. |
| 400 | `invalid_value` | The items file was not uploaded with purpose batch or fine-tune. |
| 404 | `not_found` | No such eval in your account. Another account's ids answer the same. |
| 404 | `not_found` | No such run of this eval. |
| 404 | `not_found` | No such output item. They are kept 30 days after the run ends. |
| 409 | `invalid_state` | Deleting an eval, or a run, while a run is still open. |
| 409 | `invalid_state` | Cancelling a run that has already ended. |
| 429 | `insufficient_quota` | Your credit is below the run's quoted maximum charge. |
| 429 | `rate_limit_exceeded` | Your account holds the most evals allowed. Delete one first. |
| 429 | `rate_limit_exceeded` | Your account has the most runs open. Wait for one to end. |
| 429 | `rate_limit_exceeded` | This eval keeps the most runs allowed. Delete an old run first. |

---

Source: https://gridlinegpu.com/docs/evals