Cookbook

# Evaluate a fine-tune

Grade a fine-tuned model and its base on the same held-out rows with the Evals API.

[All cookbooks](https://gridlinegpu.com/docs/cookbooks)

- Evals
- Batch inference

## What you'll build

Compare your fine-tune with its base and inspect the tickets it missed.

The script creates an eval that passes a row when the model's answer is exactly the expected queue, runs it on your fine-tuned model and on its base, and prints how many rows each passed and which ones the fine-tuned model missed.

Each run's model calls are one batch, billed as any batch.

### How it works

1. Held-out tickets
2. Base and fine-tune runs
3. Passes and misses

Held-out tickets and expected queues feed eval runs on the base and fine-tuned model. An exact-match grader checks answers; the script prints pass counts and lists missed tickets.

## What you need

Comfort with

Python and terminal basics; classifier recipe first

Models

`qwen3.5-9b` (see [Models](https://gridlinegpu.com/docs/models))

Jobs and cost

It creates two eval runs, each of them one batch. Fine-tuning and batch jobs are paid for from your prepaid credits.

Time

The script waits for its jobs to finish. A time estimate is not available.

You need a Gridline token and credits: the [quickstart](https://gridlinegpu.com/docs#access) shows how to get both. Run [Fine-tune a classifier](https://gridlinegpu.com/docs/cookbooks/fine-tune-a-classifier) first, in the same folder: it writes `test.jsonl` and prints your model. Set `FT_MODEL` to that model's id.

What a credit is worth, the credit packs and each model's rates are on the [pricing page](https://gridlinegpu.com/pricing).

## Run the complete recipe

Run these commands in your terminal. The download contains every step below, in order. Running it creates the paid jobs described above.

```
python -m pip install -U openai
export GRIDLINE_TOKEN="gla_..."
export FT_MODEL="ft:qwen3.5-9b:..."
curl -O https://gridlinegpu.com/cookbooks/evaluate-a-fine-tune/evaluate_fine_tune.py
python evaluate_fine_tune.py
```

The whole script is one file: [`evaluate_fine_tune.py`](https://gridlinegpu.com/cookbooks/evaluate-a-fine-tune/evaluate_fine_tune.py). The rest of this page is that file, a step at a time.

To understand or change the script, follow the walkthrough below. Read each step in order; later steps use the files and variables from earlier ones.

## Set up

The script starts with the OpenAI client, pointed at Gridline's base URL with your token as the key, and the names the steps below use.

```
"""Grade a fine-tuned model against its base on rows it never trained on.

Run fine_tune_classifier.py first: it prints your ft: model and writes test.jsonl.

  python -m pip install -U openai
  export GRIDLINE_TOKEN="gla_..."
  export FT_MODEL="ft:qwen3.5-9b:..."
  python evaluate_fine_tune.py

It creates two eval runs. Each one is a batch, charged to your prepaid credits.
"""

import json
import os
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://api.gridlinegpu.com/openai/v1",
    api_key=os.environ["GRIDLINE_TOKEN"],
    timeout=60,
)

TUNED = os.environ["FT_MODEL"]
BASE = TUNED.split(":")[1]  # ft:<base>:<org>:<job>
```

## 1. Load the held-out rows

Use tickets kept out of training to check the model's answers.

An eval item is any JSON object. Here each one holds a ticket and the queue you expect, read from rows the model never trained on.

```
# Chat rows in the fine-tuning format: system, user, then the answer you expect.

rows = [json.loads(line)["messages"] for line in open("test.jsonl")]
system = rows[0][0]["content"]
items = [
    {"item": {"ticket": user["content"], "queue": answer["content"]}}
    for _, user, answer in rows
]
```

**What you should see** The script loads tickets and expected queues from `test.jsonl`; this step prints nothing.

## 2. Say how an answer is graded

Define what counts as a correct answer before running either model.

A `string_check` grader compares the model's answer, `{{sample.output_text}}`, with a field of the row. `text_similarity` and `score_model` graders are offered too, and an eval holds up to 8 criteria.

```
# A row passes when the model's answer is exactly the expected queue.

routing_eval = client.evals.create(
    name="ticket-routing",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {"ticket": {"type": "string"}, "queue": {"type": "string"}},
        },
        "include_sample_schema": True,
    },
    testing_criteria=[
        {
            "type": "string_check",
            "name": "exact queue",
            "input": "{{sample.output_text}}",
            "reference": "{{item.queue}}",
            "operation": "eq",
        }
    ],
)

def run_request(model):
    return {
        "name": model,
        "data_source": {
            "type": "completions",
            "model": model,
            "input_messages": {
                "type": "template",
                "template": [
                    {"role": "system", "content": system},
                    {"role": "user", "content": "{{item.ticket}}"},
                ],
            },
            "sampling_params": {"temperature": 0, "max_completion_tokens": 8},
            "source": {"type": "file_content", "content": items},
        },
    }
```

**What you should see** The eval is created with an exact-queue grader; model runs have not started yet.

## 3. Ask what it can cost

Read the maximum charge before starting the model runs.

`dry_run` is a Gridline field: it answers the most a run can charge and creates nothing.

```
# dry_run answers the most a run can charge and creates nothing.

quote = client.post(
    f"/evals/{routing_eval.id}/runs",
    body={**run_request(TUNED), "dry_run": True},
    cast_to=object,
)
print("each run costs at most $", quote["quote"]["max_charge_usd"])
```

**What you should see** The `dry_run` quote prints without creating a run.

## 4. Run it on both models

Use the same rows and grader for both models.

A run renders each row into chat messages, sends them to the model as one batch, then grades the answers. A run grades up to 500 rows; here they go inline as `file_content`.

```
runs = [client.evals.runs.create(routing_eval.id, **run_request(model)) for model in (BASE, TUNED)]
while any(run.status in ("queued", "in_progress") for run in runs):
    time.sleep(30)
    runs = [client.evals.runs.retrieve(run.id, eval_id=routing_eval.id) for run in runs]
```

**What you should see** This step waits without printing until the runs leave their queued or in-progress states.

## 5. Compare

Read the scores and missed tickets before deciding what to change.

`result_counts` gives the rows that passed, failed and errored. The script then lists the tickets the fine-tuned model got wrong.

```
for run in runs:
    counts = run.result_counts
    print(f"{run.name}: {run.status}, {counts.passed} of {counts.total} passed")

failed = client.evals.runs.output_items.list(runs[1].id, eval_id=routing_eval.id, status="fail")
for item in failed:
    print("missed:", item.datasource_item["ticket"])
```

**What you should see** Each model's status and passed-row count print, followed by tickets the fine-tune missed.

## Next steps

Start by changing the sample data, then use these guides to go further.

- Use your own test set: any JSONL of chat rows the model did not train on.
- Grade free text with a `score_model` grader, where a catalog model scores each answer: see [Graders](https://gridlinegpu.com/docs/evals#graders).

### More recipes

---

Source: https://gridlinegpu.com/docs/cookbooks/evaluate-a-fine-tune
