Skip to content
Cookbook

Evaluate a fine-tune

Grade a fine-tuned model and its base on the same held-out rows with the Evals API.

All cookbooks
  • Evals
  • Batch inference

What you'll build

Compare your fine-tune with its base and inspect the tickets it missed.

The script creates an eval that passes a row when the model's answer is exactly the expected queue, runs it on your fine-tuned model and on its base, and prints how many rows each passed and which ones the fine-tuned model missed.

Each run's model calls are one batch, billed as any batch.

How it works

Held-out tickets and expected queues feed eval runs on the base and fine-tuned model. An exact-match grader checks answers; the script prints pass counts and lists missed tickets.

What you need

Comfort with
Python and terminal basics; classifier recipe first
Models
qwen3.5-9b (see Models)
Jobs and cost
It creates two eval runs, each of them one batch. Fine-tuning and batch jobs are paid for from your prepaid credits.
Time
The script waits for its jobs to finish. A time estimate is not available.

You need a Gridline token and credits: the quickstart shows how to get both. Run Fine-tune a classifier first, in the same folder: it writes test.jsonl and prints your model. Set FT_MODEL to that model's id.

What a credit is worth, the credit packs and each model's rates are on the pricing page.

Run the complete recipe

Run these commands in your terminal. The download contains every step below, in order. Running it creates the paid jobs described above.

python -m pip install -U openai
export GRIDLINE_TOKEN="gla_..."
export FT_MODEL="ft:qwen3.5-9b:..."
curl -O https://gridlinegpu.com/cookbooks/evaluate-a-fine-tune/evaluate_fine_tune.py
python evaluate_fine_tune.py

The whole script is one file: evaluate_fine_tune.py. The rest of this page is that file, a step at a time.

To understand or change the script, follow the walkthrough below. Read each step in order; later steps use the files and variables from earlier ones.

Set up

The script starts with the OpenAI client, pointed at Gridline's base URL with your token as the key, and the names the steps below use.

"""Grade a fine-tuned model against its base on rows it never trained on.

Run fine_tune_classifier.py first: it prints your ft: model and writes test.jsonl.

  python -m pip install -U openai
  export GRIDLINE_TOKEN="gla_..."
  export FT_MODEL="ft:qwen3.5-9b:..."
  python evaluate_fine_tune.py

It creates two eval runs. Each one is a batch, charged to your prepaid credits.
"""

import json
import os
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://api.gridlinegpu.com/openai/v1",
    api_key=os.environ["GRIDLINE_TOKEN"],
    timeout=60,
)

TUNED = os.environ["FT_MODEL"]
BASE = TUNED.split(":")[1]  # ft:<base>:<org>:<job>

1. Load the held-out rows

Use tickets kept out of training to check the model's answers.

An eval item is any JSON object. Here each one holds a ticket and the queue you expect, read from rows the model never trained on.

# Chat rows in the fine-tuning format: system, user, then the answer you expect.

rows = [json.loads(line)["messages"] for line in open("test.jsonl")]
system = rows[0][0]["content"]
items = [
    {"item": {"ticket": user["content"], "queue": answer["content"]}}
    for _, user, answer in rows
]

What you should see The script loads tickets and expected queues from test.jsonl; this step prints nothing.

2. Say how an answer is graded

Define what counts as a correct answer before running either model.

A string_check grader compares the model's answer, {{sample.output_text}}, with a field of the row. text_similarity and score_model graders are offered too, and an eval holds up to 8 criteria.

# A row passes when the model's answer is exactly the expected queue.

routing_eval = client.evals.create(
    name="ticket-routing",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {"ticket": {"type": "string"}, "queue": {"type": "string"}},
        },
        "include_sample_schema": True,
    },
    testing_criteria=[
        {
            "type": "string_check",
            "name": "exact queue",
            "input": "{{sample.output_text}}",
            "reference": "{{item.queue}}",
            "operation": "eq",
        }
    ],
)


def run_request(model):
    return {
        "name": model,
        "data_source": {
            "type": "completions",
            "model": model,
            "input_messages": {
                "type": "template",
                "template": [
                    {"role": "system", "content": system},
                    {"role": "user", "content": "{{item.ticket}}"},
                ],
            },
            "sampling_params": {"temperature": 0, "max_completion_tokens": 8},
            "source": {"type": "file_content", "content": items},
        },
    }

What you should see The eval is created with an exact-queue grader; model runs have not started yet.

3. Ask what it can cost

Read the maximum charge before starting the model runs.

dry_run is a Gridline field: it answers the most a run can charge and creates nothing.

# dry_run answers the most a run can charge and creates nothing.

quote = client.post(
    f"/evals/{routing_eval.id}/runs",
    body={**run_request(TUNED), "dry_run": True},
    cast_to=object,
)
print("each run costs at most $", quote["quote"]["max_charge_usd"])

What you should see The dry_run quote prints without creating a run.

4. Run it on both models

Use the same rows and grader for both models.

A run renders each row into chat messages, sends them to the model as one batch, then grades the answers. A run grades up to 500 rows; here they go inline as file_content.

runs = [client.evals.runs.create(routing_eval.id, **run_request(model)) for model in (BASE, TUNED)]
while any(run.status in ("queued", "in_progress") for run in runs):
    time.sleep(30)
    runs = [client.evals.runs.retrieve(run.id, eval_id=routing_eval.id) for run in runs]

What you should see This step waits without printing until the runs leave their queued or in-progress states.

5. Compare

Read the scores and missed tickets before deciding what to change.

result_counts gives the rows that passed, failed and errored. The script then lists the tickets the fine-tuned model got wrong.

for run in runs:
    counts = run.result_counts
    print(f"{run.name}: {run.status}, {counts.passed} of {counts.total} passed")

failed = client.evals.runs.output_items.list(runs[1].id, eval_id=routing_eval.id, status="fail")
for item in failed:
    print("missed:", item.datasource_item["ticket"])

What you should see Each model's status and passed-row count print, followed by tickets the fine-tune missed.

Next steps

Start by changing the sample data, then use these guides to go further.

  • Use your own test set: any JSONL of chat rows the model did not train on.
  • Grade free text with a score_model grader, where a catalog model scores each answer: see Graders.

More recipes

Contact us

Whether you have a question about fine-tuning or batch, need help with a job, or want to share feedback, send us a note. We’ll make sure it reaches the right person.

New messageGridline / contact
0/5,000