Cookbook

# Fine-tune a classifier

Train a small model to route support tickets to the right queue, then test it against its base on tickets it has not seen.

[All cookbooks](https://gridlinegpu.com/docs/cookbooks)

- Fine-tuning
- Batch inference

## What you'll build

Build a support-ticket classifier and compare it with its base model.

The script trains `qwen3.5-9b` on 240 labelled tickets and tests it on 60 more, written in wordings the training set does not contain. It runs the test on the fine-tuned model and on the base model in one batch, and prints how many each got right.

The tickets are synthetic, so the script runs as it is. The same steps fit any task with a short answer and examples of it: routing, tagging, moderation, yes-or-no checks.

### How it works

1. Labelled tickets
2. Fine-tuning job
3. Your model
4. Compare with base

Labelled tickets train your model. A batch runs it and the base on tickets kept out of training; the script compares their answers.

## What you need

Comfort with

Python and terminal basics

Models

`qwen3.5-9b` (see [Models](https://gridlinegpu.com/docs/models))

Jobs and cost

It creates one fine-tuning job and one batch. Fine-tuning and batch jobs are paid for from your prepaid credits.

Time

The script waits for its jobs to finish. A time estimate is not available.

You need a Gridline token and credits: the [quickstart](https://gridlinegpu.com/docs#access) shows how to get both.

What a credit is worth, the credit packs and each model's rates are on the [pricing page](https://gridlinegpu.com/pricing).

## Run the complete recipe

Run these commands in your terminal. The download contains every step below, in order. Running it creates the paid jobs described above.

```
python -m pip install -U openai
export GRIDLINE_TOKEN="gla_..."
curl -O https://gridlinegpu.com/cookbooks/fine-tune-a-classifier/fine_tune_classifier.py
python fine_tune_classifier.py
```

The whole script is one file: [`fine_tune_classifier.py`](https://gridlinegpu.com/cookbooks/fine-tune-a-classifier/fine_tune_classifier.py). The rest of this page is that file, a step at a time.

To understand or change the script, follow the walkthrough below. Read each step in order; later steps use the files and variables from earlier ones.

## Set up

The script starts with the OpenAI client, pointed at Gridline's base URL with your token as the key, and the names the steps below use.

```
"""Fine-tune a small model to route support tickets, then measure it against its base.

  python -m pip install -U openai
  export GRIDLINE_TOKEN="gla_..."
  python fine_tune_classifier.py

It creates one fine-tuning job and one batch, both charged to your prepaid credits.
"""

import json
import os
import random
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://api.gridlinegpu.com/openai/v1",
    api_key=os.environ["GRIDLINE_TOKEN"],
    timeout=60,
)

BASE = "qwen3.5-9b"
SYSTEM = (
    "Route the support ticket to one queue: billing, shipping, returns, account, "
    "technical or sales. Answer with the queue name only."
)
```

## 1. Make the data

Keep test tickets separate to check answers the model did not learn during training.

Each training row is one conversation in OpenAI's chat format: the instruction, the ticket, and the queue as the assistant's answer. A training file needs at least 10 rows. The script writes `train.jsonl` and `test.jsonl`; the test rows stay on your machine.

```
# Synthetic tickets, so the script runs as it is. Replace TICKETS with your own labelled rows.

TICKETS = {
    "billing": [
        "I was charged twice for my {plan} plan this month.",
        "My invoice says {amount} but the {plan} plan costs less than that.",
        "Please refund the {amount} you took on {day}. I cancelled before the renewal.",
        "Why did the price of the {plan} plan go up without any notice?",
        "I paid for order {order} and it never arrived. I want my money back.",
        "I need a VAT invoice for the {amount} payment on {day}.",
        "My card was declined, but {amount} still left my bank account.",
        "You billed me for a full year. I chose monthly on the {plan} plan.",
    ],
    "shipping": [
        "Order {order} was due on {day} and it still has not arrived.",
        "The tracking page for order {order} has not changed in a week.",
        "Can you send order {order} to a different address? I moved.",
        "My {product} shows as delivered but nothing is at my door.",
        "How long does delivery of the {product} take to Canada?",
        "The courier left a card for order {order}. How do I book a new delivery?",
        "I paid for express delivery of my {product} and it came by standard post.",
        "Only one of the two boxes for order {order} has turned up.",
    ],
    "returns": [
        "The {product} arrived with a cracked screen. I want to send it back.",
        "How do I return the {product}? It is the wrong size.",
        "I sent my {product} back on {day} and have heard nothing since.",
        "You sent a {product} that I did not order. What do I do with it?",
        "Can I swap my {product} for the newer model?",
        "The {product} stopped working after two days. I want a replacement.",
        "Where do I print the return label for order {order}?",
        "Is the {product} still inside the 30 day return window? I bought it on {day}.",
    ],
    "account": [
        "I cannot sign in. The password reset email never comes.",
        "Please change the email address on my account.",
        "How do I turn on two-step sign-in?",
        "I want to close my account and have my data deleted.",
        "Somebody signed in to my account from another country on {day}.",
        "I cannot sign in to pay my invoice. It says my account is locked.",
        "How do I add a colleague to our {plan} account?",
        "My name is spelled wrong on my profile and I cannot edit it.",
    ],
    "technical": [
        "The app closes every time I open the {product} settings page.",
        "My {product} will not connect to wifi since the last update.",
        "Exports from the dashboard fail with error 500.",
        "The {product} battery runs out within an hour of a full charge.",
        "Sync between my phone and my {product} has stopped working.",
        "I get a blank white page after I sign in on Safari.",
        "The API returns a timeout on every request since {day}.",
        "The firmware update for my {product} stops at 80 percent.",
    ],
    "sales": [
        "Do you offer a discount for schools on the {plan} plan?",
        "What is the difference between the {plan} plan and the one above it?",
        "We need 200 units of the {product}. Can you send a quote?",
        "Is there a free trial of the {plan} plan?",
        "Does the {product} work with an iPhone before I buy one?",
        "Can I pay for the {plan} plan by bank transfer and not by card?",
        "When will the {product} be back in stock?",
        "I would like a demo for my team before we choose a plan.",
    ],
}
FILLERS = {
    "plan": ["Starter", "Team", "Pro", "Business"],
    "product": ["headphones", "smart speaker", "fitness band", "tablet", "router", "camera"],
    "amount": ["$19", "$49", "$120", "$245", "$980"],
    "day": ["Monday", "3 March", "last Friday", "the 14th", "1 June"],
}
OPENERS = ["", "", "Hi, ", "Hello, ", "Good morning. "]
CLOSERS = ["", "", " Thanks.", " Please help.", " This is urgent."]

rng = random.Random(7)

def tickets(templates, count):
    made = set()
    while len(made) < count:
        text = rng.choice(templates).format(
            order=rng.randint(10000, 99999),
            **{name: rng.choice(values) for name, values in FILLERS.items()},
        )
        made.add(rng.choice(OPENERS) + text + rng.choice(CLOSERS))
    return sorted(made)

# The last two wordings of each queue never appear in training: the test is on unseen phrasings.
train, test = [], []
for queue, templates in TICKETS.items():
    train += [(ticket, queue) for ticket in tickets(templates[:6], 40)]
    test += [(ticket, queue) for ticket in tickets(templates[6:], 10)]
rng.shuffle(train)

def chat_row(ticket, queue):
    return {
        "messages": [
            {"role": "system", "content": SYSTEM},
            {"role": "user", "content": ticket},
            {"role": "assistant", "content": queue},
        ]
    }

for name, part in (("train.jsonl", train), ("test.jsonl", test)):
    with open(name, "w") as out:
        for ticket, queue in part:
            out.write(json.dumps(chat_row(ticket, queue)) + "\n")
print(f"{len(train)} training rows, {len(test)} held out for the test")
```

**What you should see** The script writes `train.jsonl` and `test.jsonl`, then prints the training and held-out row counts.

## 2. Start the fine-tune

Upload your examples to start training and see the job's maximum charge.

Upload the file with `purpose="fine-tune"` and create the job. Creating a job starts it, and the response carries its quote: `max_charge_usd` is the most the job can cost. `n_epochs` is how many times the job reads the training file; the default is 1.

```
# Creating the job starts it. The response carries the quote: the most the job can cost.

training = client.files.create(file=open("train.jsonl", "rb"), purpose="fine-tune")
job = client.fine_tuning.jobs.create(
    model=BASE,
    training_file=training.id,
    hyperparameters={"n_epochs": 3},
)
print(job.id, "costs at most $", job.gridline["quote"]["per_token"]["max_charge_usd"])
```

**What you should see** A job id and its maximum charge print when the job is created.

## 3. Wait for the model

Wait for training to finish before using your new model.

Poll until the job ends. A job that succeeds reports its eval loss before and after training in `gridline.result`, measured on training rows it held out. Lower after than before means the model fits your examples better. `fine_tuned_model` is your model's id.

```
while job.status not in ("succeeded", "failed", "cancelled"):
    time.sleep(30)
    job = client.fine_tuning.jobs.retrieve(job.id)
if job.status != "succeeded":
    raise SystemExit(f"{job.id} ended {job.status}: {job.error}")

result = job.gridline["result"]
print("eval loss", result["eval_loss_before"], "->", result["eval_loss"])
print("your model:", job.fine_tuned_model)
```

**What you should see** On success, the script prints eval loss before and after training, then your ft: model id.

## 4. Run the held-out tickets on both models

Give both models the same held-out tickets so their answers can be compared.

A batch can mix a base model with fine-tunes of it, so both run in one batch. `custom_id` records which model and which ticket each answer belongs to.

```
# One batch can mix a base model with fine-tunes of it, so both run on the same GPU.

MODELS = {"base": BASE, "tuned": job.fine_tuned_model}
with open("requests.jsonl", "w") as out:
    for arm, model in MODELS.items():
        for n, (ticket, _) in enumerate(test):
            body = {
                "model": model,
                "messages": chat_row(ticket, "")["messages"][:2],
                "max_tokens": 8,
                "temperature": 0,
            }
            line = {
                "custom_id": f"{arm}-{n}",
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": body,
            }
            out.write(json.dumps(line) + "\n")

batch_input = client.files.create(file=open("requests.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
    input_file_id=batch_input.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)
while batch.status not in ("completed", "failed", "expired", "cancelled"):
    time.sleep(30)
    batch = client.batches.retrieve(batch.id)
if batch.status != "completed":
    raise SystemExit(f"{batch.id} ended {batch.status}: {batch.errors}")
```

**What you should see** This step waits for the batch without printing; scoring starts after it completes.

## 5. Score both

Count exact queue matches to see how each model did on the test set.

Results come back in any order, one line per request. The script matches them by `custom_id` and counts the answers that are exactly the expected queue.

```
answers = {}
for line in client.files.content(batch.output_file_id).text.splitlines():
    row = json.loads(line)
    answers[row["custom_id"]] = row["response"]["body"]["choices"][0]["message"]["content"]

for arm in MODELS:
    right = sum(
        answers.get(f"{arm}-{n}", "").strip().lower() == queue
        for n, (_, queue) in enumerate(test)
    )
    print(f"{arm}: {right} of {len(test)} right")
```

**What you should see** A base line and a tuned line show how many tickets each routed correctly.

## Next steps

Start by changing the sample data, then use these guides to go further.

- Use your own rows: replace `TICKETS` with your labelled examples, and keep some of them out of training for the test.
- Try another base: change `BASE`. [Fine-tuning](https://gridlinegpu.com/docs/fine-tuning#bases) lists the bases and the settings you can change.
- Grade the model with the Evals API, not your own loop: [Evaluate a fine-tune](https://gridlinegpu.com/docs/cookbooks/evaluate-a-fine-tune).
- The model is yours. [Your ft: model](https://gridlinegpu.com/docs/fine-tuning#model) shows how to download it.

### More recipes

---

Source: https://gridlinegpu.com/docs/cookbooks/fine-tune-a-classifier
