Cookbook

# Distill a large model into a small one

Have a large model answer your prompts, train a small model on those answers, and count how often the small one agrees.

[All cookbooks](https://gridlinegpu.com/docs/cookbooks)

- Distillation
- Fine-tuning
- Batch inference

## What you'll build

Train a smaller model on a teacher's answers, then check their agreement.

The script sends 300 unlabelled support tickets to `qwen3.8-27b`, which answers each with a JSON triage record: queue, urgency and a one-line summary. `POST /files/distill` turns those answers into a training file and a held-out file. The script trains `qwen3.5-9b` on the first, then runs the held-out tickets on it and on its untrained base, and counts how often each gives the large model's queue and urgency.

Nobody labels anything: the large model's answers are the labels. Use it when a large model does the job well and you want the same answers from a smaller, cheaper one.

You are responsible for having the right to train on what you upload, including model outputs.

### How it works

1. Your prompts
2. Teacher answers
3. Training files
4. Train student
5. Check agreement

A teacher batch answers your prompts. Those answers become training data for a smaller student, which is checked against the teacher on held-out rows.

## What you need

Comfort with

Python, JSON and terminal basics

Models

`qwen3.8-27b` and `qwen3.5-9b` (see [Models](https://gridlinegpu.com/docs/models))

Jobs and cost

It creates two batches and one fine-tuning job. Fine-tuning and batch jobs are paid for from your prepaid credits.

Time

The script waits for its jobs to finish. A time estimate is not available.

You need a Gridline token and credits: the [quickstart](https://gridlinegpu.com/docs#access) shows how to get both.

What a credit is worth, the credit packs and each model's rates are on the [pricing page](https://gridlinegpu.com/pricing).

## Run the complete recipe

Run these commands in your terminal. The download contains every step below, in order. Running it creates the paid jobs described above.

```
python -m pip install -U openai
export GRIDLINE_TOKEN="gla_..."
curl -O https://gridlinegpu.com/cookbooks/distill-a-large-model/distill.py
python distill.py
```

The whole script is one file: [`distill.py`](https://gridlinegpu.com/cookbooks/distill-a-large-model/distill.py). The rest of this page is that file, a step at a time.

To understand or change the script, follow the walkthrough below. Read each step in order; later steps use the files and variables from earlier ones.

## Set up

The script starts with the OpenAI client, pointed at Gridline's base URL with your token as the key, and the names the steps below use.

```
"""Distill a large model into a small one: the large model answers, the small one learns.

  python -m pip install -U openai
  export GRIDLINE_TOKEN="gla_..."
  python distill.py

It creates two batches and one fine-tuning job, all charged to your prepaid credits.
"""

import json
import os
import random
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://api.gridlinegpu.com/openai/v1",
    api_key=os.environ["GRIDLINE_TOKEN"],
    timeout=60,
)

TEACHER = "qwen3.8-27b"
STUDENT = "qwen3.5-9b"
SYSTEM = (
    "You triage support tickets. Reply with JSON only: queue (billing, shipping, returns, "
    "account, technical or sales), urgency (low, normal or high) and summary (one short "
    "sentence)."
)
TRIAGE = {
    "type": "json_schema",
    "json_schema": {
        "name": "triage",
        "schema": {
            "type": "object",
            "properties": {
                "queue": {
                    "type": "string",
                    "enum": ["billing", "shipping", "returns", "account", "technical", "sales"],
                },
                "urgency": {"type": "string", "enum": ["low", "normal", "high"]},
                "summary": {"type": "string", "maxLength": 200},
            },
            "required": ["queue", "urgency", "summary"],
            "additionalProperties": False,
        },
    },
}

def wait(batch):
    while batch.status not in ("completed", "failed", "expired", "cancelled"):
        time.sleep(30)
        batch = client.batches.retrieve(batch.id)
    if batch.status != "completed":
        raise SystemExit(f"{batch.id} ended {batch.status}: {batch.errors}")
    return batch

def run_batch(path):
    batch_input = client.files.create(file=open(path, "rb"), purpose="batch")
    return wait(
        client.batches.create(
            input_file_id=batch_input.id,
            endpoint="/v1/chat/completions",
            completion_window="24h",
        )
    )
```

## 1. Write the prompts

Give the teacher a common answer format to use its responses as training examples.

Each line is a batch request to the large model, the teacher. Its `response_format` holds the teacher to a JSON schema, so every finished answer parses. Set `max_tokens` high enough for a whole answer: one cut off at the limit is dropped in step 3.

```
# Unlabelled tickets: no human wrote the answers. Replace these with your own inputs.

SUBJECTS = [
    "I was charged twice for my {plan} plan",
    "order {order} has not arrived",
    "my {product} came with a cracked screen",
    "I cannot sign in and the reset email never comes",
    "the app closes when I open the {product} settings",
    "we want a quote for 200 units of the {product}",
    "my invoice for the {plan} plan is wrong",
    "the tracking for order {order} has not moved in a week",
    "I want to send back the {product}",
    "someone else signed in to my account",
    "the {product} will not connect since the update",
    "is there a discount on the {plan} plan for schools",
]
CONTEXTS = [
    "",
    " and I need it sorted before a trip tomorrow",
    ", no rush",
    " and this is the third time I have written",
    " and our whole team is blocked",
    ", just checking",
    " and I am about to cancel",
]
PLANS = ["Starter", "Team", "Pro", "Business"]
PRODUCTS = ["headphones", "smart speaker", "fitness band", "tablet", "router", "camera"]

rng = random.Random(11)
tickets = set()
while len(tickets) < 300:
    subject = rng.choice(SUBJECTS).format(
        plan=rng.choice(PLANS),
        product=rng.choice(PRODUCTS),
        order=rng.randint(10000, 99999),
    )
    tickets.add(f"Hello, {subject}{rng.choice(CONTEXTS)}.")

with open("teacher-requests.jsonl", "w") as out:
    for n, ticket in enumerate(sorted(tickets)):
        body = {
            "model": TEACHER,
            "messages": [
                {"role": "system", "content": SYSTEM},
                {"role": "user", "content": ticket},
            ],
            "max_tokens": 200,
            "temperature": 0,
            "response_format": TRIAGE,
        }
        line = {
            "custom_id": f"ticket-{n}",
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": body,
        }
        out.write(json.dumps(line) + "\n")
```

**What you should see** `teacher-requests.jsonl` holds the prompts and schema; this step prints nothing.

## 2. Run the teacher

Collect the teacher's answers before preparing the student's training data.

Upload the requests with `purpose="batch"`, create the batch and poll until it is `completed`.

```
teacher_batch = run_batch("teacher-requests.jsonl")
print("teacher answered", teacher_batch.request_counts.completed, "tickets")
```

**What you should see** The script prints how many tickets the teacher answered after the batch completes.

## 3. Turn its answers into training files

Separate training answers from held-out answers before the student learns from them.

`POST /files/distill` takes the batch's `output_file_id`. Each result becomes one chat row: the request's messages, then the teacher's answer. `require_json` drops an answer that is not valid JSON, and `eval_fraction` holds out a fifth of the rows in a validation file. The call is not charged. The OpenAI client has no method for it, so the script uses `client.post`.

```
# One row per answer: the request's messages, then the teacher's answer. A fifth is held out.

distilled = client.post(
    "/files/distill",
    body={
        "results_file": teacher_batch.output_file_id,
        "require_json": True,
        "eval_fraction": 0.2,
    },
    cast_to=object,
)
print(distilled["train_rows"], "training rows,", distilled["eval_rows"], "held out")
print("dropped:", distilled["dropped"])
```

**What you should see** Training and held-out row counts print, followed by the dropped-answer counts.

## 4. Fine-tune the student

Use the teacher's answers as examples for the smaller model.

The job trains the small model, the student, on the training file and measures its eval loss on the validation file.

```
job = client.fine_tuning.jobs.create(
    model=STUDENT,
    training_file=distilled["training_file"]["id"],
    validation_file=distilled["validation_file"]["id"],
    hyperparameters={"n_epochs": 3},
)
print(job.id, "costs at most $", job.gridline["quote"]["per_token"]["max_charge_usd"])
while job.status not in ("succeeded", "failed", "cancelled"):
    time.sleep(30)
    job = client.fine_tuning.jobs.retrieve(job.id)
if job.status != "succeeded":
    raise SystemExit(f"{job.id} ended {job.status}: {job.error}")
print("student:", job.fine_tuned_model)
```

**What you should see** The job id and maximum charge print first; the student's model id prints on success.

## 5. Check the student against the teacher

Compare the student and its base with the teacher on rows kept out of training.

The validation file's rows were not trained on, and each ends with the teacher's answer. The script downloads it, runs the same tickets on the student and on its base, and compares queue and urgency. Agreement is not correctness: read some of the teacher's answers yourself before you use them as labels.

```
# The held-out rows were never trained on. Run them on the student and on its untrained base,
# then count how often each one gives the teacher's queue and urgency.

held_out = [
    json.loads(line)["messages"]
    for line in client.files.content(distilled["validation_file"]["id"]).text.splitlines()
]
MODELS = {"base": STUDENT, "student": job.fine_tuned_model}
with open("student-requests.jsonl", "w") as out:
    for arm, model in MODELS.items():
        for n, messages in enumerate(held_out):
            body = {
                "model": model,
                "messages": messages[:-1],
                "max_tokens": 200,
                "temperature": 0,
            }
            line = {
                "custom_id": f"{arm}-{n}",
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": body,
            }
            out.write(json.dumps(line) + "\n")

student_batch = run_batch("student-requests.jsonl")
answers = {}
for line in client.files.content(student_batch.output_file_id).text.splitlines():
    row = json.loads(line)
    answers[row["custom_id"]] = row["response"]["body"]["choices"][0]["message"]["content"]

def triage(text):
    try:
        reply = json.loads(text)
        return reply["queue"], reply["urgency"]
    except (ValueError, KeyError, TypeError):
        return None

for arm in MODELS:
    agree = sum(
        triage(answers.get(f"{arm}-{n}", "")) == triage(messages[-1]["content"])
        for n, messages in enumerate(held_out)
    )
    print(f"{arm}: agrees with the teacher on {agree} of {len(held_out)}")
```

**What you should see** The `base` and `student` lines report agreement with the teacher, not whether the teacher was correct.

## Next steps

Start by changing the sample data, then use these guides to go further.

- Send your own prompts: replace the tickets, the instruction and the schema.
- Choose another teacher or student. [Models](https://gridlinegpu.com/docs/models) lists what runs in a batch and what can be fine-tuned.
- Already have answers from another provider? `POST /files/distill` also reads an uploaded results file in OpenAI's or Anthropic's batch format: see [From batch results](https://gridlinegpu.com/docs/fine-tuning#files).

### More recipes

---

Source: https://gridlinegpu.com/docs/cookbooks/distill-a-large-model
