Skip to content
Cookbook

Distill a large model into a small one

Have a large model answer your prompts, train a small model on those answers, and count how often the small one agrees.

All cookbooks
  • Distillation
  • Fine-tuning
  • Batch inference

What you'll build

Train a smaller model on a teacher's answers, then check their agreement.

The script sends 300 unlabelled support tickets to qwen3.8-27b, which answers each with a JSON triage record: queue, urgency and a one-line summary. POST /files/distill turns those answers into a training file and a held-out file. The script trains qwen3.5-9b on the first, then runs the held-out tickets on it and on its untrained base, and counts how often each gives the large model's queue and urgency.

Nobody labels anything: the large model's answers are the labels. Use it when a large model does the job well and you want the same answers from a smaller, cheaper one.

You are responsible for having the right to train on what you upload, including model outputs.

How it works

A teacher batch answers your prompts. Those answers become training data for a smaller student, which is checked against the teacher on held-out rows.

What you need

Comfort with
Python, JSON and terminal basics
Models
qwen3.8-27b and qwen3.5-9b (see Models)
Jobs and cost
It creates two batches and one fine-tuning job. Fine-tuning and batch jobs are paid for from your prepaid credits.
Time
The script waits for its jobs to finish. A time estimate is not available.

You need a Gridline token and credits: the quickstart shows how to get both.

What a credit is worth, the credit packs and each model's rates are on the pricing page.

Run the complete recipe

Run these commands in your terminal. The download contains every step below, in order. Running it creates the paid jobs described above.

python -m pip install -U openai
export GRIDLINE_TOKEN="gla_..."
curl -O https://gridlinegpu.com/cookbooks/distill-a-large-model/distill.py
python distill.py

The whole script is one file: distill.py. The rest of this page is that file, a step at a time.

To understand or change the script, follow the walkthrough below. Read each step in order; later steps use the files and variables from earlier ones.

Set up

The script starts with the OpenAI client, pointed at Gridline's base URL with your token as the key, and the names the steps below use.

"""Distill a large model into a small one: the large model answers, the small one learns.

  python -m pip install -U openai
  export GRIDLINE_TOKEN="gla_..."
  python distill.py

It creates two batches and one fine-tuning job, all charged to your prepaid credits.
"""

import json
import os
import random
import time

from openai import OpenAI

client = OpenAI(
    base_url="https://api.gridlinegpu.com/openai/v1",
    api_key=os.environ["GRIDLINE_TOKEN"],
    timeout=60,
)

TEACHER = "qwen3.8-27b"
STUDENT = "qwen3.5-9b"
SYSTEM = (
    "You triage support tickets. Reply with JSON only: queue (billing, shipping, returns, "
    "account, technical or sales), urgency (low, normal or high) and summary (one short "
    "sentence)."
)
TRIAGE = {
    "type": "json_schema",
    "json_schema": {
        "name": "triage",
        "schema": {
            "type": "object",
            "properties": {
                "queue": {
                    "type": "string",
                    "enum": ["billing", "shipping", "returns", "account", "technical", "sales"],
                },
                "urgency": {"type": "string", "enum": ["low", "normal", "high"]},
                "summary": {"type": "string", "maxLength": 200},
            },
            "required": ["queue", "urgency", "summary"],
            "additionalProperties": False,
        },
    },
}


def wait(batch):
    while batch.status not in ("completed", "failed", "expired", "cancelled"):
        time.sleep(30)
        batch = client.batches.retrieve(batch.id)
    if batch.status != "completed":
        raise SystemExit(f"{batch.id} ended {batch.status}: {batch.errors}")
    return batch


def run_batch(path):
    batch_input = client.files.create(file=open(path, "rb"), purpose="batch")
    return wait(
        client.batches.create(
            input_file_id=batch_input.id,
            endpoint="/v1/chat/completions",
            completion_window="24h",
        )
    )

1. Write the prompts

Give the teacher a common answer format to use its responses as training examples.

Each line is a batch request to the large model, the teacher. Its response_format holds the teacher to a JSON schema, so every finished answer parses. Set max_tokens high enough for a whole answer: one cut off at the limit is dropped in step 3.

# Unlabelled tickets: no human wrote the answers. Replace these with your own inputs.

SUBJECTS = [
    "I was charged twice for my {plan} plan",
    "order {order} has not arrived",
    "my {product} came with a cracked screen",
    "I cannot sign in and the reset email never comes",
    "the app closes when I open the {product} settings",
    "we want a quote for 200 units of the {product}",
    "my invoice for the {plan} plan is wrong",
    "the tracking for order {order} has not moved in a week",
    "I want to send back the {product}",
    "someone else signed in to my account",
    "the {product} will not connect since the update",
    "is there a discount on the {plan} plan for schools",
]
CONTEXTS = [
    "",
    " and I need it sorted before a trip tomorrow",
    ", no rush",
    " and this is the third time I have written",
    " and our whole team is blocked",
    ", just checking",
    " and I am about to cancel",
]
PLANS = ["Starter", "Team", "Pro", "Business"]
PRODUCTS = ["headphones", "smart speaker", "fitness band", "tablet", "router", "camera"]

rng = random.Random(11)
tickets = set()
while len(tickets) < 300:
    subject = rng.choice(SUBJECTS).format(
        plan=rng.choice(PLANS),
        product=rng.choice(PRODUCTS),
        order=rng.randint(10000, 99999),
    )
    tickets.add(f"Hello, {subject}{rng.choice(CONTEXTS)}.")

with open("teacher-requests.jsonl", "w") as out:
    for n, ticket in enumerate(sorted(tickets)):
        body = {
            "model": TEACHER,
            "messages": [
                {"role": "system", "content": SYSTEM},
                {"role": "user", "content": ticket},
            ],
            "max_tokens": 200,
            "temperature": 0,
            "response_format": TRIAGE,
        }
        line = {
            "custom_id": f"ticket-{n}",
            "method": "POST",
            "url": "/v1/chat/completions",
            "body": body,
        }
        out.write(json.dumps(line) + "\n")

What you should see teacher-requests.jsonl holds the prompts and schema; this step prints nothing.

2. Run the teacher

Collect the teacher's answers before preparing the student's training data.

Upload the requests with purpose="batch", create the batch and poll until it is completed.

teacher_batch = run_batch("teacher-requests.jsonl")
print("teacher answered", teacher_batch.request_counts.completed, "tickets")

What you should see The script prints how many tickets the teacher answered after the batch completes.

3. Turn its answers into training files

Separate training answers from held-out answers before the student learns from them.

POST /files/distill takes the batch's output_file_id. Each result becomes one chat row: the request's messages, then the teacher's answer. require_json drops an answer that is not valid JSON, and eval_fraction holds out a fifth of the rows in a validation file. The call is not charged. The OpenAI client has no method for it, so the script uses client.post.

# One row per answer: the request's messages, then the teacher's answer. A fifth is held out.

distilled = client.post(
    "/files/distill",
    body={
        "results_file": teacher_batch.output_file_id,
        "require_json": True,
        "eval_fraction": 0.2,
    },
    cast_to=object,
)
print(distilled["train_rows"], "training rows,", distilled["eval_rows"], "held out")
print("dropped:", distilled["dropped"])

What you should see Training and held-out row counts print, followed by the dropped-answer counts.

4. Fine-tune the student

Use the teacher's answers as examples for the smaller model.

The job trains the small model, the student, on the training file and measures its eval loss on the validation file.

job = client.fine_tuning.jobs.create(
    model=STUDENT,
    training_file=distilled["training_file"]["id"],
    validation_file=distilled["validation_file"]["id"],
    hyperparameters={"n_epochs": 3},
)
print(job.id, "costs at most $", job.gridline["quote"]["per_token"]["max_charge_usd"])
while job.status not in ("succeeded", "failed", "cancelled"):
    time.sleep(30)
    job = client.fine_tuning.jobs.retrieve(job.id)
if job.status != "succeeded":
    raise SystemExit(f"{job.id} ended {job.status}: {job.error}")
print("student:", job.fine_tuned_model)

What you should see The job id and maximum charge print first; the student's model id prints on success.

5. Check the student against the teacher

Compare the student and its base with the teacher on rows kept out of training.

The validation file's rows were not trained on, and each ends with the teacher's answer. The script downloads it, runs the same tickets on the student and on its base, and compares queue and urgency. Agreement is not correctness: read some of the teacher's answers yourself before you use them as labels.

# The held-out rows were never trained on. Run them on the student and on its untrained base,
# then count how often each one gives the teacher's queue and urgency.

held_out = [
    json.loads(line)["messages"]
    for line in client.files.content(distilled["validation_file"]["id"]).text.splitlines()
]
MODELS = {"base": STUDENT, "student": job.fine_tuned_model}
with open("student-requests.jsonl", "w") as out:
    for arm, model in MODELS.items():
        for n, messages in enumerate(held_out):
            body = {
                "model": model,
                "messages": messages[:-1],
                "max_tokens": 200,
                "temperature": 0,
            }
            line = {
                "custom_id": f"{arm}-{n}",
                "method": "POST",
                "url": "/v1/chat/completions",
                "body": body,
            }
            out.write(json.dumps(line) + "\n")

student_batch = run_batch("student-requests.jsonl")
answers = {}
for line in client.files.content(student_batch.output_file_id).text.splitlines():
    row = json.loads(line)
    answers[row["custom_id"]] = row["response"]["body"]["choices"][0]["message"]["content"]


def triage(text):
    try:
        reply = json.loads(text)
        return reply["queue"], reply["urgency"]
    except (ValueError, KeyError, TypeError):
        return None


for arm in MODELS:
    agree = sum(
        triage(answers.get(f"{arm}-{n}", "")) == triage(messages[-1]["content"])
        for n, messages in enumerate(held_out)
    )
    print(f"{arm}: agrees with the teacher on {agree} of {len(held_out)}")

What you should see The base and student lines report agreement with the teacher, not whether the teacher was correct.

Next steps

Start by changing the sample data, then use these guides to go further.

  • Send your own prompts: replace the tickets, the instruction and the schema.
  • Choose another teacher or student. Models lists what runs in a batch and what can be fine-tuned.
  • Already have answers from another provider? POST /files/distill also reads an uploaded results file in OpenAI's or Anthropic's batch format: see From batch results.

More recipes

Contact us

Whether you have a question about fine-tuning or batch, need help with a job, or want to share feedback, send us a note. We’ll make sure it reaches the right person.

New messageGridline / contact
0/5,000