Guide

# Distillation

Run a batch on a large model, turn its answers into fine-tuning files, and train a small model on them. Then run your batches on the small model.

## Overview

Distillation trains a small model (the student) on the answers of a large model (the teacher). It takes five calls to the Batch and fine-tuning APIs, at Gridline's base URL with a Gridline token as the API key (see the [quickstart](https://gridlinegpu.com/docs#access) for the token):

```
https://api.gridlinegpu.com/openai/v1
```

1. Upload your prompts as batch input, `purpose=batch`.
2. Run a batch on the teacher, `qwen3.8-27b`.
3. Call `POST /files/distill` on the batch's output file. It writes a training file and a validation file.
4. Create a fine-tuning job on the student with those files.
5. Run batches on the job's `ft:` model.

This guide uses `qwen3.8-27b` (Apache-2.0) as the teacher and `qwen2.5-1.5b-instruct` as the student. Any of these fine-tuning bases can be the student: qwen2.5-0.5b-instruct, qwen2.5-1.5b-instruct, qwen2.5-7b-instruct, mistral-7b-instruct-v0.3, qwen3-8b and llama-3.1-8b-instruct. You are responsible for having the right to train on what you upload, including model outputs.

## 1. Upload prompts

Each line is a batch request to `/v1/chat/completions` on the teacher, as on the [Batch API](https://gridlinegpu.com/docs/batch-api). Its last message must be a user message. On `qwen3.8-27b` a system message may only come first.

```
{"custom_id": "q-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen3.8-27b", "max_tokens": 512, "messages": [{"role": "system", "content": "Answer in one short paragraph."}, {"role": "user", "content": "Why does ice float on water?"}]}}
```

Set `max_tokens` high enough for a whole answer (`qwen3.8-27b` writes at most 1,024 tokens a request): an answer that stops at the limit is dropped in step 3 unless you pass `keep_truncated`.

```
curl https://api.gridlinegpu.com/openai/v1/files \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -F purpose=batch \
  -F [[email protected]](https://gridlinegpu.com/cdn-cgi/l/email-protection)
```

## 2. Run the teacher

```
curl https://api.gridlinegpu.com/openai/v1/batches \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input_file_id": "file-...", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'
```

Poll `GET /batches/{batch_id}` until its status is `completed`. Its `output_file_id` is the results file for the next step:

```
{"id": "batch_...", "object": "batch", "status": "completed", "input_file_id": "file-...", "output_file_id": "file-...", "request_counts": {"total": 1000, "completed": 1000, "failed": 0}, ...}
```

## 3. Make training files

```
curl https://api.gridlinegpu.com/openai/v1/files/distill \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"results_file": "file-..."}'
```

Each result becomes one training row: the request's messages, then the teacher's answer as the assistant message. The rows go to a training file and, held out, a validation file, both `purpose=fine-tune`. The same files and options always give the same rows. The body takes:

| Field | Default | What it does |
| --- | --- | --- |
| `results_file` | Required | The batch's output_file_id. An uploaded results file (purpose=batch) also works, with its requests_file. |
| `requests_file` | The batch's input file | The requests the results answer. Needed with an uploaded results file, and once the batch's input file is gone (it is kept 7 days after the batch). |
| `provider` | "auto" | The results file's format: "openai" (a Gridline batch's output file) or "anthropic". "auto" reads it from the file. |
| `require_json` | false | true drops answers that are not valid JSON. |
| `keep_truncated` | false | true keeps answers that stopped at max_tokens. |
| `eval_fraction` | 0.1 | The share of rows held out for the validation file, from 0 up to (not including) 1. |
| `seed` | 0 | Which rows are held out: an integer from 0 to 2,147,483,647. |

Any other field is refused by name. The answer:

```
{
  "object": "gridline.distill",
  "converter": "gridline-distill/1",
  "provider": "openai",
  "training_file": {"id": "file-...", "object": "file", "purpose": "fine-tune", "filename": "distill-train.jsonl", ...},
  "validation_file": {"id": "file-...", "object": "file", "purpose": "fine-tune", "filename": "distill-validation.jsonl", ...},
  "results": 1000,
  "missing_results": 0,
  "dropped": {"truncated": 4},
  "train_rows": 897,
  "eval_rows": 99,
  "min_training_rows": 10
}
```

A result that cannot become a row is dropped and counted in `dropped` by reason: invalid_line, duplicate, unmatched, errored, refused, unsupported, truncated, empty, invalid_json and too_long. `validation_file` is null when no row is held out. Each file names its sources in `gridline.derived_from`.

| Status | Code | When |
| --- | --- | --- |
| 400 | `invalid_request` | The body is not a JSON object, a field is unknown or out of range, or requests_file is missing with an uploaded results file. |
| 400 | `invalid_value` | results_file is not a batch output or batch file, or requests_file is not a batch file. |
| 404 | `not_found` | A file does not exist, or the batch's input file is gone: upload the requests and name them as requests_file. |
| 400 | `invalid_file_format` | The results file's format cannot be told (pass provider), the requests file is not valid, or no result could become a row. |
| 400 | `file_too_large` | The rows are over the 64 MiB training-file limit, or a file written would be over 200 MB: split the batch. |
| 413 | `file_too_large` | A file has over 50,000 lines, or the conversion ran over 120 seconds: split the batch. |
| 429 | `storage_limit_exceeded` | Your account holds the most files allowed, or its file storage is full: delete some files. |

## 4. Fine-tune the student

```
curl https://api.gridlinegpu.com/openai/v1/fine_tuning/jobs \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5-1.5b-instruct", "training_file": "file-...", "validation_file": "file-...", "hyperparameters": {"n_epochs": 2}}'
```

The job needs at least 10 training rows (the answer's `min_training_rows`). Its quote, status and errors are as on the [Fine-tuning](https://gridlinegpu.com/docs/fine-tuning) page. When it is `succeeded`, its `fine_tuned_model` is the student, for example `ft:qwen2.5-1.5b-instruct:o3f9a1c2d4e5:7b2c9e41a0d3`.

To skip step 3, name the batch's `output_file_id` as the job's `training_file`: the job converts it with the defaults first, and uses the held-out rows as its validation file when you name none.

## 5. Run the student

Batches on the `ft:` model work like any other: the same lines, with its name as `model`.

```
{"custom_id": "q-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "ft:qwen2.5-1.5b-instruct:o3f9a1c2d4e5:7b2c9e41a0d3", "max_tokens": 512, "messages": [{"role": "system", "content": "Answer in one short paragraph."}, {"role": "user", "content": "Why does ice float on water?"}]}}
```

Create the batch as in step 2. When it is `completed`, download the results:

```
curl https://api.gridlinegpu.com/openai/v1/files/file-.../content \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -o results.jsonl
```

## Costs

The two batches and the fine-tuning job are each charged at their model's rates; `POST /files/distill` is not charged. For example, with every prompt 200 tokens and every answer 300:

| Step | Tokens | Charge |
| --- | --- | --- |
| Teacher batch, 1,000 prompts | 200,000 input, 300,000 output | $0.306 |
| Fine-tuning, 900 rows for 2 epochs | 900,000 trained | $0.50 (the minimum per job) |
| Student batch, 50,000 prompts | 10,000,000 input, 15,000,000 output | $1.025 |

$1.831 in all. The same 50,000 prompts on `qwen3.8-27b` would be $15.30. The student's answers are not the teacher's: check its eval loss and your own tests before you switch.

What a credit is worth, the credit packs and each model's rates are on the [pricing page](https://gridlinegpu.com/pricing).

## Limits

- A results or requests file holds at most 50,000 lines and 200 MB.
- The training file is at most 64 MiB, and a conversion runs at most 120 seconds. Past either, split the batch.
- The files `POST /files/distill` writes are kept 30 days and count toward your file limits.
- A batch's input file is kept 7 days after the batch ends; after that, name your uploaded requests as `requests_file`.

---

Source: https://gridlinegpu.com/docs/distillation