Distillation

Overview

Distillation trains a small model (the student) on the answers of a large model (the teacher). It takes five calls to the Batch and fine-tuning APIs, at Gridline's base URL with a Gridline token as the API key (see the quickstart for the token):

https://api.gridlinegpu.com/openai/v1

1. Upload your prompts as batch input, purpose=batch.
2. Run a batch on the teacher, qwen3.8-27b.
3. Call POST /files/distill on the batch's output file. It writes a training file and a validation file.
4. Create a fine-tuning job on the student with those files.
5. Run batches on the job's ft: model.

This guide uses qwen3.8-27b (Apache-2.0) as the teacher and qwen2.5-1.5b-instruct as the student. Any of these fine-tuning bases can be the student: qwen2.5-0.5b-instruct, qwen2.5-1.5b-instruct, qwen2.5-7b-instruct, mistral-7b-instruct-v0.3, qwen3-8b and llama-3.1-8b-instruct. You are responsible for having the right to train on what you upload, including model outputs.

1. Upload prompts

Each line is a batch request to /v1/chat/completions on the teacher, as on the Batch API. Its last message must be a user message. On qwen3.8-27b a system message may only come first.

{"custom_id": "q-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "qwen3.8-27b", "max_tokens": 512, "messages": [{"role": "system", "content": "Answer in one short paragraph."}, {"role": "user", "content": "Why does ice float on water?"}]}}

Set max_tokens high enough for a whole answer (qwen3.8-27b writes at most 1,024 tokens a request): an answer that stops at the limit is dropped in step 3 unless you pass keep_truncated.

curl https://api.gridlinegpu.com/openai/v1/files \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -F purpose=batch \
  -F [email protected]

2. Run the teacher

curl https://api.gridlinegpu.com/openai/v1/batches \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"input_file_id": "file-...", "endpoint": "/v1/chat/completions", "completion_window": "24h"}'

Poll GET /batches/{batch_id} until its status is completed. Its output_file_id is the results file for the next step:

{"id": "batch_...", "object": "batch", "status": "completed", "input_file_id": "file-...", "output_file_id": "file-...", "request_counts": {"total": 1000, "completed": 1000, "failed": 0}, ...}

3. Make training files

curl https://api.gridlinegpu.com/openai/v1/files/distill \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"results_file": "file-..."}'

Each result becomes one training row: the request's messages, then the teacher's answer as the assistant message. The rows go to a training file and, held out, a validation file, both purpose=fine-tune. The same files and options always give the same rows. The body takes:

Any other field is refused by name. The answer:

{
  "object": "gridline.distill",
  "converter": "gridline-distill/1",
  "provider": "openai",
  "training_file": {"id": "file-...", "object": "file", "purpose": "fine-tune", "filename": "distill-train.jsonl", ...},
  "validation_file": {"id": "file-...", "object": "file", "purpose": "fine-tune", "filename": "distill-validation.jsonl", ...},
  "results": 1000,
  "missing_results": 0,
  "dropped": {"truncated": 4},
  "train_rows": 897,
  "eval_rows": 99,
  "min_training_rows": 10
}

A result that cannot become a row is dropped and counted in dropped by reason: invalid_line, duplicate, unmatched, errored, refused, unsupported, truncated, empty, invalid_json and too_long. validation_file is null when no row is held out. Each file names its sources in gridline.derived_from.

4. Fine-tune the student

curl https://api.gridlinegpu.com/openai/v1/fine_tuning/jobs \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen2.5-1.5b-instruct", "training_file": "file-...", "validation_file": "file-...", "hyperparameters": {"n_epochs": 2}}'

The job needs at least 10 training rows (the answer's min_training_rows). Its quote, status and errors are as on the Fine-tuning page. When it is succeeded, its fine_tuned_model is the student, for example ft:qwen2.5-1.5b-instruct:o3f9a1c2d4e5:7b2c9e41a0d3.

To skip step 3, name the batch's output_file_id as the job's training_file: the job converts it with the defaults first, and uses the held-out rows as its validation file when you name none.

5. Run the student

Batches on the ft: model work like any other: the same lines, with its name as model.

{"custom_id": "q-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "ft:qwen2.5-1.5b-instruct:o3f9a1c2d4e5:7b2c9e41a0d3", "max_tokens": 512, "messages": [{"role": "system", "content": "Answer in one short paragraph."}, {"role": "user", "content": "Why does ice float on water?"}]}}

Create the batch as in step 2. When it is completed, download the results:

curl https://api.gridlinegpu.com/openai/v1/files/file-.../content \
  -H "Authorization: Bearer $GRIDLINE_TOKEN" \
  -o results.jsonl

Costs

The two batches and the fine-tuning job are each charged at their model's rates; POST /files/distill is not charged. For example, with every prompt 200 tokens and every answer 300:

$1.831 in all. The same 50,000 prompts on qwen3.8-27b would be $15.30. The student's answers are not the teacher's: check its eval loss and your own tests before you switch.

What a credit is worth, the credit packs and each model's rates are on the pricing page.

Limits

• A results or requests file holds at most 50,000 lines and 200 MB.
• The training file is at most 64 MiB, and a conversion runs at most 120 seconds. Past either, split the batch.
• The files POST /files/distill writes are kept 30 days and count toward your file limits.
• A batch's input file is kept 7 days after the batch ends; after that, name your uploaded requests as requests_file.