Evals

Overview

1. Create an eval: how each row is graded (its testing criteria) and the shape of your rows.
2. Start a run on a model, a catalog model or one of your ft: models, over a file of rows. Gridline sends the rows to the model as a batch, then grades each answer.
3. Read the run's pass and fail counts, then each graded row.

The routes and objects are OpenAI's, under Gridline's base URL, with a Gridline token as the API key (see the quickstart for the token). Code written for OpenAI's client.evals runs with the base URL changed, within the differences below.

https://api.gridlinegpu.com/openai/v1

Create an eval

An eval holds up to 8 testing criteria. Templates read the row with {{item.<field>}} and the model's answer with {{sample.output_text}}. With the OpenAI Python SDK:

import os
from openai import OpenAI

client = OpenAI(base_url="https://api.gridlinegpu.com/openai/v1", api_key=os.environ["GRIDLINE_TOKEN"])

support_eval = client.evals.create(
    name="support-answers",
    data_source_config={
        "type": "custom",
        "item_schema": {
            "type": "object",
            "properties": {"question": {"type": "string"}, "ideal": {"type": "string"}},
        },
        "include_sample_schema": True,
    },
    testing_criteria=[
        {
            "type": "string_check",
            "name": "exact",
            "input": "{{sample.output_text}}",
            "reference": "{{item.ideal}}",
            "operation": "eq",
        },
        {
            "type": "text_similarity",
            "name": "close",
            "input": "{{sample.output_text}}",
            "reference": "{{item.ideal}}",
            "evaluation_metric": "fuzzy_match",
            "pass_threshold": 0.8,
        },
        {
            "type": "score_model",
            "name": "judge",
            "model": "qwen2.5-7b-instruct",
            "input": [
                {"role": "system", "content": "Score from 0 to 1 how well the answer matches the ideal answer. Reply with the number only."},
                {"role": "user", "content": "Question: {{item.question}}\nIdeal: {{item.ideal}}\nAnswer: {{sample.output_text}}"},
            ],
            "range": [0, 1],
            "pass_threshold": 0.7,
        },
    ],
)

A row passes when every criterion passes. Update an eval's name or metadata with POST /evals/{eval_id}; its criteria are fixed.

Items

One JSON object per line, each with an item. Upload the file with purpose="batch", or send up to 500 rows inline as file_content:

{"item": {"question": "How do I reset my password?", "ideal": "Use the Forgot password link on the sign-in page."}}
{"item": {"question": "Can I change my plan mid-month?", "ideal": "Yes. The change applies from the next billing day."}}

items = client.files.create(file=open("support-items.jsonl", "rb"), purpose="batch")

A run grades the first 500 usable rows. A row that is not valid JSON, has no item, or makes a request the batch API would refuse is skipped and counted. To grade answers you already have, give each row a sample ({"output_text": "…"}) and run with a jsonl data source: nothing is sent to a model.

Run it

A completions run renders each row into chat messages, sends them to the model as one batch, then grades the answers. Ask for the quote first with dry_run (a Gridline field: it answers the run's maximum charge and writes nothing):

run_request = {
    "name": "base model",
    "data_source": {
        "type": "completions",
        "model": "qwen2.5-7b-instruct",
        "input_messages": {
            "type": "template",
            "template": [
                {"role": "system", "content": "Answer the customer in one sentence."},
                {"role": "user", "content": "{{item.question}}"},
            ],
        },
        "sampling_params": {"temperature": 0, "max_completion_tokens": 128},
        "source": {"type": "file_id", "id": items.id},
    },
}

quote = client.post(
    f"/evals/{support_eval.id}/runs",
    body={**run_request, "dry_run": True},
    cast_to=object,
)
print(quote["quote"]["max_charge_usd"])

run = client.evals.runs.create(support_eval.id, **run_request)

Poll the run until its status is completed, failed or canceled. It carries OpenAI's result_counts, per_testing_criteria_results and per_model_usage, and a gridline object with its batch ids, its results file and its quote:

import time

while run.status in ("queued", "in_progress"):
    time.sleep(30)
    run = client.evals.runs.retrieve(run.id, eval_id=support_eval.id)

print(run.result_counts)  # passed, failed, errored, total
for item in client.evals.runs.output_items.list(run.id, eval_id=support_eval.id, status="fail"):
    print(item.datasource_item["question"], item.results)

Cancel a run with POST /evals/{eval_id}/runs/{run_id} (client.evals.runs.cancel); its batches are cancelled. Output items are kept 30 days after the run ends.

Evaluate a fine-tune

Run the same eval twice over the same items: once on your ft: model and once on its base model. Compare the two runs' result_counts and per_testing_criteria_results.

for model in ["ft:qwen2.5-7b-instruct:o3f9a1c2d4e5:7b2c9e41a0d3", "qwen2.5-7b-instruct"]:
    request = {**run_request, "name": model}
    request["data_source"] = {**run_request["data_source"], "model": model}
    client.evals.runs.create(support_eval.id, **request)

Use your model's own id from GET /models or the succeeded job's fine_tuned_model. Hold out rows you did not train on, or the comparison measures memory, not quality.

Graders

string_check

Compares the rendered input with the rendered reference. Passes or fails (1 or 0).

text_similarity

A score from 0 to 1, passing at or above pass_threshold. Metrics: fuzzy_match, bleu, rouge_1, rouge_2, rouge_3, rouge_4, rouge_5, rouge_l. Not supported yet: gleu, meteor, cosine. fuzzy_match and rouge_l compare at most the first 1000 characters or words of each side. BLEU and ROUGE split text into lower-cased runs of letters and digits, so scores can differ slightly from OpenAI's on punctuation-heavy text.

score_model

A catalog chat model reads your grading messages for each row and replies with a number within range (default 0 to 1); the row passes at or above pass_threshold. A reply that is not a number in range errors the row; it is never guessed. Each eval uses at most 2 distinct score models, and a score model cannot be an ft: model. Its reply is at most 512 tokens (default 64).

What it costs

• A run's model calls are ordinary batches in your account: one for the answers, and one per score model for the grading. Each is billed at the published batch prices on /pricing, as any batch, and appears in GET /batches with its receipt.
• Each batch costs at least its model's per-batch minimum ($0.20 on qwen2.5-7b-instruct). So a small run on one model costs at least one minimum, and two with a score model.
• A jsonl run graded only by string_check and text_similarity sends nothing to a model and costs nothing.
• The quote (dry_run, and the run's gridline.quote) is an upper bound: every answer at its full max_completion_tokens. You pay what the batches actually use.
• A run is refused before anything is created when your credit is below its quote. A batch that fails or is cancelled ends the run and is billed as any batch with that end.

Limits

• 500 rows per run; 500 rows inline.
• 8 testing criteria per eval, 2 distinct score models.
• Answers up to 4096 tokens (max_completion_tokens, default 1024).
• 100 evals per account, 20 runs kept per eval, and 2 runs open at a time per account.
• Output items are kept 30 days after a run ends; the run itself is deleted 90 days after it ends. An eval stays until you delete it.

Differences from OpenAI

• Data sources are jsonl and completions. The responses data source is refused.
• data_source_config is custom. The logs and stored_completions types are refused: Gridline does not store completions.
• Graders are string_check, text_similarity and score_model. The python and label_model graders are refused.
• A score_model grader runs on a catalog chat model, not an ft: model.
• Upload items with purpose batch (or use a fine-tune file). Files with purpose evals are not accepted yet.
• item_schema is stored and echoed; items are not checked against it.
• report_url is null: there is no dashboard view of a run yet.

Errors

Refusals use OpenAI's error body; error.message says what to change.