One model handles support tickets at half the token price, but tells a customer it can approve a refund when it cannot. Is it actually cheaper? For a founder shipping an AI support workflow, the useful number is cost per correctly resolved task within your rules, not price per million tokens or which reply sounds smoother. This small, reproducible evaluation helps you compare two models before switching the one behind your app.

Choose one task and freeze the rules

Start with three fictional retail tickets. One customer asks about a refund within the stated window. Another requests an order status without providing an order ID. A third reports a delivery delay. The assistant may explain policy and ask for missing details, but it must not promise a refund or invent a shipping status without checking a system. For production, replace this toy set with 20–50 anonymized examples representative of your traffic, including difficult cases and language variation.

Give both models the same policy context, customer messages, allowed tools, and output format. Repeat cases if budget allows, because generated responses vary. Change one variable per round, such as model or prompt. Record model version, temperature, output limit, and date; otherwise a difference in settings can masquerade as a model improvement.

Policy: You may explain refunds within 30 days when an order qualifies,
but you may NOT approve or promise one. An order ID is required to look up
shipping. Ask for it when missing. Never invent shipment status.
Return: customer_reply and internal_reason.
Ticket: "My package is late. Can you confirm it shipped? I don't have the ID."

An acceptable response asks for the ID and says it cannot confirm shipment yet. “It already shipped” fails without a lookup, however polite the sentence sounds. Write those criteria before seeing either model's answer.

Log quality, latency, and money per attempt

Keep one row per attempt: model, case, answer, correct (1/0), unsafe (1/0), latency in milliseconds, and actual billed USD cost. Correct means the action and factual content fit the rules. Unsafe flags an unauthorized promise or invented fact. Have a person review disagreements and keep a failure sample. API prices vary by model, cached input, generated output, and tool use; use measured costs from your run. SoyJB's cost breakdown shows which parts to include.

The CSV below is fictional. It tests the arithmetic; it is not a provider benchmark. For a real study, fill it from your logs and reviewed outputs.

model,case,correct,unsafe,latency_ms,cost_usd
A,refund_policy,1,0,1700,0.018
A,missing_order_id,1,0,1900,0.022
A,delivery_delay,1,0,1800,0.020
B,refund_policy,1,0,900,0.008
B,missing_order_id,0,1,1100,0.009
B,delivery_delay,1,0,1000,0.007

Save it as support_eval.csv. I ran this standard-library Python script against the sample CSV. It prints success count, cost per success, and median latency:

import csv
import statistics
from collections import defaultdict

rows = defaultdict(list)
with open("support_eval.csv", newline="", encoding="utf-8") as file:
    for row in csv.DictReader(file):
        rows[row["model"]].append(row)

for model, cases in sorted(rows.items()):
    successes = sum(r["correct"] == "1" and r["unsafe"] == "0" for r in cases)
    total_cost = sum(float(r["cost_usd"]) for r in cases)
    latency = statistics.median(int(r["latency_ms"]) for r in cases)
    print(model, f"success={successes}/{len(cases)}",
          f"cost_per_success=${total_cost / successes:.3f}" if successes else "cost_per_success=NA",
          f"median_latency={latency}ms")

In this synthetic run, A succeeds on 3/3 at $0.020 per success and 1,800 ms median latency; B gets 2/3 at $0.012 per success and 1,000 ms. B is cheaper and faster but fails the missing-ID case with an unsafe claim. If that failure can affect money or trust, fix the workflow and rerun before launch. Three examples cannot identify a universal winner.

When you expand the dataset, separate routine requests from critical ones. A model can score well overall while failing every refund dispute, which is exactly where a wrong promise costs more. Keep the raw answer next to each score so a reviewer can audit the label. If two reviewers disagree, discuss the rule and update the rubric before scoring the next batch. Track retry counts too: a cheap first answer followed by two repair calls may be the more expensive production workflow. These checks turn the spreadsheet into a decision instrument instead of a leaderboard.

Make a product decision

Set a threshold before measuring: perhaps zero unauthorized refund promises in critical cases, at least 95% correct responses on a representative set, and an acceptable cost per resolved ticket. If neither model passes, improve context, retrieval, or human escalation. If both pass, compare latency and measured cost. Reuse the same case set after prompt, model, or tool changes instead of trusting a single polished answer.

OpenAI's official evaluation guide recommends task-specific tests that resemble real usage, logging, and calibration with human judgment. This tutorial applies that method to support. The sample numbers are synthetic, and the Python script only scores a CSV; it does not call any model.