A newly released model wins an impressive public benchmark. Your product has a smaller and more concrete job: route incoming messages to sales, support or spam. The decision is which model makes fewer mistakes on your traffic, especially the mistakes that upset customers. Start with twenty examples from your workflow, a human-approved label for each and a small local grader.
OpenAI's evaluation guidance recommends defining success before measuring, using representative production cases alongside expert examples and avoiding overly generic metrics. This tutorial applies that method without requiring an evaluation service from any model vendor. The code below scores answers you collect yourself. It does not call either model, and the sample numbers are synthetic.
1. Build a dataset that resembles your work
Select twenty recent messages, remove names, email addresses and other private details, and confirm you may use them for an internal test. Include straightforward and borderline examples: a pricing question, a bug report, a complaint that also asks about an upgrade, and disguised promotional spam. If actual traffic is mostly support, keep roughly that mix or say explicitly that your sample was balanced for testing. Twenty examples can uncover obvious failures; they cannot establish broad statistical superiority.
Write the expected label before you read either model's output. Define tie-breaking rules. For example, an existing customer's broken feature goes to support even if the message also asks about a paid plan. Have another person review ambiguous cases. If two people disagree, fix the rubric before you compare models. A shifting “gold” answer makes the score meaningless.
2. Give both models the same work
Use the same instruction, allowed labels and case texts for A and B. Keep output settings as comparable as the providers permit, and record model version, date, temperature, token limit, cost and latency when available. Do not tune B's prompt after seeing its errors unless you rerun A under the same change. Ask for one label per message. If a model returns an explanation anyway, normalize the label manually for the CSV while preserving the raw answer for diagnosis.
This four-row synthetic sample makes the scoring reproducible. Replace it with your twenty human-labeled cases and both model outputs. The label set is sales, support and spam:
case_id,expected,model_a,model_b
ex-01,sales,sales,sales
ex-02,support,support,sales
ex-03,spam,spam,spam
ex-04,support,sales,sales

3. Grade locally and inspect disagreements
Save the CSV as results.csv, save this file as compare_models.py and run python3 compare_models.py results.csv. The script uses only Python's standard library:
import csv
import sys
from collections import Counter
LABELS = {'sales', 'support', 'spam'}
REQUIRED = {'case_id', 'expected', 'model_a', 'model_b'}
def main(path):
with open(path, newline='', encoding='utf-8') as handle:
reader = csv.DictReader(handle)
if not REQUIRED.issubset(reader.fieldnames or []):
raise ValueError(f'Missing columns: {sorted(REQUIRED - set(reader.fieldnames or []))}')
rows = list(reader)
if not rows:
raise ValueError('No test cases')
ids = [row['case_id'].strip() for row in rows]
if any(not case_id for case_id in ids) or len(ids) != len(set(ids)):
raise ValueError('Empty or duplicate case_id')
for row in rows:
for field in ('expected', 'model_a', 'model_b'):
row[field] = row[field].strip().lower()
if row[field] not in LABELS:
raise ValueError(f"{row['case_id']}: invalid {field}={row[field]!r}")
for model in ('model_a', 'model_b'):
correct = sum(row[model] == row['expected'] for row in rows)
by_label = Counter(row['expected'] for row in rows)
print(f'{model}: {correct}/{len(rows)} correct ({correct/len(rows):.1%})')
print(' gold labels:', ', '.join(f'{label}={by_label[label]}' for label in sorted(LABELS)))
disagreements = [row['case_id'] for row in rows if row['model_a'] != row['model_b']]
print('disagreements for review:', ', '.join(disagreements) or 'none')
if __name__ == '__main__':
if len(sys.argv) != 2:
raise SystemExit('Usage: python3 compare_models.py results.csv')
main(sys.argv[1])
We ran the script on the synthetic CSV. It reported A at 3/4 correct (75%), B at 2/4 (50%) and ex-02 as a disagreement. It checks for missing columns, repeated IDs and invalid labels. These numbers verify the calculation, not either real model's quality. With twenty cases, one mistake moves the score by five percentage points. Read the individual errors before announcing a winner.
4. Choose based on the failure that matters
Misrouting a customer who needs urgent support into a sales queue may cost more than sending a sales question to support. Make a small error table by category and mark severe failures. Prefer a model only if it clears the critical-quality bar, fits your budget and responds quickly enough for the workflow. If scores are close, collect more representative cases and rerun the same inputs to see whether the result is stable. Keep the CSV and prompt version so future model updates can be checked for regressions.
This grader measures exact label matches. It does not measure writing quality, privacy, safety or business outcomes. For open-ended answers, use a human rubric with examples of acceptable and unacceptable responses. In either case, choose using the work you will actually ship, not a benchmark headline.
Source: OpenAI evaluation best practices. The CSV, script and diagram are original SoyJB examples; no paid model test was performed.
