AI - Vercel - Decision-1.1 - automation
4 min lectura

Decision‑1.1 on Vercel: benchmark before you migrate

A practical shadow-mode comparison using historical cases, Brier score, human review, and real account costs to decide whether to switch.

Decision‑1.1 on Vercel: benchmark before you migrate

If an AI model decides which support tickets need a person, a model swap can send a sensitive case down the wrong path. Microsoft Decision‑1.1 became available on Vercel AI Gateway on November 17, 2026. Vercel says it is more accurate, better calibrated, and faster than Decision‑1. It accepts the same request format, so the model identifier can change to microsoft/decision-1.1. That makes a comparison easy to set up; it does not establish a win on your own queue.

Make migration a measurable decision

Start with resolved, anonymized tickets. Define one observable binary question before running either model, such as “Did this case require a human?” For every case, record the known outcome, both model probabilities, response time, API errors, and actual account cost. Keep the input text and question definition fixed. Changing the prompt, threshold, and model together would make the result hard to interpret.

The decision interface supports Boolean, Choice, and Score outputs. Boolean is enough for an initial comparison: one probability per ticket and an outcome labeled 0 or 1. Remove names, email addresses, payment details, and other unnecessary personal data before uploading a case. Keep human review for regulated or high-impact decisions under your own domain policy.

A small comparison you can run this week

  1. Pick about 30 closed cases that include straightforward, urgent, and ambiguous examples. This small set can reveal obvious regressions but cannot establish a production accuracy claim.
  2. Run the same cases against Decision‑1 and Decision‑1.1. The announcement says the request format is shared; still inspect the returned fields and current AI Gateway pricing before wiring either model into a live workflow.
  3. Compare a calibration measure such as Brier score, then inspect every false negative: a ticket that needed a person but received a low probability. Compare account latency and cost as well.
  4. Select a conservative threshold on historical cases. Run the candidate in shadow mode on new tickets, logging predictions without automating routing. Review the shadow results before a migration.

This short Python function scores probabilities against known outcomes. Lower is better and zero is perfect. The values below are invented to demonstrate the calculation; they are not model benchmark results:

def brier(probabilities, labels):
    if len(probabilities) != len(labels) or not probabilities:
        raise ValueError('Aligned, non-empty lists required')
    if any(not 0 <= p <= 1 for p in probabilities):
        raise ValueError('Probabilities must be 0..1')
    if any(y not in (0, 1) for y in labels):
        raise ValueError('Labels must be 0 or 1')
    return sum((p - y) ** 2 for p, y in zip(probabilities, labels)) / len(labels)

print(brier([0.9, 0.2, 0.7], [1, 0, 0]))  # 0.18

I ran that example locally and got 0.18. An aggregate score can hide a severe failure, so inspect the case table, particularly urgent tickets. A 0.49 probability against a 0.50 threshold is not a confident negative. Route borderline cases and independently flagged risks to a person.

Diagram of cases, shared input, two probabilities and human review
Comparison workflow; bars are not real measurements.

Access, price, and what remains unverified

Decision‑1.1 is offered through AI Gateway with one key and the model ID above. Vercel describes budgets, request logs, and fallbacks. Effective access and billing depend on your project. The model page lists token-based pricing, which can change; measure input size before calling it a per-ticket cost. The claims about accuracy, calibration, and speed are Vercel's. No paid model calls or real support-ticket benchmark were run for this article, so it does not name a winner.

Sources: Vercel announcement, Nov 17, 2026; AI Gateway model page.

Comentarios

Aún no hay comentarios. Sé el primero.

Deja un comentario