LLM Evaluation Metrics: Which Ones to Track Before You Ship
The LLM evaluation metrics worth tracking before you ship are the few that tell you whether one feature does its job for a user. That usually means a format pass rate, an accuracy check against known answers, a rubric score for open-ended text, and product numbers like task success, refusal rate, latency and cost per answer. At the early stage, most other scores are noise. This post shows you how to pick three to five metrics for one feature and turn them into a release gate.
Age of AI Entrepreneur teaches you how to build, launch and grow a business with AI in short hands-on lessons with real sources, and evals have a course of their own. In Evals: Shipping AI That Works you get three lessons: why your AI product needs evals, how to build an eval set and judge outputs, and how to run AI in production. Sign in with your email and an emailed code to open it.
Why most LLM evaluation metrics mislead early AI products
Public benchmarks measure general skill, not your task. A model can top a reasoning leaderboard and still break your JSON one time in twenty. A single average also hides the one failure that costs you a customer.
Each metric should answer one question: can we ship this version? If a number can't change that answer, skip it for now.
Here is the example used in the rest of this post. You are building a feature that reads an inbound customer email and returns a JSON object. The object has three fields: a category (billing, bug, account or other), a priority (low, normal or urgent), and a reply that holds a two-sentence draft answer. You have labeled 80 real emails with the right category and priority.
Deterministic metrics: exact match, format checks and pass rates
Start with checks a script can run with no judgment at all: cheap, fast and the same every time.
- Format check: does the output parse as JSON, and does it contain all three fields? Something like
json.loads(output)followed by a key check is enough. - Allowed values: is
categoryone of your four labels? Models sometimes invent a fifth one, like "shipping." - Exact match: does the predicted category equal your label? Do the same for priority.
- Pass rate: the share of cases that pass every check.
Report each check on its own line, not only the combined pass rate. "Format 100%, category 91%, priority 84%" tells you where to work next. "Pass rate 78%" doesn't. Model outputs can vary from run to run, so run the full set more than once before you trust a small difference.
Reference metrics like BLEU, ROUGE and semantic similarity, and when they fail
Reference metrics compare the model's text to a "gold" answer you wrote. BLEU came from machine translation (Papineni and colleagues, 2002) and counts overlapping word sequences. ROUGE came from summarization (Lin, 2004) and measures how much of the reference shows up in the output. Semantic similarity embeds both texts as vectors and compares how close they are.
For open-ended answers, they fail in two directions. Say the reference reads "We have issued a refund for the duplicate payment." A good draft that says "Sorry about the double charge, your money is on its way back" shares almost no words with it, so BLEU scores it low. Now take a draft that says "We have issued a refund of $4" when the real amount was $40. It looks nearly identical to a correct reply, so similarity scores it high. The metric punishes a right answer and rewards a wrong one.
Use reference metrics when the output should stay close to a fixed text, such as a template or a translation, or as a cheap drift alarm between versions, not as the main score for a chat or writing feature.
Rubric scores and model-graded metrics for open-ended answers
For the draft reply, write a short rubric of yes or no questions. Yes or no criteria give steadier results than a 1 to 10 scale, because "is this a 6 or a 7?" has no clear answer.
- Does the reply address the customer's actual problem?
- Does it avoid promising anything the email doesn't support, such as a refund amount or a date?
- Is it two sentences or fewer?
- Is the tone polite and plain?
You can grade a few dozen cases by hand. To grade hundreds, give a second model the rubric and ask it to act as a judge:
You are grading a draft support reply. Here is the customer email, then the draft. For each of the four criteria below, answer YES or NO and give one sentence of reason. Do not reward length. Criteria: ...
Don't trust the judge until you have checked it. Grade 30 cases yourself, run the judge on the same 30, and read every case where you disagree. Research on model judges (Zheng and colleagues, 2023) examines biases toward longer answers, toward the position an answer appears in and toward a model's own answers, while reporting that strong judges agreed with human preferences over 80% of the time. Fix the prompt until the judge agrees with you on the cases that matter. Our post on LLM as a judge goes further.
Product metrics that matter: task success, refusal rate, latency and cost per answer
Quality scores tell you whether the text is good. Product metrics tell you whether the feature is worth running.
- Task success: did the user get what they came for? After launch, this might mean the agent sent the draft with few edits. Before launch, use the share of test cases that pass every check and every rubric item.
- Refusal rate: the share of in-scope emails the model declines or answers with a vague hedge. Track the opposite failure too: emails it should have flagged but answered anyway.
- Latency: track the slow tail, such as the 95th percentile, not the average. Users remember the slowest response they got.
- Cost per answer: input tokens plus output tokens, multiplied by your model's price, averaged over the eval set. The AI Product Stack course has a lesson on model APIs, tokens and what a request costs if you need to work this out.
How to choose three to five metrics for one AI feature
Use this short process for each feature:
- Write down how the feature can fail a user, in plain words.
- Give each failure one metric.
- Pick a deterministic check over a judged one wherever you can.
- Add one product metric for cost or speed.
- Cut any metric that wouldn't change your ship decision.
For the email triage feature, the failures are broken output, wrong routing, a missed urgent case, a bad draft and runaway cost. That gives you five metrics:
- Format pass rate
- Category accuracy
- Urgent recall, meaning the share of truly urgent emails marked urgent
- Reply rubric pass rate
- Cost per answer

Latency was dropped: triage runs in the background, so a two-second delay hurts no one. Urgent recall got its own line because a missed urgent email does far more harm than a misfiled billing question, and overall priority accuracy would hide that.
Turning metrics into a release gate your team trusts
A release gate is a short list of thresholds a new version must meet before it ships. Teams trust it when the rules are set in advance and the gate never changes to fit the result.
- Freeze and version the eval set. If the set keeps changing, you can't compare two scores.
- Set thresholds before you run. For example: format 100%, category at least 90%, no urgent email missed, rubric pass at least 85%, and cost per answer under the budget you set.
- Compare against the current version, not only the threshold. Someone reads every case that passed before and fails now, even if the totals still clear the gate.
- Grow the set from real failures. Each bad output a user reports becomes a new labeled case.
The comparison step is regression testing, covered in this guide to LLM regression testing. Most of all, read the failures: a number tells you something broke, and the failed cases tell you why.
Frequently asked questions
What is the most important LLM evaluation metric?
No single metric fits every feature. For most products, task success matters most, because it measures whether the user got what they needed. Build it from the checks and rubric items that define success for your feature.
Are BLEU and ROUGE still useful for evaluating LLMs?
They help when the output should match a fixed text closely, as in translation or templated answers, and they work as a cheap alarm for changes between versions. For open-ended answers they often score correct text low and wrong text high, so use a rubric instead.
How many test cases do I need for metrics to be reliable?
Do the arithmetic for your set: with 50 cases, each case moves a score by 2 points, so a 2 point change is a single case. Start with real examples that cover every category and your worst failure, then add cases from production until differences between versions are larger than a handful of cases.
Should I track the same metrics in production and before release?
Keep the same definitions so the numbers line up. In production you lose exact labels, so lean on format checks, refusal rate, latency, cost and user signals like edits, and feed failures back into your eval set.
Get started
Evals: Shipping AI That Works is one of the Advanced courses. Its three lessons, each about 9 minutes, take you from why evals matter, to building an eval set and judging outputs, to running AI in production. Every lesson lists its sources. Questions from lessons you finish come back later in spaced review, so the ideas stick. Sign in with your email and an emailed code to open the course.
Start Age of AI Entrepreneur: short hands-on lessons on how to build a business in the age of AI, with every side of the debate and the sources to read next.
Building a business in the age of AI, from a first project to a company.
Visit aientrepreneur.vlvd.net↗


Comments
No comments yet.