LLM as a Judge: How to Grade AI Outputs Without Fooling Yourself

LLM as a judge means you use one language model to grade the outputs of another, following a written rubric, so you can check hundreds of answers without reading each one yourself. It works well when the judge gets narrow, concrete questions and you check it against a few human labels. It fails when you ask for a vague score and trust it blindly.

This post shows how to set up a judge you can trust, with a worked example you can copy. The same ideas are taught in the Evals: Shipping AI That Works course, where you build an eval set and practice judging outputs. Open the Evals: Shipping AI That Works course (you sign in first, with your email and an emailed code) to find its three lessons: why your AI product needs evals, building an eval set and judging outputs, and running AI in production.

Age of AI Entrepreneur gives you short hands-on lessons on building and launching a business with AI, and the evals course is where it covers the quality side of that work.

What LLM as a judge means and when it beats human review

Say your product produces text: support replies, summaries or answers from your own documents. You cannot unit test text the way you test a function, because there are many correct answers and many ways to be wrong.

You have three options: read every output yourself (accurate, does not scale), use string rules such as "contains the word refund" (scales, misses meaning), or ask a model to read each output and answer questions about it.

The model grader wins in a specific spot. You have a few hundred outputs to check after every prompt change, the question is about meaning rather than exact wording, and a human reading all of them each time would be too slow or too costly. It does not replace human review. It extends a small amount of human judgment across a large number of cases.

Keep human review for the first pass, the cases the judge is unsure about, and checking the judge itself.

Where a model grader fails: position, verbosity and self-enhancement bias

A judge is a model, so it has habits. The MT-Bench paper on model judges names position, verbosity and self-enhancement biases, plus limited reasoning ability, and reports that strong judges such as GPT-4 agreed with human preferences more than 80% of the time. Agreement is good, not perfect, so know the habits before you trust any score.

Three judge biases and a fix for each.

• Position bias. Shown two answers, the judge can lean toward one position. Run each pair twice with the order swapped and only count results that agree.
• Verbosity bias. Judges can favor longer answers even when the short one is right. If your product should be brief, say so in the rubric.
• Self-enhancement. A model can favor text in its own style. When you can, use a judge from a different model family than the writer.

There is a fourth failure that is your fault, not the model's: a vague question. "Is this a good answer?" lets the judge decide what good means, and it will decide differently on different days.

Write a grading rubric a model can follow

A rubric is the instruction text you give the judge. Think of it as a brief for a careful new hire who has never seen your product. Say what the product does, what the input was, what the output was, and exactly what to look for.

Here is a worked example. Say your product drafts replies to customer emails using your help documents. One check might be: does the reply make a claim that is not in the provided documents?

You are checking a drafted customer reply. You will see the customer email, the help document excerpts the writer was given, and the draft reply. Question: does the draft state any fact, price, deadline or policy that does not appear in the excerpts? Answer FAIL if it does, and quote the unsupported sentence. Answer PASS if every factual claim is supported or the draft makes no factual claims. Give your reason in one sentence first, then the verdict on its own line.

This rubric names the evidence the judge may use, defines both outcomes, asks for a quote so the judge points at something real, and asks for the reason before the verdict.

Write one rubric per quality you care about. A reply can be accurate and rude, or polite and wrong; cram both into one question and you cannot tell which broke.

Pick pass/fail checks over vague 1 to 10 scores

A score of 7 out of 10 means nothing until you define the difference between 6, 7 and 8. A 7 today may be a 6 tomorrow, and you will chase noise.

Binary checks are easier to write, audit and act on. Either the reply invented a policy or it did not. You can then report a clean number such as "41 of 50 replies passed the grounding check" and watch it move as you change the prompt.

If you need a scale, keep it to three levels with a written example of each, such as "fully answers", "partly answers" and "does not answer". To compare two prompt versions, a pairwise question ("which reply follows the policy better, A or B?") works if you swap the order and run both ways.

Calibrate the judge against a small set of human labels

This is the step most people skip, and it is the one that stops you fooling yourself. Before you rely on the judge, check that it agrees with you.

1. Pick 30 or so outputs from your eval set. Include some obvious passes, some obvious fails and some borderline ones.
2. Label each one yourself with PASS or FAIL, without looking at the judge's answer.
3. Run the judge on the same outputs.
4. Line them up and count where you disagree.

Look at the kind of disagreement, not just the count. If the judge passes things you would fail, it is too lenient, and bad outputs will slip through. If it fails things you would pass, it is too strict, and you will waste time fixing non-problems. Read each disagreement, find the sentence in the rubric that caused it, and tighten it. Add a short example of a borderline case to the rubric if that helps.

Repeat until the disagreements are rare and explainable. Then keep those labeled examples. They become the test for your judge, so whenever you change the judge prompt or its model, you can rerun them and see if it still agrees with you.

Cut cost and noise: cheaper judge models, sampling and repeat runs

Grading everything with your most expensive model gets costly fast. A few habits keep it under control.

• Try a smaller judge. Narrow yes/no checks are often easy enough for a cheaper model. Test it against your human labels. If agreement holds, keep it. If you are unsure how to compare models, this guide on how to choose an LLM for your AI product covers the trade-offs.
• Sample in production. Grade a random slice of live outputs each day and watch the trend.
• Repeat borderline cases. If a judge flips between runs on the same output, take the majority of three runs, or send the case to a human.
• Keep settings steady. Use the same judge model, prompt and low randomness, so a score change comes from your product and not the grader.

Log the judge's reason with the verdict, so a score drop explains itself. For what else to record, see LLM observability: what small AI teams should log and track.

Plug the judge into your eval set and release checks

A judge is only useful when it runs on a fixed set of cases every time. Build an eval set from real inputs: typical ones, hard ones, and every failure a customer has reported. Each time you change a prompt, a model or your retrieval setup, run the set through the judge and compare the pass rates with the last run.

Set a rule before you look at results. For example: the grounding check must pass on at least as many cases as the previous release, and no case that passed before may now fail without someone reading it. That turns the judge into a gate, next to the manual steps in this checklist for testing LLM prompts before release. Once the product is live, keep sampling real outputs and add new failures to the eval set.

Frequently asked questions

Can a model grade its own output?

It can, but it tends to favor its own style and miss its own mistakes. Use a different model as the judge when you can, and check it against human labels either way.

How many human labels do I need to trust a judge?

There is no magic number. Start with about 30 examples that include clear passes, clear fails and borderline cases, and add more wherever the judge and you disagree.

Is LLM as a judge good enough for production decisions?

It is good enough to catch regressions and rank versions when you have calibrated it and use narrow pass/fail checks. For high-stakes cases, keep a human in the loop and treat the judge as a filter.

Which model should I use as the judge?

Pick one that agrees with your human labels on your own cases, not the one with the best reputation. Start with a capable model, then try a cheaper one and keep it only if agreement holds.

Get started: turn your eval set into a quality gate with the Evals and Quality course

You now have the pieces: a narrow rubric, pass/fail checks, a small set of human labels, and a rule for when a release is allowed. Start small. Pick one quality that matters for your product, write one rubric, label 30 outputs and see how far the judge is from you.

To go through this step by step, open the Evals: Shipping AI That Works course. Sign in with your email and an emailed code to open it. If you are still deciding what to build, the 12 course path from zero to launch shows where evals fit.

Start Age of AI Entrepreneur: short hands-on lessons on how to build a business in the age of AI, with every side of the debate and the sources to read next.