Confession: I ran an LLM judge for six weeks before I noticed it was rewarding the longer answer roughly every time. Not the better one. The longer one.
I found out on a Tuesday. I was pairing my eval pipeline against a colleague’s spreadsheet of hand-graded outputs, and I ran a quick sanity check. What happens if I ask the judge to compare a good answer to a padded version of the same good answer, with three extra sentences of restated context? The judge picked the padded one 74% of the time. I had shipped that pipeline. Live traffic was routing through it. Cool cool cool.
That week broke my faith in LLM-as-a-judge as a standalone metric, and it made me rebuild what I actually track. If you run any kind of LLM evaluation in production, or you are thinking about it, this is what I have kept and what I have thrown away.
The two biases that made me stop trusting LLM judges
The bias I hit is not new. It is just easy to miss because judge scores look clean, and clean numbers are seductive. A recent paper on Bayesian active top-k ranking with LLM judges puts it plainly: LLM judges are noisy and systematically biased. They favor verbose answers. They favor answers shown in specific positions. The paper’s pitch is a Bayesian model that treats verbosity and position as covariates so you can back them out, which is smart, but the message under the math is the important part. If you aggregate raw judge votes, you are ranking presentation, not quality.
The original LLM-as-a-judge paper already flagged position bias in 2023. It did not stop me from ignoring it, because most tutorials I read shipped a “just call GPT with this prompt” example and moved on. That is the trap. The trick is easy. The interpretation is hard.
Two biases I saw in my own runs. Verbosity bias: the longer answer wins even when it is padding. Add three sentences that restate the question, and the judge scores it higher. Position bias: swap the order of the two candidates and the winner flips in roughly a third of comparisons on my data. If your eval does not randomize order or run both orderings, your ranking is partly a coin flip.
LLM evaluation metrics I actually keep on the wall
I stopped treating judge scores as the metric. Now they are one signal in a stack. Here is what I track for a text generation pipeline in production, in rough order of how much I trust each one.
Task-specific programmatic checks. These are boring and cheap. Does the JSON parse. Does the SQL run without error. Does the citation URL return a 200. Does the returned month name match the input date. If your output has any structure you can validate, validate it. This is the single highest-value metric I run, and it has zero LLM in the loop.
Reference-based similarity when I have references. BLEU and ROUGE get a bad rap and they deserve some of it for open-ended text, but if I am evaluating a translation or a paraphrase task where the answer space is narrow, they still catch drift. Semantic similarity via a small embedding model is my default when references exist and are open-form.
Human review on a rotating sample. Not one big review. A tiny one, every day. I sample ten to twenty completions, hand-grade them against a rubric, and log the results in the same table as my automated scores. Trends matter more than absolute numbers. If human agreement with the judge slips week over week, I know before the users do. I write about this the same way I catch hallucinations: a small habit beats a big audit.
Refusal and safety triggers. Counts of “I can’t help with that”, counts of hedged answers, counts of empty outputs. This is not fancy. A regex catches most of it.
Judge scores, but only after I have calibrated them. More on that below.
Latency and cost per successful completion. If your judge says two systems are tied on quality but one costs three times as much, you do not have a tie. This is the metric I forget most often and regret most.
The five-minute judge audit I run before I trust any score
Before I trust a judge score for anything, I run a quick audit. Total time is under five minutes if the harness is already wired up. If you do not have this, wire it up before your next eval.
First, the swap test. Take fifty comparison pairs. Score each pair in both orders (A vs B and B vs A). Count how often the winner flips. If it is above 15% you have serious position bias, and you should either force randomization plus both-order scoring, or find a different judge.
Second, the padding test. Take twenty answers your judge scored highly. For each one, append two sentences of restated context that add no real content. Rescore. If the padded versions consistently win by a wider margin, you have verbosity bias. This one is depressingly common.
Third, the null comparison. Give the judge an answer paired with a slightly reworded version of itself. It should call it a tie most of the time. If it strongly prefers one, your judge is grasping at surface features.
Fourth, the reference disagreement check. Take twenty items where you have a human label. Compare judge agreement with human labels. Below 65% is where I start getting nervous. Below 55% and I would say the judge is doing something else, not what I asked.
Fifth, the cost math. Multiply your per-item judge call cost by the number of items you want to evaluate per week. If that number makes you flinch, batch or downsample now, not in three months when the finance team asks.
Mixing human review, programmatic checks, and judges without overpaying
Here is the split I settled on for a mid-sized eval, say a few thousand outputs per week.
Programmatic checks run on everything. They are free-ish and they catch the biggest failures first. If the JSON will not parse you do not need a judge to tell you it is a bad answer.
The judge runs on a stratified sample. Maybe 15 to 25 percent of items, biased toward the interesting cases. Anything where a programmatic check failed. Anything where confidence signals were low. Anything from a new deployment. I do not run the judge on everything. It is expensive and, since I know it has biases, running it on more data does not produce more truth. It produces more noise.
Human review sits on the tail. A small daily sample of the hard cases, plus a weekly sample of items where the judge and the programmatic checks disagreed. That last bucket is gold. Disagreements are where I learn.
I have written more about this trade-off in how I picked between a fine-tuned small model and a GPT-class one. Short version: paying the judge tax on every item is rarely worth it, for the same reason paying a giant model for every inference is rarely worth it.
If you want to see how I string this together in real projects, I keep a couple of write-ups in my work portfolio that show the pipeline end to end.
Where LLM-as-a-judge still earns its keep
I am not going to pretend judges are useless. I still use them. Here is where they hold up.
Comparing two prompt variants on the same task with strong controls. Both-orders scoring. Fixed rubric. Held-out data. If the judge picks A over B by a wide margin under those controls, I trust it enough to ship the variant.
Catching obvious quality regressions after a model swap. If yesterday’s scores were 8s and today’s are 5s, something changed. The absolute number is fuzzy. The delta is useful.
Grading long-form outputs where nothing else fits. Reference-based metrics do not help me on open-form writing tasks. A calibrated judge is still better than nothing there, as long as I trust the calibration.
Cases where judges fail, and where you should stop trusting them, are basically anywhere the answer’s length or presentation could vary independent of quality. Summaries. Explanations. Anything with room to pad.
Try this one check this week
If you already have an eval pipeline, run the swap test on your last hundred comparisons. Score every pair in both orders and count the flips. It takes about ten minutes to code. If more than 15% flip, you do not have a ranking. You have a ranking plus a lottery.
If you do not have an eval pipeline yet, do not start with a judge. Start with programmatic checks and a hand-graded sample of twenty outputs a day. That is more signal than most teams get from their fancy dashboards, and you will build the intuition you need to know when your judge is lying to you later.
