Okay, this is going to sound like a small thing, but I spent an afternoon last week discovering that my eval harness had been putting the model name in the judge prompt. Not deliberately. It was in a metadata block I’d added for logging, and the logging block got interpolated into the same template the judge saw.
So for about six weeks, every score my grader produced knew which model wrote the answer.
I don’t have a clean before-and-after number to show you, because I didn’t think to keep the old runs separate. That’s the honest version. What I do have is a strong suspicion that some of my “model B is clearly better” conclusions were partly the judge reading a label.
There’s now research pointing the same direction, and it’s worth reading before you ship an eval pipeline you plan to make decisions with.
The thing everyone forgets about a judge
An LLM judge is a measurement instrument. Instruments have systematic error. We know this about scales and thermometers and we act accordingly, and then we wire up GPT-as-grader and treat the output like ground truth because it comes back as a tidy number between one and five.
The number being tidy is exactly the problem. A human rater giving a 4 out of 5 comes with an implied shrug. A model giving 4.0 comes with false precision, and that precision propagates into every dashboard and every “we improved 12% on helpfulness” claim downstream.
I’ve been running judges for two years and I still catch myself doing this. Score comes back, score goes in the spreadsheet, spreadsheet becomes a decision. The step where I ask what the instrument’s error bars look like keeps getting skipped because there’s no obvious place to put it.
Identity disclosure is a real confound
A recent paper called “Who Verifies the Benchmark? Decentralizing Trust in Large Language Model Evaluation” attacks this directly. The setup: seven verifier models, including GPT-OSS 120B, Llama 3.3 70B, DeepSeek V4 Pro and Mistral Large3, scoring responses from three models across 58 questions spanning factual, reasoning, political and preference-based categories. Each response is scored twice, once anonymised and once with the source model disclosed.
That’s the experiment I should have been running on my own harness and wasn’t.
The framing in the paper’s introduction stuck with me more than the numbers did. It points at how much rides on vendor-reported benchmarks, and uses the January 2025 episode where unverified claims about DeepSeek R1 outperforming o1 preceded Nvidia shedding a very large amount of market value in a day. Whatever you think about the causality there, benchmark claims are not academic trivia any more. They move money.
Two caveats before anyone quotes this at their team. 58 questions is small. And the categories are lumped together in a way that makes me want per-category breakdowns before I generalise. I’d read it as a strong reason to blind your own judge rather than a settled effect size.
Verbal confidence is not a probability
The other half of the problem is that models are bad at knowing how sure they are, and they express that badness in extremely confident-sounding English.
REHEARSE tackles this with a training-free approach: run the model through a scored confidence game with a proper scoring rule, summarise what it got systematically wrong into a trajectory prefix, and feed that back at inference time. Across four models, three benchmarks and five seeds, they report a 58% reduction in expected calibration error against the uncalibrated baseline.
What I take from this isn’t “use REHEARSE.” It’s that a training-free prefix can cut calibration error by more than half, which tells you how much slack was in the original numbers. If your pipeline asks a model “how confident are you, 0 to 100,” you are reading a stylistic tic, not a probability. I’ve built things on top of those numbers. I’d like them back.
What I changed in my own harness
Four things, in the order I did them.
Blinded the prompt. The judge now receives the response text and nothing else. No model name, no run ID, no metadata block, no filename. I wrote a test that fails if any known model name appears in the rendered judge prompt, which felt paranoid and caught a regression within a fortnight.
Randomised presentation order. For pairwise comparisons, whether a response appears first or second shifts the outcome. Coin flip per comparison, and I log which side each response was on so I can check for order effects afterwards.
Added a human-graded holdout. About forty items I graded myself, kept out of everything, rerun whenever I change the judge model or the rubric. It’s tedious and it’s the only thing standing between me and a slow drift I’d never otherwise notice.
Stopped reporting a single number. The judge score now travels with the disagreement rate against the holdout. If those two diverge, the score means less, and the format makes that visible instead of burying it.
None of that is clever. All of it is the sort of thing that feels like overhead until the first time it catches something.
The uncomfortable part
Here’s what I keep circling back to. Every mitigation above makes the eval slower and more expensive, and the entire appeal of an LLM judge was that it was fast and cheap. Blinding costs nothing. Order randomisation costs nothing. The human holdout costs real hours, and it’s the one that actually protects you.
So there’s a floor on how cheap honest evaluation gets, and I don’t think the floor is as low as most of us were hoping. That’s not a reason to go back to hand-grading everything. It’s a reason to be specific about which decisions your judge is good enough for. Ranking two prompt variants where the gap is large: fine. Deciding whether a 3% regression is real: not fine, get a human.
I made a related argument about eval tooling missing long-horizon behaviour in why LLM evaluation tools keep missing the long game, and this is the same failure from a different angle. We keep building measurement around what’s easy to measure.
One thing to do this week
Open your judge prompt template and render it with real data. Not the template, the rendered output, the exact string the model receives.
Then read it and ask what a motivated observer could infer about the source of the response. Model names are the obvious leak. Response formatting is a subtler one, since models have house styles and a judge that has seen a lot of text can recognise them without being told. Timestamps and run IDs leak more than you’d think.
If the answer is “nothing,” good, you’re in better shape than I was. If it’s “quite a lot,” you’ve just found out that some of your recent conclusions need a second look. That’s a bad afternoon and a much better position than not knowing.
I build and audit this kind of evaluation plumbing as part of client work, and the pattern is consistent: the harness is almost always the least-tested code in an AI project despite being the thing every decision routes through. More on that side of what I do is on my portfolio.
