The judge that invented deletions

Almost everyone building with models now uses a model to grade the model. It is the cheapest evaluation there is, it is what “LLM as a judge” means, and it is the part of my own stack I trust least. A judge that is told what it is looking for will find it. I can show you that from my own harness, on a set of seventy-two rewrites in which nothing was wrong at all.

I built the harness because I wanted to fine-tune small models on my own data without believing anything I could not check. Part of it makes preference pairs: take a real answer from a shop’s support log, ask a model to write a worse version in one named way, “promise more than the source allows”, and keep the pair only if a judge model agrees the rewrite is worse in that way and adds no new fact. The judge is a 3.7-billion-parameter model running locally on one consumer GPU. For a day I believed it was doing its job, because it said KEEP to twenty pairs out of twenty and every one of those pairs deserved it.

Then I planted ten defects. Ten rewrites built in code, from a closed list, with no model anywhere in the truth path: a dropped within 30 days, a dropped only, cannot turned into can, a 6 doubled to 12. Mixed with ten real pairs, shuffled under a fixed seed. The judge caught eight of the ten with its reasoning turned on and two of the ten with it turned off. So it could say DROP. That was the good news, and it was the last uncomplicated number I got that day.

The number I withdrew

The same run said the judge had thrown away five of the ten legitimate pairs. I wrote “a 50% false-positive rate” and called the judge trigger-happy. An hour later I withdrew both. The ten “legitimate” pairs were rows the same judge had already said KEEP to, pulled from the kept output of an earlier run. I had measured the judge against its own opinion.

When the KEEP side was rebuilt by construction instead, thirteen rewrites that each delete one restricting span from the source’s own text, so the degradation is real and nothing can have been added, the false-drop rate was one of thirteen. Not merely uncertain. Wrong in direction.

The set with nothing in it

Then I built the set I should have built first: seventy-two rewrites with no degradation at all. A final full stop removed. two written as 2. it is written as it's. Every row’s correct verdict is DROP, because the judge’s own instructions say a rewrite that is “merely different rather than more permissive” is not a valid pair.

I ran all seventy-two, reasoning on. Nineteen came back KEEP. Every one of the nineteen justified itself with a clause the rewrite still contained, word for word. On a row whose only edit was a missing full stop, the judge wrote: the rewrite drops the “over 25.00” condition, making the recommendation unconditional. Nothing had been dropped. Of the eighteen kept punctuation rows, eighteen cited a deletion that did not happen.

The cause was in my own prompt. Before showing the two answers, it told the judge what it was about to see: the rewrite was supposed to be worse in exactly this way. The judge was told the answer before it was asked the question, and it found the named degradation whether or not it was there. On real pairs a fabricated reason and a true one read exactly the same. Only a set with nothing in it could tell them apart.

What replaced it

Not a better prompt. A diff.

Whether a rewrite removed something, substituted something, reordered sentences or changed punctuation is a question about two strings, and a program answers it with no opinion at all. I put that program in front of the judge. It refused all seventy-two non-degradations and thirty-four surface-level rewrites by rule, with zero judge calls, and admitted every one of thirty-nine planted defects, of which the judge then caught thirty-five.

The judge still decides the one thing a diff cannot: whether a genuine removal widened a promise. It just never sees a pair where nothing was removed. The standing rule on the judge’s record page now reads: a verdict is a bit; the reason is the deterministic diff, computed in code and printed beside it.

What I think this means

Three things I would now say to anyone using a model as a grader, and I would have said none of them a week ago.

Never give a judge the hypothesis. Every prompt I have seen in the wild, including my own, tells the judge what kind of flaw to look for. That is the whole defect. The judge is not lying; it is answering the question you asked, which was a leading one.

A verdict can come from a model, a reason cannot. If the reason matters, and it does, because the reason is how you audit the verdict later, then compute it. A diff, a parse, a lookup. Anything that cannot invent.

Build the control set with nothing in it. It is the cheapest test in evaluation and almost nobody runs it. A set where every correct answer is “no” costs an afternoon to build and it is the only thing that separates a judge that works from a judge that agrees with you.

I am not claiming the corpus this judge grades is good. Thirteen and seventy-two are small numbers, the planted edits are mechanical, and a person would argue about some of the deletions. Earlier that same morning I found two generators building the same edit, deleting the word only, and labelling it oppositely, because the rubric was silent on implicature. What I am claiming is narrower and I think it generalises: the instrument grading my data was telling me what I had told it, and the way I found out was building a set with no defect in it.

Since then the diff runs in front of the judge on every pair, the rule is on the judge’s record page, and the four checks I now refuse to run an experiment without are a small library anyone can install. That is the part I would want back if I lost everything else here.

Sources

Commits in the harness repository, 2026-09-05, in the order the article uses them.

The judge is a granite-family model, 3.7 billion parameters, 2.24 GB, run locally through Ollama. The planted-defect and control sets are built by scripts in the repository with no model in the truth path.