Measuring an automated annotator

I use a small local model to label training data, and before trusting a single number it produces I wanted to know something cheaper than whether it is right: whether it gives the same answer twice. It does not, and the shape of its disagreement turned out to be the useful part. Then a later run showed that making it repeat itself is available and is not the fix it looks like, which is now section 6. This is written as a methods note rather than a story, because the protocol is the part worth copying.

1. The question

A model was used to label preference pairs: given a customer-service answer and a rewrite of it, decide whether the rewrite is a genuinely worse version, which makes the pair usable, or something else, which makes it noise. Every downstream number rests on those labels: how many rows survive, what fraction of them are lies, whether a corpus is large enough to train on.

The question this note answers is not is the annotator accurate, which needs ground truth and a human. It is prior to that and cheaper: does the annotator give the same answer twice? An annotator that disagrees with itself puts a ceiling on every measurement made with it, and the ceiling is knowable without knowing the truth.

2. The setup

Two sets of rows, and the difference between them is the whole design.

The sentinel set, 72 rows. Every row is a rewrite that changes nothing that matters: a full stop removed, a numeral spelled out, it is to it’s. 48 punctuation, 18 numeral, 6 synonym. The correct label on all 72 is the same one, and a deterministic rule already decides them at zero model calls. They are sent to the annotator anyway, precisely because they carry no signal, which makes them the place where any disagreement is the instrument and not the material.

The real set, 188 rows. Every pair that reached the annotator in a production run: real rewrites of real answers, a mixture of genuine degradations, invented facts and near-misses.

The protocol, both times. The same rows in the same order, the same prompt bytes, the same model, the same context window, nothing else between the passes and nothing carried from one to the next. Two passes, and the comparison is per row.

3. What was registered before the runs

Written down and committed before the first call, because a self-agreement number is trivially easy to read charitably after the fact:

• the band: 58 to 68 of 72 agreeing on the sentinel set, described in its own registration as if anything, generous;
• the flip prediction: of the 19 rows the annotator had wrongly kept in an earlier single pass, at least 8 would not be kept in a second;
• the refuters: 72 of 72, which would mean the wire is deterministic and the earlier variation came from somewhere else; or below 50, which would mean the instrument is too unstable to floor anything.

4. The numbers

On the sentinel set the registered band was missed on the low side. 52 of 72; the band was 58 to 68; neither refuter fired, 52 clearing the lower one by two rows. By edit kind: punctuation 32 of 48, numerals 15 of 18, synonyms 5 of 6.

The flip prediction held at double its floor. Of the 19 rows kept in the earlier single pass, 16 were not kept in at least one of the two passes and 12 in neither; only 3 were kept in both. And in the other direction, 18 rows the single pass had dropped were kept in one of the two passes: the kept set moves, it does not merely shrink.

The verdict that ships a row is the unstable one. Of the 16 rows the first pass kept, 11 changed (69%). Of the 56 it dropped, 9 changed (16%). A drop is a refusal and costs one row; a keep puts a row into training data. The instability sits on the decision with the consequence.

On real rows the disagreement is smaller and its direction is informative. 22 of 188 rows changed: 14 from drop to keep, 8 from keep to drop. A third reader classified all 22 by hand: ten are false statements about the subject matter, five are genuine degradations, seven change nothing either way. Where the second pass dropped a row it was right on five of eight; where it kept one it was right on two of fourteen.

The two numbers are not in tension. 72% on the sentinel set is not a worse measurement of the same thing; it is a measurement of a different thing. The sentinel rows are where the annotator has nothing to go on, so their instability is an upper bound on its noise, not an estimate of it. Anyone quoting the 72% as this annotator’s reliability on real work would be quoting a ceiling as a rate.

5. What is not established

• Accuracy. None of this says the annotator is right, only how often it is consistent. The 22-row hand classification is one reader’s, and where two readers of the same 68 rows were compared earlier they agreed on 52 and differed on 16, always in one direction, over where a widened promise becomes a false one.
• The cause of the variation is now named - sampling under default decoding settings, section 6 - but not the cause of the judgement: why this annotator, held still, keeps these particular eight sentinel rows is not addressed here and is not answered by making it repeat them.
• A remedy. The obvious one, making decoding deterministic, was tried and cannot be used on this runtime: sending a temperature of zero makes the server stop answering about one call in seven, measured at 4 hangs in 30 calls against 0 in 834 with default settings, intervals disjoint. Whether a seed alone is harmless is registered and unrun. So the instability stands with no known remedy on this wire, and any acceptance threshold has to be set inside it rather than argued away. Amended 2026-09-06 15:20, and left above struck rather than deleted. That paragraph was wrong in both halves. What was tried and cannot be used is temperature zero, which is not the same thing as deterministic decoding; the run that was registered and unrun when this note was written has since gone, and a seed alone does not hang and does make the annotator repeat itself exactly. There is a remedy and it is one parameter. What it buys, and what it does not, is now section 6 - and what it does not buy is the more useful half.
• Generality. One model, one prompt, one machine, two sets. Nothing here transfers to another annotator without being run again.

6. What a seed does, and what it does not

Added 2026-09-06 15:20. The seeded pair is a run this note registered and had not made; the four one-change conditions now sit in one table, and the result changes what section 5 could say.

The registered band for a seed alone was 0 to 1 hangs; it came back at 0, and the stop rule written into that registration - abandon at one hang in twenty over the first forty calls - never fired. Temperature is what hangs this runtime; the seed was carried along and blamed with it.

And under a seed the annotator repeats itself exactly: 72 of 72 on the same sentinel rows, Wilson 0.949 to 1.000, no unreadable rows, 100% on each of the three edit kinds. That is the refuter this note registered in section 3 - 72 of 72 would mean the wire is deterministic and the earlier variation came from somewhere else - firing in the condition that had not been run. The somewhere-else has a name now, and it is sampling.

Which is where the useful part starts. Every sentinel row’s correct label is DROP; the set was built so that a KEEP is always an error. Under a seed the annotator returns 64 DROP and 8 KEEP, and the same eight rows every time: 8 of 72 wrong, 11.1%, Wilson 5.7 to 20.4. Under default sampling the same set produced 16 wrong keeps on one pass and 14 on the other.

So the seed did not remove the errors. It removed the detector:

A second pass is the whole method of this note, and against a seeded annotator it returns a clean bill every time. Fewer errors, none of them findable.

Nor is the seeded run the same annotator with the noise taken out. On 10 of the 72 rows its verdict matched neither unseeded pass - five rows it keeps that both passes dropped, five it drops that both passes kept - and only 3 of its 8 wrong keeps are among the 25 rows the unseeded annotator ever kept. A seed does not reveal what the model really thought. It fixes one draw and throws away the evidence that there were others.

What this does to the 52 of 72. It was never a claim about the annotator’s judgement; it is a ceiling on the noise of the annotator as it was actually run, and that configuration is the one that labelled the corpus. Every label already in that training data was produced under sampling, so the ceiling still applies to all of them. What has to go is any suggestion that the noise was forced on us: it was a default nobody set deliberately. The honest statement is that the instability was a configuration, not a property, and that fixing the configuration hides it rather than fixing it.

Not established here. The seeded pair is the 72 sentinel rows only. The 166 of 188 on real rows has not been re-run under a seed, and nothing in this section says what a seeded annotator would agree with itself at on rows that carry signal - though 72 of 72 on a deterministic wire makes the answer arithmetic rather than interesting. Why these eight rows: unaddressed.

7. What somebody else would have to do to check it

1. Build a sentinel set for your own task: rows whose correct label is not in question and which carry no signal, so that any disagreement is the instrument. Send them through the same path as real rows, not a shortcut.
2. Register the band, the flip prediction and both refuters before the first call, in a file with a timestamp. A self-agreement number is easy to read charitably afterwards.
3. Run two passes, same rows, same order, same prompt bytes, and the same decoding configuration, written down with the rest. Record every row’s verdict, not a summary.
4. Report agreement with an interval, and report it separately for the sentinel set and for real rows; the first is a ceiling on noise and the second is what you have.
5. Split the flip rate by verdict. If one verdict ships a row and the other refuses it, the asymmetry is the finding.
6. Write out the disagreement set as its own file and have somebody read it. It is small, it is the instrument’s own error set, and it is where the useful surprises are.
7. Count distinct rows, not rows. The corpus these labels built was counted at 164, 156 and 95 in three places and was 127, 119 and 60 distinct; one pair appeared seven times.
8. Run the seeded pair too, and run it second. It separates the wire from the annotator: if it comes back at 100% you have learned that your runtime is deterministic when you ask it to be, which is worth knowing and is not a fact about your annotator. Do not then seed your production labelling and call the problem solved. A seeded annotator makes the same mistakes every time, and the two-pass check in step 3 - the one that caught eleven of sixteen bad keeps here - catches none of them. Seed to reproduce a result; sample to find out whether the result is any good.

8. What this cost

144 calls for the sentinel pair, 188 for the second pass and 144 more for the seeded pair, about 51 seconds a call on one consumer GPU: roughly two hours, three hours, and two hours. The registrations, the disagreement file and the hand classification took longer than the runs.

Sources