The statistic that rewarded failure

For about three hours on Saturday morning my teaching benchmark was rewarding failure. A run whose lessons broke scored higher than a run whose lessons worked, and nothing in the number, the report or the first three explanations I wrote said so. This is that morning: a prediction I registered and lost, and four readings of why it lost, each one a layer under the last until the fourth found the measure itself.

The prediction, and losing it

On Friday afternoon a scanner fix added 251 dependency edges to Sequence’s own repository, one edge in eleven that it had been missing, all of them carried by re-exports through barrel files. Before any run, the product’s agent registered what those edges should do to the teaching bench across the thirteen conversations that run against that repository. Concepts with nothing to draw should fall. Turns with a grounded chart should rise. The comprehension check should not move, because a gate and the model decide it and neither reads an edge count. It wrote the refutation clause too: refuted if the first two fail on both runs. Two runs a side, because one run gives the other side no range to fall outside of.

The four runs held the card from 21:14 to 01:31, with a stale-build guard stopping the first attempt and eleven five-minute hangs stretching the rest.

Both clauses fired. The visual fell instead of rising, on both runs. The concepts with nothing to draw did not fall on either. The check-in sat inside the old range, as registered. The registration had named the honest reading in advance and it stood: the 251 edges are not where these lessons look.

Four readings of why

The first was mine, and it was the absence of one. I wrote that the fall was not explained on the record and carried two confounds: the new arm’s first run had spent forty of its fifty-eight minutes on hung requests that the retry re-sent, and the tree had grown between runs. Neither pointed the visual downward by any mechanism I could name, so I left it open.

The second was the product agent’s, seven minutes later, and it moved the fall upstream of the picture. The chart drew on 94, 100, 93 and 100 percent of the turns that were given a concept. Whatever fell, it was not the drawing. It was how many turns were given a concept at all: 18 and 17 on the old scan against 14 and 15 on the new. Reporting that the scan lowered the visual rate would have been a real conclusion drawn off the wrong quantity.

The third was mine again, and it was wrong. From the thirteen lesson files I counted the concepts: 24 picked across the run, not one of them a barrel file, so the new edges never became the subject of a lesson. Four conversations never got a concept on any arm, which caps the visual at nine of thirteen. And I found where two concepts had gone: three conversations were given one on both turns under the old scan and on one turn under the new, in both runs, deterministically. I wrote that this pointed at the concept queue’s second pick.

The fourth, from the same files, showed there was no second pick. The queue is byte-identical across all four arms for all thirteen conversations. The conversations I had named have a one-entry queue on both scans, because their asks never match the phrase that reaches the walk where the new edges live. So what had I counted? A stub. When a turn produces nothing, its concept is not consumed, and the next turn is counted as having been given one. Pooled over sixty turn-pairs, a one-entry queue survived a stub turn four times out of four and a real turn seven times out of twenty. The old arms stubbed nine and five times; the new arms three and four. That difference of about three and a half is the entire drop in concepts given.

The new scan scored lower because it failed less.

The corrected statistic

The fix counts concepts only over turns that produced a lesson, and reports stubs in their own column, where a failure is a result rather than a hidden multiplier. The first planted case states the defect as a fact: two runs of one lesson, one failing its first turn, and the uncorrected statistic scores the failing run higher, three concepts against two.

Identical, not merely overlapping. The prediction stays refuted, and there is no longer a fall to explain, because there never was one. The uncorrected fields stay in the report beside the corrected ones, since the prediction was registered and judged under them and the record has to show why the judgement changed.

Three sentences

The last commit of that stretch asked why six of the thirteen lessons had a queue of one. Three asks naming the same file got 1, 1 and 6 concepts: why does orders.ts use the retry flag at both ends, one; teach me what orders.ts does here, one; teach me how orders.ts works, bit by bit, six. The middle one is a request to be taught and was getting a single file for want of a phrase. A request to be taught now gets its neighbours whether or not it says so, and a pointed question still gets exactly one, because padding a pointed question with neighbours invents a syllabus the learner never asked for and cannot see to correct.

The agent fixing it turned down the condition it had been handed, a walk for any ask whose concept names a node, because that would have broken the single-question case and moved none of the six lessons, which name no node at all. The signal that separates them is whether teaching was requested. I record that as a good refusal.

What I think this means

A measure can reward the failure it exists to detect, and it will do it silently. Nothing about this looked like a defect. The number was computed correctly from a field that was populated correctly, and it credited a broken turn with a concept it never used. If a statistic can go up when your product gets worse, no amount of care in reading it will save you, because the reading is not where the error is.

When a number moves the wrong way, check what it counts before you explain why it moved. Two of my four readings were explanations for a fall that had not happened. They were careful, they cited real confounds, and they were answers to a question that did not exist. That is the expensive kind of wrong: it produces work, and the work looks like progress.

I am not claiming the 251 edges are irrelevant to teaching. Six more lessons can now reach them, and whether that moves the picture is registered as its own condition with a ratio and bands rather than settled by re-reading this one. Nor that the corrected statistic is the last one that needs correcting: the coverage figure on the same bench counts something else wearing a grounding label, and it stays as it is for now because it is the instrument these arms were measured with. And my two confounds were real. Eleven hangs and two growing trees are on the record. They just were not the cause.

What saved this was registering the prediction with its refutation clause before the run existed. Losing a prediction I had written down sent me looking for a mechanism, and the fourth look found one in the measure. Had I not written it down, I would have read the same table as noise and moved on, and the benchmark would still be paying my product for failing.

I would like to say I saw it quickly. It took four readings, three of them mine, and the one I argued hardest for was the wrong one. What I have instead of speed is a record that made my own bad reading visible within the hour, and I would not trade it for being faster.

Sources