Running what nothing ran

Every test runner reports what it ran. None reports what it was supposed to run, and the gap between those two numbers is where defects sit undisturbed for months. Having been bitten by that once, I predicted it would happen again in my other project, wrote the prediction down first, and watched it be refuted in seventeen minutes and un-refuted nineteen minutes later. The reversal is the part worth reading, because it turned on which number I had written down.

The prediction

My fine-tuning harness has had a counting gate for a week: one discovery of the test suite drives both the count and the run, and if the number that ran is not the number discovered, the gate exits with its own code before it looks at pass or fail. The first time it counted, it found thirty-five tests collapsing into two placeholders on a clean clone, under a green tick.

So when I wrote the design for the same gate in my other product, a code-understanding tool with nine packages, I wrote the metric down first — declarations counted on disk against what ran — and predicted the same thing: the first count finds at least one package where fewer tests ran than were declared.

The refutation

The gate was built seventeen minutes after the page and run. All eight packages using the node test runner agreed: 331 discovered, 331 ran, zero mismatches. Refuted, cleanly.

Two real defects turned up anyway, both in packages the standing CI gate had never run: a cap of 250 callers that my own change that afternoon had pushed exactly one over, silently dropping a caller from the busiest file in the repository; and a package whose every spawn test failed for a correct reason, a guard refusing to touch the real user store under test, which had been read as a server that would not start.

The gate itself could not fail until it was made to. The runner emits a wrapper event per file, so a file with no tests counted as having run. Planting an empty one proved it: discovered 18, ran 18, exit 0. Fixed so that “ran” means at least one real test, the same tree gives 18, 17, exit 2, and names the file.

The reversal

Nineteen minutes later the same agent re-read the design page and reported that its own refutation had used the wrong metric: it had compared the files a glob matched against the files that ran, both sides from the same glob, so “ran equals discovered” held by construction for anything the glob missed. The page’s metric was declarations on disk, and under that metric the prediction was confirmed. One test file in the schema package compiled to a directory no glob covered: built, and run by nothing, while the gate reported 37 of 37 green over 38.

Run for the first time, two of its three tests failed, and one was a shipped product defect: a built-in template the product’s own validator rejects, two halves of a composite each defining a node called check, concatenated without namespacing, under a docstring calling the result validator-clean. It had been invalid since the day it was written.

The gate now walks each package’s built test tree independently of its glob and names any test file no glob reaches: 333 of 333, 3,007 tests, zero unreached.

The inventory

Then I had the whole tree measured.

Of 574 test files on disk, CI runs 544, the counting gate sees 331, and 30 are run by nothing at all: two in the desktop package, fourteen end-to-end specs that need a browser, thirteen under tools, and the one file the glob missed. Both defects from the first run were in the 30. The third was in the 331, hidden by the metric.

Eighty more tests across the repository report green while asserting nothing, because their skip is a return statement rather than a skip. Seventeen of those are the repository’s own security locks, which return on my Windows machine before reaching a single assertion.

What I think this means

The prediction was right on the metric I wrote down and wrong on the one that was used, and the difference between those two metrics was itself a shipped defect. I would not have had that without writing the metric down before anything was built.

I am not claiming the gate found much. Three defects in a day is unremarkable, only one came from the count, and the eighty vacuous tests are a different problem it does not touch.

What I am claiming holds anywhere. A green suite is a claim about what ran, and nothing in your toolchain checks that claim. A gate has to be able to fail before its green means anything, and the cheapest way to find out is to plant an empty file and see whether it notices. And a skip that returns is a pass, so the number on your screen is counting tests that decline to test.

The gate is a single file with no dependencies now, and run against three unfamiliar repositories it found each of them refusing to answer for a different reason. That is the part I would keep if I lost the rest: not the number, but the habit of asking a suite what it was supposed to do.

Sources