Every number anyone publishes about an agent is read out of that agent’s own log. The completion rates, the token counts, the sandbox-escape figures in the papers: all of them come from a record the system wrote about itself, and no benchmark I have found checks whether that record is true. I could not do much about the benchmarks, so I did the small version. I took four things I had learned the hard way to assert about a run, and I asked whether three agent systems’ own logs could answer them at all.
The four checks, and why these four
They are not a framework. They are the four ways I have been wrong on this machine, in the order the failures found me.
A count with its denominator. My test suite printed Ran 3315 tests ... OK on a tree where discovery found 3,822. A runner reports what it ran and never what it was supposed to run.
A holder on a shared device. Four agents share one consumer GPU here. A liveness check answered alive for a process id belonging to the desktop shell, and a lock reader called a live lane dead within twenty minutes of the protocol being written.
A witness that outlives the bytes. A chart was written to a canvas file and wiped by the next write before anything saved it. A path is a location, not a version.
A verdict that can be false. A judge kept 19 of 72 rewrites that had nothing wrong with them and explained every one with a deletion that never happened.
What I read
Three systems, all on this machine, all chosen for being here: a command-line coding agent from one vendor, another from a second, and the model server that owns the GPU. 186,546 records; 402,769 records; 27,838 requests inside 833,147 log lines. I have not summed them, because two of those corpora count records and the third counts requests, and a total across two units is not a number.
| coding agent A | coding agent B | the model server | |
|---|---|---|---|
| a count with its denominator | holds | holds | completions only |
| a holder on a shared device | absent | absent | present, and useless |
| a witness that outlives the bytes | absent | absent | absent |
| a verdict that can be false | absent | fails 2,455 times | fails 538 times |
The two that are missing everywhere
Not one of the three records the identity of an artifact in a form that survives the artifact changing. Zero digests as a field value in any of them. Digests do appear — 582 in one, 1,820 in another — and every one is inside prose the model typed, not a field the system wrote. Files are named by path. When the bytes at that path change, nothing in the record notices, and a log read a week later cannot tell you whether the file it names is the file that was there.
And not one of them records who holds a shared resource. Two have no such field at all. The third is the interesting one, and it is the one arbitrating the card my four agents contend for: it stamps a client identity on every single request, 27,838 of them, and the value is 127.0.0.1 every time. It has the dimension. It answers the same thing to everyone. A populated field that cannot discriminate looks like an answer, which is worse than the two systems that simply do not have it, because absence is legible and a wrong answer in the right shape is not.
The one that surprised me
I went in assuming the counting check was the scarce one. It is the most ordinary of the four: all three pair or count, two of them by an identifier that survives their own log being rewritten mid-stream. A quarter of what I treat as discipline is standard practice, and I would rather find that out than keep saying it.
But the third system narrowed it for me. Its access log records completions, not arrivals — a request that never returns writes no line at all. That is not a hypothetical: I have measured this server hanging, thirteen times across two conditions, and its own log carries six client-closed responses. At most six of thirteen are visible, and I cannot prove those six are those hangs. A record of completions is a record of the successes and the fast failures.
What I am not claiming
Three subjects on one machine, chosen for being on it, is not a sample of anything. The hang correspondence is an upper bound on visibility rather than a match. Nothing here says any of these systems is bad — the pairing in two of them is meticulous, and one records failure 2,455 times when it would have been easier not to. And the four checks are mine: they are the failures I have had, not a survey of the failures available.
What I think this means
The gap is not carelessness, it is a missing dimension. Nobody filled these fields in badly. Two of the four are simply not in the record to be asked, in three systems from three vendors, which is what makes it a fact about agent logs rather than about anyone’s product.
And the field that answers 127.0.0.1 to everyone is the one I would fix first. Not because it is the worst engineering, but because it is the shape of defect I keep meeting: a check that returns something true and useless, in the right format, at the right time. My lock read a live lane as dead. My suite reported 3,315 of 3,822 and called it OK. My latency parser could not read 1m45s and reported two hours of silence that never happened. Every one of those was a system telling a true-looking story about itself.
I write these four assertions into everything now, and the whole of what they buy is that a green run has to say what it ran. That is a low bar. Three systems from three vendors clear half of it.