The lock that read a live lane as dead

Almost everything written about running agents assumes a cluster. I have one consumer graphics card, and on Friday four agents wanted it at once. That is not a scaled-down version of somebody’s data centre, it is a different problem, because the thing being protected is not throughput. It is another lane’s five-hour run. This is what my lock got wrong about that within twenty minutes of my changing it, and who broke it first.

The arrangement, since it is unusual

Two products, one card, four agents working on them at the same time, and no cluster behind any of it. The rule for sharing is a lock file holding one line: which lane has the card, since when, and what for. Every agent reads that line before a batch of model calls and refuses to run if the card belongs to somebody else. Staleness is the file’s age together with whether the runtime reports a model loaded, both at once, because a lane can legitimately hold the card for hours.

That is the whole mechanism. It is a text file, and I would recommend it to anyone in the same position, which is most people building this way alone.

Friday, from 20:18

At 20:18 a judge run of two hundred rows finished its five hours on the card and wrote its files. Its runner then crashed printing a non-breaking hyphen the Windows console could not encode. Nothing was lost, because releasing the lock was chained to the command rather than to the script finishing. Five minutes later another agent took the card for a bench run and wrote the same one-line lock.

At 20:46 the agent building the card’s future owner, a process meant to hold the lock on everyone’s behalf, tried to construct two of its planted cases: a holder whose process is dead, and a holder whose process id has been recycled, which is a pid that exists but started after the lock was written and now belongs to something else. Both need a process identity, and the line had none. So it built the cases as far as the file allowed and stopped. Its liveness check answers alive, dead, or cannot decide, and every lock the lanes had written answered cannot decide. Cannot decide is not dead. A caller that read dead as free would take a card in use.

Two writers, twenty minutes

The protocol changed at 20:50: the line gained a pid and the process’s start time, with older lines still readable and undecidable rather than free. At 21:07 the harness wrote its own lock in the new form. At 21:09 the other product’s writer landed, a wrapper that is itself the holder, and it found two defects by being run. The first version wrote the lock after spawning the run, leaving a window in which the card was unlocked, and it named the child’s pid, whose start time is not readable the instant it is spawned, so it wrote the birth time as unknown. That is precisely the lock the change existed to stop anyone writing. Both were fixed the same way, by making the wrapper the holder.

21:31, a live lane read as dead

The first lock written under the new rule was the other product’s, carrying an ISO timestamp for the birth time, because the note asked for one and a protocol a person has to read should be readable. The harness’s reader compared it against the function it actually had, which returns a Windows file time: an integer counting hundred-nanosecond intervals since 1601. Two notations in one field, compared as strings, never equal. The reader answered dead about a running lane. A version that acted on that answer would have taken a card in use, which is the exact collision the lock exists to prevent, arriving through the lock.

The fix split one question into two. A dead holder is decided by the pid alone, so gone means dead whatever the birth field says. A recycled pid compares birth times as instants, parsing either notation, with a two-second tolerance, because a hand-written lock carries whole seconds and no operating system recycles a pid inside two seconds. And two values that cannot be compared answer cannot decide, never dead. Verified against the live foreign lock: alive, and the harness’s own job still refused, which is the correct pair.

21:38, the author who did not follow it

Then the harness read its own lock. It had written the note asking for an ISO time at 20:50 and had written its own lock as a file time minutes later, because that was the function the lane actually had to hand. The protocol failed within twenty minutes and its author broke it first. A protocol its own author does not follow inside the hour was written from the reader’s side only. The note now takes both forms, and the one reader parses both and compares instants.

What I think this means

A field that two writers fill in two notations is not one field. It is two fields with one name, and nothing about the file’s shape says so. If a format is written by a person in one place and a program in another, the format has already forked and you will find out from a wrong answer rather than from a parse error.

A check with two answers will eventually pick the dangerous one. Alive and dead force a reader to guess when it cannot tell, and it will guess in whichever direction its default falls. The third answer is the safety property: cannot decide, never the value that invites a caller to act. On a shared card the cost of the wrong answer is not a failed run of mine. It is somebody else’s five hours, corrupted at hour four.

I am not claiming the lock is finished. The owner meant to hold it for every lane still reads and refuses and writes nothing, by design, until every lane can stop writing by hand in a single change. And one text file on one machine is not a distributed lock, and I would not pretend otherwise to anyone running more than one box.

What convinced me the evening was worth it is not the fix. It is that a change to a shared file produced a wrong answer about a live lane in nineteen minutes, and the only reason anyone saw it was that reading the real lock was part of the work rather than a formality. Had the check been mocked, it would have passed.

Four agents on one card is not a compromise I am waiting to grow out of. It is the constraint that has taught me more about building carefully than any amount of capacity would have.

Sources

Commits in the harness repository and the other product’s, 2026-09-05, with the times they were written. Four of the harness’s were rebased by its own push after this was drafted; the hashes below are the ones on main, which are the ones a reader can fetch.