A monitor is supposed to be the safest thing you write, because all it does is look. In practice a liveness probe is a request like any other, and a request can change the state it was sent to observe. On 9 September mine took down the job it existed to protect, and then filed a report blaming the server.
The probe asked Ollama for a two-word completion and checked that an answer came back within thirty seconds. Reasonable: /api/tags proves the process is listening, not that it can actually generate, and I wanted the stronger signal. To ask for a completion you must name a model, so it named the house model. What I had not connected is that OLLAMA_MAX_LOADED_MODELS on this box is 1 — deliberately, because the card is 8 GB and two resident models do not fit. Naming a model that is not the resident one does not queue behind it. It evicts it.
At 12:03 the probe fired while Bonsai 27B was mid-evaluation. ollama ps went to “Stopping…”, the run died, and the watchdog logged OLLAMA DOWN — about a server that answered /api/tags in four milliseconds throughout. The outage was real, the report was accurate about what it measured, and the cause was the thing doing the measuring. On a five-minute schedule, it would have done that forever.
The fix is a smaller probe: /api/tags lists what is installed and loads nothing. It is a weaker check, and that is the correct trade rather than a compromise. A probe strong enough to prove generation works is strong enough to change which model is loaded, and on a single-slot card those are the same capability. The honest version of “is it healthy” here is “is it listening”, plus a separate look at whether the work itself is progressing — which is a different question asked of a different thing, and should not be smuggled into a five-minute cron.
The watchdog existed at all because of a worse morning. At 05:05 the same day, Windows Update restarted this machine three times in four minutes. Ollama died, the harness engine died, every session died, and nothing brought any of it back; a run that had been going for forty-eight hours stopped at 04:52 and did not resume for six hours. That is the kind of failure you plan for once and then assume is handled.
Except it had been planned for. Two recovery loops were on record as armed, and neither existed. Get-ScheduledTask matched nothing. The cron listing returned “No scheduled jobs”. Both had been asserted in a status file and never verified against the system that would have to run them, and the reboot is simply when that stopped being survivable. A safety net you have described is not a safety net you have. That is the more embarrassing finding of the two, and it is the reason the watchdog is a file on disk registered as a scheduled task rather than a claim in a document — so that anyone who doubts it can check, with one command, without believing me.
It also reaps orphans, which is the other thing that day taught. Killing Ollama does not kill the llama-server.exe children it spawned; they outlive the parent and keep their allocation. Two of them held 7.8 GB of the 8 GB card while every subsequent model load wedged at “Stopping…”, which reads exactly like a broken model and is actually a bookkeeping problem. Order matters when clearing that: the children hold the memory, so they go first.
What it deliberately does not do is start or prompt a session. A session is an interactive process owned by an application, and a watchdog that tried to drive one would be claiming a capability it does not have — which is precisely the fault it was written to correct. So the honest statement of coverage is narrow: it keeps two services alive across a reboot, and it cannot wake a lane. Every scheduled job still dies with the session that owns it. The six-hour gap from that morning is not closed by any of this. The state is merely legible now to whoever arrives next.
The judgement I would keep from it is about monitors generally. The instinct that a stronger check is a better one is wrong in any system where observing costs something, and it is wrong in a specific, findable way: ask what resource the probe consumes, and whether the work it watches competes for that same resource. A GPU with one model slot, a database with one connection, a rate limit shared between the health check and the thing being checked — same shape every time. The probe that can prove the most is the probe most likely to become the incident, and a monitor that cannot distinguish its own footprint from a fault will eventually report itself and be believed.