A generation cap is the standard fix for a model that will not stop, and it is the right fix. In practice a cap does not always turn a runaway into a shortened answer. On a model that reasons before it replies it can turn the answer into nothing at all, and the difference matters more than the cap does — because a truncated reply is a visible bad answer and an empty string is indistinguishable from an error.
The problem it was solving is real. Bonsai declares no parameters whatsoever — ollama show --parameters returns empty — so generation is unbounded, and it spent 719 tokens answering explain what a load balancer does in two sentences. That is the same shape that wedged this machine the day before with a different model: something reasons without a ceiling, the GPU sits at 99%, every caller blocks, and ollama ps still reports the server as healthy. A hang and hard work look identical from outside, which is why an explicit bound is worth having at all.
So num_predict 4096 went into the Modelfile with a paragraph explaining why. The check behind it was a run that stopped at 341 tokens with done_reason: stop. That reads like proof and is not: finishing early with a stop reason is exactly what an uncapped model does when it is done. The check could not distinguish the cap working from the cap being irrelevant, and it was shipped anyway with the reasoning written confidently beside it. That is the more common failure than an untested change — a test that runs, passes, and cannot fail.
Proving it properly took a control. Two providers registered on the harness, one eval file, and a second model identical to the first except for a single value: same base, same context length, num_predict 32 instead of 4096. Same row, same grader, same path. At 32 the first row returned 0 characters. At 4096 it returned 258 characters of well-formed JSON. All four graded rows of the capped arm came back empty. The cap reaches the eval path, which was the thing genuinely in doubt — the harness itself never sends one, because its Ollama adapter builds its options with a context length and nothing else, and neither num_predict nor max_tokens appears anywhere in the application outside tests. On that path the Modelfile is the only bound that exists.
And the result corrected the sentence I had written to justify the cap. A visible bad answer beats an invisible infinite one assumed the caller would see a cut-off reply. At 32 tokens the caller sees an empty string. Bonsai spends its budget inside a thinking block, and the adapter strips that block before the caller ever sees it, so a cap that expires before the block closes returns nothing — not a partial answer, not a malformed one, nothing. That is worse for a caller than truncation, not better, because a truncated answer tells you what happened and an empty one looks like a failed request.
The cap stays at 4096, and the reason is now narrower than it was. It is not that capping is safe; it is that 4096 on this model outlives the deliberation, and a bound only functions as a safety feature while that remains true. Set it below the thinking budget and you have not made the failure visible, you have made it silent and moved it from the GPU to the caller.
Two limits belong in the same breath as those numbers. The controlled pair is one row — the 4096 arm graded row 0 and the request ended before row 1 — so the direction is unambiguous and the magnitude is not measured; 0 against 258 characters tells you the cap fires and does not tell you what a well-chosen cap costs in quality. And the per-row timings from that run are worthless, because another lane was running its own evaluation on the same card at the time, so 190 seconds for 32 tokens is contention rather than generation. Only the character counts are reading anything real, and I would not quote the seconds anywhere.
That second caveat turned out to be honest and insufficient, which I did not see when I wrote it. Three runs from two lanes were on that card inside twenty minutes, and the reason none of them collided visibly is that the card is shared through a lock file governing lanes that run models — and all three were lanes asking a server to run one, which the lock cannot see. So the contention was not bad luck to be disclosed; it was a hole in a protocol sitting on disk two directories away. Disclosing it did not save the seconds. A caveat like that should read as evidence a run needs repeating rather than as the run being finished — the lock that read a live lane as dead has the full account.
The judgement I would keep is about what a bound is for. A limit is a promise that failure will be legible, and that promise is only kept when the limit sits above the model’s own working space rather than inside it. On a model that answers immediately, any cap truncates and every caller can see it. On a model that thinks first, a low cap does not produce a worse answer — it produces no answer, wearing the shape of a network error, and the calling code will retry it. The number to choose is not the smallest one that prevents a hang. It is the smallest one that still lets the model finish thinking, and you cannot find that by reasoning about it, because the model will not tell you where its deliberation ends.