I wrote 2,456 tokens of tool instructions into Sequence’s prompt, then proposed deleting them to save money — the tools they described were being called about once in a hundred turns, so the tokens looked inert. Deleting them made the model call its tools fifteen times more often. Four clauses written in advance to catch the damage found none. I still cannot tell you why it happens.
The proposal, and the shape of the bet
Sequence sends a TOOLS section with every ask. It is about 2,456 tokens and it is the largest single block in the prompt. The tools it covers were being called roughly once in 104 turns, and tokens cost money on every request, so the obvious move was to trim it and keep the behaviour.
That is a claim, so it got a registration before it got a run. Four clauses, each with a number written down first:
- tool calls must not rise — a ceiling of 2 in 104 turns;
- fabrication must not rise — unsupported technologies per claim-checked turn;
- refusals must not rise — within the control’s range;
- charts must not fall — a fall voids the run.
The point of writing the ceiling on clause 1 first was to stop myself explaining away a small rise afterwards. It turned out not to be a small rise.
What deleting it did
30 tool calls in 104 turns, against a registered ceiling of 2. The clause was refuted by a factor of fifteen, in the direction nobody proposed. Control arms made 1 and 1 tool calls across 52 turns; the trimmed arms made 16 and 14 across 9 and 8.
A block of instructions is supposed to be either earning its tokens or dead weight. This is a third thing: a block whose presence works against the behaviour it is about. Removing it did not free the model to ignore the tools. It preceded the model using them.
The clause that stood unmeasured for a day
Clause 2 could not be evaluated at all on the first run. The bench never recorded what the pipeline already returns, so fabrication must not rise had no number beside it — and publishing “removing instructions made the model use its tools” without knowing whether it also made things up is a trade I was not willing to make.
Three instrument fixes later — recording the claims at all, keeping each arm’s file, and recording whether the check ran — it has a number.
| arm | flag | conversations | clean turns | claim check ran on | unsupported technologies | refusals |
|---|---|---|---|---|---|---|
| control | off | 13 | 52 | 52 | 0 | 19.2% |
| treatment | trim | 13 | 52 | 52 | 0 | 3.8% |
| control | off | 13 | 52 | 52 | 0 | 21.2% |
| treatment | trim | 13 | 52 | 52 | 0 | 3.8% |
52 of 52 claim-checked in every arm. That column is the one that matters, and it is why the zeros mean anything: the check ran on every clean turn and found nothing, rather than nobody looking. A zero without its denominator is an upper bound wearing the clothes of a measurement.
And the reading was fixed before the numbers existed: no detectable rise, never no rise. The smallest difference this design can honestly see was registered in advance at 3 or more in both treatment arms, derived from a measured base rate of 0–1 per 52 turns. At 0 and 0 the test saw nothing, and it could only ever have seen something large. Discovering that limit after a null result would have been motivated reasoning; writing it down first cost nothing.
The four clauses, complete
| clause | registered | measured | |
|---|---|---|---|
| tool calls must not rise | ≤ 2 of 104 | 30 of 104 | refuted |
| fabrication must not rise | 3+ in both treatment arms | 0, 0 | no detectable rise |
| refusals must not rise | within the control’s range | 3.8%, 3.8% vs 19.2%, 21.2% | passes |
| charts must not fall | a fall voids the run | 71.2%, within the control’s range | passes |
The refusal fall is reported and not banked. Clause 3 was written one-sided — refusals must not rise — and reading a fall out of a clause written for a rise is how a registration becomes decoration. If I want that result I have to register for it and run it again.
The story I told myself, and why it is wrong
For about half an hour I had a tidy explanation: the section documents the tools, and documenting a tool is what suppresses it. It is a good line and it does not survive reading the section.
TOOLS never documents the chart tool at all — that example lives in the teach contract, which the trim keeps. It opens by instructing tool use, in as many words: Retrieve before you answer. And every tool call in both arms was a chart or a plan, with zero retrievals either way — so removing the section that says “retrieve” produced no retrievals, and the section that says “retrieve” produced none either.
The effect is real and reproduced. The mechanism is unknown, and the mechanism I proposed is not merely unproven — it is refuted by its own subject’s text.
What this does not establish
It is one model, granite42-hermes, run locally. One product, one prompt, one bench tier.
It is not a claim that an engineer gets a better answer. The arms ran on the teach bench, where the derived chart already fires, so 20 extra chart calls moved the picture rate only from 67 and 69 percent to 71 and 71. That was my framing when this started, and it was wrong: wrong tier, and a second mechanism supplying the same outcome. Anything about an engineer needs a different measurement, on a different path, and I have not run it.
And it licenses nothing further on its own terms. More arms after seeing this number would be optional stopping with extra steps.
Where each claim comes from
| claim | commit |
|---|---|
| the belt refuted, tool calls 30 of 104 against a ceiling of 2 | 7c591f1c |
| clause 2 unmeasurable; fabrication now recorded | 9a599904 |
| the section’s own text read, and the tidy mechanism refuted | 464874bd |
| the re-run registered, with its threshold and power limit written first | c1e767b4 |
| the kill and the power limit reconciled before any arm ran | 935b67d9 |
| four arms, zero unsupported technologies, the full verdict | 9cb48c0f |
What I think this means
I have stopped believing I can tell which parts of a prompt are doing work by reading it. I wrote this section. I knew what was in it. I estimated its value at roughly zero and proposed deleting it for the token saving, and the one thing I was confident about — that removing it would not change behaviour — is the only thing the experiment actually refuted.
The honest limit is that I do not know why, and I want to be plain that this is not a modest hedge attached to a solid finding. The mechanism is missing, and the explanation I found convincing was refuted by the text of the section it was about. What is left is an effect with a number on it and no story: remove 2,456 tokens of tool instructions, and this model on this bench calls its tools fifteen times more often, invents nothing measurable, refuses less, and draws just as much.
What I would defend is the shape of the work rather than the result. The number that mattered was written down before the run, so a fifteenfold rise could not be talked down into noise. The reading of a null was fixed before the null existed, so I could not discover after the fact that my test was too weak to see anything. The denominator was recorded, so a zero means the check ran rather than that nobody looked. Three instrument fixes went into being able to say that last sentence, and they are most of what the day cost.
The thing I keep coming back to is that this was supposed to be a cost saving. I went looking for 2,456 tokens I could stop paying for, and found out I had been paying them to make my own product worse at the thing the tokens were about.