The list it was told to use

I spent about two hours of rented card proving that a feature I built does not work. Sequence carries a list of file names from one turn of a conversation to the next, so the model can answer a follow-up question without fetching anything again. It arrives intact. I moved it, I pointed at it, I finally told the model to use it. Across fifteen runs it named none of the three files only that list could have given it.

What the feature is

Ask Sequence what depends on scan.ts and it calls its own who_calls, which finds 84 distinct callers. The answer you read names a handful. The carry keeps the first eight in a block and hands them to the next turn, so that when you ask “of those, which would I change first?” the pronoun has something to point at.

That is the whole idea, and it is a reasonable one. A follow-up question that needs a second full scan is slow, and the model has already been given the answer once.

The measurement, and why three names carry it

Five of the eight names in the block were also spoken aloud in turn 1’s own answer, and Sequence carries prior chat prose independently of any carry slot. So if the model says one of those five, it proves nothing — there are two paths to it and no way to tell which one it took.

Three of the eight are not in the prose: eval/grade.ts, functions/repoFunctionGraph.ts, harness/designDrawBaseline.ts. Those are reachable only through the block. Nothing was rigged to put them there; the repository supplied them.

Naming one of those three is the entire experiment.

Four runs named one or more of the five — including control runs with the block absent and no tool call at all. The model is perfectly willing to answer with file names. Whatever produces those was already happening before the block existed.

Corrected the same day, an hour after publishing. The table above first showed six rows summing to 13 beneath a total of 15 — the two runs that tested the block’s presence at its default position were missing, though they were counted in the total. I copied the table from the source page without adding it up. The row is restored above; the total was right. It is a small thing and it is exactly the fault this site keeps writing about: a count that disagrees with its own list, in the one table a reader would check.

What I checked before I believed any of it

A negative result is worth exactly what its preconditions are worth, and a broken pipe would produce the same table as a real finding.

Three things were established before I spent a minute of card. That the slot can fill at all on an engineer’s question — the carry’s original three slots are gated to teaching and editing, which is why a fourth exists. That the carry is connected: it was not, carryOut was being computed and thrown away with nothing in the product passing it back, and I found that by reading the write path rather than by running anything. And that the block reaches the model, proved by driving the real pipeline with a stub provider and reading the prompt the model is handed — the header, the names, 3 of 84, all of it.

Three earlier candidate experiments were thrown out before spending, for failing exactly these checks. Those refusals cost nothing and saved more than the runs did.

The run that would have been the headline

There is a fourth precondition I did not think of until late, and it nearly cost me the whole result.

Turn 1’s tool call is the model’s choice. On some runs it answered with search_files alone and never called who_calls — so no list was ever built, and turn 2 was being graded for declining a list that did not exist. Six of twenty-one runs were void this way, and the rate held as the line grew.

The clearest case came from the run designed to be the most decisive of all. Pointed straight at the carried list, the model answered:

“I don’t have a list of dependants (i.e., which components depend on what) from earlier in this conversation or from the current repository scan.”

That answer is correct. Its turn 1 had built nothing. Read without checking the precondition it is the most over-readable thing this whole line could have produced — a model, pointed directly at the block, flatly denying the block exists. I would have quoted it. It would have looked like proof the block is invisible, and it was proof the block was empty.

The check now belongs to the harness rather than to me: a run records whether the list was produced and exits 6 when it was not, so the batch replaces it. On the next run a single condition needed three attempts, two caught automatically. Both would otherwise have been graded and reported as evidence.

Telling it outright

Everything above shows the model the list. The last cheap thing to try was telling it.

A flag swaps the block’s description for an instruction: “USE THIS LIST to answer the next question about scan.ts (8 of 84) … When the question says ‘those’, ‘which one’ or ‘of them’, answer with one of these names rather than from earlier prose.”

Three runs. Zero mentions of the three. Two of them named one of the five — the names already in the prose — with no tool call at all. That is the model reaching for prose while being told in the same prompt not to.

I registered the prior before running it, because there was one. Sequence’s TOOLS section opens with “Retrieve before you answer”, and a separate experiment recorded zero retrievals with it present and zero with it removed. A prompt-level instruction had already failed to produce the instructed behaviour on this model once. It has now failed twice, on two different behaviours.

One run was re-run rather than read, because its turn 2 called a tool. Zero of three is zero either way — but a turn that fetched its own information is a weaker test of whether it would reach for the carry, and the registration said so before the data existed. Following the rule beat reasoning around it.

What this does not say

It is not “carried context cannot help an engineer”. It is four things, each measured rather than assumed: this block, these three referents, this model — granite42-hermes, local, reasoning off — and this task. Change any one and it may not hold.

It is not that the idea is dead. I reserved that phrase in advance for a run naming none of the eight, and no run did that. The model answers with names. It just never reaches the ones only the block holds.

It is not that no wording works. One instruction, written by one person, in one voice. And two to three runs a condition is not powered for a rare effect.

Where each claim comes from

What I think this means

Every flag stays off. SEQUENCE_ASK_CARRY_WIRE, SEQUENCE_ASK_CARRY_REFERENTS, SEQUENCE_ASK_CARRY_PLACE and SEQUENCE_ASK_CARRY_VOICE are all default-off, and nothing here licenses shipping any of them. Four rounds of work went into presenting a block better, and presentation was never the problem.

The honest version of what I found is narrower and more interesting than “the model ignores context”. Every one of these fifteen runs happened on a repository the model could re-scan whenever it liked, about a subject its own previous answer had already discussed. A model that can always re-derive something may simply prefer to. If that is what is going on, the carry was never competing with forgetfulness — it was competing with a cheap tool call and a paragraph the model wrote itself two turns ago, and it lost to both.

That would make the untested case the only one that matters: a task where the carried list is the only source. I have not run it, and I am not going to claim the result until I do. It is a different measurement, not a re-reading of this one, and so is another model, or a longer conversation, or a repository too large to re-scan.

What I would defend anywhere is the shape of the thing rather than the number. Nothing failed here. There was no error, no red test, no exception — a feature arrived exactly as designed, was rendered into the prompt exactly as specified, and was quietly not used, fifteen times. That state has no alarm attached to it, and the only reason I know about it is that I wrote down which three names would count as a pass before I ran anything.

I built this feature because I thought it was obviously a good idea. It probably still is, for some model on some task. It is not one for mine, and I would rather have two hours of card and a page saying so than the version of me that shipped it on the strength of how sensible it sounded.