The picture that answered its own question

Showing someone a diagram teaches them very little. Having them commit to what the diagram will say teaches them a great deal. I did not know that when I built the teaching part of my product on the opposite belief, and I found out by going to look for the evidence behind something I had been saying out loud for months. What follows is that evidence, the defect it exposes in a feature I was pleased with, and the change I am making, written down before I build it.

What it is, and why teaching is the third thing it does

Sequence is for a software engineer with a hard system in front of them. It draws the system from the code, keeps the drawing honest against the repository, and puts a harness around the design so the shape you end up with is one you chose rather than one you drifted into. Teaching is the third tier of that, behind the engineer and behind the ordinary builder, and it exists for a plain reason: the drawing that helps one person design a system is the drawing that helps another understand one.

So teach mode takes a repository, a level and a question about one file, and answers with a lesson, a chart of that part of the system, and a check that you followed. On Friday it began ending every turn with a question built from the chart’s own links: Looking at the picture: if brief.ts changed what it returns, which of cli.ts, repoServer.ts or brief.test.ts do you think would break first? It is generated in code, so it cannot be wrong about the repository, and it is graded from the learner’s next message. I was pleased with it.

What the evidence says

Retrieval practice works: about half a standard deviation across 222 classroom studies and 48,478 students, and more with feedback than without. Guessing before you know works too, even when the guess is always wrong, provided the answer follows. And a prediction helps most when the outcome can surprise you. In one study the learner’s surprise response appeared only after a prediction had been made, and the size of the surprise tracked the learning.

The one I keep coming back to is a physics lecture. Students who watched a demonstration understood the concept no better than students who never saw it: 24 percent could explain it afterwards, against 22. Students who predicted the outcome before watching: 30. The demonstration was worth nothing. The two minutes of commitment in front of it were worth everything.

What my question actually does

It asks about a picture already on the screen, whose arrows already answer it. A learner who can read an arrow answers correctly without predicting anything, and a learner who cannot is not helped by being handed three options. There is no moment at which the picture can surprise, because the picture came first. I had built the demonstration and then asked the students what they had just watched.

The change, registered before I build it

Ask about the picture that comes next. The product already knows what that is: the concept queue and the chart are both built in code before the learner sees either. So the closing question becomes Next, Sequence draws what depends on brief.ts. Of cli.ts, repoServer.ts and brief.test.ts, which do you think breaks first if brief.ts changes what it returns? The next turn’s chart is the answer, revealed exactly as it is now, with one line added naming the arrow that decided it.

The prediction, written down before the build. With the picture on screen, the derived question is answered correctly at least nine times in ten, because it is a reading. About the next picture, between three and seven in ten. If the second number is also above nine in ten, the prose gave the answer away and the picture adds nothing. If it is under two in ten, the question is unanswerable from the lesson and I have built a quiz.

What I think this means

The order is the feature. Show-then-ask and ask-then-show use the same chart, the same words and the same model call. The only difference is which side of the picture the question sits on, and in that study the difference is the entire effect. Anyone building something that explains a system to a person has already made this choice, and I would guess most of them made it the way I did, without noticing there was one.

So the cheapest change available to a builder this week is moving the question in front of the diagram. It costs no model calls and no new interface. If you are writing documentation, an onboarding tour, or an agent that walks somebody through an unfamiliar codebase, there is already a moment where you show a thing, and you can ask about it one paragraph earlier than you do now.

I am not claiming the picture teaches by being shown. The 24 against 22 says it does not. Nor that I am a visual learner and this is therefore right for me: the review that went looking for that kind of interaction found virtually none. What survives is narrower than what I used to say and I think it is stronger. Interactive is supported. Visual is supported when the learner commits to something about the picture before or while reading it. Visual on its own, with no commitment, is the 24 percent.

None of that is settled by my having registered a prediction about it, and the run that tests the change has not happened yet. The bands exist so the product can tell me I was wrong a second time rather than let me be pleased twice.

The belief was mine, the product was built on it, and nothing except going to look was ever going to move it. I would rather have it narrowed in public than kept and flattering.

Sources