A First Taste of Algorithms · Lab 9: The Engine

after training (provided), print the attention matrix for a name

✦ ✦ ✦

seq = [stoi[c] for c in “.” + “riemann”] _, A = forward(seq) labels = list(“.” + “riemann”) print(” ” + ” “.join(f”{c:>5}” for c in labels)) for i, row in enumerate(A): print(f”{labels[i]:>4} ” + ” “.join(f”{x:5.2f}” for x in row)) ```

You will see real structure: a consonant attending to the vowel it follows, the second “n” looking back at the first. These weights were not set by anyone — descent found them, your backward sweep priced them, and now you can read the model’s own opinion about which letters matter for what comes next. This is Figure 9.3, computed in your hands.

Break it

Predict first. Run second. Explain any gap third.

1. Blame that never resets. In §4, delete the w.grad = 0.0 reset loop and run. Predict what happens to the loss before running — converges faster? slower? diverges? wildly? Then read the result and explain it through the += in every blame-rule: across steps, what is each knob’s grad accumulating, and why does that poison the descent? (This exact bug — forgotten gradient reset — is the single most common mistake in real deep-learning code; you have now made it deliberately and watched it bite.)

2. The engine versus the calculus. Build the softmax-plus-cross-entropy loss for three scores out of exp, log, and division, on Values, and run backward. Separately, compute the gradient by the deep water’s formula p − [i = t] by hand. Predict they match; verify they do to several decimals. This is the Diamond made executable — and the deepest point of the chapter. State in one sentence what the agreement proves about the relationship between your engine and Book 2.

3. Watching it memorize. In §5, shrink the training set hard — keep only the first 15 names — and retrain. Predict both curves before running: where will the held-out loss go, and how early will it turn up? Then generate from this starved model and report what fraction of samples are exact copies. Connect all of it to one sentence: what does a model do when it has more knobs than pattern to learn? (You are reproducing, deliberately, the overfitting the chapter says no one fully understands at scale.)

4. Temperature to the limits. Generate 10 names each at temp = 0.1, temp = 1.0, and temp = 3.0. Predict the character of each batch first. At very low temperature the model becomes nearly deterministic — predict what it converges toward (is it always the same name? why?). At very high temperature, predict whether the output is still namelike at all. Then connect temperature to Chapter 8: dividing all scores by T before softmax does what, geometrically, to the gap between the likeliest letter and the rest?

5. Breaking the mask. In §7’s forward, delete the two masking lines (the mask = ... and the scores = np.where(...)). The model can now attend to future characters — including the very one it is trying to predict. Predict what happens to the training loss before running: does it improve, and is that improvement real or fraudulent? Then check by trying to generate from the unmasked model and explaining why generation falls apart even though the loss looked wonderful. State the one-sentence moral about a model that is allowed to see the answer it is being graded on. (This connects directly to Chapter 8’s parrot exam — what is leakage of the test answer into the training input?)

All 32 chapters of A First Taste of Algorithms · Foundations · All courses