From model to agent

Learners will notice that the model multiplies big numbers with no tool. It is a good question, and it is not memorisation, because the model cannot have seen that exact sum before. Two things are going on.

1. It learned something like a method, not a lookup table. Anthropic’s interpretability research on addition (“Tracing the thoughts of a large language model”) found parallel pathways inside the network: one estimates roughly how big the answer should be, another works out exactly what the last digit must be, and the two are combined. Nobody programmed this; it emerged because getting arithmetic right helps predict the next word. The same research found that when the model explains how it did a sum (the school method of carrying digits), its explanation does not match what it actually did internally. A model’s account of its own working is not reliable evidence.

2. It writes out its working. In a typical reply the model splits the sum into small steps, for example “953 × 300 = 285,900; 953 × 47 = 44,791; total = 330,691”. Each step is easy, and what it has already written acts as its working memory for the next. Asked for a long answer in one go with no working, models do noticeably worse. Recent models are also trained to reason before answering.

Where it breaks. Accuracy falls as the numbers get longer and the steps multiply, and a wrong answer looks exactly as confident as a right one. A calculator is guaranteed exact. So the lesson is “usually right, with no guarantee”, and a tool is needed when that is not good enough.

A demo for the class. In Lab 3, with the calculator switched off, ask for multiplications with more and more digits (try 12 digits by 12 digits), then check each answer by switching the calculator on. Where it starts to get them wrong varies by model, so keep this as a demonstration rather than a scored challenge.