Reward prediction error

Midbrain dopamine neurons fire when a monkey receives unexpected juice. Train the monkey so a tone predicts the juice, and the firing moves to the tone; the juice itself, now expected, produces nothing. Omit the juice after the tone and the cells go below baseline at exactly the moment it was due.

That is not a reward signal. It is an error signal — reward minus prediction, signed.

The crossing

This is the clearest case on the site of a formalism arriving first and turning out to be what the biology was already doing.

The problem is one of delayed feedback, familiar to anyone who has tuned a controller: if the consequence of an action arrives long after the action, how do you assign credit to the right earlier state? Temporal-difference methods answer it the way a control engineer would — do not wait for the outcome, correct on the discrepancy between successive predictions:

$$\delta_t = r_t + \gamma V(s_{t+1}) - V(s_t)$$

Schultz, Dayan and Montague pointed out that $\delta_t$ — a quantity invented to make an algorithm converge — has the same signature as the dopamine response, including the shift to the predictive cue and the dip below baseline on omission. The formalism was not built to explain dopamine. It already existed, and the neurons matched it.

That is a stronger claim than an analogy. The dip on omission is the part that convinces: a system merely reporting reward has no reason to go negative when nothing happens. A system reporting error has no choice.

Caveats worth stating

The story is cleaner in review articles than in the data. Dopamine also responds to novelty, salience, and stimulus intensity in ways a pure scalar error term does not predict; there are multiple dopamine populations with different tuning; and the distributional picture that has emerged more recently suggests neurons encode a distribution over rewards rather than one expected value. The prediction-error account remains the best organising idea in the area, which is not the same as being complete.

The reason this sits close to my own work: an error signal is only worth having if something can act on it. A controller needs an actuator, and for the nervous system that means stimulation you can shape and time rather than the fixed pulse trains a conventional implant delivers open-loop.