The artificial neuron

This entry is about an origin, not a field. What crossed from neuroscience into engineering in 1943 was a single idea — that a neuron could be treated as a computing element — and that idea is still the unit inside anything called a network today. Where it went afterwards is somebody else’s subject.

The abstraction

McCulloch and Pitts asked what remains of a neuron if you keep only its logic. Their answer: a unit that sums its inputs with weights and produces an output when the sum crosses a threshold.

$$y = \Theta\!\left(\sum_i w_i x_i - \theta\right)$$

Here $x_i$ is 1 if input $i$ is active and 0 if not, $w_i$ is that input’s weight, $\theta$ the threshold, and $\Theta$ a step that gives 1 for a positive argument and 0 otherwise. In 1943 the weights were whole numbers — a cell making two synapses on another counted twice — and inhibition was not a negative weight but a veto: one active inhibitory synapse kept the cell silent whatever else arrived.

They then showed that networks of such units could compute any logical proposition. That was the point of the paper — not to model neurons well, but to establish that something this simple was already sufficient for computation.

Strictly, the theorems cover every proposition that is false when all its inputs are silent: a unit fires only when something excites it, so no net can announce that nothing happened. That a net given a tape could compute exactly what a Turing machine can was stated in a closing remark, without proof.

It is worth being blunt about how little biology survived the abstraction. There is no time, no spike, no dendrite, no adaptation, no refractory period. Time survives only as a clock tick one synaptic delay long, and a unit carries nothing from one tick to the next. Set beside the Hodgkin–Huxley model, written nine years later and describing an actual membrane, the McCulloch–Pitts neuron is not a competing model of a neuron. It is a different kind of object: a claim about computation that borrowed a neuron’s name.

Borrowed: a threshold and a sum

The rules the unit obeys came from the physiology that the paper’s introduction summarises. A neuron fires all or none. No single synapse was known to fire one, but enough of them excited within about a quarter of a millisecond would, and the impulse followed after a synaptic delay of more than half a millisecond.

The two ideas underneath — an output of fixed size, and inputs summed against a threshold — were both set out in Nobel lectures in December 1932, when Charles Sherrington and Edgar Adrian shared the Nobel Prize in Physiology or Medicine. Adrian, following Keith Lucas, had shown that a nerve fibre’s impulse is the same whatever the stimulus, so a fibre can vary its message only by how often and how many times it fires — rate coding. Sherrington described subliminal excitations adding up to threshold, and excitation and inhibition summing algebraically at a single neuron. Together that is the neuron read as a thresholded sum of its inputs: the physiologists’ reading, before McCulloch and Pitts made it a model.

What they added was the step to logic. One of them, the paper says, had long taken a neuron’s response as equivalent to a proposition that its adequate stimulus had occurred, and the all-or-none law lets every firing stand for a proposition, true or false.

Why it counts as engineering

Because the second step made it a machine. Rosenblatt’s perceptron added a rule for adjusting the weights from examples, so the unit could be trained rather than specified — and he built it. The Mark I Perceptron was analog hardware: an array of photocells for input, and weights held as potentiometer settings driven by electric motors. Learning was a physical process, literally turning knobs.

That is the part that makes this belong here rather than in a history of computer science. The first artificial neuron that learned was an electrical machine, and the reason it was built that way is the reason neuromorphic designers give now: the arithmetic is nearly free if you let the physics do it.

Rosenblatt called his neuron model a direct descendant of McCulloch and Pitts’, and said his approach had been shaped by Hebb and Hayek. The Mark I was built at the Cornell Aeronautical Laboratory in Buffalo with funding from the Office of Naval Research, its engineering done mainly, he wrote, by Charles Wightman and Francis Martin.

Its operators’ manual of February 1960 gives the numbers. The eye was 400 photoresistors in a 20 × 20 grid at the film plane of a modified view camera, wired pseudo-randomly to 512 association units. Each was a transistor amplifier and a relay that closed when the algebraic sum of its input voltages exceeded a threshold — the McCulloch–Pitts unit, with inhibition subtracting instead of vetoing, a swap their fourth theorem had shown to be equivalent. Each fed one of eight response units through a potentiometer turned by a small DC motor, and the manual is plain that the wiper positions of those 512 potentiometers were the perceptron’s memory. At full drive a shaft turned about a sixteenth of a revolution a minute.

The weight was the hard part

A McCulloch–Pitts net remembers only by keeping activity going round a loop — the 1943 paper replaced synapses altered by learning with circles of neurons — and Rosenblatt drew the consequence: nothing survives the net falling silent. A learning machine wants a part that holds an analog value with the power off and changes it on command. In 1961 Rosenblatt named it as one of two obstacles: physical perceptrons of significant size were waiting on a cheap, mass-producible integrator, and on a cheap way to wire large networks. The Mark I’s motor-driven potentiometers and patch panel, he wrote, would not do for very large systems.

At Stanford, Bernard Widrow and his doctoral student Ted Hoff had reached the same unit from switching circuits. Their 1960 Adaline — for adaptive linear — was about the size of a lunch pail, with 16 inputs set on a 4 × 4 array of toggle switches, and an operator trained it by adjusting all 17 gains, threshold included, until the error on each pattern was zero. That procedure became the least-mean-squares learning rule: move each weight by the error times its input.

That October, Widrow described a weight that changed electrically: the memistor, a resistor with memory, made from an ordinary pencil lead in a bath of copper sulphate and sulphuric acid. Direct current plated copper on or off, so the conductance followed the time integral of the control current; a small alternating current sensed it; and each memistor plated on or off by the sign of error times input. It was an integrator in the strict sense, pitched as cheap and mass-producible, and the report’s list of next steps ended with memistors made by thin-film techniques. A weight held as a conductance that the training current sets is the idea that memristive synapses pursue today.

What the first machines could not do

A single thresholded sum can only split its inputs with a flat cut: a line for two inputs, a plane for three. Widrow and Hoff put it in switching terms — one unit realises only the linearly separable functions of its inputs, a vanishing fraction as inputs are added. McCulloch and Pitts never faced this, since they wired units in layers by design; the learning machines adjusted only their last layer. Rosenblatt put the alphabet, in a fixed position, at about the limit of what the Mark I could learn, and all 26 letters in two typefaces at once defeated it.

The memories had physical faults too. The Mark I’s motors would not run below a minimum voltage, so a decayed memory kept the sign of the old one, and a reset had to drive every value negative first. The memistors held a training for days, but only in a bath rich in copper sulphate, since the acid dissolved thin platings, and an operator still held a switch for each adaptation.

Where this entry stops

At the origin, deliberately. The subsequent history — hidden layers, gradient methods, convolutional architectures, the whole modern apparatus — is computer science, and following it would swamp a site about circuits and nervous systems.

One thing from that history is worth recording, because it reflects back on the crossing itself: the methods that eventually made these networks work are not ones a neuron could plausibly use. So the honest reading is that biology did not hand engineering a working design. It handed over a framing — computation as many simple adaptive elements in parallel, rather than one fast sequential processor — and the framing outlived every detail that came with it.