When a phone recognizes a face in a photo or translates a sentence into English, no programmer has written a rule like "eyes above nose." The software itself draws rules from many examples. The most popular learning method today is the artificial neural network, and its secret is quite simple: adjust millions of small numbers called weights, a little at a time, until the result is correct.

A neuron can only multiply and add

According to MIT News, neural networks loosely simulate the brain, consisting of many simple processing nodes connected together, often arranged in layers. Each line connecting a node carries a number called a weight. As data flows through, the node multiplies each input value by the corresponding weight and then adds it all up into a single number. If this sum passes a threshold, the node passes the result to the next layer.

A large weight means that input is valued, a negative weight drags the result down. An image recognition network has an input layer that receives pixels, several layers in between, and an output layer that answers "cat" or "dog". Everything the network learns is within those weights.

An artificial neuron multiplies each input by a weight and then adds it together. The multi-layer network guesses the result, compares it with the answer, then transmits the error back to adjust the weight

Measuring error using loss function

According to MIT News, at the beginning of training, all weights and thresholds are assigned random values, so the network guesses almost haphazardly. People include pre-labeled examples, such as thousands of pictures clearly labeled as cars, houses or cups. Each time the network guesses, the result is compared with the correct answer, and the distance between the two is calculated into a number called a loss function, or loss. The more wrong you guess, the bigger your loss. Learning means making this number smaller and smaller.

Tracing the way down to the valley

Imagine the loss is the height of a mountainous area, and the weights are the position of the person standing on it. If you want to get to the lowest place in the fog, the simple way is to see which way the ground below you slopes down and then take a step in that direction, repeating many times. That is the gradient descent method. According to Wikipedia, it is often credited to French mathematician Augustin-Louis Cauchy, who proposed it in 1847.

Gradient descent: with each step, the weights are nudged in a direction that reduces the error, like groping down a valley

The gradient shows how much the loss will increase or decrease when each weight is slightly increased. At each step, the weights are adjusted in the direction opposite to the gradient. Step length is called learning speed: small steps mean slow learning, large steps make it easier to jump over the bottom.

Backpropagation: divide the error by each weight

A network can contain millions of weights, and the error in the output is the cumulative result of them all. The backpropagation algorithm, or backpropagation, deals with error sharing. After calculating the loss in the last layer, it goes back from the output to the input, layer by layer, using the rule of derivatives of composite functions to calculate how much each weight contributes to the error. With this number, the downhill method knows which weights to adjust and in which direction.

According to Wikipedia, the modern form of this technique was published by Seppo Linnainmaa in 1970. A 1986 Nature article by David Rumelhart, Geoffrey Hinton, and Ronald Williams popularized it, making it possible to effectively train multilayer networks. A training round therefore consists of four steps: feed the data forward, measure the error, propagate the error back, then nudge the weights, repeating millions of times.

Advertisement

From a machine with eyes in 1958

According to the Cornell Chronicle, in July 1958, the US Office of Naval Research demonstrated psychologist Frank Rosenblatt's perceptron: a 5-ton IBM 704 machine, after 50 tries, learned to distinguish the card marked on the left from the card marked on the right. Rosenblatt and his colleagues then built the specialized Mark I Perceptron machine at the Cornell Aeronautical Laboratory in Buffalo. The machine is currently in the collection of the National Museum of American History (Smithsonian).

The Mark I Perceptron in the photo released by the US Navy. The machine has an "eye" consisting of 400 photoelectric cells, which are taught to recognize pre-printed letters placed in front of it

According to Wikipedia, the Mark I has 400 photovoltaic cells arranged in a 20 x 20 grid. The weights are stored in potentiometers, and as the machine learns, electric motors rotate the potentiometers to adjust the weights. The weight shifting that today's network is done by calculation, Mark I is done by mechanics. The perceptron only has one layer of learnable weights, so its learning power is limited. We have to wait for backpropagation, then the GPU graphics chip calculates in parallel, and the neural network can stack dozens of layers, what is now called deep learning.

Today's AI models have billions of weights, but the basic learning method is still the same: guess, measure the error, find a way to reduce the error, then nudge each weight a little.

Source:

MIT News: Explained: Neural networks

,

Wikipedia: Gradient descent

,

Wikipedia: Backpropagation

,

Wikipedia: Perceptron

,

Cornell Chronicle: Professor's perceptron paved the way for AI

,

Smithsonian NMAH: Electronic Neural Network, Mark I Perceptron

Advertisement