Math is in the Air

Attention explained!!

Backpropagation is easiest to understand when the network is small enough that you can follow every number by hand. The example below has one input, two hidden ReLU units and a linear output. We begin with a normal forward pass. The input is multiplied by the hidden-layer weights, ReLU produces the two hidden activations, and the output layer combines them into a prediction. Comparing that prediction with the target gives a loss of 0.5. Now comes the part we actually need for training. We want to know how sensitive the loss is, locally, to each weight in the network. A hidden-layer weight does not touch the loss directly, so we have to follow the route through which it affects the result. It changes a hidden unit, that hidden unit changes the prediction, and the prediction changes the loss. The chain rule gives us a way to combine those effects. This is what backpropagation is doing as it moves through the computation graph. At each step it takes the gradient arriving from later in the network and combines it with the local derivative of the operation at that point. If a value contributes to several later computations, the gradient contributions from those paths are added before propagation continues. The ReLU units in this example also show why activation derivatives matter. Both hidden pre-activations are positive, so the local ReLU slope is 1 and the gradient can pass through those activations without additional scaling. Had one of those pre-activations been negative, the local ReLU slope would have been zero and that path would have contributed no gradient to the weight behind it for this example. After working backward through the network, we have a gradient for each of the four weights. Backpropagation has done its job at that point. Plain SGD then uses those gradients to update the parameters. The important detail is that all four gradients are computed at the same original parameter values, before any of the weights are changed. Running the network again with the updated weights gives a prediction of -0.49 and a loss of 0.12005, down from 0.5 in this particular step. The decrease is useful for the example, although a gradient update with an arbitrary learning rate is not guaranteed to reduce the loss on every step. The networks used in practice are much larger, but the underlying calculation is the same. Forward computation produces the loss, reverse-mode differentiation propagates sensitivities backward through the graph, and the optimizer uses the resulting gradients to change the parameters. Backpropagation computes the gradients. SGD is what changes the weights.