Welcome back
Last time, in the bakery, we learned that networks learn from batches, small random handfuls of examples, taking one step downhill per batch. That’s stochastic gradient descent.
Today we return to a thought I asked you to tuck away back in Lesson 10, with the dominoes: that multiplying many numbers together can do strange things. It’s time to see why that matters. Let’s play a game.
The whisper game
You probably played this as a child. A group of friends sits in a long line. The first person whispers a message, say “The cat sat by the window,” to the second. The second whispers what they heard to the third, and so on, down the line.
With three or four people, the message usually arrives intact. But with twenty people, something happens. Maybe each person speaks a little too softly. The message fades and fades until the last child hears only a breath and shrugs. Or maybe each person exaggerates a little. “The cat sat” becomes “The cat leapt,” then “The tiger leapt,” until the last child shouts something wild about a tiger on the roof.
Either way, the original message doesn’t survive a long line.
The same problem, inside a network
Remember backpropagation. The blame for a mistake starts at the output and travels backward, layer by layer, toward the earliest weights. And remember the chain rule: at each layer, the “effect so far” is multiplied by that layer’s own small handoff.
Now picture a very deep network with, say, fifty layers. That’s a long line of whisperers.
If each handoff is a bit smaller than one, say a half, then by the time the signal reaches the early layers, it has been multiplied by a half fifty times. That’s an almost unimaginably tiny number. The early layers receive a whisper so faint it’s practically silence. Their weights barely change, and they stop learning. This is the vanishing gradient problem.
If each handoff is a bit bigger than one, say two, the opposite happens. Two multiplied by itself fifty times is enormous. The early layers receive a shout so loud their weights leap wildly, and training falls apart. This is the exploding gradient problem.
For many years, this is why deep networks were so hard to train. The front of the line simply never heard the message clearly.
How we learned to keep the message alive
People found several remedies, and you’ll meet some of them properly in coming lessons.
- Better activation functions. Remember the musical note from Lesson 4, the rule that decides how strongly a neuron fires? Older choices squashed signals so much that they shrank at every layer. A simpler rule called ReLU (it just passes positive signals through unchanged and silences negatives) lets the message travel much further without fading.
- Careful starting weights. Choosing where the weights begin, so the handoffs start close to one. That’s next lesson.
- Keeping signals steady. Rebalancing signals as they pass between layers (Lesson 19).
- Clipping. Simply capping how big a step can be, so an exploding shout can’t knock everything over.
What to carry with you
In a deep network, the learning signal is multiplied at every layer on its way back. If it shrinks each time, it vanishes. If it grows each time, it explodes. Either way, early layers struggle to learn.