Welcome back
Last time we stood in front of a cathedral and saw how it was built: bricks into walls, walls into arches, arches into something that takes your breath away. A network looking at a picture works the same way, building big ideas out of small ones.
But a picture arrives all at once. You can see the whole thing in a single glance. Today we sit down beside a fire and think about things that don’t arrive all at once at all.
Around the fire
Imagine a circle of friends around a campfire, late in the evening. Someone begins a story. “There was an old cabin at the edge of the lake…”
Each new sentence only makes sense because of the ones before it. When the storyteller whispers, “…and the door was open again,” you shiver. But you only shiver because you remember that, ten minutes ago, she told you she’d locked it.
A story is a sequence: things that come one after another, where order matters and the past gives meaning to the present. Spoken words, music, the weather day by day, the price of bread month by month: all of these are sequences.
The trouble with forgetting
The networks we’ve met so far have a peculiar habit. They look at their input, produce an answer, and then forget everything. Show them the next word of a story and they meet it as a total stranger, with no memory of the cabin or the lake.
For a single photograph, that’s fine. For a story, it’s hopeless. Hearing the word “again” tells you nothing unless you remember what happened the first time.
A network that listens as it goes
A recurrent neural network, or RNN, solves this with one simple, lovely idea: it keeps a little note to itself.
It reads a sequence one piece at a time, perhaps one word at a time. At each step it does two things:
- It looks at the new word in front of it.
- It looks at its own note from the step before, a small bundle of numbers summarising what it has heard so far. We call this note the hidden state.
It blends the two together and writes a fresh note, which it carries forward to the next word. Then the next. Then the next.
So the note after “There was an old cabin” holds something cabin-ish. By the time the network reaches “the door was open again,” that note still carries a faint memory of the locked door. Like a listener by the fire, the network hears each new sentence in the light of all the others.
The word recurrent simply means “happening again and again.” The same small set of weights is used at every step, in a loop, passing the note along.
What to carry with you
A recurrent network reads a sequence one step at a time, carrying a small note of what came before. That note lets the past give meaning to the present, though over long stories, it tends to fade.