Welcome back
Last time, we went running. We saw that a network learns through a simple loop (guess, measure, trace, adjust), repeated over many epochs, and that we watch the loss fall as it goes.
But a falling loss raises a quiet question. How do we know the network has really learned something, and not just got very good at the particular examples we showed it? To answer that, let’s step into a classroom.
A teacher’s oldest rule
Picture a maths teacher preparing a class for an end-of-year exam. All term, she gives them practice problems. They work through them, make mistakes, see the corrections, and improve.
Now imagine that, on the day of the final exam, she hands out the exact same problems they practised on. Every student scores perfectly. Has the class mastered maths? Not necessarily. Some may truly understand. Others may simply remember the answers. From the scores alone, she can’t tell the difference.
A wise teacher keeps the final exam secret. She tests her students on problems they’ve never seen. Only then does a good score mean real understanding.
Splitting the examples three ways
With neural networks, we follow the same wise rule. Before training begins, we divide our examples into three separate piles:
- Training data, the practice problems. This is the big pile, often around 70 to 80 percent of everything. The network learns from these, adjusting its weights in the training loop.
- Validation data, the mock exams. A smaller pile the network never learns from directly. Every so often, perhaps after each epoch, we quietly check how well it does on these. It’s like a teacher setting a practice test halfway through term to see how the class is really getting on.
- Test data, the final exam. Another small pile, locked away until the very end. We use it once, after all training and tinkering is done, to get an honest measure of how well the network will do on examples it has truly never met.
Why we need the middle pile
You might ask: if we have a final exam, why bother with mock exams too?
Because along the way, we make decisions. How big should the learning rate be? How many layers? When should we stop training? Each time we check our choices against some data, we learn a little about that data, and our choices start to fit it. If we used the final exam for all those checks, it would slowly stop being a fair test.
So the validation pile is where we do our checking and adjusting. The test pile stays sealed, so that at the end it can tell us the truth.
Reading the signs
During training, we watch two numbers side by side: the loss on the training data and the loss on the validation data.
When both fall together, all is well. The network is learning patterns that hold up on new examples. But sometimes the training loss keeps falling while the validation loss stalls, or even starts to rise. That’s the classroom’s warning bell. It means the network is getting better at its practice problems but worse at fresh ones. It has started memorising.
What to carry with you
We split our examples into training, validation and test data. The network learns from the first, we check our choices with the second, and the third, kept sealed until the end, tells us honestly how well it has learned.