Machine Learning from First Principles

Project 3 · Part VII: Projects

A Handwritten Digit Recognizer

A neural network written in NumPy, from forward pass to backpropagation and a gradient checker, then rebuilt in PyTorch and beaten by a small convolutional network. Draw your own digit at the bottom of the page.

After this project you will be able to

  • Implement an MLP and backpropagation in NumPy for any depth.
  • Verify backpropagation with finite differences.
  • Train with SGD, momentum, and weight decay.
  • Rebuild the model in PyTorch and improve it with a CNN.

Code: projects/p3_digit_recognizer/. Background: Chapters 15, 16, and 17.

P3.1The Problem

Recognize handwritten digits 0–9 from small grayscale images. Metric: test accuracy and the confusion matrix.

P3.2The Data

scikit-learn's load_digits: 1,797 images of $8\times8$ pixels with values 0–16, made by counting "on" pixels in $4\times4$ blocks of $32\times32$ bitmaps. We divide by 16, flatten to 64 numbers, and split 75/25 (1,347 / 450), stratified.

P3.3Building It from Scratch

def forward(params, X):
    hs, zs = [X], []
    for i, layer in enumerate(params):
        z = hs[-1] @ layer["W"] + layer["b"]
        zs.append(z)
        hs.append(np.maximum(z, 0) if i < len(params) - 1 else z)
    logits = hs[-1] - hs[-1].max(1, keepdims=True)
    return hs, zs, np.exp(logits) / np.exp(logits).sum(1, keepdims=True)

def loss_and_grads(params, X, y, weight_decay=0.0):
    B = len(y)
    hs, zs, P = forward(params, X)
    loss = -np.mean(np.log(P[np.arange(B), y] + 1e-12))
    dZ = P.copy(); dZ[np.arange(B), y] -= 1; dZ /= B
    grads = [None] * len(params)
    for i in reversed(range(len(params))):
        grads[i] = {"W": hs[i].T @ dZ + weight_decay * params[i]["W"], "b": dZ.sum(0)}
        if i > 0:
            dZ = (dZ @ params[i]["W"].T) * (zs[i - 1] > 0)
    return loss, grads

Gradient check first: 20 random entries against centered finite differences give a worst relative error of about $7\times10^{-9}$. Training: 64–64–10, He initialization, batches of 32, SGD with momentum 0.9, learning rate 0.1, weight decay $10^{-4}$, 60 epochs: under two seconds.

P3.4Using Libraries

mlp = nn.Sequential(nn.Linear(64, 64), nn.ReLU(), nn.Linear(64, 10))
cnn = nn.Sequential(
    nn.Unflatten(1, (1, 8, 8)),
    nn.Conv2d(1, 32, 3, padding=1), nn.ReLU(),
    nn.Conv2d(32, 32, 3, padding=1), nn.ReLU(), nn.MaxPool2d(2),
    nn.Flatten(), nn.Linear(32 * 16, 64), nn.ReLU(), nn.Linear(64, 10))

P3.5Results and Discussion

ModelParametersTest accuracy
NumPy MLP 64–64–104,81097.6%
NumPy MLP 64–32–10 (used below)2,41097.1%
Small CNN, PyTorch43,05098.7%

Accuracy rises from 90% after one epoch to 97% after 30, then plateaus while training loss keeps falling. The 11 remaining errors are scattered; the most common is an 8 and a 1 mistaken for each other. The CNN halves the error rate with the same data, because its assumptions (locality, weight sharing) match images.

Your drawings differ from the training writers in pen width, slant, and position: a small, real distribution shift. Expect lower accuracy than 97%. Matching the preprocessing (crop, center, block-count to $8\times8$) is what makes the demo work at all.

P3.6Extensions

  1. Implement Adam yourself and compare training curves.
  2. Add dropout to the NumPy network.
  3. Train a CNN on MNIST; how large must it be to exceed 99%?
  4. Show the first-layer weights as $8\times8$ images.
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.