Project 9 · Part VII: Projects
A CIFAR-10 Image Classifier
Classify tiny color photographs into ten classes with five models of increasing power, from nearest neighbors on raw pixels to a convolutional network, and watch test accuracy climb from 28% to 86%.
After this project you will be able to
- Load and preprocess an image data set, and set up training, validation, and test splits.
- Implement $k$-nearest neighbors with vectorized distances and choose $k$ by cross-validation.
- Implement the multiclass SVM and softmax losses with vectorized gradients, and verify them numerically.
- Train a two-layer neural network from scratch, and a small convolutional network in PyTorch.
- Compare the five models and explain the gaps between them.
Code: projects/p9_cifar10/ (knn.py, linear_classifier.py, two_layer_net.py, cnn_torch.py, tests/). Background: Chapters 17, 16, 15, and 9.
P9.1The Problem
Classify $32\times32$ color images into ten classes: airplane, automobile, bird, cat, deer, dog, frog, horse, ship, and truck. Chance is 10%. The project retraces the path of Chapter 17: a nonparametric baseline, linear classifiers, a fully connected network, and a convolutional network, each implemented and evaluated on the same splits, so that the effect of every modeling idea can be measured.
P9.2The Data
CIFAR-10 (Krizhevsky 2009) has 50,000 training and 10,000 test images, 6,000 per class, small enough that every model in this project trains on a laptop CPU. data_utils.py downloads the Python version of the data set (about 170 MB) from its authors’ page on first use; it is not shipped with the book’s code. We hold out the last 1,000 training images as a validation set for choosing hyperparameters, and touch the test set only to report final numbers (Chapter 12).
For the NumPy models, each image is flattened to a vector of 3,072 pixel values and the mean training image is subtracted, which centers the data without changing its scale. For the CNN, images stay as $3\times32\times32$ arrays and each color channel is standardized with the channel means and standard deviations of the training set.
P9.3Building It from Scratch
$k$-nearest neighbors. The expensive part is the matrix of distances between every test image and every training image. Written with two loops it takes minutes; using $\norm{\va-\vb}^2=\norm\va^2-2\va\T\vb+\norm\vb^2$, it is one matrix product, as in Section 9.5. Predictions take a majority vote among the $k$ nearest training images. Because every query is compared with all stored images, we use a subset of 5,000 training and 500 test images, and choose $k$ by 5-fold cross-validation (Section 5.5), computing the distances once per fold and reusing them for every candidate $k$.
Linear classifiers. With a bias column appended, a linear classifier is a $3073\times10$ weight matrix $\mW$, and the scores of a mini-batch are $\mX\mW$. Both losses of Section 17.2 and their gradients are fully vectorized; the SVM gradient puts $+\vx_i$ in the column of every class that violates the margin and minus the number of violations times $\vx_i$ in the correct class’s column (Section 9.8):
import numpy as np
def svm_loss(W, X, y, reg):
"""Multiclass hinge loss (Eq. 9.9 with margin 1) + reg * ||W||^2. W: (D, C), X: (N, D)."""
N = len(X)
scores = X @ W
correct = scores[np.arange(N), y][:, None]
margins = np.maximum(0, scores - correct + 1.0)
margins[np.arange(N), y] = 0
loss = margins.sum() / N + reg * np.sum(W * W)
# gradient: +x_i for every violating class, -(number of violations) x_i for the true class
G = (margins > 0).astype(X.dtype)
G[np.arange(N), y] = -G.sum(1)
return loss, X.T @ G / N + 2 * reg * W
def softmax_loss(W, X, y, reg):
"""Softmax cross-entropy + reg * ||W||^2, computed stably from the scores (Section 6.4)."""
N = len(X)
scores = X @ W
scores = scores - scores.max(1, keepdims=True)
logp = scores - np.log(np.exp(scores).sum(1, keepdims=True))
loss = -logp[np.arange(N), y].mean() + reg * np.sum(W * W)
P = np.exp(logp)
P[np.arange(N), y] -= 1 # dL/dscores = p - onehot
return loss, X.T @ P / N + 2 * reg * W
rng = np.random.default_rng(0)
X, y, W = rng.normal(size=(5, 4)), rng.integers(0, 10, 5), 1e-3 * rng.normal(size=(4, 10))
print("softmax loss with tiny weights:", round(softmax_loss(W, X, y, 0.0)[0], 3), "~ log 10 = 2.303")
print("SVM loss with tiny weights:", round(svm_loss(W, X, y, 0.0)[0], 3), "~ K - 1 = 9")
These two sanity checks of initial loss (Section 16.6) are worth running before any training: with near-zero weights all scores are equal, so the softmax loss is $\log10\approx2.303$ and every one of the nine wrong classes violates the margin by 1. A centered finite-difference check at random entries of $\mW$ then confirms the gradients, with relative errors around $10^{-11}$ for the SVM and $10^{-7}$ for the softmax. Training uses mini-batch SGD with batches of 200 images, and a grid search over learning rate and regularization strength picks the configuration with the best validation accuracy.
A two-layer network. TwoLayerNet computes $\vs=\mW_2\relu(\mW_1\vx+\vb_1)+\vb_2$ with 200 hidden units (616,610 parameters), the softmax loss, and the backward pass of Section 15.3. Pixels are additionally divided by their overall standard deviation so that the default initialization scale works. It is trained for 4,000 steps of SGD with momentum 0.9, decaying the learning rate by 5% after every epoch, and its gradients are checked numerically on a small version of the network.
P9.4Using Libraries
The convolutional network in cnn_torch.py applies the recipe of Section 17.7: three stages of two $3\times3$ convolutions with batch normalization and ReLU, max pooling after the first two stages, global average pooling, and a linear classifier (288,746 parameters). Training augments each image with a random crop from a 4-pixel reflection-padded copy and a random horizontal flip, uses SGD with Nesterov momentum, weight decay $5\times10^{-4}$, label smoothing 0.1, a half-epoch warm-up, and cosine decay of the learning rate from 0.1, for 8 epochs. The trained weights are saved for Project 12.
import torch.nn as nn
def block(cin, cout):
return nn.Sequential(nn.Conv2d(cin, cout, 3, padding=1, bias=False), nn.BatchNorm2d(cout), nn.ReLU(inplace=True))
class SmallCNN(nn.Module):
def __init__(self, width=32, num_classes=10):
super().__init__()
w = width
self.features = nn.Sequential(
block(3, w), block(w, w), nn.MaxPool2d(2), # 32 -> 16
block(w, 2 * w), block(2 * w, 2 * w), nn.MaxPool2d(2), # 16 -> 8
block(2 * w, 4 * w), block(4 * w, 4 * w)) # 8x8 maps, used by Grad-CAM
self.head = nn.Sequential(nn.AdaptiveAvgPool2d(1), nn.Flatten(), nn.Linear(4 * w, num_classes))
def forward(self, x):
return self.head(self.features(x))
print(sum(p.numel() for p in SmallCNN().parameters()), "parameters")
P9.5Results and Discussion
All numbers are test accuracies on the 10,000 test images (500 for $k$-NN), measured on a 4-core CPU:
| Model | Validation | Test | Notes |
|---|---|---|---|
| $k$-NN on pixels, $k=10$ | 28.0% (CV) | 28.2% | 5,000 training images; seconds |
| Linear SVM | 39.6% | 38.1% | learning rate $10^{-7}$, $\lambda=5\times10^3$ |
| Softmax classifier | 40.5% | 39.4% | learning rate $3\times10^{-7}$, $\lambda=10^3$ |
| Two-layer network (NumPy) | 52.4% | 52.0% | 200 hidden units; about one minute |
| Small CNN (PyTorch) | — | 86.4% | 8 epochs, about 80 s per epoch |
Nearest neighbors reach 28%, almost three times chance but far from useful. Cross-validation accuracy was flat between $k=5$ and $k=20$: on raw pixels, the nearest images share colors and backgrounds more often than objects, so adding more voters helps little (Section 17.1).
Linear classifiers gain 10 points by learning one template per class. The SVM and softmax losses perform within about one point of each other, as Section 17.2 anticipated; the regularization strength mattered more than the choice of loss.
The two-layer network adds another 13 points. Its 200 hidden units act as 200 learned templates that the output layer combines, so a class can be recognized from several different-looking prototypes. Validation accuracy was still creeping up after 16 epochs, and it would improve somewhat with longer training and tuning, but fully connected networks on raw pixels plateau around 55%.
The convolutional network reaches 86% with fewer than half the parameters of the two-layer network, the largest jump of the project. Weight sharing and locality (Section 17.3) let it reuse the same edge and texture detectors at every position, and the augmentation teaches it that a shifted or mirrored car is still a car. Test accuracy rose from 64% after the first epoch to 86% after the eighth; the dip after epoch 2 is typical of the early, high-learning-rate phase of the schedule, and the largest gains came as the cosine schedule lowered the learning rate. Larger networks (ResNets) trained for longer reach 95% and more on CIFAR-10.
P9.6Extensions
- Replace the raw pixels given to the linear classifier by color histograms and histograms of oriented gradients (HOG). How much do hand-designed features help?
- Add dropout and a third layer to the NumPy network and tune it with random search (Section 16.6); can you reach 55%?
- Train the CNN for 30 epochs, then double its width. Plot test accuracy against training time for both.
- Plot the confusion matrix of the CNN. Which pairs of classes are confused most often, and why?