Machine Learning from First Principles

Chapter 17 · Part IV: Deep Learning

Convolutional Neural Networks

Images are large, structured, and full of nuisance variation. Convolutional networks build in locality and translation equivariance, learn a hierarchy of features from edges to objects, and turned image classification from a research problem into a commodity.

After this chapter you will be able to

  • Describe image classification as a data-driven problem and explain why nearest neighbors on raw pixels fail.
  • Interpret a linear classifier on images as templates and as hyperplanes, and train it with the multiclass SVM or softmax loss.
  • Compute a convolution, its output size, parameter count, and cost, and explain weight sharing, 1-by-1 and depthwise convolutions.
  • Compute receptive fields of stacked convolution and pooling layers.
  • Derive the backward pass of a convolutional layer and implement convolution as a matrix product (im2col).
  • Trace the evolution of CNN architectures from LeNet to ConvNeXt and explain the key idea of each.
  • Train and fine-tune convolutional networks with augmentation and transfer learning.

Computer vision was the field in which deep learning first proved itself, and convolutional neural networks (CNNs) remain the backbone of most vision systems, from phone cameras to medical imaging. This chapter follows the path of Stanford’s CS231n course: it starts from image classification, the task that drove progress, builds up from linear classifiers to convolutional layers, and ends with the architectures and training practices used today. Detection and segmentation (Chapter 21), visualization (Chapter 22), and video and 3D (Chapter 23) build on this chapter.

17.1Image Classification

A color image is an array of numbers: height $\times$ width $\times$ 3 channels (red, green, blue), each an integer from 0 to 255. A $32\times32$ image is a vector of 3,072 numbers; a $224\times224$ image has over 150,000. Image classification asks for a label from a fixed set, such as {airplane, car, bird, cat, …}. Humans do this effortlessly, but the semantic gap between the numbers and the label is wide. The same object produces completely different arrays under changes of viewpoint, illumination, scale, deformation (a cat can curl up or stretch), occlusion, and background clutter, and members of one class vary enormously (intra-class variation: think of all the breeds of dog). There is no obvious way to write a rule-based classifier.

The data-driven approach of this book replaces rules with examples: collect a large labeled data set, train a model, evaluate on held-out images. Three data sets shaped the field:

  • MNIST: 70,000 grayscale $28\times28$ handwritten digits (LeCun et al. 1998).
  • CIFAR-10: 60,000 color $32\times32$ images in 10 classes, 50,000 for training and 10,000 for testing (Krizhevsky 2009). Project 9 builds classifiers for it from scratch.
  • ImageNet: over a million training images in 1,000 classes, the basis of the ImageNet Large Scale Visual Recognition Challenge (ILSVRC) held from 2010 to 2017 (Deng et al. 2009; Russakovsky et al. 2015). Results are reported as top-1 and top-5 error: a prediction counts as correct under top-5 if the true class is among the model’s five highest-scoring classes.

Nearest neighbors on pixels. The simplest data-driven classifier is $k$-nearest neighbors (Chapter 8) with a pixel distance, for example $d_1(I_1,I_2)=\sum_p|I_1^p-I_2^p|$ or the Euclidean distance. On CIFAR-10, with $k$ chosen by cross-validation, it gets about 28% of the test images right (Project 9), far better than the 10% of chance but far from useful. Two flaws are fundamental. First, pixel distances do not reflect meaning: shifting an image by a few pixels, darkening it, or covering part of it can each move it as far from the original as an image of a different class, and neighbors tend to share backgrounds and colors rather than objects. Second, prediction compares the query with every training image, so it is slow exactly where speed matters, at test time. The curse of dimensionality (Section 8.2) makes both problems worse.

17.2Linear Classifiers

A parametric classifier summarizes the training data in parameters. The simplest is linear: flatten the image into $\vx\in\R^D$ and compute one score per class,

$$\vs=f(\vx;\mW,\vb)=\mW\vx+\vb,\qquad \mW\in\R^{K\times D},\ \vb\in\R^K . \tag{17.1}$$

For CIFAR-10, $D=3072$ and $K=10$, so the classifier has about 31,000 parameters, and prediction is a single matrix–vector product. (Appending a constant 1 to $\vx$ and a column $\vb$ to $\mW$ folds the bias into the weights.) There are three useful ways to look at Equation (17.1):

  • Algebraic. Each class score is a weighted sum of all pixel values, with its own weights.
  • Visual. Each row of $\mW$ has the shape of an image and acts as a template; the score is the inner product of the image with the template. Learned templates look like blurry averages of their class: the car template is a red-ish blob on gray, and the horse template often shows a horse with two heads, one facing each way, because a single template must match horses facing left and right.
  • Geometric. Each class score is zero on a hyperplane in the 3,072-dimensional pixel space, and the classifier carves the space into $K$ regions with linear boundaries.

Losses. Training picks $\mW$ to make correct scores high and incorrect ones low, measured by a loss averaged over the training set plus a regularizer:

$$\Loss(\mW)=\frac1N\sum_{i=1}^NL_i\big(f(\vx_i;\mW),y_i\big)+\lambda R(\mW),\qquad R(\mW)=\sum_{k,l}W_{kl}^2 . \tag{17.2}$$

Two choices of $L_i$ dominate. The multiclass SVM loss (Equation (9.9)) $L_i=\sum_{j\ne y_i}\max(0,s_j-s_{y_i}+1)$ asks the correct score to beat every other score by a margin of 1, and is then satisfied. The softmax or cross-entropy loss (Chapter 6) $L_i=-\log\big(e^{s_{y_i}}/\sum_je^{s_j}\big)$ interprets the scores as unnormalized log-probabilities and is never fully satisfied: it keeps pushing the correct class’s probability toward 1. In practice they give similar accuracy; softmax is more common because its outputs are probabilities. Both are optimized with mini-batch SGD and the gradients of Chapters 9 and 6, and Project 9 checks them numerically.

Limits of linearity. Linear classifiers reach about 40% test accuracy on CIFAR-10 (Project 9). One template per class cannot represent a class whose images look different (a horse facing left versus right, a red car versus a blue car), and a hyperplane cannot separate classes whose regions in pixel space are not convex. A two-layer network $\vs=\mW_2\max(\mathbf 0,\mW_1\vx)$ (Chapter 15) fixes the first problem, since its first layer can learn many templates that the second combines, and reaches about 50%. But a fully connected layer on images is still wasteful and blind to their structure, which is what convolutions address.

17.3Convolutional Layers

A fully connected layer on a $224\times224\times3$ image would need $150{,}528$ weights per hidden unit, and it would ignore two facts about images: nearby pixels are related (locality), and a cat is a cat wherever it appears (translation equivariance). Convolutional layers (LeCun et al. 1998) build both facts in.

The operation. A convolutional layer slides a small kernel (filter) $\mathbf K\in\R^{k\times k}$ over the input and computes a weighted sum at each position. For a single-channel input $\mX$, the output, or feature map, is

$$Y_{ij} = b + \sum_{u=0}^{k-1}\sum_{v=0}^{k-1}K_{uv}\,X_{i+u,\,j+v} . \tag{17.3}$$

(Mathematicians would flip the kernel and call this cross-correlation; deep learning calls it convolution, and since the kernel is learned, the difference does not matter.) The same kernel is used at every position: this weight sharing means a $3\times3$ layer has 10 parameters regardless of the image size, and a feature detected in one place is detected everywhere. Shifting the input shifts the output by the same amount, which is what equivariance means.

Channels. Real layers take $C_{\text{in}}$ input channels (RGB, or the feature maps of the previous layer) and produce $C_{\text{out}}$ output channels. Each output channel has its own $C_{\text{in}}\times k\times k$ kernel that sums over all input channels, so a layer has $C_{\text{out}}(C_{\text{in}}k^2+1)$ parameters. Each output channel is a learned detector for one pattern, and the stack of output channels is a new image whose “colors” are features.

Size bookkeeping. With input size $n$, kernel size $k$, zero-padding $p$ on each side, and stride $s$ (how far the kernel moves between positions), the output size is

$$n_{\text{out}} = \left\lfloor\frac{n+2p-k}{s}\right\rfloor + 1 . \tag{17.4}$$

“Same” padding, $p=(k-1)/2$ with stride 1, keeps the size unchanged. Stride 2 halves it. The cost of a layer is $H_{\text{out}}W_{\text{out}}C_{\text{out}}C_{\text{in}}k^2$ multiply–adds, which dominates the compute of a CNN.

Special convolutions. A $1\times1$ convolution applies the same small fully connected layer to the channel vector at every position: it mixes channels without looking at neighbors, and is used to change the number of channels cheaply. A dilated (atrous) convolution spaces the kernel taps $d$ pixels apart, enlarging the region it covers without more parameters. A depthwise convolution gives each channel its own $k\times k$ filter and does not mix channels; followed by a $1\times1$ convolution it forms a depthwise separable convolution, which costs $C_{\text{in}}k^2+C_{\text{in}}C_{\text{out}}$ instead of $C_{\text{in}}C_{\text{out}}k^2$ per position, about 8 to 9 times less for $3\times3$ kernels and many channels.

17.4Pooling and Receptive Fields

Max pooling takes the maximum over small windows (typically $2\times2$ with stride 2), halving the spatial size and making the representation tolerant to small shifts; average pooling takes the mean. Strided convolutions are a learnable alternative to pooling, and global average pooling averages each channel over all positions at the end of a network, producing one feature per channel for the classifier regardless of the input size.

Each unit in a deep layer depends on a region of the input, its receptive field. Track two numbers through the network: the receptive field size $r_l$ (in input pixels) and the jump $j_l$, the distance in input pixels between adjacent units of layer $l$. Starting from $r_0=j_0=1$, a layer with kernel size $k_l$ (dilation included: a dilated kernel covers $d(k-1)+1$ positions) and stride $s_l$ gives

$$r_l=r_{l-1}+(k_l-1)\,j_{l-1},\qquad j_l=j_{l-1}\,s_l . \tag{17.5}$$

Without downsampling, each $3\times3$ layer adds 2 pixels, so $L$ layers see $(2L+1)\times(2L+1)$. Every stride-2 layer doubles the jump and therefore doubles the growth rate of everything after it, which is why networks downsample periodically: after five stride-2 stages, a single $3\times3$ layer adds 64 pixels. Two caveats: the effective receptive field, where the input actually influences the unit, is concentrated near the center and roughly Gaussian-shaped, much smaller than the theoretical one (Luo et al. 2016); and late layers in classification networks typically have receptive fields covering the whole image, which is how they recognize whole objects. Visualizations (Chapter 22) show early layers detecting edges and color blobs, middle layers textures and parts, and late layers object-like patterns (Zeiler and Fergus 2014).

17.5Backpropagation Through Convolutions

A convolutional layer is a linear layer with a special, sparse, weight-shared matrix, so backpropagation (Section 15.3) applies unchanged; it is still worth writing the gradients out. For the single-channel layer Equation (17.3) with upstream gradient $G_{ij}=\partial\Loss/\partial Y_{ij}$, the chain rule sums over every output that a parameter or input touched:

$$\pd{\Loss}{K_{uv}}=\sum_{i,j}G_{ij}\,X_{i+u,\,j+v},\qquad \pd{\Loss}{X_{pq}}=\sum_{u,v}G_{p-u,\,q-v}\,K_{uv},\qquad \pd{\Loss}{b}=\sum_{i,j}G_{ij}, \tag{17.6}$$

with $G$ taken as zero outside its range. The kernel gradient is the correlation of the input with the upstream gradient: each kernel weight collects the evidence from every position where it was used. The input gradient is a “full” convolution of the upstream gradient with the kernel, that is, a correlation with the kernel flipped in both directions. Both are themselves convolutions, which is why the backward pass of a CNN costs about twice its forward pass. Max pooling routes each upstream gradient to the input that achieved the maximum and sends zero to the rest; average pooling spreads it evenly.

Convolution as matrix multiplication. Computing Equation (17.3) with nested loops is slow. The im2col trick copies every $C_{\text{in}}\times k\times k$ input patch into a column of a matrix $\mathbf X_{\text{col}}$ of shape $C_{\text{in}}k^2\times H_{\text{out}}W_{\text{out}}$ and reshapes the kernels into a matrix $\mW_{\text{row}}$ of shape $C_{\text{out}}\times C_{\text{in}}k^2$. Then the whole layer is one matrix product, $\mathbf Y=\mW_{\text{row}}\mathbf X_{\text{col}}$, which runs at the speed of highly optimized matrix multiplication libraries. The backward pass is two more matrix products and a scatter-add (col2im) back onto the image. The price is memory: each input pixel is copied up to $k^2$ times. Production libraries use this idea, or FFT- and Winograd-based algorithms for some kernel sizes, behind interfaces such as torch.nn.Conv2d.

17.6CNN Architectures

The ILSVRC competition turned architecture design into an empirical science. The table below summarizes the milestones; ImageNet top-5 error fell from 26% before deep learning to under 4% within four years, below the roughly 5% estimated for a trained human annotator (Russakovsky et al. 2015).

NetworkYearKey ideaParametersTop-5 error
LeNet-51998conv + pool + fully connected60 thousand(digits)
AlexNet2012ReLU, dropout, GPUs, augmentation60 million15.3%
VGG-162014stacks of $3\times3$ convolutions138 million7.3%
GoogLeNet2014Inception modules, $1\times1$ bottlenecksabout 6 million6.7%
ResNet2015residual connections, 152 layers60 million3.6%

LeNet-5 (LeCun et al. 1998) established the pattern used ever since: convolution and subsampling layers that shrink the image while increasing the number of channels, followed by fully connected layers. It read handwritten digits on bank checks in the 1990s.

AlexNet (Krizhevsky et al. 2012) scaled this pattern to ImageNet with five convolutional and three fully connected layers, and won ILSVRC 2012 with 15.3% top-5 error against 26.2% for the runner-up. Its ingredients were ReLU activations, dropout, heavy data augmentation, and training on two GPUs. Most of its parameters sat in the fully connected layers.

VGG (Simonyan and Zisserman 2015) used only $3\times3$ convolutions, stacked deeply. Two stacked $3\times3$ layers have the receptive field of one $5\times5$ layer (Equation (17.5)) but $2\cdot9C^2=18C^2$ instead of $25C^2$ parameters, and an extra nonlinearity; three of them match a $7\times7$ layer with $27C^2$ instead of $49C^2$. VGG’s uniform design made it a popular feature extractor, but its 138 million parameters and heavy compute made it expensive.

GoogLeNet (Szegedy et al. 2015) introduced the Inception module: parallel $1\times1$, $3\times3$, and $5\times5$ convolutions and a pooling branch whose outputs are concatenated along channels, with $1\times1$ “bottleneck” convolutions reducing the channel count before the expensive branches. It replaced the large fully connected layers by global average pooling and won ILSVRC 2014 with 12 times fewer parameters than AlexNet.

ResNet (He et al. 2016) made depth itself work through residual connections (Section 16.2). Deeper ResNets use bottleneck blocks, $1\times1$ (reduce channels) $\to3\times3\to1\times1$ (restore channels), so that a 50-layer ResNet has only 25.6 million parameters. ResNet won ILSVRC 2015 and is still the default baseline backbone.

After ImageNet. Later work optimized accuracy per unit of compute. DenseNet (Huang et al. 2017) feeds every layer the concatenated outputs of all previous layers in a block, encouraging feature reuse. MobileNets (Howard et al. 2017) build networks from depthwise separable convolutions for phones. EfficientNet (Tan and Le 2019) scales depth, width, and input resolution together by fixed ratios (compound scaling) from a small base network found by architecture search. ConvNeXt (Liu et al. 2022) revisited the ResNet design with lessons from Vision Transformers (larger $7\times7$ depthwise kernels, an inverted bottleneck, LayerNorm, GELU, fewer normalization and activation layers, and Transformer-style training recipes) and matched the accuracy of contemporary Vision Transformers, showing that much of their advantage came from training methods rather than attention itself.

17.7Training Convolutional Networks in Practice

Preprocessing and augmentation. Subtract the per-channel mean and divide by the per-channel standard deviation of the training set. Augment with random crops (often from an image padded by a few pixels), horizontal flips, and color jitter; stronger policies such as RandAugment, mixup, and CutMix (Section 16.4) help large models. Only use transformations that preserve the label: flipping a photo of a cat is fine, flipping a digit or a road sign may not be.

Transfer learning. Networks pretrained on ImageNet (available in torchvision and similar libraries) are the usual starting point for a new vision task (Section 16.8). How much to retrain depends on the amount of data and its similarity to ImageNet:

Similar to ImageNetVery different (e.g. medical)
Little datalinear classifier on top featureslinear classifier on earlier features
A lot of datafine-tune the top layersfine-tune most or all layers

Use the same preprocessing as during pretraining, a smaller learning rate for the pretrained layers than for the new head, and remember that BatchNorm statistics adapt only in training mode.

A recipe. A standard modern recipe for training an image classifier from scratch: a ResNet-style network with BatchNorm, SGD with momentum $0.9$ or AdamW, a short warm-up and cosine decay, weight decay around $5\times10^{-4}$ for SGD (or $0.05$ for AdamW), label smoothing, random crops and flips, and mixed precision. Section 16.6 gives the tuning process.

17.8From Convolutions to Vision Transformers

Convolutions hard-wire locality and translation equivariance. These inductive biases help most when data are scarce. Vision Transformers (Section 19.9) instead cut an image into patches, treat them as tokens, and let attention learn which positions to combine; with enough data or pretraining they match or beat CNNs, and they scale naturally to multimodal models (Chapter 23). In practice the two families have converged: ConvNeXt borrows Transformer design and training, hybrid models use convolutions in early layers and attention later, and both families share the training toolkit of Chapter 16.

17.9Implementation

Convolution as one matrix multiplication per image, with its backward pass, checked against finite differences and PyTorch:

import numpy as np

def im2col(x, k, stride=1, pad=0):
    """x: (N, C, H, W) -> columns (N, C*k*k, H_out*W_out) holding every k-by-k patch."""
    N, C, H, W = x.shape
    x = np.pad(x, ((0, 0), (0, 0), (pad, pad), (pad, pad)))
    Ho, Wo = (H + 2 * pad - k) // stride + 1, (W + 2 * pad - k) // stride + 1
    cols = np.empty((N, C, k, k, Ho, Wo))
    for u in range(k):
        for v in range(k):
            cols[:, :, u, v] = x[:, :, u:u + stride * Ho:stride, v:v + stride * Wo:stride]
    return cols.reshape(N, C * k * k, Ho * Wo), (Ho, Wo)

def col2im(cols, shape, k, stride=1, pad=0):
    """Adjoint of im2col: scatter-add the columns back onto the (padded) image."""
    N, C, H, W = shape
    Ho, Wo = (H + 2 * pad - k) // stride + 1, (W + 2 * pad - k) // stride + 1
    cols = cols.reshape(N, C, k, k, Ho, Wo)
    x = np.zeros((N, C, H + 2 * pad, W + 2 * pad))
    for u in range(k):
        for v in range(k):
            x[:, :, u:u + stride * Ho:stride, v:v + stride * Wo:stride] += cols[:, :, u, v]
    return x[:, :, pad:pad + H, pad:pad + W]

def conv_forward(x, Wt, b, stride=1, pad=0):
    """Wt: (C_out, C_in, k, k). One matrix multiplication per image (Eq. 17.3 for all positions)."""
    Cout, _, k, _ = Wt.shape
    cols, (Ho, Wo) = im2col(x, k, stride, pad)
    y = Wt.reshape(Cout, -1) @ cols + b[:, None]          # (N, C_out, Ho*Wo)
    return y.reshape(x.shape[0], Cout, Ho, Wo), (x.shape, cols, Wt, stride, pad)

def conv_backward(dy, cache):
    shape, cols, Wt, stride, pad = cache
    Cout, Cin, k, _ = Wt.shape
    dy2 = dy.reshape(dy.shape[0], Cout, -1)                # (N, C_out, Ho*Wo)
    dW = np.einsum("nop,nqp->oq", dy2, cols).reshape(Wt.shape)
    db = dy2.sum(axis=(0, 2))
    dcols = Wt.reshape(Cout, -1).T @ dy2                   # (N, C_in*k*k, Ho*Wo)
    return col2im(dcols, shape, k, stride, pad), dW, db

rng = np.random.default_rng(0)
x = rng.standard_normal((2, 3, 7, 7))
Wt, b = rng.standard_normal((4, 3, 3, 3)), rng.standard_normal(4)
y, cache = conv_forward(x, Wt, b, stride=2, pad=1)
dy = rng.standard_normal(y.shape)
dx, dW, db = conv_backward(dy, cache)
print("output shape:", y.shape)                            # (2, 4, 4, 4)

def num_grad(f, a, h=1e-6):
    g = np.zeros_like(a)
    for idx in np.ndindex(*a.shape):
        old = a[idx]; a[idx] = old + h; fp = f(); a[idx] = old - h; fm = f(); a[idx] = old
        g[idx] = (fp - fm) / (2 * h)
    return g
loss = lambda: np.sum(dy * conv_forward(x, Wt, b, stride=2, pad=1)[0])
rel = lambda a, n: np.max(np.abs(a - n)) / np.max(np.abs(n))
print(f"relative errors: dx {rel(dx, num_grad(loss, x)):.1e}, dW {rel(dW, num_grad(loss, Wt)):.1e}")

import torch                                               # cross-check against PyTorch
yt = torch.nn.functional.conv2d(torch.tensor(x), torch.tensor(Wt), torch.tensor(b), stride=2, padding=1)
print("matches torch.conv2d:", np.allclose(yt.numpy(), y))

A small residual CNN in PyTorch with the techniques of Chapter 16, trained on scikit-learn’s $8\times8$ digits so that it runs in seconds on a CPU (Project 9 trains a larger one on CIFAR-10):

import torch, torch.nn as nn, torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.model_selection import train_test_split

X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32).view(-1, 1, 8, 8)
y = torch.tensor(y)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.25, random_state=0)

class ResBlock(nn.Module):
    def __init__(self, c):
        super().__init__()
        self.conv1, self.bn1 = nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c)
        self.conv2, self.bn2 = nn.Conv2d(c, c, 3, padding=1, bias=False), nn.BatchNorm2d(c)
    def forward(self, x):
        h = F.relu(self.bn1(self.conv1(x)))
        return F.relu(x + self.bn2(self.conv2(h)))      # residual connection

model = nn.Sequential(
    nn.Conv2d(1, 32, 3, padding=1), nn.BatchNorm2d(32), nn.ReLU(),
    ResBlock(32), ResBlock(32), nn.MaxPool2d(2),        # 8x8 -> 4x4
    ResBlock(32), nn.AdaptiveAvgPool2d(1), nn.Flatten(),
    nn.Dropout(0.2), nn.Linear(32, 10))

steps, warmup = 600, 50
opt = torch.optim.AdamW(model.parameters(), lr=3e-3, weight_decay=0.05)
sched = torch.optim.lr_scheduler.LambdaLR(opt, lambda t: min(1, (t + 1) / warmup)
              * 0.5 * (1 + torch.cos(torch.tensor(torch.pi * t / steps)).item()))
for t in range(steps):
    model.train()
    idx = torch.randint(len(Xtr), (64,))
    xb = torch.roll(Xtr[idx], shifts=tuple(torch.randint(-1, 2, (2,)).tolist()),
                    dims=(2, 3))                         # augmentation: small shifts
    loss = F.cross_entropy(model(xb), ytr[idx], label_smoothing=0.05)
    opt.zero_grad(); loss.backward()
    nn.utils.clip_grad_norm_(model.parameters(), 1.0)   # gradient clipping
    opt.step(); sched.step()

model.eval()
with torch.no_grad():
    print("test accuracy:", (model(Xte).argmax(1) == yte).float().mean().item())

17.10Common Pitfalls

Pitfall: Wrong output sizes

Mismatched shapes between the last convolution and the first fully connected layer are the most common CNN bug. Track sizes with Equation (17.4) on paper, or pass a dummy input through the network and print the shapes; better, end with global average pooling so the classifier does not depend on the input size.

Pitfall: Mismatched preprocessing for pretrained models

A pretrained network expects inputs normalized exactly as during its training (often the ImageNet channel means and standard deviations, in a specific channel order and value range). Feeding it differently scaled images silently costs accuracy.

Pitfall: Training a big network from scratch on a small data set

A network with millions of parameters trained on a few thousand images will overfit. Start from pretrained weights, use strong augmentation, or use a smaller network.

17.11Summary

  • Image classification maps pixel arrays to labels despite viewpoint, illumination, deformation, occlusion, and intra-class variation; nearest neighbors on pixels fail because pixel distances do not reflect meaning.
  • A linear classifier $\mW\vx+\vb$ is $K$ templates or $K$ hyperplanes, trained with the multiclass SVM or softmax loss plus regularization; it reaches about 40% on CIFAR-10.
  • Convolutions share small kernels across positions: locality, equivariance, and few parameters. Output size is $\lfloor(n+2p-k)/s\rfloor+1$; $1\times1$, dilated, and depthwise separable convolutions trade off cost and context.
  • Receptive fields grow by $(k-1)j$ per layer, and downsampling multiplies the jump.
  • The gradients of a convolution are convolutions; im2col turns the layer into a matrix product.
  • Architectures progressed from LeNet and AlexNet through VGG, GoogLeNet, and ResNet to efficient and modernized designs such as MobileNet, EfficientNet, and ConvNeXt.
  • In practice: normalize inputs, augment with label-preserving transformations, and fine-tune pretrained networks.

17.12Exercises

  1. MathA layer maps a $32\times32\times64$ input to $32\times32\times128$ with $3\times3$ kernels. How many parameters and multiply–adds does it have? How many parameters would a fully connected layer with the same input and output sizes have? And a depthwise separable layer?
  2. MathUse Equation (17.5) to compute the receptive field of the output of: (a) five $3\times3$ convolutions with stride 1; (b) [$3\times3$ conv, $2\times2$ max pool] repeated three times; (c) three $3\times3$ convolutions with dilations 1, 2, 4.
  3. MathDerive Equation (17.6) for a one-dimensional convolution with a kernel of size 3, writing the layer as multiplication by a banded matrix. Show that the input gradient is multiplication by the transpose of that matrix.
  4. MathShow that a linear classifier on images is not translation invariant: give an example of an image and a one-pixel shift of it that receive very different scores. Why does a convolutional layer followed by global average pooling not have this problem?
  5. CodingExtend the im2col implementation with max pooling (forward and backward) and assemble a two-layer CNN in NumPy. Check all gradients numerically, then train it on the $8\times8$ digits.
  6. CodingTrain the residual CNN of the implementation with and without the residual connections (replace x + ... by ...) after increasing the depth to 10 blocks. Compare training curves.
  7. CodingFine-tune an ImageNet-pretrained ResNet-18 from torchvision on a small data set of your choice (for example 10 classes with 50 images each), comparing feature extraction with full fine-tuning.
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.