Machine Learning from First Principles

Chapter 16 · Part IV: Deep Learning

Training Deep Networks

Depth makes networks powerful and hard to train. This chapter explains, with derivations, the ideas that make training reliable: initialization that preserves signal, residual connections, normalization, regularization, schedules, and a disciplined process for tuning and debugging.

After this chapter you will be able to

  • Derive how the variance of activations and gradients evolves with depth, and choose initializations that keep it stable.
  • Explain why residual connections make very deep networks trainable, using the Jacobian of a residual block.
  • Derive the backward pass of batch normalization, and choose between batch, layer, group, and RMS normalization.
  • Apply the regularizers of deep learning: decoupled weight decay (AdamW), dropout, data augmentation, mixup, label smoothing, and early stopping.
  • Use warm-up and cosine decay, gradient clipping, and mixed precision for stable, fast training.
  • Tune hyperparameters with random search and debug training with sanity checks and learning curves.
  • Describe what is known about the loss landscapes of deep networks.

Chapter 15 built networks from fully connected layers and trained them with backpropagation. Between 2010 and today, a handful of further ideas turned such networks into systems with billions of parameters that train reliably. This chapter covers the ones every practitioner uses, together with the theory that explains why they work. They apply to every architecture in the rest of the book: the convolutional networks of Chapter 17, the recurrent networks of Chapter 18, and the Transformers of Chapter 19.

16.1Signal Propagation and Initialization

Section 15.4 introduced the rule that weights should be scaled to preserve the variance of the signal. We now derive it properly, for both the forward and the backward pass, because the same calculation explains vanishing and exploding gradients, the choice of activation functions, and why normalization layers help.

Forward pass. Consider layer $l$ of an MLP with $n_{l-1}$ inputs, $z_i^{(l)}=\sum_jW_{ij}^{(l)}h_j^{(l-1)}$ and $h^{(l)}=\phi(z^{(l)})$, with biases zero. At initialization the weights are independent with mean zero and variance $v_l$, independent of the activations. Then each pre-activation has mean zero and

$$\Var\big(z^{(l)}\big)=n_{l-1}\,v_l\,\E\big[(h^{(l-1)})^2\big]. \tag{16.1}$$

For ReLU and a pre-activation symmetric around zero, half of the mass is zeroed, so $\E[(h^{(l-1)})^2]=\frac12\Var(z^{(l-1)})$, and therefore $\Var(z^{(l)})=\frac12n_{l-1}v_l\Var(z^{(l-1)})$. Over $L$ layers the variance is multiplied by $\prod_l\frac12n_{l-1}v_l$. If each factor is $1.1$, a 50-layer network amplifies the variance by $1.1^{50}\approx117$; if each factor is $0.9$, it shrinks it by a factor of about 190. Only $v_l=2/n_{l-1}$, the He initialization (He et al. 2015), keeps it constant.

Backward pass. Write $\delta^{(l)}=\partial\Loss/\partial z^{(l)}$. Backpropagation gives $\delta^{(l-1)}_j=\phi'(z^{(l-1)}_j)\sum_iW^{(l)}_{ij}\delta^{(l)}_i$, a sum over the $n_l$ units of the next layer. The same argument gives

$$\Var\big(\delta^{(l-1)}\big)=n_l\,v_l\,\E\big[\phi'(z^{(l-1)})^2\big]\Var\big(\delta^{(l)}\big), \tag{16.2}$$

and for ReLU $\E[\phi'^2]=\frac12$. Backward stability needs $v_l=2/n_l$: the fan-out instead of the fan-in. When widths are equal the two conditions coincide. Glorot and Bengio (Glorot and Bengio 2010) proposed the compromise $v_l=2/(n_{l-1}+n_l)$ for tanh networks, where $\phi'(0)=1$ and the factor $\frac12$ is absent. The weight gradients are $\partial\Loss/\partial\mW^{(l)}=\vdelta^{(l)}\vh^{(l-1)\top}$, so they inherit the scale of both the forward activations and the backward signal: if either vanishes, the layer stops learning.

What goes wrong in practice. With tanh and weights that are too small, activations shrink toward zero layer by layer, every unit operates in the linear regime, and the gradients reaching the first layers are tiny. With weights that are too large, pre-activations are huge, tanh saturates at $\pm1$, $\phi'\approx0$, and gradients vanish for the opposite reason. Sigmoid adds a second problem: its outputs are always positive, so the gradients of all weights entering a unit share a sign, which makes optimization zigzag. These observations are why ReLU-family activations, He initialization, and (next) normalization and residual connections became standard.

Key idea: Dynamical isometry

Variance preservation only controls the average size of the signal. A stronger requirement, studied in the theory of signal propagation, is that the input–output Jacobian of the whole network have all singular values near 1, so that every direction is transmitted without distortion. Orthogonal initialization of linear layers achieves this exactly for linear networks, and residual connections and normalization approximate it for nonlinear ones.

16.2Residual Connections

Before 2015, making plain networks deeper beyond about 20 layers made them worse, even on the training set. This was not overfitting but an optimization failure: gradients passing through many layers vanish or explode, and even an identity mapping is hard for a stack of nonlinear layers to learn. Residual networks (ResNets) (He et al. 2016) fixed this with one change. Instead of $\vh_{l+1}=F_l(\vh_l)$, each block computes

$$\vh_{l+1} = \vh_l + F_l(\vh_l) \tag{16.3}$$

where $F_l$ is a small stack of layers (for example two convolutions with normalization and ReLU). The block only has to learn the residual $F_l$, the change to its input; doing nothing is as easy as setting $F_l\approx0$.

Why gradients flow. The Jacobian of one block is $\mI+\partial F_l/\partial\vh_l$. Through $L$ blocks, the chain rule gives

$$\pd{\vh_L}{\vh_l} = \prod_{m=l}^{L-1}\Big(\mI+\pd{F_m}{\vh_m}\Big) = \mI + \sum_{m}\pd{F_m}{\vh_m} + \dots$$

The identity term means that the gradient arriving at layer $l$ always contains an unattenuated copy of the gradient at the top, plus corrections. In a plain network the product has no identity terms, and a product of many factors with norm below one vanishes geometrically. Residual connections turn the network into a sum of many paths of different lengths, and the short paths keep gradients healthy. In terms of Equation (16.1), the variance of $\vh_l$ now grows additively with depth rather than multiplicatively, which is why residual blocks are often initialized with their last layer (or the scale of their last normalization) set to zero, so that every block starts as the identity.

Residual connections are now everywhere: ResNets with over 100 layers train easily, and every Transformer block (Chapter 19) wraps both its attention and its feed-forward sublayers in residual connections.

16.3Normalization

Normalization layers rescale activations to a standard range during training, not just at initialization. They keep the signal in the well-behaved part of the activation functions, make the network insensitive to the scale of the weights that precede them, and make the loss surface better conditioned, which allows larger learning rates.

Batch normalization (Ioffe and Szegedy 2015) normalizes each feature (each channel, for convolutions) with statistics of the current mini-batch of $m$ examples, then applies a learned scale $\gamma$ and shift $\beta$:

$$\mu_B=\frac1m\sum_{i=1}^mx_i,\qquad \sigma_B^2=\frac1m\sum_{i=1}^m(x_i-\mu_B)^2,\qquad \hat x_i = \frac{x_i-\mu_B}{\sqrt{\sigma_B^2+\epsilon}},\qquad y_i = \gamma\hat x_i+\beta . \tag{16.4}$$

The learned $\gamma$ and $\beta$ let the layer undo the normalization if that is useful, so no expressive power is lost. At test time, batch statistics are unavailable or meaningless (a batch of one), so the layer uses running averages of $\mu_B$ and $\sigma_B^2$ collected during training.

The backward pass. Every $\hat x_i$ depends on every $x_j$ through $\mu_B$ and $\sigma_B$, so the gradient is not just $\gamma/\sigma_B$ times the upstream gradient. With upstream gradients $\partial\Loss/\partial y_i$, let $g_i=\gamma\,\partial\Loss/\partial y_i$ be the gradient with respect to $\hat x_i$. Differentiating through $\mu_B$ and $\sigma_B^2$ and simplifying (an exercise) gives

$$\pd{\Loss}{x_i}=\frac{1}{m\sqrt{\sigma_B^2+\epsilon}}\Big(m\,g_i-\sum_{j=1}^mg_j-\hat x_i\sum_{j=1}^mg_j\hat x_j\Big),\qquad \pd{\Loss}{\gamma}=\sum_i\pd{\Loss}{y_i}\hat x_i,\quad \pd{\Loss}{\beta}=\sum_i\pd{\Loss}{y_i}. \tag{16.5}$$

The formula has a clean geometric meaning: from the vector $\vg$ it removes its component along the all-ones vector and its component along $\hat\vx$, then rescales. Those are exactly the two directions in which changing the input $\vx$ does not change $\hat\vx$ (shifting all inputs, or scaling them about their mean), so the gradient never points along them. One consequence is that the weights feeding a normalized unit are scale-invariant: multiplying them by $c$ does not change the output, and their gradient is orthogonal to the weights themselves.

Layer, group, and RMS normalization. Batch normalization couples the examples in a batch, which causes problems with small batches (noisy statistics), with sequence models (variable lengths), and with distributed training (statistics must be synchronized). Layer normalization (Ba et al. 2016) normalizes each example on its own, across its $d$ features; it behaves identically at training and test time, which is why Transformers use it. Group normalization (Wu and He 2018) is the convolutional counterpart: it splits the channels of one example into groups (typically 32) and normalizes within each group over channels and spatial positions; with one group it is layer normalization, with one channel per group it is instance normalization. It works with batches of one or two images, common in detection and segmentation (Chapter 21). RMSNorm drops the mean subtraction and divides by the root mean square, $y=\gamma\odot\vx/\sqrt{\frac1d\norm{\vx}^2+\epsilon}$; it is slightly cheaper and is the norm of choice in most current large language models.

Where to put it. The original Transformer normalized after each residual addition (post-norm). Modern networks normalize the input of each residual branch instead (pre-norm), $\vh+F(\mathrm{Norm}(\vh))$, which leaves the identity path untouched and trains stably without long warm-ups (Xiong et al. 2020).

Pitfall: Forgetting evaluation mode

Layers that behave differently in training and evaluation (batch normalization, dropout) must be switched: model.train() before training and model.eval() before validation or inference. Evaluating in training mode gives noisy, often much worse, results; training in evaluation mode silently disables dropout and freezes the running statistics.

16.4Regularization for Deep Networks

Weight decay and AdamW. Weight decay shrinks every weight toward zero at each step, $\vtheta\gets\vtheta-\eta\lambda\vtheta$. For plain SGD this is the same as adding $\frac\lambda2\norm{\vtheta}^2$ to the loss, i.e. ridge regularization (Section 5.3). For Adam it is not: the penalty’s gradient $\lambda\vtheta$ passes through Adam’s per-coordinate rescaling, so weights with a large gradient history are barely regularized. AdamW (Loshchilov and Hutter 2019) decouples the decay, applying it directly to the weights outside the adaptive update:

$$\vtheta_t = \vtheta_{t-1} - \eta\Big(\frac{\hat\vm_t}{\sqrt{\hat\vs_t}+\epsilon} + \lambda\,\vtheta_{t-1}\Big). \tag{16.6}$$

AdamW is the standard optimizer for Transformers. Biases and normalization parameters are usually excluded from decay. For weights followed by normalization, which are scale-invariant, weight decay does not regularize the function at all; instead it keeps the weight norm small, which raises the effective learning rate. This is one of several ways in which hyperparameters of deep networks interact.

Dropout (Srivastava et al. 2014) sets each activation to zero with probability $p$ during training and scales the survivors by $1/(1-p)$, so the expected activation is unchanged; at test time it does nothing. Each step trains a different random subnetwork, which discourages units from relying on specific partners and behaves like an average over an ensemble of $2^{\text{units}}$ networks sharing weights.

Data augmentation creates new training examples by transformations that should not change the label: random crops, flips, small rotations, and color changes for images; noise or time shifts for audio. It encodes prior knowledge about invariances and is often the single most effective regularizer for vision.

Mixup and CutMix. Mixup (Zhang et al. 2018) trains on convex combinations of pairs of examples and their one-hot labels,

$$\tilde\vx=\lambda\vx_i+(1-\lambda)\vx_j,\qquad \tilde\vy=\lambda\vy_i+(1-\lambda)\vy_j,\qquad \lambda\sim\mathrm{Beta}(\alpha,\alpha), \tag{16.7}$$

with $\alpha$ around $0.2$. It encourages the network to behave linearly between training examples, which smooths decision boundaries and improves calibration. CutMix (Yun et al. 2019) pastes a rectangle from one image into another and mixes the labels in proportion to the area, which keeps the images locally natural.

Label smoothing replaces one-hot targets with $1-\epsilon$ on the true class and $\epsilon/(K-1)$ elsewhere, discouraging overconfident logits.

Early stopping monitors the validation loss and keeps the parameters from the best epoch. For gradient descent on least squares starting from zero, stopping after $t$ steps with step size $\eta$ behaves much like ridge regression with $\lambda\approx1/(\eta t)$: training time is a regularization parameter. Early stopping costs nothing and should almost always be used, at least in the form of saving the best checkpoint.

Ensembles and weight averaging. Averaging the predictions of several independently trained networks reliably improves accuracy and calibration, at a multiple of the cost. A cheap relative is to average the weights along the training trajectory, $\bar\vtheta\gets\beta\bar\vtheta+(1-\beta)\vtheta_t$ (an exponential moving average, EMA, with $\beta\approx0.999$), and evaluate with $\bar\vtheta$; it is standard in diffusion models (Chapter 25).

16.5Learning-Rate Schedules and Stable Training

Warm-up and decay. At initialization, Adam’s second-moment estimates are poor and the network may sit in a region of high curvature, so large steps can blow up training. A warm-up increases the learning rate linearly from near zero over the first few hundred or thousand steps. After warm-up, the rate is decayed so that SGD noise shrinks and the iterate settles (Section 3.5). A common schedule is cosine decay:

$$\eta_t = \eta_{\min} + \tfrac12(\eta_{\max}-\eta_{\min})\Big(1+\cos\frac{\pi(t-T_w)}{T-T_w}\Big),\qquad T_w\le t\le T . \tag{16.8}$$

Large language model training often uses a warm-up–stable–decay schedule instead: a long constant phase followed by a short final decay, which makes it easy to continue training or branch off checkpoints.

Gradient clipping. If the gradient norm exceeds a threshold $c$, rescale it to norm $c$: $\vg\gets\vg\min(1,c/\norm{\vg})$. Clipping prevents a single bad batch from throwing the parameters far away, and is standard for Transformers and recurrent networks (Chapter 18).

Mixed precision. Modern accelerators compute much faster in 16-bit floating point. Mixed-precision training (Micikevicius et al. 2018) runs matrix multiplications in 16-bit while keeping a 32-bit master copy of the weights and accumulating sums in 32-bit. The bfloat16 format keeps the 8-bit exponent of float32 (so it has the same range) with a shorter mantissa, which makes it the default for training large models; the older float16 format needs loss scaling to keep small gradients from underflowing to zero.

16.6Hyperparameter Search and Babysitting Training

A deep network has many hyperparameters: learning rate and schedule, weight decay, batch size, width and depth, dropout, augmentation strength. They interact, and each training run is expensive. The following process, refined over a decade of practice, avoids most wasted runs.

  1. Check the initial loss. With $K$ balanced classes and small final-layer weights, the predicted distribution is nearly uniform, so the initial cross-entropy should be about $\log K$ ($2.30$ for ten classes). A very different value signals a bug in the loss, the labels, or the initialization.
  2. Overfit a tiny subset. Train on 10–20 examples with regularization off. A correct model and training loop drive the training loss nearly to zero within a few hundred steps. If not, debug before going further: no amount of tuning fixes a broken pipeline.
  3. Find a learning rate that makes the loss go down. With the full training set and small weight decay, try learning rates on a logarithmic grid ($10^{-1},10^{-2},\dots,10^{-5}$) for a few hundred steps each. Pick the largest one that decreases the loss quickly without instability.
  4. Search coarse to fine, randomly. Sample hyperparameters at random, on logarithmic scales for learning rate and weight decay, train each configuration briefly, and narrow the ranges around the best results. Random search beats grid search (Bergstra and Bengio 2012): when only a few hyperparameters matter, a grid wastes most of its runs repeating the same values of the important ones, while random samples try a new value of every hyperparameter in every run.
  5. Train the best configurations longer, with the full schedule, and pick the winner on validation data.

Reading the curves. Plot the training loss (smoothed), the validation loss, and the validation metric against steps. A loss that stays flat at the start points to a bad initialization or a learning rate that is too small; a loss that decreases and then explodes, to a learning rate that is too large or a missing warm-up. A large gap between training and validation accuracy means overfitting: add regularization or data. Small gap but both poor means underfitting: make the model bigger or train longer. Another useful diagnostic is the ratio of the size of each update to the size of the weights, $\norm{\Delta\mW}/\norm{\mW}$, which is typically around $10^{-3}$ per step for well-tuned training; much larger or smaller values in some layer point to a learning rate that is wrong for that layer.

Scaling up hyperparameters. For very large models, tuning at full size is unaffordable. Parameterizations such as $\mu$P (Yang et al. 2021) rescale initializations and per-layer learning rates with width so that the optimal hyperparameters of a small model transfer to a large one.

16.7The Loss Landscape

The training loss of a deep network is a nonconvex function of millions or billions of parameters. Classical optimization theory would predict trouble, yet training reliably reaches low loss. What is known about this landscape?

Symmetries. Permuting the hidden units of a layer, together with the corresponding rows and columns of the adjacent weight matrices, gives a different parameter vector with exactly the same function. A network with $L$ hidden layers of width $n$ has $(n!)^L$ such copies of every minimum. With ReLU, scaling the weights into a unit by $c>0$ and those out of it by $1/c$ also leaves the function unchanged. Minima therefore come in large families, not isolated points.

Saddle points, not bad local minima. At a random critical point of a high-dimensional function, the Hessian has many eigenvalues, and it is unlikely that all are positive. Empirically and in random-matrix models, critical points with high loss are almost always saddle points with many descent directions, while true local minima concentrate near the global minimum value (Dauphin et al. 2014). Noisy gradient methods escape saddles, so the practical difficulties are plateaus and ill-conditioning rather than getting stuck.

Flat and sharp minima. Different minima of the training loss can generalize differently. Minima in wide, flat basins are robust to perturbations of the weights, and small-batch SGD, whose noise kicks the iterate out of narrow basins, tends to find flatter minima than large-batch training (Keskar et al. 2017). The picture needs care: sharpness depends on the parameterization, and a reparameterization can make any minimum look sharp without changing the function (Dinh et al. 2017). Flatness measured in a scale-invariant way remains a useful, though imperfect, predictor of generalization, and it connects to the PAC-Bayesian bounds of Section 11.7.

Connected minima. Independently trained networks reach minima that look isolated along the straight line between them (the loss rises in the middle), yet they can be joined by simple curved paths along which the training loss stays low (Garipov et al. 2018). The low-loss set is better pictured as a connected valley network than as separate holes.

The edge of stability. Chapter 3 showed that gradient descent on a quadratic is stable only if $\eta<2/\lambda_{\max}$. On neural networks trained with full-batch gradient descent, the sharpness $\lambda_{\max}$ of the Hessian typically rises during training until it reaches $2/\eta$ and then hovers there, while the loss keeps decreasing non-monotonically (Cohen et al. 2021). The learning rate thus selects how sharp a region the optimizer can stay in, one more reason why larger learning rates often generalize better.

Architecture shapes the landscape. Loss surfaces visualized along random directions are smooth and nearly convex around the minima of residual networks and chaotic for deep plain networks (Li et al. 2018), the landscape counterpart of the gradient-flow argument of Section 16.2.

16.8Transfer Learning

Training a large network from scratch needs a large data set. But features learned on one large task (edges, textures, object parts for ImageNet; syntax and facts for text) are useful for many others. Transfer learning starts from a network pretrained on a large data set and adapts it:

  • Feature extraction. Freeze the pretrained network, remove its last layer, and train only a new linear head on its features. Fast, and works with a few hundred examples.
  • Fine-tuning. Update all (or most) weights on the new task with a small learning rate, so that the useful pretrained features are adjusted rather than destroyed. With little data, fine-tune only the top layers; with more data, fine-tune more.

This pattern, pretrain once at great expense, adapt cheaply many times, is the foundation of modern AI. Models pretrained on very broad data and adapted to many tasks are called foundation models; large language models (Chapter 24) are the most prominent example, and parameter-efficient fine-tuning methods such as LoRA make adapting them affordable. Chapter 20 discusses when transfer helps and when it does not.

16.9Implementation

The first listing implements batch normalization by hand and checks the backward pass Equation (16.5) against finite differences, the way every new layer should be tested:

import numpy as np

def batchnorm_forward(x, gamma, beta, eps=1e-5):
    mu = x.mean(0)
    var = x.var(0)
    xhat = (x - mu) / np.sqrt(var + eps)
    return gamma * xhat + beta, (xhat, var, gamma, eps)

def batchnorm_backward(dy, cache):
    xhat, var, gamma, eps = cache
    m = dy.shape[0]
    dgamma, dbeta = (dy * xhat).sum(0), dy.sum(0)
    dxhat = dy * gamma
    # Eq. (16.5): remove the components along 1 and along xhat, then rescale
    dx = (m * dxhat - dxhat.sum(0) - xhat * (dxhat * xhat).sum(0)) / (m * np.sqrt(var + eps))
    return dx, dgamma, dbeta

rng = np.random.default_rng(0)
x = rng.normal(3.0, 2.0, (16, 5))
gamma, beta = rng.normal(size=5), rng.normal(size=5)
dy = rng.normal(size=(16, 5))                       # upstream gradient of L = sum(dy * y)
y, cache = batchnorm_forward(x, gamma, beta)
dx, dgamma, dbeta = batchnorm_backward(dy, cache)

num = np.zeros_like(x); h = 1e-6                     # centered finite differences
for idx in np.ndindex(*x.shape):
    xp, xm = x.copy(), x.copy(); xp[idx] += h; xm[idx] -= h
    num[idx] = (np.sum(dy * batchnorm_forward(xp, gamma, beta)[0])
                - np.sum(dy * batchnorm_forward(xm, gamma, beta)[0])) / (2 * h)
print("max relative error in dx:", np.max(np.abs(dx - num) / (np.abs(dx) + np.abs(num) + 1e-12)))
print("dx sums to zero over the batch:", np.allclose(dx.sum(0), 0))

The second listing follows the tuning process of Section 16.6 on the digits data with a four-layer MLP: check the initial loss, overfit 20 examples, then run a short random search over the learning rate and weight decay.

import math, torch, torch.nn as nn, torch.nn.functional as F
from sklearn.datasets import load_digits

torch.manual_seed(0)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32); y = torch.tensor(y)
Xtr, ytr, Xval, yval = X[:1400], y[:1400], X[1400:], y[1400:]

def make_model(width=128, depth=4):
    layers, d = [], 64
    for _ in range(depth):
        lin = nn.Linear(d, width)
        nn.init.kaiming_normal_(lin.weight, nonlinearity="relu")   # He initialization
        nn.init.zeros_(lin.bias)
        layers += [lin, nn.LayerNorm(width), nn.ReLU()]
        d = width
    layers.append(nn.Linear(d, 10))
    return nn.Sequential(*layers)

# Check 1: the initial loss should be close to log(10) = 2.303 for 10 balanced classes
model = make_model()
with torch.no_grad():
    model[-1].weight.mul_(0.01)                      # small last layer: near-uniform predictions
    print(f"initial loss {F.cross_entropy(model(Xtr), ytr).item():.3f} (log 10 = {math.log(10):.3f})")

# Check 2: the model must be able to overfit 20 examples
opt = torch.optim.AdamW(model.parameters(), lr=1e-3, weight_decay=0.0)
for _ in range(200):
    loss = F.cross_entropy(model(Xtr[:20]), ytr[:20])
    opt.zero_grad(); loss.backward(); opt.step()
print(f"loss on 20 memorized examples {loss.item():.4f}")

# Random search over log-learning-rate and log-weight-decay, short runs on the full data
def short_run(lr, wd, steps=300):
    torch.manual_seed(1)
    m = make_model()
    opt = torch.optim.AdamW(m.parameters(), lr=lr, weight_decay=wd)
    for t in range(steps):
        idx = torch.randint(len(Xtr), (64,))
        loss = F.cross_entropy(m(Xtr[idx]), ytr[idx])
        opt.zero_grad(); loss.backward(); opt.step()
    with torch.no_grad():
        return (m(Xval).argmax(1) == yval).float().mean().item()

g = torch.Generator().manual_seed(2)
trials = []
for _ in range(8):
    lr = 10 ** (-4 + 3 * torch.rand(1, generator=g).item())      # 1e-4 .. 1e-1
    wd = 10 ** (-5 + 4 * torch.rand(1, generator=g).item())      # 1e-5 .. 1e-1
    trials.append((short_run(lr, wd), lr, wd))
for acc, lr, wd in sorted(trials, reverse=True)[:3]:
    print(f"val acc {acc:.3f}  lr {lr:.1e}  weight decay {wd:.1e}")

On this run the initial loss is $2.305$, the 20 examples are memorized (loss below $0.01$), and the best short runs reach about 95% validation accuracy with learning rates between $10^{-3}$ and $5\times10^{-3}$. Chapter 17 applies the same recipe to a convolutional network.

16.10Common Pitfalls

Pitfall: Tuning on the test set

Every decision based on a data set (architecture, learning rate, epoch to stop) fits that data set a little. Use a validation set for all tuning and touch the test set once, at the end (Chapter 12).

Pitfall: Comparing runs with different budgets

A configuration that looks better after 300 steps may be worse after 30,000, especially for learning rate and schedule. Compare candidates with the same number of steps, and confirm the winner with a full-length run.

Pitfall: Trusting a single seed

Two runs of the same configuration with different random seeds can differ by a percentage point or more. Before claiming that a change helps, repeat both configurations over several seeds.

16.11Where the Field Is Heading

The ingredients of this chapter (careful initialization, residual connections, normalization, AdamW, warm-up and decay schedules, mixed precision, heavy augmentation or very large data, and transfer from pretrained models) have become a shared toolkit across architectures, and they carry over unchanged to the large language models and diffusion models of Part VI. Current work aims to make training more predictable at scale: parameterizations that transfer hyperparameters across model sizes, optimizers that exploit the matrix structure of weights, and scaling laws that predict the final loss of a large run from small ones (Chapter 24).

16.12Summary

  • Activation variance is multiplied by $\frac12n v$ per ReLU layer forward and backward; He initialization $v=2/n$ keeps it constant, Xavier $v=2/(n_{\text{in}}+n_{\text{out}})$ suits tanh.
  • Residual blocks $\vh+F(\vh)$ add identity paths to the Jacobian product, so gradients reach early layers and very deep networks train.
  • BatchNorm normalizes over the batch; its gradient removes the components along $\mathbf 1$ and $\hat\vx$. LayerNorm, GroupNorm, and RMSNorm normalize within each example; pre-norm is the modern placement.
  • AdamW decouples weight decay; dropout, augmentation, mixup, label smoothing, early stopping, and weight averaging regularize further.
  • Warm-up, cosine or warm-up–stable–decay schedules, gradient clipping, and bfloat16 mixed precision make large-scale training stable and fast.
  • Tune with sanity checks (initial loss, overfitting a tiny subset) and random search on log scales; read learning curves to diagnose problems.
  • Deep loss landscapes have many symmetric, connected minima and many saddles; flatter minima and the edge of stability link optimization to generalization.

16.13Exercises

  1. MathFor a tanh network with weights of variance $v$ and inputs of unit variance, use the linearization $\tanh z\approx z$ near zero to show that the pre-activation variance after $L$ layers of width $n$ is about $(nv)^L$. What happens for $v=1/n$, $v=0.5/n$, and $v=2/n$ with $L=20$?
  2. MathDerive the batch normalization backward pass Equation (16.5). Then show that $\sum_i\partial\Loss/\partial x_i=0$ and $\sum_i(\partial\Loss/\partial x_i)\hat x_i=0$, and interpret both identities.
  3. MathShow that dropout with the $1/(1-p)$ rescaling leaves the expected value of each activation unchanged. Does it leave the variance unchanged?
  4. MathFor plain SGD, show that weight decay $\vtheta\gets(1-\eta\lambda)\vtheta-\eta\nabla\Loss$ equals gradient descent on $\Loss+\frac\lambda2\norm{\vtheta}^2$. Explain why the equivalence fails for Adam.
  5. MathSuppose that only one of two hyperparameters affects validation accuracy. With a budget of 9 runs, how many distinct values of the important hyperparameter does a $3\times3$ grid try? How many does random search try? Generalize to $k$ hyperparameters of which one matters.
  6. CodingTrain the MLP of the implementation with mixup ($\alpha=0.2$) and without, on 300 training digits only. Compare validation accuracy and the expected calibration error of the predicted probabilities.
  7. CodingUsing the init-activations widget’s setup in PyTorch (ten tanh layers of width 256), plot the standard deviation of activations and of gradients per layer for $v=0.5/n,\ 1/n,\ 2/n$, then add LayerNorm after every linear layer and repeat.
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.