Machine Learning from First Principles

Chapter 20 · Part IV: Deep Learning

Deep Learning in Practice and at Scale

A working model is the start of a project, not the end. This chapter covers the strategy of deep learning projects (metrics, error analysis, data mismatch) and the engineering of training and serving models too big for one device.

After this chapter you will be able to

  • Set up a project with a single evaluation metric and train, dev, and test sets that reflect the target distribution.
  • Use human-level performance to separate avoidable bias from variance, and choose the next action accordingly.
  • Run an error analysis and estimate the value of fixing each category of errors.
  • Diagnose and address mismatch between training and test distributions.
  • Decide when transfer, multitask, and end-to-end learning help.
  • Estimate the memory and compute of training, and explain data, tensor, and pipeline parallelism, all-reduce, ZeRO sharding, and activation checkpointing.
  • Make models cheaper to serve with quantization, pruning, and distillation.

The previous chapters explained how networks learn. Building a system that works in the real world involves two further skills. The first is strategy: deciding what to try next among dozens of possible improvements, each of which can take weeks. The second is scale: training models whose parameters, activations, and data do not fit on one accelerator, and serving them cheaply. The first half of this chapter follows the practical methodology popularized by Andrew Ng (Ng 2018); the second half covers the systems techniques behind today’s large models. Chapter 12 covered the general machine learning workflow; this chapter adds what is specific to deep learning.

20.1Setting Goals: Metrics and Data Splits

One number. Progress is fastest when every experiment ends with a single number that says whether it helped. If you care about several quantities, combine them (for example the $F_1$ score instead of precision and recall), or designate one optimizing metric and turn the others into satisficing constraints: “maximize accuracy subject to latency below 100 ms and model size below 50 MB.”

Dev and test sets from the target distribution. The development (validation) set and the test set should come from the distribution you expect in deployment, and from the same distribution as each other. If an app’s users upload blurry phone photos, a dev set of crisp web images will steer you to the wrong model. The training set may come from elsewhere if that is where the data are (Section 20.4).

Sizes. With 10,000 examples, the classical 60/20/20 split is reasonable. With 10 million, a dev set of 10,000 already pins down an error rate to within about 1.4 percentage points with 95% confidence by Hoeffding’s inequality (Chapter 11), and more tightly in practice, so splits like 98/1/1 give the training set almost everything. Make the dev set large enough to detect the differences you care about between models.

Change the metric when it is wrong. If a model ranks better on the metric but users prefer the other one, the metric (or the dev set) does not capture what matters. A classic example is a filter that is more accurate overall but lets through a few unacceptable images; weighting those errors more heavily fixes the metric. Fix the target first, then optimize toward it.

20.2Human-Level Performance, Bias, and Variance

For perception tasks (vision, speech, language), humans are very good, and the error of an expert human, or a team of experts, is a useful proxy for the Bayes error, the lowest achievable error (Chapter 8). It splits the gap between where you are and where you could be:

QuantityDefinitionWhat reduces it
Avoidable biastraining error $-$ human-level errorbigger model, longer training, better optimizer, better architecture
Variancedev error $-$ training errormore data, regularization, augmentation

With human error 1%, training error 8%, and dev error 10%, the avoidable bias (7%) dominates the variance (2%): work on fitting the training set. With human error 7.5% on the same numbers, the avoidable bias is only 0.5% and the variance becomes the target. The same training and dev errors call for opposite actions depending on what is achievable. Two caveats: once a model surpasses human performance, this guide runs out and progress becomes harder to direct; and in domains where humans are poor (ad click prediction, product recommendations, many structured-data problems), human-level error is not a useful reference.

20.3Error Analysis

Before investing a month in an idea, measure how much it could possibly help. Take around 100 misclassified dev examples and look at them, tabulating the causes in a spreadsheet: for a cat detector, “dog,” “great cat (lion),” “blurry,” “mislabeled.” If 5% of the errors are dogs, a perfect dog fix improves the error from 10% to 9.5% at best; if 50% are blurry images, the ceiling for fixing blur is 5 percentage points. These ceilings tell you where to spend effort, and looking at the data often reveals categories nobody thought of.

Mislabeled data. Deep networks are robust to a small rate of random label errors in the training set, but not to systematic ones (all white dogs labeled as cats). In the dev and test sets, if errors caused by wrong labels are a significant fraction of the remaining errors, relabel; and apply any correction process to dev and test alike so that they stay comparable.

Build the first system quickly. For a new problem, set up the data, metric, and a simple baseline within days, then let bias–variance analysis and error analysis direct the next steps. Elaborate systems designed in advance tend to solve the wrong problems.

20.4Mismatched Training and Test Distributions

Often the plentiful data come from a different distribution than the one you care about: 200,000 web images and 10,000 photos from the app, or clean speech plus a little in-car audio. Put the web data and part of the target data in the training set, and use the rest of the target data for dev and test. Then a gap between training and dev error has two possible causes, variance and the change of distribution, and you cannot tell them apart. Hold out a training-dev set: data from the training distribution not used for training. The errors now decompose as

GapBetweenMeaning
Avoidable biashuman level and training errorunderfitting
Variancetraining and training-dev erroroverfitting the training data
Data mismatchtraining-dev and dev errorthe target distribution is different
Dev overfittingdev and test errortoo much tuning on the dev set

There is no systematic fix for data mismatch, but there are good strategies: understand the differences by error analysis (noise? lighting? accents?), collect more data like the target, or make training data more similar to it, for example by synthesizing in-car audio from clean speech and recorded car noise. Synthetic data must be varied enough: if the network sees the same hour of car noise a thousand times, it may overfit to that hour, and the synthesized examples may cover only a tiny corner of the real space while looking convincing to people.

20.5Transfer, Multitask, and End-to-End Learning

Transfer learning (Section 16.8) helps when the source task has much more data than the target, both share the input type, and low-level features of the source are useful for the target: ImageNet pretraining for radiology, a large speech model for a wake-word detector. It does not help when the target already has more data than the source.

Multitask learning trains one network with several output heads, for example detecting pedestrians, cars, signs, and traffic lights in the same image, with a loss that sums over tasks (and ignores missing labels). It helps when the tasks share low-level features, have similar amounts of data, and the network is large enough for all of them; a small network can do worse on each task than separate networks.

End-to-end learning replaces a pipeline of hand-designed stages with one network trained from raw input to final output: audio to transcript without phonemes, pixels to steering angle. It needs a lot of data, because it must discover what the stages encoded, but it avoids the errors and information loss of hand-designed intermediate representations. With limited data, or when the stages have abundant data of their own (face detection, then face recognition), a pipeline of learned components is often better.

20.6Training at Scale

How much compute and memory? Training a dense Transformer with $N$ parameters on $D$ tokens costs about $6ND$ floating-point operations: two per parameter per token for the forward pass and four for the backward pass (Kaplan et al. 2020). A 7-billion-parameter model trained on 2 trillion tokens needs about $8.4\times10^{22}$ operations, weeks of a thousand accelerators. Memory is the other limit. With mixed-precision Adam, each parameter needs

$$\underbrace{2}_{\text{bf16 weights}}+\underbrace{2}_{\text{bf16 gradients}}+\underbrace{4+4+4}_{\text{fp32 master weights, } \vm,\ \vs}=16\ \text{bytes}, \tag{20.1}$$

so the 7-billion-parameter model needs 112 GB for its model state alone, before any activations, more than one accelerator holds. Activations scale with batch size, sequence length, width, and depth and are often larger still. Four techniques, usually combined, make such training possible.

Data parallelism. Every one of $P$ workers holds a full copy of the model, processes a different slice of the mini-batch, and computes its gradient; the gradients are then averaged across workers so that all copies take the same step. This is mathematically identical to training with the full batch. The averaging is an all-reduce operation. A naive implementation sends every gradient to one server, whose link becomes the bottleneck. Ring all-reduce (Patarasuk and Yuan 2009) arranges the workers in a ring and splits each gradient into $P$ chunks. In a reduce-scatter phase of $P-1$ steps, each worker passes one chunk to its neighbor, which adds its own copy, until every chunk has been fully summed on some worker; in an all-gather phase of $P-1$ more steps, the finished chunks circulate around the ring. Each worker sends and receives

$$2\,\frac{P-1}{P}\,N\ \text{numbers}, \tag{20.2}$$

which is almost independent of the number of workers, so the communication time stays constant as the cluster grows. Data parallelism with large global batches needs care with the learning rate: scaling it linearly with the batch size, with a warm-up, works up to batch sizes of several thousand images (Goyal et al. 2017). When a large batch does not fit in memory, gradient accumulation sums the gradients of several smaller micro-batches before taking one step.

Sharding the model state: ZeRO and FSDP. Data parallelism replicates the 16 bytes per parameter on every worker. ZeRO (Rajbhandari et al. 2020) removes the redundancy in three stages: partition the optimizer states across workers (stage 1), then also the gradients (stage 2), then also the parameters themselves (stage 3), so that each worker stores $16N/P$ bytes. Parameters are gathered layer by layer just before they are needed and discarded afterward, which costs about 50% more communication than plain data parallelism. PyTorch’s fully sharded data parallel (FSDP) implements the same idea.

Tensor parallelism. A single layer can be split across devices. For an MLP block $\mathbf Y=\phi(\mX\mA)\mathbf B$, split $\mA$ by columns and $\mathbf B$ by rows: each device computes its slice of $\phi(\mX\mA)$ independently (the nonlinearity acts elementwise) and a partial product with its rows of $\mathbf B$, and one all-reduce sums the partial outputs (Shoeybi et al. 2019). Attention heads split the same way. Tensor parallelism needs very fast links and is used within a server.

Pipeline parallelism. Assign consecutive groups of layers to different devices, like stations on an assembly line. To keep all stations busy, split each mini-batch into $m$ micro-batches that flow through the pipeline one after another (Huang et al. 2019). At the start and end of each step some stations are idle; with $P$ stages the idle fraction, the bubble, is

$$\frac{P-1}{m+P-1}, \tag{20.3}$$

small when there are many more micro-batches than stages.

Activation checkpointing. Instead of storing every layer’s activations for the backward pass, store only some of them (for example the input of every Transformer block) and recompute the rest during the backward pass. With checkpoints every $\sqrt L$ layers, activation memory falls from $O(L)$ to $O(\sqrt L)$ for about one extra forward pass of compute (Chen et al. 2016).

Large training runs combine all of these, 3D parallelism: tensor parallelism within a server, pipeline parallelism across a few servers, and sharded data parallelism across the rest, with checkpointing and mixed precision throughout.

20.7Efficient Inference and Deployment

A trained model must answer requests within a latency budget (time per request) at an acceptable throughput (requests per second per device) and cost. Batching requests raises throughput but adds waiting time; large language models use continuous batching and the KV cache (Section 19.8). Several techniques make the model itself cheaper.

Quantization stores weights, and sometimes activations, in low-precision integers. With symmetric per-channel quantization, each output channel of a weight matrix gets a scale $s=\max|w|/127$, and each weight is stored as the 8-bit integer $q=\mathrm{round}(w/s)$, recovered as $\hat w=sq$:

$$q=\mathrm{round}\!\left(\frac{w}{s}\right)\in\{-127,\dots,127\},\qquad s=\frac{\max_{\text{channel}}|w|}{127}. \tag{20.4}$$

Int8 weights take a quarter of the memory of float32, and integer arithmetic is faster and more energy-efficient. Post-training quantization needs only a little calibration data and usually costs little accuracy at 8 bits (Jacob et al. 2018); at 4 bits or fewer, quantization-aware training, which simulates rounding during training, or quantization methods that account for outliers become necessary. The QLoRA method of Chapter 24 fine-tunes on top of 4-bit weights.

Pruning removes weights. Unstructured pruning zeroes individual small-magnitude weights and can remove 90% of the weights of many networks with little loss after fine-tuning (Han et al. 2016), but sparse matrices speed up computation only on hardware that supports them. Structured pruning removes whole channels, heads, or layers and yields a smaller dense model. The lottery ticket hypothesis (Frankle and Carbin 2019) observes that dense networks contain sparse subnetworks that, trained in isolation from the original initialization, match the full network.

Knowledge distillation (Hinton et al. 2015) trains a small student to match the output distribution of a large teacher, typically with a softened softmax (temperature $T>1$) whose “dark knowledge” (which wrong classes are almost right) carries more information per example than one-hot labels. Many small deployed language and vision models are distilled from larger ones.

Efficient architectures and compilers. Depthwise separable convolutions (Section 17.6), smaller vocabularies, and fewer layers reduce compute directly. Export formats and compilers (ONNX, TensorRT, torch.compile, and similar tools) fuse operations and choose fast kernels for the target hardware. After deployment, monitor inputs and outputs for distribution shift (Chapter 12).

20.8Implementation

The listing simulates ring all-reduce on four workers and checks both its result and its communication cost Equation (20.2), then applies post-training int8 quantization Equation (20.4) to a trained MLP:

import numpy as np

def ring_allreduce(grads):
    """Sum equal-shaped arrays held by P workers using only neighbor-to-neighbor messages.
    Phase 1 (reduce-scatter): after P-1 steps, worker p holds the full sum of chunk (p+1) % P.
    Phase 2 (all-gather): P-1 more steps circulate the finished chunks to everyone."""
    P = len(grads)
    chunks = [np.array_split(g.astype(float).copy(), P) for g in grads]
    sent = 0
    for step in range(P - 1):                              # reduce-scatter
        msgs = [(p, (p - step) % P, chunks[p][(p - step) % P].copy()) for p in range(P)]
        for p, c, data in msgs:                            # worker p sends chunk c to p+1
            chunks[(p + 1) % P][c] += data; sent += data.size
    for step in range(P - 1):                              # all-gather
        msgs = [(p, (p + 1 - step) % P, chunks[p][(p + 1 - step) % P].copy()) for p in range(P)]
        for p, c, data in msgs:
            chunks[(p + 1) % P][c] = data; sent += data.size
    return [np.concatenate(c) for c in chunks], sent / P   # result per worker, numbers sent per worker

rng = np.random.default_rng(0)
P, N = 4, 1_000_000
grads = [rng.standard_normal(N) for _ in range(P)]
out, per_worker = ring_allreduce(grads)
print("all workers agree with the true sum:", all(np.allclose(o, sum(grads)) for o in out))
print(f"numbers sent per worker: {per_worker:,.0f} = 2(P-1)/P * N = {2 * (P - 1) / P * N:,.0f}")

# Post-training int8 quantization of a trained MLP's weights (per output channel, symmetric)
from sklearn.datasets import load_digits
from sklearn.neural_network import MLPClassifier
X, y = load_digits(return_X_y=True); X = X / 16.0
mlp = MLPClassifier((128, 64), max_iter=400, random_state=0).fit(X[:1400], y[:1400])

def quantize(W):
    scale = np.abs(W).max(axis=0, keepdims=True) / 127.0   # one scale per output unit
    return np.round(W / scale).astype(np.int8), scale

def predict(weights, biases, X):
    h = X
    for i, (W, b) in enumerate(zip(weights, biases)):
        h = h @ W + b
        if i < len(weights) - 1:
            h = np.maximum(h, 0)
    return h.argmax(1)

q = [quantize(W) for W in mlp.coefs_]
W_deq = [Wq.astype(float) * s for Wq, s in q]             # dequantize for this NumPy demo
acc_fp = np.mean(predict(mlp.coefs_, mlp.intercepts_, X[1400:]) == y[1400:])
acc_q = np.mean(predict(W_deq, mlp.intercepts_, X[1400:]) == y[1400:])
bytes_fp = sum(W.size * 4 for W in mlp.coefs_); bytes_q = sum(Wq.size for Wq, _ in q)
print(f"float32 accuracy {acc_fp:.3f}, int8 accuracy {acc_q:.3f}, weight memory {bytes_fp} -> {bytes_q} bytes")

All workers end with the exact sum, each sends 1.5 million numbers for a 1-million-entry gradient, as Equation (20.2) predicts, and int8 weights keep the accuracy of the float32 model on the digits test set at a quarter of the memory.

20.9Common Pitfalls

Pitfall: A dev set that does not look like deployment

Most surprises after launch come from evaluating on data unlike real usage. Build the dev and test sets from the target distribution first, even if they are small.

Pitfall: Optimizing what is easy to measure

Tuning hard against a convenient but misaligned metric (overall accuracy when rare failures are what matter) makes the system worse. Revisit the metric when it disagrees with judgment.

Pitfall: Scaling the batch without adjusting the recipe

Moving from one GPU to many multiplies the global batch size. Without adjusting the learning rate and warm-up, accuracy can drop; and BatchNorm statistics computed per device change with the per-device batch.

20.10Summary

  • Use one optimizing metric with satisficing constraints, and dev and test sets drawn from the deployment distribution.
  • Human-level error proxies the Bayes error: avoidable bias calls for more capacity, variance for more data and regularization.
  • Error analysis on about 100 dev errors bounds the value of every possible fix.
  • A training-dev set separates variance from data mismatch; reduce mismatch by collecting or synthesizing target-like data.
  • Training costs about $6ND$ operations and 16 bytes of model state per parameter; data parallelism with ring all-reduce, ZeRO sharding, tensor and pipeline parallelism, and activation checkpointing make large models trainable.
  • Quantization, pruning, distillation, and compilers cut the cost of serving.

20.11Exercises

  1. MathFor each case, compute the avoidable bias and the variance, and name the most promising next step: (a) human 1%, training 1.5%, dev 8%; (b) human 0.5%, training 4%, dev 4.5%; (c) human 4%, training 4.5%, dev 5%, test 9%.
  2. MathProve that ring all-reduce sends $2(P-1)N/P$ numbers per worker, and compare with a parameter server that receives $N$ numbers from each of $P$ workers and sends the sum back. How long does each take with $P=64$, $N=10^9$ 2-byte gradients, and links of 50 GB/s?
  3. MathUsing Equation (20.1), how many 80 GB accelerators are needed just for the model state of a 70-billion-parameter model with plain data parallelism, and with ZeRO stage 3 on 64 workers?
  4. MathShow that splitting $\mA$ by columns and $\mathbf B$ by rows in $\phi(\mX\mA)\mathbf B$ gives the correct result with one sum, and explain why splitting $\mA$ by rows would require communication before the nonlinearity.
  5. CodingExtend the quantization code to 4 bits per weight ($q\in\{-7,\dots,7\}$) with per-channel and per-tensor scales. How much accuracy does each lose?
  6. CodingDistill the MLP of the implementation into a network with one hidden layer of 16 units: train the student once on the labels and once on the teacher’s softened probabilities ($T=4$). Compare test accuracies.
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.