Chapter 22 · Part V: Computer Vision
Visualizing and Understanding Networks
A trained network is millions of numbers that work. Visualization and attribution methods show what those numbers respond to, adversarial examples show how brittle they can be, and the same gradient tools turn classifiers into image generators for style transfer.
After this chapter you will be able to
- Visualize filters, activations, and learned feature spaces of a CNN.
- Compute occlusion maps, gradient saliency maps, integrated gradients, and Grad-CAM, and know their limitations.
- Generate images that maximize a neuron’s activation and invert feature representations.
- Construct adversarial examples with FGSM and PGD, explain why they exist, and defend with adversarial training.
- Transfer the style of one image onto another with Gram-matrix losses.
Chapter 17 trained networks that recognize thousands of object categories. What have they learned? This question matters for scientific understanding, for debugging (is the model looking at the object or at a watermark?), and for trust in high-stakes applications. This chapter collects the tools developed to answer it. Almost all of them use the same ingredient as training, the gradient, but with respect to the input rather than the weights.
22.1Looking Inside: Filters, Activations, and Features
First-layer filters. The first convolutional layer operates directly on pixels, so its kernels can be displayed as small images. In virtually every network trained on natural images, they are oriented edge detectors at various angles and frequencies, and color-opponent blobs, much like the receptive fields of neurons in the primary visual cortex and the filters found by ICA on image patches (Section 14.8). Deeper filters cannot be displayed this way, because their inputs are feature maps, not pixels.
Activations and maximally activating patches. For a unit in a deeper layer, run the network on many images and collect the image patches (the unit’s receptive field) that activate it most. Units in middle layers respond to textures and parts, such as eyes, wheels, and text, and units in late layers to whole objects or scenes (Zeiler and Fergus 2014). Occlusion experiments slide a gray square over the image and record how the class score drops, revealing which regions the decision depends on.
The feature space. The vector of the last hidden layer (for example 2,048 numbers in a ResNet-50) is a learned representation of the image. Nearest neighbors in this space are semantically similar images even when their pixels differ greatly (a dog facing left and right), unlike the pixel-space neighbors of Section 17.1. To see the structure of the whole space, embed the feature vectors of many images in two dimensions with PCA (Chapter 14) or with nonlinear methods such as t-SNE (van der Maaten and Hinton 2008) and UMAP, which preserve local neighborhoods: images of the same class form clusters.
22.2Which Pixels Matter? Attribution Maps
Gradient saliency. The gradient of the class score $s_c$ with respect to the input, $\nabla_{\vx}s_c$, says how much each pixel would change the score if it changed slightly. Its magnitude, maximized over color channels, gives a saliency map (Simonyan et al. 2014):
$$M_{ij}=\max_{\text{channel }k}\Big|\pd{s_c}{x_{ijk}}\Big| . \tag{22.1}$$One backward pass computes it. Raw gradients are noisy; averaging them over several noisy copies of the input (SmoothGrad) gives cleaner maps.
Integrated gradients. The gradient is a local quantity: if a pixel’s effect has saturated (a ReLU already off, a sigmoid already flat), its gradient is zero even though it matters. Integrated gradients (Sundararajan et al. 2017) accumulate gradients along a straight path from a baseline $\vx'$ (such as a black image) to the input:
$$\mathrm{IG}_i(\vx)=(x_i-x_i')\int_0^1\pd{s_c}{x_i}\Big(\vx'+\alpha(\vx-\vx')\Big)\dd\alpha . \tag{22.2}$$By the fundamental theorem of calculus for path integrals, the attributions sum exactly to $s_c(\vx)-s_c(\vx')$ (completeness), a property plain gradients lack. In practice the integral is approximated with 20 to 300 points.
Grad-CAM. Class activation mapping uses the last convolutional layer, whose feature maps $A^k$ ($k=1,\dots,K$ channels) are still spatial but already semantic. Grad-CAM (Selvaraju et al. 2017) weights each channel by the average gradient of the class score over the spatial positions and keeps the positive part:
$$\alpha_k^c=\frac1Z\sum_{i,j}\pd{s_c}{A^k_{ij}},\qquad L^c_{\text{Grad-CAM}}=\relu\Big(\sum_k\alpha_k^cA^k\Big). \tag{22.3}$$The result is a coarse heat map (for example $7\times7$, upsampled to the image) of where the evidence for class $c$ lies. It is class-discriminative: the map for “cat” and the map for “dog” in an image with both highlight different animals.
Pitfall: Attribution maps that ignore the model
Some attribution methods produce nearly the same maps for a trained network and for a network with randomized weights, so they reflect the input image’s edges more than the model’s reasoning (Adebayo et al. 2018). Before trusting a method, check that its maps change when the model’s weights are randomized and when the labels it was trained on are permuted.
22.3What Does a Neuron Want? Synthesizing Images
Activation maximization. Instead of searching a data set for images that excite a unit, optimize an image to excite it. Start from noise and run gradient ascent on the input:
$$\vx^*=\argmax_{\vx}\ s_c(\vx)-\lambda R(\vx), \tag{22.4}$$with a regularizer $R$ that keeps the image natural: an $\ell_2$ penalty, penalties on high-frequency content, blurring or jittering the image between steps, or a learned image prior. Without regularization the result is high-frequency noise that excites the unit but looks like nothing; with it, class visualizations show recognizable fragments of the class (dog faces, bird beaks) arranged in unnatural ways.
Feature inversion. Given the features $\phi(\vx_0)$ of an image at some layer, find an image whose features match: $\argmin_{\vx}\norm{\phi(\vx)-\phi(\vx_0)}^2+\lambda R(\vx)$ (Mahendran and Vedaldi 2015). Inverting early layers recovers the image almost exactly; inverting late layers preserves the content and layout but loses color and fine detail, showing which information the network keeps.
DeepDream amplifies whatever a chosen layer already detects in a real image: gradient ascent on the norm of the layer’s activations makes faint patterns stronger, producing the famous psychedelic images full of eyes and dog faces.
22.4Adversarial Examples
Szegedy et al. (2014) discovered that for essentially every correctly classified image, a perturbation too small for a human to notice changes the network’s prediction to almost any class the attacker chooses. These adversarial examples are not rare failures; they are a systematic property of high-dimensional models.
Why they exist: linearity in high dimensions. Consider a linear score $\vw\T\vx$ and a perturbation $\vdelta$ with $\norm{\vdelta}_\infty\le\varepsilon$, so no pixel changes by more than $\varepsilon$. Choosing $\vdelta=\varepsilon\sign(\vw)$ changes the score by $\varepsilon\norm{\vw}_1$, which grows linearly with the dimension: in a $224\times224\times3$ image, a change of $1/255$ per pixel, invisible to humans, can move the score by about 590 times the average absolute weight ($150{,}528/255$). Deep networks are piecewise linear (ReLU) and are trained to be roughly linear in their inputs, so the same effect applies (Goodfellow et al. 2015).
FGSM. The fast gradient sign method applies this idea to the loss of a network:
$$\vx_{\text{adv}}=\mathrm{clip}\Big(\vx+\varepsilon\,\sign\big(\nabla_{\vx}\Loss(\vtheta,\vx,y)\big)\Big), \tag{22.5}$$one gradient computation, with clipping to the valid pixel range.
PGD. Projected gradient descent (Madry et al. 2018) is the iterative version: take several smaller signed steps of size $\alpha$, and after each, project back onto the allowed set $\{\vx':\norm{\vx'-\vx}_\infty\le\varepsilon\}$ and the valid pixel range:
$$\vx^{(t+1)}=\Pi_{\varepsilon}\Big(\vx^{(t)}+\alpha\,\sign\big(\nabla_{\vx}\Loss(\vtheta,\vx^{(t)},y)\big)\Big). \tag{22.6}$$PGD from random starts is the standard strong attack against which defenses are evaluated. A targeted attack instead descends the loss of a chosen wrong class.
Beyond the threat model. Adversarial perturbations often transfer: examples crafted against one network fool others trained on similar data, enabling black-box attacks on models whose weights are secret. Attacks also work in the physical world: printed patches and stickers on road signs can fool detectors under varying viewpoints.
Adversarial training. The most reliable empirical defense trains on adversarial examples, solving a min–max problem:
$$\min_{\vtheta}\ \E_{(\vx,y)}\Big[\max_{\norm{\vdelta}_\infty\le\varepsilon}\Loss(\vtheta,\vx+\vdelta,y)\Big], \tag{22.7}$$approximating the inner maximum with PGD at every training step. It makes training several times more expensive and typically lowers accuracy on clean inputs, and robustness to one kind of perturbation does not imply robustness to others. Many proposed defenses that seemed to work were later broken by stronger, adaptive attacks; certified defenses such as randomized smoothing (Cohen et al. 2019) instead prove that no perturbation within a given radius can change the prediction. Adversarial robustness remains unsolved and matters well beyond images: the jailbreaks of language models (Chapter 24) are adversarial examples in text.
22.5Neural Style Transfer
Gradient-based image synthesis can also make art. Neural style transfer (Gatys et al. 2016) renders the content of one photograph in the style of a painting by optimizing a new image $\vx$ against a pretrained CNN (typically VGG). Content is captured by the feature maps $F^l(\vx)$ of a deep layer, and the content loss is $\norm{F^l(\vx)-F^l(\vx_{\text{content}})}^2$. Style, texture without layout, is captured by the correlations between channels, the Gram matrix of the feature maps of layer $l$ with $C_l$ channels at $N_l$ positions:
$$G^l_{ij}=\sum_{p=1}^{N_l}F^l_{ip}F^l_{jp},\qquad \Loss_{\text{style}}=\sum_lw_l\frac{1}{4C_l^2N_l^2}\big\Vert G^l(\vx)-G^l(\vx_{\text{style}})\big\Vert_F^2 . \tag{22.8}$$Summing over positions throws away where features occur and keeps which features co-occur, which is what texture is. Minimizing a weighted sum of the content and style losses over $\vx$ by gradient descent produces the stylized image. Optimization takes many forward and backward passes per image; Johnson et al. (2016) trained a feed-forward network to produce the result in one pass using the same losses as training objectives (perceptual losses), which also improved super-resolution.
Frontier note: Interpretability of large models
The questions of this chapter now extend to language and multimodal models. Feature attribution highlights which input tokens drive an output; mechanistic interpretability tries to reverse-engineer the algorithms implemented by individual circuits of attention heads and MLP neurons; and sparse autoencoders (Section 14.9) trained on internal activations decompose them into many interpretable features, some of which can be amplified or suppressed to steer the model’s behavior. These methods are young, and a complete account of how a large network computes remains out of reach.
22.6Implementation
Saliency, FGSM, PGD, and adversarial training for an MLP on the digits (pixels scaled to $[0,1]$), in PyTorch:
import torch, torch.nn as nn, torch.nn.functional as F
from sklearn.datasets import load_digits
torch.manual_seed(0)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32); y = torch.tensor(y) # pixels in [0, 1]
Xtr, ytr, Xte, yte = X[:1400], y[:1400], X[1400:], y[1400:]
def train(adv_eps=0.0, steps=1500):
torch.manual_seed(0)
model = nn.Sequential(nn.Linear(64, 128), nn.ReLU(), nn.Linear(128, 10))
opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for _ in range(steps):
idx = torch.randint(len(Xtr), (64,))
xb, yb = Xtr[idx], ytr[idx]
if adv_eps > 0: # adversarial training: fit PGD examples
xb = pgd(model, xb, yb, adv_eps, steps=5)
loss = F.cross_entropy(model(xb), yb)
opt.zero_grad(); loss.backward(); opt.step()
return model
def input_grad(model, x, y):
x = x.clone().requires_grad_(True)
F.cross_entropy(model(x), y).backward()
return x.grad
def fgsm(model, x, y, eps):
return (x + eps * input_grad(model, x, y).sign()).clamp(0, 1) # one signed step
def pgd(model, x, y, eps, steps=10):
alpha, xa = 2.5 * eps / steps, x.clone()
for _ in range(steps): # signed steps, projected to the eps-ball
xa = xa + alpha * input_grad(model, xa, y).sign()
xa = torch.min(torch.max(xa, x - eps), x + eps).clamp(0, 1)
return xa.detach()
def accuracy(model, x, y):
with torch.no_grad():
return (model(x).argmax(1) == y).float().mean().item()
model = train()
sal = input_grad(model, Xte[:1], yte[:1]).abs().view(8, 8) # saliency map of one test digit
print("most salient pixels (row, col):", [divmod(int(i), 8) for i in sal.flatten().argsort(descending=True)[:3]])
for eps in [0.0, 0.05, 0.1, 0.2]:
print(f"eps={eps:.2f}: clean model FGSM {accuracy(model, fgsm(model, Xte, yte, eps), yte):.2f}"
f" PGD {accuracy(model, pgd(model, Xte, yte, eps), yte):.2f}")
robust = train(adv_eps=0.1)
print(f"adversarially trained (eps=0.1): clean {accuracy(robust, Xte, yte):.2f}, "
f"PGD at eps=0.1 {accuracy(robust, pgd(robust, Xte, yte, 0.1), yte):.2f}")
The most salient pixels of the first test digit lie on its strokes. With perturbations of at most $0.1$ per pixel (one tenth of the intensity range), FGSM and PGD cut test accuracy from 91% to below 50%, and at $0.2$ to almost zero. Adversarial training with PGD at $\varepsilon=0.1$ keeps clean accuracy and raises accuracy under the same attack from 44% to about 70%. Project 12 repeats these experiments, together with Grad-CAM, on the CIFAR-10 CNN of Project 9.
22.7Common Pitfalls
Pitfall: Evaluating a defense with a weak attack
A defense that only resists FGSM, or only the attack its authors tried, is not robust. Evaluate with PGD from several random starts, with adaptive attacks that know the defense, and check that accuracy falls to zero as $\varepsilon$ grows; if it does not, the gradients are probably masked rather than the model robust.
Pitfall: Reading too much into a heat map
A saliency map shows sensitivity, not reasoning. It can highlight the right object for the wrong reason and changes with small, irrelevant input perturbations. Use attribution maps for debugging hypotheses, not as explanations to end users.
22.8Summary
- First-layer filters are edge and color detectors; deeper units respond to textures, parts, and objects; the last-layer feature space places semantically similar images close together.
- Saliency maps use input gradients; integrated gradients satisfy completeness; Grad-CAM weights last-layer feature maps by averaged gradients for class-specific heat maps. Sanity checks guard against maps that ignore the model.
- Gradient ascent on the input with image priors visualizes what units want; feature inversion shows what information a layer keeps.
- Adversarial examples arise from approximate linearity in high dimensions; FGSM takes one signed gradient step, PGD many projected ones; adversarial training is the strongest empirical defense, at a cost in accuracy and compute.
- Style transfer matches deep features for content and Gram matrices for style.
22.9Exercises
- MathFor a linear classifier with weights $\vw\in\R^d$ whose entries all have magnitude about $m$, compute the change of the score under the perturbation $\varepsilon\sign(\vw)$. How large must $\varepsilon$ be to change the score by 10 when $d=3072$ and $m=0.01$?
- MathProve the completeness property of integrated gradients, $\sum_i\mathrm{IG}_i(\vx)=s_c(\vx)-s_c(\vx')$, using the chain rule along the path $\alpha\mapsto\vx'+\alpha(\vx-\vx')$.
- MathShow that the Gram matrix of Equation (22.8) is unchanged by any permutation of the spatial positions. What does this imply about what the style loss can and cannot capture?
- MathShow that for a linear model with logistic loss, the FGSM perturbation Equation (22.5) is the exact solution of the inner maximization in Equation (22.7), and that adversarial training then becomes logistic regression in which every margin $y\,\vw\T\vx$ is reduced by $\varepsilon\norm{\vw}_1$.
- CodingAdd a targeted PGD attack to the implementation that turns every test digit into a 3. What fraction of digits can be turned into a 3 with $\varepsilon=0.1$? Visualize the perturbations: do they look like strokes of a 3?
- CodingImplement integrated gradients for the digits MLP with a black baseline and 50 steps, verify completeness numerically, and compare its maps with plain gradient saliency.