Chapter 21 · Part V: Computer Vision
Object Detection and Segmentation
Classification says what is in an image. Detection says where each object is, and segmentation labels every pixel. These dense prediction tasks power self-driving cars, medical imaging, and photo editing, and they reuse the backbones of the previous chapters with new heads, losses, and evaluation.
After this chapter you will be able to
- Distinguish semantic segmentation, object detection, instance segmentation, and panoptic segmentation, and their evaluation metrics.
- Build fully convolutional segmentation networks with transposed convolutions and U-Net skip connections.
- Compute intersection over union, apply non-maximum suppression, and compute average precision.
- Explain the two-stage detectors (R-CNN, Fast R-CNN, Faster R-CNN with anchors and region proposals) and the one-stage detectors (YOLO, SSD, RetinaNet with the focal loss).
- Describe set-based detection with Transformers (DETR) and instance segmentation with Mask R-CNN.
- Explain promptable segmentation foundation models such as SAM.
Chapter 17 assigned one label to a whole image. Most applications need more: a self-driving car must know where every pedestrian is, a radiologist needs the outline of a tumor, a photo editor needs to separate a person from the background. This chapter covers the tasks that answer “where,” in roughly the order the field developed them.
21.1From Classification to Dense Prediction
Four tasks go beyond classification:
- Semantic segmentation assigns a class to every pixel (road, car, person, sky). It does not distinguish two cars from one big car.
- Object detection outputs a list of objects, each with a class, a confidence score, and an axis-aligned bounding box.
- Instance segmentation outputs a pixel mask for each detected object, so that two adjacent people get separate masks.
- Panoptic segmentation (Kirillov et al. 2019) unifies the two: every pixel gets a class, and pixels of countable “things” (people, cars) also get an instance identity, while amorphous “stuff” (sky, road) does not.
The standard benchmark is COCO (Lin et al. 2014), with about 118,000 training images annotated with boxes and masks for 80 object categories, plus stuff classes for panoptic segmentation.
21.2Semantic Segmentation
A naive approach classifies each pixel by running a CNN on a patch around it, which recomputes almost the same features for neighboring pixels millions of times. A fully convolutional network (FCN) (Long et al. 2015) instead applies convolutions to the whole image at once and produces a $K$-channel score map, one channel per class, with a softmax at every pixel. The loss is the cross-entropy summed over pixels. Because convolutions work on any input size, the network can be trained on crops and applied to full images.
To see large context, the network must downsample, as classification networks do; but the output must be at full resolution. So segmentation networks have an encoder that downsamples and a decoder that upsamples back to the input size.
Metrics. Pixel accuracy is dominated by large classes such as road and sky. The standard metric is the intersection over union of the predicted and true pixel sets for each class $c$, averaged over the $K$ classes:
$$\mathrm{IoU}_c=\frac{|\text{pred}_c\cap\text{true}_c|}{|\text{pred}_c\cup\text{true}_c|}=\frac{TP_c}{TP_c+FP_c+FN_c},\qquad \mathrm{mIoU}=\frac1K\sum_{c=1}^K\mathrm{IoU}_c . \tag{21.1}$$21.3Upsampling and Transposed Convolution
Fixed upsampling methods include nearest-neighbor copying, bilinear interpolation, and max unpooling, which remembers which input position won each max-pooling window in the encoder and places the value back there, filling the rest with zeros. A learnable alternative is the transposed convolution. Each input value multiplies a $k\times k$ kernel, and the scaled kernels are placed in the output at a spacing of $s$ pixels, overlapping ones being summed. It is called “transposed” because, if a convolution is written as multiplication by a sparse matrix $\mathbf C$ (Section 17.5), this operation is multiplication by $\mathbf C\T$; it is exactly the backward pass of a convolution with respect to its input. (The name “deconvolution” is also used but is misleading: it does not invert the convolution.) With input size $n$, the output size is
$$n_{\text{out}}=(n-1)\,s-2p+k, \tag{21.2}$$the inverse of Equation (17.4). When the kernel size is not divisible by the stride, some output pixels receive more overlapping contributions than others, which produces checkerboard artifacts in generated images; using $k$ divisible by $s$ (for example $k=4$, $s=2$), or resizing with interpolation followed by an ordinary convolution, avoids them.
21.4U-Net and Skip Connections
Upsampling cannot recover detail that the encoder threw away: the boundaries of a segmentation from a heavily downsampled feature map are blurry. U-Net (Ronneberger et al. 2015) adds skip connections from each encoder stage to the decoder stage of the same resolution, concatenating the high-resolution encoder features with the upsampled decoder features. Coarse layers say what is there, fine layers say where the edges are, and the decoder combines both. Drawn with the encoder on the left and the decoder on the right, the architecture forms a U. Designed for biomedical images with few training examples, U-Net became the standard segmentation architecture and, later, the standard denoising network of diffusion models (Chapter 25). An alternative keeps resolution high with dilated convolutions (Section 17.3), which enlarge the receptive field without downsampling. Project 11 trains a small U-Net on synthetic shapes.
21.5Object Detection: Boxes, IoU, and Non-Maximum Suppression
A bounding box is four numbers, either corners $(x_1,y_1,x_2,y_2)$ or center, width, and height $(x,y,w,h)$. A detector outputs many candidate boxes with class scores, and two operations turn them into a final answer.
Intersection over union. The overlap of two boxes $A$ and $B$ is measured by
$$\mathrm{IoU}(A,B)=\frac{\mathrm{area}(A\cap B)}{\mathrm{area}(A\cup B)}\in[0,1]. \tag{21.3}$$An IoU of 0.5 is a rough match, 0.7 a good one, and 0.9 nearly perfect.
Non-maximum suppression (NMS). Detectors typically fire several overlapping boxes on each object. Greedy NMS sorts the boxes of each class by score, keeps the highest-scoring box, deletes all remaining boxes whose IoU with it exceeds a threshold (typically 0.5), and repeats with the next surviving box. NMS can wrongly suppress a true object that heavily overlaps another, such as people in a crowd, a weakness that set-based detectors avoid (Section 21.9).
21.6Evaluating Detectors: Average Precision
A detection counts as a true positive if its IoU with an unmatched ground-truth box of the same class is at least a threshold; otherwise it is a false positive, and so is a second detection of an already matched object. Sorting all detections of a class by score and sweeping a threshold gives a precision–recall curve (Chapter 6). The average precision (AP) is the area under it, after replacing each precision value by the maximum precision at any higher recall (which makes the curve monotone):
$$\mathrm{AP}=\int_0^1p_{\text{interp}}(r)\dd r,\qquad p_{\text{interp}}(r)=\max_{r'\ge r}p(r'). \tag{21.4}$$The mean average precision (mAP) averages AP over classes. PASCAL VOC used an IoU threshold of 0.5 (mAP@0.5); COCO averages mAP over IoU thresholds from 0.5 to 0.95 in steps of 0.05, rewarding precise localization. Note that duplicates hurt: detecting the same object twice adds a false positive, which is why NMS improves AP.
21.7Two-Stage Detectors: From R-CNN to Faster R-CNN
R-CNN (Girshick et al. 2014) brought CNNs to detection in three steps: generate about 2,000 class-agnostic region proposals per image with a classical algorithm (selective search), warp each region to a fixed size and run a CNN on it, and classify the features while also regressing corrections to the box. It improved accuracy dramatically but ran one CNN per proposal, taking tens of seconds per image.
Fast R-CNN (Girshick 2015) runs the backbone once on the whole image and extracts a fixed-size feature for each proposal from the shared feature map by RoI pooling: the region is divided into a grid of, say, $7\times7$ cells and each cell is max-pooled. Classification and box regression are trained jointly with a multitask loss. The proposals, still computed by selective search, became the bottleneck.
Faster R-CNN (Ren et al. 2015) learns the proposals too, with a small region proposal network (RPN) that slides over the backbone’s feature map. At each position it considers $A$ anchor boxes of several sizes and aspect ratios (typically $A=9$) and predicts, for each anchor, an objectness score and four offsets that turn the anchor $(x_a,y_a,w_a,h_a)$ into a box $(x,y,w,h)$. The offsets are parameterized to be scale-invariant:
$$t_x=\frac{x-x_a}{w_a},\qquad t_y=\frac{y-y_a}{h_a},\qquad t_w=\log\frac{w}{w_a},\qquad t_h=\log\frac{h}{h_a}. \tag{21.5}$$Anchors with IoU above 0.7 with a ground-truth box are positives and those below 0.3 negatives. The RPN’s top proposals after NMS go to the second stage, which classifies them and refines their boxes. Both stages are trained with a classification loss plus a regression loss on the offsets (a smooth version of $\ell_1$, only for positive anchors):
$$\Loss=\frac{1}{N_{\text{cls}}}\sum_i\ell_{\text{cls}}(p_i,p_i^*)+\lambda\frac{1}{N_{\text{reg}}}\sum_ip_i^*\,\ell_{\text{reg}}(\vt_i,\vt_i^*). \tag{21.6}$$Feature pyramid networks (Lin et al. 2017) add a top-down path with lateral connections that builds strong features at every scale, so small objects are detected on high-resolution maps and large objects on coarse ones.
21.8One-Stage Detectors
One-stage detectors skip the proposal step and predict classes and boxes directly on a dense grid. YOLO (Redmon et al. 2016) divides the image into an $S\times S$ grid; each cell predicts a few boxes with confidences and class probabilities, all in one forward pass, fast enough for real-time video. SSD (Liu et al. 2016) predicts from anchors on feature maps of several resolutions. Project 11 builds a small YOLO-style grid detector.
The focal loss. A dense detector evaluates on the order of $10^5$ anchors per image, almost all of them easy background. Even with a small loss each, the easy negatives dominate the total loss and the gradient. RetinaNet (Lin et al. 2017) reweights the cross-entropy so that well-classified examples contribute little. With $p_t$ the predicted probability of the true class,
$$\mathrm{FL}(p_t)=-\alpha_t\,(1-p_t)^\gamma\log p_t, \tag{21.7}$$and $\gamma=2$: an example classified with $p_t=0.9$ has its loss reduced 100-fold relative to the cross-entropy, while hard examples keep nearly their full weight. With the focal loss, one-stage detectors matched the accuracy of two-stage ones. Later one-stage designs dropped anchors altogether and predict, at each location, the distances to the four sides of the object box.
21.9Detection with Transformers
DETR (Carion et al. 2020) treats detection as set prediction. A CNN or ViT backbone produces image features, a Transformer encoder processes them, and a Transformer decoder takes a fixed number $M$ (say 100) of learned object queries, each of which attends to the image and outputs one box and class, or “no object.” Training matches the $M$ predictions to the ground-truth objects (padded with “no object”) by the Hungarian algorithm, which finds the one-to-one assignment $\hat\sigma$ minimizing the total matching cost,
$$\hat\sigma=\argmin_{\sigma}\sum_{i=1}^M\mathcal C_{\text{match}}\big(y_i,\hat y_{\sigma(i)}\big), \tag{21.8}$$where the cost combines class probability and box distance; the loss is then computed on matched pairs. Because each object is matched to exactly one prediction, the model learns not to produce duplicates: there are no anchors and no NMS. The original DETR trained slowly and struggled with small objects; follow-up work with deformable and multi-scale attention fixed both, and DETR-style detectors are now among the most accurate.
21.10Instance Segmentation
Mask R-CNN (He et al. 2017) adds a third head to Faster R-CNN: for every detected region, a small FCN predicts a $28\times28$ binary mask for each class, and the mask of the predicted class is resized to the box. A small but important change was RoIAlign: RoI pooling rounds region coordinates to the feature-map grid, misaligning masks by up to half a stride; RoIAlign samples the feature map at exact fractional positions with bilinear interpolation. The same framework predicts human keypoints by treating each keypoint as a one-hot mask.
21.11Foundation Models for Segmentation
Segmentation models trained on a fixed set of classes cannot segment anything else. Segment Anything (SAM) (Kirillov et al. 2023) made segmentation promptable: given an image and a prompt (a point, a box, or a rough mask), it returns a mask of the object the prompt refers to, for any kind of object. It has a heavy ViT image encoder that runs once per image, a light prompt encoder, and a light mask decoder that runs in milliseconds for each new prompt, so it is interactive. Because a single point is ambiguous (does a click on a shirt mean the shirt or the person?), it outputs several masks with confidence scores. SAM was trained on SA-1B, a data set of 1.1 billion masks on 11 million images collected with a data engine: the model assisted human annotators, its improved versions annotated automatically, and the cycle repeated. SAM 2 (Ravi et al. 2024) extended the approach to video with a streaming memory that tracks the prompted object across frames. In parallel, open-vocabulary detectors use text embeddings from vision–language models (Chapter 23) instead of a fixed classifier, so that they can detect categories named in a text prompt.
21.12Implementation
The core operations of detection evaluation in NumPy: vectorized IoU, greedy NMS, and average precision with all-point interpolation, applied to a simulated detector that fires several jittered boxes per object plus two false alarms:
import numpy as np
def iou(a, b):
"""IoU between boxes a (n, 4) and b (m, 4) in (x1, y1, x2, y2) format -> (n, m)."""
x1 = np.maximum(a[:, None, 0], b[None, :, 0]); y1 = np.maximum(a[:, None, 1], b[None, :, 1])
x2 = np.minimum(a[:, None, 2], b[None, :, 2]); y2 = np.minimum(a[:, None, 3], b[None, :, 3])
inter = np.clip(x2 - x1, 0, None) * np.clip(y2 - y1, 0, None)
area = lambda r: (r[:, 2] - r[:, 0]) * (r[:, 3] - r[:, 1])
return inter / (area(a)[:, None] + area(b)[None, :] - inter)
def nms(boxes, scores, thresh=0.5):
"""Greedy non-maximum suppression: keep the best box, drop boxes overlapping it, repeat."""
order, keep = np.argsort(-scores), []
while order.size:
i = order[0]; keep.append(i)
order = order[1:][iou(boxes[[i]], boxes[order[1:]])[0] < thresh]
return np.array(keep)
def average_precision(pred_boxes, scores, gt_boxes, thresh=0.5):
"""AP for one class in one image set: match predictions to ground truth in score order."""
order = np.argsort(-scores); matched = np.zeros(len(gt_boxes), bool); tp = []
for i in order:
ious = iou(pred_boxes[[i]], gt_boxes)[0] if len(gt_boxes) else np.array([])
j = int(np.argmax(ious)) if ious.size else -1
hit = j >= 0 and ious[j] >= thresh and not matched[j]
if hit:
matched[j] = True
tp.append(hit)
tp = np.array(tp, float); fp = 1 - tp
recall = np.cumsum(tp) / len(gt_boxes); precision = np.cumsum(tp) / (np.cumsum(tp) + np.cumsum(fp))
# all-point interpolation: make precision monotone, then integrate over recall
prec = np.concatenate([[0], precision, [0]]); rec = np.concatenate([[0], recall, [1]])
for k in range(len(prec) - 2, -1, -1):
prec[k] = max(prec[k], prec[k + 1])
idx = np.where(rec[1:] != rec[:-1])[0]
return np.sum((rec[idx + 1] - rec[idx]) * prec[idx + 1])
rng = np.random.default_rng(0)
gt = np.array([[10, 10, 50, 60], [70, 20, 120, 80], [30, 90, 90, 140]], float)
# a detector's raw output: several jittered boxes per object plus two false alarms
cands = [g + rng.normal(0, 4, 4) for g in gt for _ in range(5)] + [np.array([130, 120, 160, 150.]), np.array([0, 100, 20, 130.])]
cands = np.array(cands)
scores = np.concatenate([rng.uniform(0.6, 0.95, 15), [0.7, 0.4]])
keep = nms(cands, scores, 0.5)
print(f"{len(cands)} candidate boxes -> {len(keep)} after NMS")
print(f"AP@0.5 before NMS: {average_precision(cands, scores, gt):.3f}")
print(f"AP@0.5 after NMS: {average_precision(cands[keep], scores[keep], gt):.3f}")
print("IoU of each ground-truth box with its best kept detection:",
np.round(iou(gt, cands[keep]).max(1), 2))
NMS reduces the 17 raw boxes to one per object plus the false alarms, and AP@0.5 rises from 0.83 to 1.0 because duplicate detections no longer count as false positives. Project 11 uses these functions to evaluate a trained detector.
21.13Common Pitfalls
Pitfall: Inconsistent box formats
Mixing $(x_1,y_1,x_2,y_2)$ with $(x,y,w,h)$, normalized with pixel coordinates, or row–column with $x$–$y$ order is the most common detection bug. Convert at the data-loading boundary and assert the format everywhere else.
Pitfall: Augmenting images but not labels
Random crops, flips, and resizes must transform the boxes and masks too. Visualize a few augmented examples with their labels before training.
Pitfall: Comparing mAP numbers from different protocols
mAP@0.5 and COCO’s mAP@[0.5:0.95] differ by 20 points or more for the same detector. State the protocol, the IoU thresholds, and the data split.
21.14Summary
- Semantic segmentation labels pixels, detection boxes objects, instance segmentation masks objects, and panoptic segmentation does both; mIoU and mAP are the standard metrics.
- Fully convolutional encoder–decoder networks segment images; transposed convolutions upsample (beware checkerboards), and U-Net’s skip connections restore detail.
- IoU measures box overlap, NMS removes duplicates, and AP integrates the interpolated precision–recall curve.
- Two-stage detectors propose regions (with anchors and an RPN in Faster R-CNN) and then classify them; one-stage detectors predict densely and use the focal loss against easy negatives.
- DETR casts detection as set prediction with Hungarian matching and needs neither anchors nor NMS; Mask R-CNN adds a mask head with RoIAlign.
- SAM segments any object from a point or box prompt, trained on a billion masks gathered with a model-in-the-loop data engine.
21.15Exercises
- MathShow that IoU is symmetric, that $1-\mathrm{IoU}$ satisfies the triangle inequality for boxes (you may assume it, but test it numerically on random boxes), and compute the IoU of two unit squares offset by $\frac12$ horizontally.
- MathUsing Equation (21.2), choose kernel size, stride, and padding for a transposed convolution that exactly doubles a $16\times16$ feature map. Show that for $k=3$, $s=2$ the number of kernel contributions to an output pixel alternates between 1, 2, and 4.
- MathInvert Equation (21.5) to recover a box from an anchor and predicted offsets. Why are $t_w$ and $t_h$ logarithmic?
- MathFor the focal loss with $\alpha_t=1$, compute the ratio $\mathrm{FL}(p_t)/\mathrm{CE}(p_t)$ for $p_t=0.5,0.9,0.99$ and $\gamma=0,1,2,5$. Derive the gradient of the focal loss with respect to the logit for a binary classifier.
- CodingExtend the implementation to compute COCO-style mAP averaged over IoU thresholds $0.5,0.55,\dots,0.95$ for several classes, and check your numbers on the simulated detector with box noise of 2, 4, and 8 pixels.
- CodingImplement soft-NMS, which decays the scores of overlapping boxes by a factor $e^{-\mathrm{IoU}^2/\sigma}$ instead of deleting them, and compare it with greedy NMS on two heavily overlapping objects.