Machine Learning from First Principles

Project 11 · Part VII: Projects

Detecting and Segmenting Shapes

Build both halves of dense prediction on synthetic images with exact labels: a tiny U-Net for semantic segmentation and a one-stage grid detector with NMS and mean average precision.

After this project you will be able to

  • Generate a data set with pixel-accurate masks and bounding boxes, including occlusion.
  • Train an encoder–decoder segmentation network and measure what skip connections contribute, with mIoU and boundary accuracy.
  • Encode boxes as grid-cell targets, train a YOLO-style detector with a multitask loss, and decode its predictions.
  • Evaluate detections with IoU, non-maximum suppression, and mean average precision at several IoU thresholds.

Code: projects/p11_detection/ (shapes_data.py, det_utils.py, unet_seg.py, grid_detector.py, tests/). Background: Chapters 21 and 17.

P11.1The Problem

Given an image containing a few objects, (1) label every pixel with its class (semantic segmentation) and (2) output a box and a class for every object (detection), the two tasks of Chapter 21. Real benchmarks such as COCO require GPUs and days of training; synthetic shapes keep every mechanism of the real systems while training in a few minutes on a CPU, and their labels are exact.

P11.2The Data

shapes_data.py draws $64\times64$ images with one to three objects of three classes (circle, square, triangle) of random size, position, and color on a smoothly shaded, noisy background. Later objects are drawn on top of earlier ones, so objects can be partly hidden; the generator removes covered pixels from each object’s mask, computes each box from the visible pixels, and drops objects with fewer than 12 visible pixels. Each image comes with a class map (0 for background) and a list of boxes. We train on 3,000 (segmentation) or 4,000 (detection) images and test on 500 images from a different random seed.

P11.3Building It from Scratch

Segmentation with a tiny U-Net. The encoder has three stages of two $3\times3$ convolutions with batch normalization and ReLU, with $2\times2$ max pooling between them ($64\to32\to16$ pixels; 16, 32, 64 channels). The decoder upsamples with $2\times2$ transposed convolutions of stride 2 (Section 21.3), concatenates the encoder features of the same resolution (the skip connections of Section 21.4), and applies two more convolutions per stage; a $1\times1$ convolution produces four class scores per pixel. The loss is the per-pixel cross-entropy. For comparison, an otherwise identical network drops the concatenations. Besides the per-class IoU and mIoU of Equation (21.1), we measure boundary accuracy: the pixel accuracy on the 6% of pixels whose $3\times3$ neighborhood contains two classes, which is where the two networks should differ.

A one-stage grid detector. A small CNN reduces the image to an $8\times8$ grid of cells of $8\times8$ pixels. Each cell predicts eight numbers: an objectness logit, the offsets $(t_x,t_y)$ of an object’s center within the cell (through a sigmoid), its width and height on a log scale relative to a 16-pixel anchor, $t_w=\log(w/16)$ and $t_h=\log(h/16)$, as in Equation (21.5), and three class scores. Each object is assigned to the cell containing its center; when two centers fall in the same cell, the larger object wins:

import numpy as np
import torch

G, CELL, ANCHOR = 8, 8, 16.0

def make_targets(boxes_list):
    """One target per grid cell; if two objects share a cell, the larger one wins."""
    T = np.zeros((len(boxes_list), G, G, 6), np.float32)   # obj, tx, ty, tw, th, class
    for b, boxes in enumerate(boxes_list):
        for cls, x1, y1, x2, y2 in sorted(boxes.tolist(), key=lambda r: (r[3] - r[1]) * (r[4] - r[2])):
            cx, cy, w, h = (x1 + x2) / 2, (y1 + y2) / 2, x2 - x1, y2 - y1
            j, i = min(int(cx // CELL), G - 1), min(int(cy // CELL), G - 1)
            T[b, i, j] = [1, cx / CELL - j, cy / CELL - i, np.log(w / ANCHOR), np.log(h / ANCHOR), cls - 1]
    return torch.tensor(T)

boxes = [np.array([[1, 10, 12, 30, 28.], [3, 40, 40, 60, 62.]])]   # (class, x1, y1, x2, y2)
T = make_targets(boxes)
print("cells with objects:", torch.nonzero(T[0, ..., 0]).tolist())

The loss is a weighted sum of three terms (Equation (21.6)): binary cross-entropy for objectness over all cells (with positives weighted 4 times, since only a few of the 64 cells contain an object), a smooth $\ell_1$ loss on the four box numbers, and cross-entropy on the class, both only for cells that contain an object. At test time, the score of a cell is its objectness probability times its most probable class probability; cells with a score above 0.05 become boxes, and class-wise NMS with an IoU threshold of 0.5 removes duplicates (Section 21.5).

Evaluation. det_utils.py contains the IoU, NMS, and average-precision functions of Section 21.6, plus mean_ap, which pools the detections of each class over all test images (shifting each image’s boxes into its own region of a large canvas so that boxes from different images can never match), computes AP per class, and averages over classes.

P11.4Using Libraries

torchvision provides Faster R-CNN, RetinaNet, and Mask R-CNN models pretrained on COCO, along with nms and box_iou operators, and pycocotools implements the official COCO evaluation. For real data, fine-tuning a pretrained detector is almost always better than training from scratch. We wrote everything ourselves here because the point is to see every piece; the tests check our NMS, IoU, and AP against hand-computed cases.

P11.5Results and Discussion

Running python unet_seg.py and python grid_detector.py (about 4 and 3 minutes on four CPU cores):

ModelCircleSquareTrianglemIoUBoundary acc.
U-Net (117,732 parameters)0.9800.9830.9820.98698.6%
No skips (106,212)0.9710.9800.9620.97896.8%

(mIoU averages over the four classes, including the background at 0.999–1.000.)

Skip connections pay off at the edges. Both networks segment the interiors of the shapes almost perfectly, so the overall mIoU differs by less than one point. The difference shows at the boundaries: without skips, 3.2% of boundary pixels are wrong, with skips 1.4%, less than half. The decoder without skips must reconstruct edges from $16\times16$ features, which cannot locate a boundary to better than a few pixels; the skip connections hand it the full-resolution features of the encoder. Triangles, with their sharp corners, suffer most without skips.

Detection. The grid detector (390,408 parameters) reaches an mAP@0.5 of 0.957 and a COCO-style mAP averaged over IoU thresholds from 0.5 to 0.95 of 0.732. The gap between the two numbers is the cost of imprecise localization: almost every object is found with a box that overlaps it by more than 50%, but fewer boxes overlap by more than 85 or 90%, and small or heavily occluded shapes are the hardest. NMS changes nothing here (mAP@0.5 is 0.957 with and without it) because each object is assigned to a single cell, and the neighboring cells learn low objectness. Detectors that place many anchors per location, and that assign each object to all anchors that overlap it, produce many duplicates, and for them NMS is essential.

P11.6Extensions

  1. Add an instance-segmentation head: for every detected box, predict a $16\times16$ mask from features cropped with bilinear sampling (as in RoIAlign), and measure mask AP.
  2. Assign each object to the $3\times3$ cells around its center, as many modern detectors do. How does mAP change, and how much does NMS matter now?
  3. Replace the focal-free objectness loss by the focal loss Equation (21.7) with $\gamma=2$ and compare on images with ten small objects each.
  4. Deepen the U-Net encoder to $8\times8$ at the bottleneck and repeat the skip-connection comparison.
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.