Project 10 · Part VII: Projects
Captioning Synthetic Scenes
Generate images of colored shapes with captions, then build two captioners from scratch: a CNN encoder with an LSTM decoder, and one with a Transformer decoder that attends to image regions. The comparison shows why spatial attention matters, and why caption metrics need care.
After this project you will be able to
- Build a data set whose captions are correct by construction, and encode captions as token sequences.
- Train a CNN–LSTM captioner with teacher forcing, and a CNN–Transformer captioner with cross-attention to image regions.
- Decode with greedy search and beam search.
- Evaluate captions with exact match and with a semantic metric that accepts paraphrases, and explain the difference.
Code: projects/p10_captioning/ (scenes.py, captioner.py, tests/). Background: Chapters 18, 19, and 17.
P10.1The Problem
Given an image, produce a sentence that describes it (Section 18.8). Real captioning data sets contain hundreds of thousands of photographs, each with five human-written captions, and models train for days. To see the mechanics in minutes on a CPU, we generate our own scenes: one or two colored shapes on a noisy background, with captions such as “a green triangle” or “a red circle left of a blue square.” The task keeps the essential difficulties of captioning: the model must recognize objects and colors, bind each color to the right shape, express spatial relations, and produce words in a grammatical order.
P10.2The Data
scenes.py draws $32\times32$ images with one object (half of the scenes) or two objects in one of four relations (left of, right of, above, below), with four colors and three shapes, random sizes, and random positions consistent with the relation. The vocabulary has 18 tokens, including the special tokens <start>, <end>, and <pad>; captions are padded to 10 tokens. We generate 8,000 training and 1,000 test scenes from different random seeds.
Two-object captions have an inherent ambiguity: “a red circle left of a blue square” and “a blue square right of a red circle” describe the same image, and the generator picks one of them at random. A captioner cannot know which phrasing the reference uses, so exact string match is the wrong metric. The function canonical maps each caption to a phrasing-independent description (objects and a normalized relation), and we report semantic match: the fraction of test images whose generated caption has the same canonical description as the reference.
def canonical(caption):
"""'A right of B' and 'B left of A' (and 'below'/'above') map to the same tuple."""
w = caption.split()
if len(w) == 3 and w[0] == "a":
return ((w[1], w[2]),)
if len(w) in (7, 8) and w[0] == "a": # 'above'/'below' captions have 7 words, 'left of'/'right of' 8
rel = " ".join(w[3:-3])
o1, o2 = (w[1], w[2]), (w[-2], w[-1])
if rel == "right of":
return (o2, "left of", o1)
if rel == "below":
return (o2, "above", o1)
if rel in ("left of", "above"):
return (o1, rel, o2)
return None
print(canonical("a red circle right of a blue square") == canonical("a blue square left of a red circle"))
P10.3Building It from Scratch
The encoder is a small CNN: three blocks of convolution, batch normalization, ReLU, and $2\times2$ max pooling reduce the $32\times32$ image to a $4\times4$ grid of 128-dimensional feature vectors, one per image region.
The LSTM captioner follows Show and Tell (Vinyals et al. 2015): the grid features are averaged into one vector, which linear layers map to the LSTM’s initial hidden and cell states. The decoder embeds the previous token, runs one LSTM step, and predicts the next token with a linear layer and softmax. Training uses teacher forcing (Section 18.6): the inputs are the reference tokens <start>, $w_1,\dots,w_{T-1}$, the targets are $w_1,\dots,w_T$, <end>, and padding is ignored in the cross-entropy.
The Transformer captioner keeps the $4\times4$ grid: the 16 region features plus learned position embeddings form the memory of a two-layer Transformer decoder (Chapter 19), whose layers apply causal self-attention over the caption so far and cross-attention from every word position to all 16 image regions. Both models have about half a million parameters, and both are trained for 15 epochs with AdamW, a one-cycle learning-rate schedule, and gradient clipping.
Decoding. Greedy decoding feeds back the most probable token at each step. Beam search (Algorithm 18.1) keeps the three best partial captions and ranks finished ones by their log-probability divided by their length to the power $0.7$. Both use a generic step(state, token) method, so the same search code serves both decoders.
P10.4Using Libraries
The models use PyTorch’s nn.LSTM and nn.TransformerDecoder (with pre-normalization, norm_first=True) rather than hand-written recurrences and attention; Chapter 18 and Project 6 implement those from scratch. For real captioning, pretrained vision–language models (Section 23.4) generate captions directly, and libraries such as pycocoevalcap compute the standard BLEU and CIDEr scores against several human references.
P10.5Results and Discussion
Running python captioner.py trains both models in about four minutes on four CPU cores:
| Decoder | Exact | Semantic | Semantic, beam 3$^*$ | Parameters |
|---|---|---|---|---|
| LSTM (pooled image vector) | 65.1% | 79.1% | 77.0% (greedy 76.3%) | 494,608 |
| Transformer (attention to 16 regions) | 74.7% | 99.8% | 100% (greedy 100%) | 498,704 |
$^*$On the first 300 test images, with greedy decoding on the same images in parentheses.
Spatial attention is the difference. Both models describe single objects perfectly. On two-object scenes the LSTM often binds colors to the wrong shapes or reverses the relation, for example “a red square below a yellow square” for “a yellow square below a red square.” Its decoder sees only the average of the region features, and averaging over positions discards exactly the information needed to tell which color belongs to which shape and which object is on top. The Transformer decoder can attend to the region containing the object it is describing at each word, which is what Show, Attend and Tell introduced for LSTMs (Xu et al. 2015), and it gets virtually every scene right with the same number of parameters.
Metrics matter. The Transformer’s exact-match score is only 74.7% although its captions are almost always correct. Half the test scenes have two objects, and for those the model and the reference each pick one of two equally valid phrasings, so exact match is capped near $50\%+50\%\times\frac12=75\%$, precisely what we observe. Real captioning metrics face the same problem in a far worse form, which is why they compare against several human references and why they correlate only loosely with human judgments (Section 18.8).
Beam search helps the LSTM slightly (from 76.3% to 77.0% on the first 300 test images) and changes nothing for the Transformer, which is already confident. With a tiny vocabulary and short captions, the greedy choice is rarely a trap; beam search pays off for longer outputs such as translations.
P10.6Extensions
- Add Show-Attend-and-Tell-style attention to the LSTM: at each step, compute attention weights over the 16 regions from the hidden state and feed the weighted feature as an extra input. How close does it get to the Transformer?
- Hold out one combination, for example every “yellow triangle,” from training, and test whether the models can describe it. This probes compositional generalization.
- Add a third object and relations such as “between,” and measure how accuracy degrades with scene complexity.
- Visualize the cross-attention weights of the Transformer for each generated word, as heat maps over the $4\times4$ grid.