Chapter 23 · Part V: Computer Vision
Video, 3D Vision, and Vision–Language Models
The world is not a stack of independent photographs. It moves, it is three-dimensional, and people describe it in language. This chapter extends the networks of the previous chapters to video, to 3D shapes and scenes, and to models that see and talk.
After this chapter you will be able to
- Compare architectures for video: late and early fusion, 3D convolutions, two-stream networks, inflated and SlowFast networks, and video Transformers.
- Choose among 3D representations (depth maps, voxels, point clouds, meshes, implicit functions) and build a permutation-invariant point-cloud network.
- Derive the volume rendering equation behind neural radiance fields and explain 3D Gaussian splatting.
- Explain contrastive vision–language pretraining and zero-shot classification.
- Describe how generative vision–language models connect an image encoder to a language model, and their failure modes.
23.1Video Understanding
A video is a four-dimensional array: $T$ frames of $H\times W\times3$ pixels, at 24 to 60 frames per second. Classifying actions (“opening a door,” “playing violin”) requires combining appearance in each frame with motion across frames. Data sets such as Kinetics (Carreira and Zisserman 2017), with hundreds of thousands of ten-second clips in hundreds of action classes, play the role of ImageNet. The difficulty is size: a ten-second clip at 30 frames per second is 300 images.
Fusion in time. The simplest model runs an image CNN on each frame (or a few sampled frames) and averages the predictions: late fusion. It recognizes actions with distinctive scenes or objects but cannot see motion. Early fusion stacks frames along the channel axis of the first layer, so only that layer sees time.
3D convolutions extend the kernel to time: a $3\times3\times3$ kernel slides over $T\times H\times W$, so every layer combines information from neighboring frames and the temporal receptive field grows with depth, exactly as the spatial one does (Section 17.4). C3D (Tran et al. 2015) built a VGG-style network from such layers. The cost is higher: a 3D layer has $k$ times the parameters and computation of its 2D counterpart.
Two-stream networks (Simonyan and Zisserman 2014) make motion explicit by feeding a second CNN the optical flow, the per-pixel displacement between consecutive frames computed by a classical algorithm, alongside an appearance CNN on single frames. Motion turned out to be very informative: the flow stream alone outperformed the appearance stream. I3D (Carreira and Zisserman 2017) combined both ideas and solved the data problem by inflating an ImageNet-pretrained 2D network into 3D: each $k\times k$ kernel is repeated $k$ times along time and divided by $k$, so the inflated network gives the same output as the 2D one on a static video and starts from good features. SlowFast networks (Feichtenhofer et al. 2019) run a slow pathway at a low frame rate with many channels for appearance and a fast pathway at a high frame rate with few channels for motion, fused by lateral connections.
Video Transformers. A ViT (Section 19.9) extends to video by cutting the clip into spatio-temporal tubelets, for example $2\times16\times16$ pixels. A 16-frame $224\times224$ clip then has $8\times196=1568$ tokens, and full attention over all of them is expensive. Factorized designs attend over space within each frame and over time at each spatial position separately (Bertasius et al. 2021; Arnab et al. 2021). For long videos, current systems sample frames sparsely or compress each frame to a few tokens before passing them to a language model (Section 23.4).
23.2Representing the 3D World
There is no single natural format for 3D data, and the choice drives the architecture:
- Depth maps store the distance to the surface at each pixel of an image ($2.5$D). They come from depth sensors and stereo, and are processed with ordinary 2D CNNs, but describe only the visible surface from one viewpoint.
- Voxel grids are 3D occupancy images processed with 3D convolutions. They are simple but cost $O(n^3)$ memory, so resolution is limited to about $128^3$.
- Point clouds are unordered sets of 3D points, the raw output of lidar and depth sensors. They are compact and precise, but have no grid for convolutions.
- Meshes of vertices and triangular faces are the standard of graphics; neural networks on them operate on the vertex graph.
- Implicit functions represent a shape by a function $f:\R^3\to\R$, such as an occupancy probability or a signed distance to the surface, with the surface as a level set. Represented by a neural network, $f$ gives a continuous, resolution-free shape.
PointNet. A network on a point cloud must not depend on the order in which the points are listed. PointNet (Qi et al. 2017) achieves this with a symmetric function: apply the same MLP $h$ to every point independently, aggregate with a coordinate-wise maximum over points, and process the result with another MLP $\gamma$:
$$f(\{\vx_1,\dots,\vx_n\})=\gamma\Big(\max_{i=1,\dots,n}h(\vx_i)\Big). \tag{23.1}$$Any permutation of the points gives the same output, and the paper proves that such networks can approximate any continuous set function. Each coordinate of the max-pooled vector is effectively determined by a few “critical points,” which makes PointNet robust to missing points. Its successors add local neighborhoods, in the spirit of convolutions.
23.3Neural Radiance Fields
Given a few dozen photographs of a scene from known camera positions, can we render it from a new viewpoint? A neural radiance field (NeRF) (Mildenhall et al. 2020) represents the scene as a function, an MLP, that maps a 3D position $\vx$ and viewing direction $\vd$ to a volume density $\sigma(\vx)$ (how much light is absorbed or emitted there) and a color $\vc(\vx,\vd)$ (which may depend on the viewing direction, to model reflections). An image is rendered by volume rendering: for the ray $\vr(t)=\vo+t\vd$ through each pixel, the expected color is
$$C(\vr)=\int_{t_n}^{t_f}T(t)\,\sigma\big(\vr(t)\big)\,\vc\big(\vr(t),\vd\big)\dd t,\qquad T(t)=\exp\Big(-\int_{t_n}^t\sigma\big(\vr(s)\big)\dd s\Big), \tag{23.2}$$where the transmittance $T(t)$ is the probability that the ray travels from the camera to $t$ without being absorbed. Sampling $N$ points $t_i$ along the ray with spacings $\delta_i=t_{i+1}-t_i$ and treating density and color as constant on each segment gives
$$\hat C(\vr)=\sum_{i=1}^NT_i\big(1-e^{-\sigma_i\delta_i}\big)\vc_i,\qquad T_i=\exp\Big(-\sum_{j<i}\sigma_j\delta_j\Big)=\prod_{j<i}\big(1-\alpha_j\big), \tag{23.3}$$with $\alpha_i=1-e^{-\sigma_i\delta_i}$ the opacity of segment $i$. This is front-to-back alpha compositing, the same rule used to stack transparent layers in image editors, and it is differentiable in $\sigma_i$ and $\vc_i$. NeRF trains the MLP by gradient descent on the squared difference between rendered and photographed pixel colors, over random rays from all training images. Nothing tells the network where the surfaces are; consistency across many views forces the density to concentrate on them.
Positional encoding. MLPs on raw coordinates are biased toward smooth functions and render blurry images. NeRF first maps each coordinate $p$ through sinusoids of increasing frequency,
$$\gamma(p)=\big(\sin(2^0\pi p),\cos(2^0\pi p),\dots,\sin(2^{L-1}\pi p),\cos(2^{L-1}\pi p)\big), \tag{23.4}$$the same idea as the sinusoidal position encodings of Transformers (Section 19.6), which lets the network represent fine detail.
3D Gaussian splatting. NeRF renders slowly, because every pixel needs hundreds of MLP evaluations. 3D Gaussian splatting (Kerbl et al. 2023) replaces the MLP by millions of explicit 3D Gaussians, each with a position, covariance (an oriented ellipsoid), opacity, and view-dependent color. To render, the Gaussians are projected onto the image as 2D ellipses (splats), sorted by depth, and alpha-composited with Equation (23.3). The rasterization runs in real time on a GPU, and the Gaussians’ parameters are optimized by gradient descent against the photographs, with heuristics that split, clone, and prune Gaussians during training. Splatting has largely replaced NeRF for reconstructing scenes, while the volume rendering principle remains the same.
23.4Vision–Language Models
Language gives vision models a far richer target than a fixed set of class labels, and gives language models eyes. Classic vision–language tasks include image captioning (Section 18.8), visual question answering (VQA) (Antol et al. 2015), which answers free-form questions about an image, visual grounding (finding the region a phrase describes), and text–image retrieval.
Contrastive pretraining. CLIP (Radford et al. 2021) trains an image encoder and a text encoder on hundreds of millions of image–caption pairs from the web so that matching pairs have similar embeddings and mismatched pairs dissimilar ones (the loss is described in Chapter 26). The result is a shared embedding space for images and text. It enables zero-shot classification: to classify an image into $K$ classes never seen as labels, embed the prompts “a photo of a {class}” and pick the class whose text embedding is most similar to the image embedding,
$$\hat y=\argmax_{k}\ \frac{f_{\text{img}}(\vx)\T f_{\text{text}}(\text{prompt}_k)}{\norm{f_{\text{img}}(\vx)}\,\norm{f_{\text{text}}(\text{prompt}_k)}} . \tag{23.5}$$CLIP’s zero-shot accuracy on ImageNet matched a ResNet-50 trained on ImageNet’s labels, and it was far more robust to distribution shift. CLIP-style encoders became the vision backbone of generative vision–language models and the text conditioning of image generators (Chapter 25).
Generative vision–language models. To answer questions and hold conversations about images, connect a pretrained vision encoder to a pretrained large language model (Chapter 24). Three designs illustrate the options:
- Flamingo (Alayrac et al. 2022) keeps both pretrained models frozen and inserts new gated cross-attention layers into the language model, through which text tokens attend to visual tokens. Trained on web pages with interleaved images and text, it learned new tasks from a few examples in the prompt.
- BLIP-2 (Li et al. 2023) trains a small Transformer with a fixed number of learned query vectors that cross-attend to the frozen image features and produce a few visual tokens for the frozen language model.
- LLaVA (Liu et al. 2023) simply maps the patch features of a CLIP ViT through a linear layer (later a small MLP) into the language model’s token-embedding space and prepends them to the text, then fine-tunes on visual instruction-following data: conversations about images generated with the help of a text-only language model from captions and boxes.
The simplest design won: most current open models follow LLaVA’s recipe of a ViT encoder, a projector, and a language model, trained first on captions to align the projector and then on instruction data, often with high-resolution images tiled into several crops. Frontier systems are trained to be natively multimodal, processing interleaved text, images, and often audio and video in one model.
Failure modes. Vision–language models hallucinate, describing objects that are not in the image, especially when the language model’s prior makes them likely (a fork next to a plate). They are often weak at counting, precise spatial relations, reading small text at low resolution, and fine-grained recognition, and their answers can change with the phrasing of the question. Evaluate them on held-out data designed to probe these weaknesses, not only on captioning benchmarks. Vision–language–action models, which add robot actions as outputs, are discussed in Chapter 27.
23.5Implementation
Volume rendering, Equation (23.3), of an analytic scene (two opaque balls) in NumPy; a NeRF would replace the function scene by an MLP and fit it to photographs:
import numpy as np
def scene(p):
"""Density and color of a toy scene: a red ball in front of a blue ball."""
d1 = np.linalg.norm(p - np.array([-0.3, 0.0, 2.6]), axis=-1)
d2 = np.linalg.norm(p - np.array([0.4, 0.1, 3.4]), axis=-1)
s1, s2 = 30.0 * (d1 < 0.5), 30.0 * (d2 < 0.6) # opaque inside each ball
sigma = s1 + s2
color = (s1[..., None] * [0.9, 0.2, 0.1] + s2[..., None] * [0.1, 0.3, 0.9]) / np.maximum(sigma, 1e-9)[..., None]
return sigma, color
def render(origins, dirs, near=1.0, far=5.0, n=128):
"""Discretized volume rendering (Eq. 23.3) along every ray."""
t = np.linspace(near, far, n)
delta = np.append(np.diff(t), 1e10)
pts = origins[:, None, :] + t[None, :, None] * dirs[:, None, :]
sigma, color = scene(pts)
alpha = 1 - np.exp(-sigma * delta) # opacity of each segment
T = np.cumprod(np.concatenate([np.ones((len(alpha), 1)), 1 - alpha[:, :-1] + 1e-10], 1), 1)
w = T * alpha # contribution of each sample
return (w[..., None] * color).sum(1), (w * t).sum(1), w.sum(1) # color, depth, opacity
H = W = 32
ys, xs = np.meshgrid(np.linspace(-0.5, 0.5, H), np.linspace(-0.5, 0.5, W), indexing="ij")
dirs = np.stack([xs, -ys, np.ones_like(xs)], -1).reshape(-1, 3)
dirs /= np.linalg.norm(dirs, axis=1, keepdims=True)
rgb, depth, acc = render(np.zeros_like(dirs), dirs)
img = rgb.reshape(H, W, 3)
print("image", img.shape, "| center pixel color", np.round(img[H // 2, W // 2], 2),
"| depth there", round(depth.reshape(H, W)[H // 2, W // 2], 2))
print("fraction of rays that hit something:", round(float((acc > 0.5).mean()), 2))
A PointNet in PyTorch that classifies point clouds sampled from spheres, cubes, and cylinders, with a check that shuffling the points does not change its output:
import torch, torch.nn as nn, torch.nn.functional as F
def sample_shape(kind, n, gen):
"""Random point clouds on the surface of a sphere, a cube, or a cylinder, randomly scaled."""
if kind == 0: # sphere
p = torch.randn(n, 3, generator=gen); p = p / p.norm(dim=1, keepdim=True)
elif kind == 1: # cube: push points to a face
p = torch.rand(n, 3, generator=gen) * 2 - 1
face = torch.randint(0, 3, (n,), generator=gen)
p[torch.arange(n), face] = torch.sign(torch.randn(n, generator=gen))
else: # cylinder side
a = torch.rand(n, generator=gen) * 6.2832
p = torch.stack([a.cos(), a.sin(), torch.rand(n, generator=gen) * 2 - 1], 1)
return p * (0.5 + torch.rand(1, generator=gen))
class PointNet(nn.Module):
def __init__(self, k=3):
super().__init__()
self.mlp = nn.Sequential(nn.Linear(3, 64), nn.ReLU(), nn.Linear(64, 128), nn.ReLU()) # shared per point
self.head = nn.Sequential(nn.Linear(128, 64), nn.ReLU(), nn.Linear(64, k))
def forward(self, pts): # pts: (batch, n_points, 3)
return self.head(self.mlp(pts).max(dim=1).values) # symmetric max-pool (Eq. 23.1)
gen = torch.Generator().manual_seed(0); torch.manual_seed(0)
def batch(B, n=256):
y = torch.randint(0, 3, (B,), generator=gen)
return torch.stack([sample_shape(int(k), n, gen) for k in y]), y
model = PointNet(); opt = torch.optim.Adam(model.parameters(), lr=1e-3)
for step in range(300):
x, y = batch(32)
loss = F.cross_entropy(model(x), y)
opt.zero_grad(); loss.backward(); opt.step()
x, y = batch(300)
with torch.no_grad():
acc = (model(x).argmax(1) == y).float().mean().item()
perm = torch.randperm(x.shape[1])
same = torch.allclose(model(x), model(x[:, perm]), atol=1e-5)
print(f"test accuracy {acc:.2f}; output unchanged when the points are shuffled: {same}")
23.6Common Pitfalls
Pitfall: Treating a video as a bag of frames
Late fusion of frame-level predictions works suspiciously well on many action benchmarks because the scene gives the action away (a swimming pool means swimming). Test on actions that require motion, or with shuffled frames, before claiming that a model understands time.
Pitfall: Wrong camera conventions
NeRF and splatting need camera poses, and different tools use different axis conventions (which axis points forward, left- or right-handed coordinates) and different intrinsic parameterizations. A flipped axis produces training that never converges; render the camera frusta and check them visually.
Pitfall: Trusting fluent answers
A vision–language model’s answer is fluent whether or not it is grounded in the image. For any application where errors matter, verify answers against the image (for example by asking for bounding boxes) and measure hallucination rates.
23.7Summary
- Video models combine appearance and motion: 3D convolutions, two-stream networks with optical flow, inflated pretrained 2D networks, SlowFast pathways, and factorized space–time attention.
- 3D data come as depth maps, voxels, point clouds, meshes, or implicit functions; PointNet handles unordered points with a shared MLP and max pooling.
- NeRF represents a scene as an MLP of density and color and renders it by differentiable volume rendering, $\hat C=\sum_iT_i\alpha_i\vc_i$; Gaussian splatting uses explicit Gaussians and the same compositing for real-time rendering.
- CLIP aligns images and text by contrastive learning and enables zero-shot classification; generative VLMs connect an image encoder to a language model through cross-attention, learned queries, or a simple projector.
- VLMs hallucinate and struggle with counting, spatial relations, and small text.
23.8Exercises
- MathA 2D convolution has $C_{\text{out}}(C_{\text{in}}k^2+1)$ parameters. How many does its $k\times k\times k$ 3D counterpart have? Show that I3D’s inflation (repeat the kernel $k$ times in time and divide by $k$) produces the same output as the 2D network on a video of identical frames.
- MathShow that Equation (23.1) is invariant to permutations of the points. Would the mean or the sum work instead of the maximum? What would change?
- MathDerive the discrete rendering formula Equation (23.3) from Equation (23.2) assuming piecewise-constant density and color, and show that the weights $w_i=T_i\alpha_i$ sum to at most 1. When do they sum to exactly 1?
- MathIn zero-shot classification with CLIP, why does averaging the text embeddings of several prompts per class (“a photo of a dog,” “a drawing of a dog,” …) usually improve accuracy?
- CodingReplace the analytic scene in the rendering code by a small MLP with positional encoding Equation (23.4), render 20 views of the analytic scene from cameras on a circle, and train the MLP to reproduce them. Compare novel views with and without positional encoding.
- CodingTrain the PointNet with mean pooling instead of max pooling, and with 10% of the points of each test cloud deleted. Which aggregation is more robust to missing points?