Machine Learning from First Principles

Chapter 26 · Part VI: Frontier Techniques

Self-Supervised Learning

Labels are expensive; raw data are abundant. Self-supervised learning manufactures supervision from the data itself and produces the general-purpose representations behind foundation models for text, images, and both together.

After this chapter you will be able to

  • Explain how pretext tasks create supervision from unlabeled data, and how learned representations are evaluated.
  • Derive the InfoNCE contrastive loss and explain the roles of augmentations, temperature, and negatives.
  • Explain how MoCo, BYOL, SimSiam, and DINO avoid representation collapse with or without negatives.
  • Describe masked image modeling (MAE) and predictive methods in representation space.
  • Explain how CLIP aligns images and text with a symmetric contrastive loss.

26.1Learning Without Labels

Labeled data are expensive; unlabeled data are abundant. Self-supervised learning (SSL) defines a pretext task whose labels come for free from the input, trains a network to solve it, and keeps the network’s internal representations for downstream tasks. Three families dominate:

  • Next-token prediction. Predict each token from the previous ones. This is how language models are pretrained (Chapter 24).
  • Masked modeling. Hide part of the input and predict it from the rest: random words of a sentence (BERT; (Devlin et al. 2019)) or random patches of an image (Section 26.5).
  • Joint embedding. Learn embeddings in which different views of the same example are close, with some mechanism that prevents all examples from collapsing to the same point (Sections 26.2 and 26.4).

In each case the pretext task itself is unimportant; what matters is that solving it well requires features that are useful for many other tasks.

Early pretext tasks. The first successful image pretext tasks were ingenious puzzles: predict which of four rotations was applied to an image (Gidaris et al. 2018), reassemble shuffled tiles of an image (Noroozi and Favaro 2016), or color a grayscale photograph (Zhang et al. 2016). Each requires recognizing objects (you cannot tell which way is up without knowing what the object is), but each also rewards shortcuts unrelated to semantics, and they were superseded by the methods below.

Evaluating representations. A self-supervised encoder is judged by how useful its features are for tasks it never saw. A linear probe freezes the encoder and trains only a linear classifier on its features with labeled data; $k$-NN evaluation classifies by nearest neighbors in feature space; fine-tuning trains the whole network on the downstream task starting from the self-supervised weights; and few-shot evaluation uses only a handful of labels per class. Good representations do well on all four and transfer across tasks such as classification, detection, and segmentation.

26.2Contrastive Learning

Take a batch of $N$ images. Create two random augmented views of each (crops, color changes, blur), run all $2N$ views through an encoder network, and normalize the outputs to unit length: $\vz_i$ and $\vz_i'$ are the embeddings of the two views of image $i$. The two views of the same image form a positive pair; all other combinations are negatives. The InfoNCE loss (van den Oord et al. 2018) treats this as an $N$-way classification problem: given $\vz_i$, pick out its partner $\vz_i'$ among all $N$ candidates, using similarity divided by a temperature $\tau$ as the logit:

$$\ell_i = -\log\frac{\exp(\vz_i\T\vz_i'/\tau)}{\sum_{j=1}^{N}\exp(\vz_i\T\vz_j'/\tau)} . \tag{26.1}$$

This is exactly the softmax cross-entropy of Section 6.4, with the “classes” being the other images in the batch. Minimizing it pulls positive pairs together and pushes all other pairs apart. InfoNCE also has an information-theoretic reading: $\log N-\ell$ is a lower bound on the mutual information between the two views, so more negatives allow a tighter bound.

Augmentations define invariances. The encoder learns to ignore whatever the augmentations change: if color jitter maps a red car and a blue car to a positive pair, the embedding learns to ignore color. SimCLR (Chen et al. 2020) found that strong random crops plus color distortion were essential, that a small projection head between the encoder and the loss (discarded after training) improved the encoder’s features, and that large batches (many negatives) helped. With these ingredients, linear probes on its features came close to supervised training on ImageNet.

Why negatives? Without them, the trivial solution maps every input to the same vector, which makes all positive pairs perfectly similar: representation collapse. Negatives make collapse costly. The danger of collapse is the central design issue of all joint-embedding methods.

26.3Momentum Contrast

SimCLR’s negatives come from the current batch, so it needs batches of thousands of images. MoCo (He et al. 2020) decouples the number of negatives from the batch size. It keeps a queue of embeddings from recent batches (65,536 in the paper) and uses them as negatives. Those embeddings were computed by an older version of the network, and if that network changed quickly they would be inconsistent with the current ones. So MoCo computes the keys with a separate momentum encoder whose weights $\vtheta_k$ are an exponential moving average of the query encoder’s weights $\vtheta_q$:

$$\vtheta_k\gets m\,\vtheta_k+(1-m)\,\vtheta_q,\qquad m=0.999 . \tag{26.2}$$

Only the query encoder is trained by backpropagation. The slowly evolving key encoder keeps the queue consistent, a trick that reappears in BYOL and DINO.

26.4Learning Without Negatives

Can collapse be avoided without negatives? Surprisingly, yes.

BYOL (Grill et al. 2020) has two networks. The online network (encoder, projector, and an extra predictor) is trained to predict the target network’s projection of a different view of the same image; the target network is an exponential moving average of the online network, as in Equation (26.2). With normalized vectors $\bar q$ (online prediction) and $\bar z'$ (target projection), the loss is

$$\ell=\big\Vert\bar q-\bar z'\big\Vert_2^2=2-2\,\bar q\T\bar z' , \tag{26.3}$$

symmetrized over the two views, with no gradient flowing into the target. The collapsed solution is a fixed point of this loss, yet training does not find it. Chen and He (2021) showed with SimSiam that the essential ingredients are the predictor and the stop-gradient on the target branch (even without the moving average): together they make the dynamics avoid collapse, although a complete theory is still debated. Other methods prevent collapse explicitly: VICReg (Bardes et al. 2022) adds penalties that keep the variance of every embedding dimension above a threshold and decorrelate the dimensions.

DINO (Caron et al. 2021) casts self-supervision as self-distillation. A student and an EMA teacher, both Vision Transformers, map views of an image to probability distributions over $K$ prototype dimensions with softmaxes at temperatures $\tau_s$ and $\tau_t$. The student is trained to match the teacher across views:

$$\ell=-\sum_{k=1}^KP_t(\vx')_k\log P_s(\vx)_k,\qquad P_t(\vx')=\softmax\big((g_t(\vx')-\vc)/\tau_t\big), \tag{26.4}$$

where the teacher’s outputs are centered by a running mean $\vc$ (which prevents one dimension from dominating) and sharpened by a low temperature $\tau_t$ (which prevents a uniform output); the two effects balance each other. With multi-crop training, the teacher sees only large global crops and the student also small local crops, so the student must infer the whole from a part. A striking finding was that the attention maps of DINO’s ViT segment objects without ever seeing a segmentation label. DINOv2 (Oquab et al. 2024) scaled the approach with a curated data set of 142 million images and produced general-purpose features that work with frozen weights for classification, segmentation, and depth estimation, and that serve as the visual backbone of many systems, including robot policies (Section 27.8).

26.5Masked Image Modeling

BERT’s masked word prediction transferred to images once Vision Transformers made images into sequences of patches. The masked autoencoder (MAE) (He et al. 2022) masks a random 75% of the patches, runs a ViT encoder on the remaining 25% only, and uses a lightweight decoder to reconstruct the pixels of the masked patches from the encoded visible ones plus mask tokens. The loss is the mean squared error on the masked patches (with pixel values normalized within each patch):

$$\ell=\frac{1}{|\mathcal M|}\sum_{p\in\mathcal M}\big\Vert\hat\vx_p-\vx_p\big\Vert^2 . \tag{26.5}$$

The high masking ratio matters: images are spatially redundant, so masking a few patches can be solved by interpolating neighbors, while masking three quarters forces the network to reason about objects and scenes. Because the encoder processes only a quarter of the patches, pretraining is several times cheaper than processing all of them. MAE features need fine-tuning to shine (their linear-probe accuracy is lower than contrastive methods’), and they scale well with model size. Related approaches predict discrete visual tokens instead of pixels (Bao et al. 2022), or predict the representations of masked regions computed by an EMA target encoder rather than their pixels (Assran et al. 2023), which avoids spending capacity on unpredictable pixel detail.

26.6Connecting Images and Text: CLIP

CLIP (Radford et al. 2021) applies contrastive learning across modalities. It trains an image encoder $f$ and a text encoder $g$ on hundreds of millions of (image, caption) pairs collected from the web. In a batch of $N$ pairs, compute the $N\times N$ matrix of cosine similarities $S_{ij}=\bar f(\vx_i)\T\bar g(t_j)/\tau$ between every normalized image embedding and every normalized text embedding. The diagonal holds the true pairs. The loss is InfoNCE in both directions, each image picking its caption among the $N$ texts and each caption picking its image:

$$\Loss=\frac{1}{2N}\sum_{i=1}^N\Big[-\log\frac{e^{S_{ii}}}{\sum_je^{S_{ij}}}-\log\frac{e^{S_{ii}}}{\sum_je^{S_{ji}}}\Big], \tag{26.6}$$

with the temperature $\tau$ learned. After training, images and texts live in a shared embedding space, which enables zero-shot classification (Section 23.4) and text–image retrieval; CLIP-style encoders also condition text-to-image generators (Section 25.8) and feed vision–language models. Variants replace the softmax by a pairwise sigmoid loss, which removes the need for a global normalization over the batch.

26.7Implementation

Contrastive learning on the $8\times8$ digits with a small MLP encoder. The views differ by random one-pixel shifts, erased pixels, and noise; no labels are used during training. Afterward, linear probes compare the learned features with raw pixels and with a randomly initialized encoder, on the clean test digits and on test digits shifted by up to one pixel:

import torch, torch.nn as nn, torch.nn.functional as F
from sklearn.datasets import load_digits
from sklearn.linear_model import LogisticRegression

torch.manual_seed(0)
X, y = load_digits(return_X_y=True)
X = torch.tensor(X / 16.0, dtype=torch.float32).view(-1, 1, 8, 8)
Xtr, ytr, Xte, yte = X[:1400], y[:1400], X[1400:], y[1400:]

def augment(x):
    """Random views of a batch of 8x8 digits: shift by up to 1 pixel, drop pixels, add noise."""
    sx, sy = torch.randint(-1, 2, (2,)).tolist()
    x = torch.roll(x, shifts=(sx, sy), dims=(2, 3))
    x = x * (torch.rand_like(x) > 0.2)                     # randomly erase 20% of the pixels
    return (x + 0.1 * torch.randn_like(x)).clamp(0, 1)

encoder = nn.Sequential(nn.Flatten(), nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 64))
head = nn.Sequential(nn.ReLU(), nn.Linear(64, 32))          # projection head, discarded after training

def info_nce(z1, z2, tau=0.1):
    """Eq. (26.1), symmetrized: each view must pick out its partner among the 2N - 1 others."""
    z = F.normalize(torch.cat([z1, z2]), dim=1)
    sim = z @ z.T / tau
    sim.fill_diagonal_(float("-inf"))                      # a view is not its own positive
    n = len(z1)
    targets = torch.cat([torch.arange(n, 2 * n), torch.arange(0, n)])
    return F.cross_entropy(sim, targets)

opt = torch.optim.Adam(list(encoder.parameters()) + list(head.parameters()), lr=1e-3)
random_encoder = nn.Sequential(nn.Flatten(), nn.Linear(64, 256), nn.ReLU(), nn.Linear(256, 64))
for step in range(3000):
    xb = Xtr[torch.randint(len(Xtr), (256,))]               # labels are never used
    loss = info_nce(head(encoder(augment(xb))), head(encoder(augment(xb))))
    opt.zero_grad(); loss.backward(); opt.step()

g = torch.Generator().manual_seed(5)                        # test digits shifted by up to 1 pixel
shifts = torch.randint(-1, 2, (len(Xte), 2), generator=g).tolist()
Xte_shift = torch.stack([torch.roll(x, shifts=tuple(s), dims=(1, 2)) for x, s in zip(Xte, shifts)])

def probe(features):
    """Linear probe: logistic regression on frozen features; accuracy on clean and shifted test digits."""
    with torch.no_grad():
        ftr, fte, fsh = (features(a).reshape(len(a), -1).numpy() for a in (Xtr, Xte, Xte_shift))
    clf = LogisticRegression(max_iter=3000).fit(ftr, ytr)
    return clf.score(fte, yte), clf.score(fsh, yte)

print(f"final contrastive loss {loss.item():.2f} (chance: log 511 = 6.24)")
for name, feat in [("raw pixels", lambda a: a), ("random encoder", random_encoder), ("contrastive encoder", encoder)]:
    clean, shifted = probe(feat)
    print(f"{name:20s} clean test {clean:.3f}   shifted test {shifted:.3f}")

The contrastive loss falls far below chance, and the learned features beat a random encoder. On this small, clean data set, raw pixels are already almost linearly separable, so the contrastive features do not beat them on the clean test set (89% against 91%). They do something more interesting: because the views differed by shifts, the encoder learned to ignore shifts, and on shifted test digits its probe keeps 76% accuracy while the raw-pixel probe drops to 43%. The augmentations chose the invariance. On natural images, where raw pixels are a poor representation, the same recipe with a convolutional or ViT encoder produces features that rival supervised pretraining.

26.8Common Pitfalls

Pitfall: Shortcut solutions

A pretext task can often be solved by low-level cues: chromatic aberration reveals a patch’s position, compression artifacts reveal rotation, and two crops of the same image share a color histogram. The network then learns the shortcut instead of semantics. Strong augmentations (color distortion for contrastive learning) exist largely to remove such shortcuts.

Pitfall: Silent collapse

A collapsed or partially collapsed representation can still have a decreasing training loss. Monitor the standard deviation of the normalized embeddings across a batch and the rank of their covariance; if they shrink toward zero, the representation is collapsing.

26.9Summary

  • Self-supervised learning creates labels from data: next-token prediction, masked modeling, and joint embeddings. Representations are evaluated by linear probes, $k$-NN, fine-tuning, and few-shot transfer.
  • InfoNCE is softmax cross-entropy over a batch: pick the positive partner among $N$ candidates. Augmentations define what the representation ignores; negatives prevent collapse.
  • MoCo keeps a queue of negatives encoded by a momentum (EMA) encoder; BYOL and SimSiam avoid negatives with a predictor and stop-gradient; DINO self-distills with centering and sharpening.
  • MAE masks 75% of image patches and reconstructs them; predicting representations instead of pixels is a promising variant.
  • CLIP aligns images and captions with a symmetric InfoNCE loss and enables zero-shot classification and retrieval.

26.10Exercises

  1. MathShow that the InfoNCE loss Equation (26.1) is bounded below by 0 and that its gradient with respect to the logits is (softmax probabilities) minus (one-hot positive), as in Equation (6.7). What is its value when all embeddings are identical?
  2. MathShow that for unit vectors $\bar q$ and $\bar z$, $\norm{\bar q-\bar z}^2=2-2\bar q\T\bar z$, so the BYOL loss Equation (26.3) is a cosine-similarity loss.
  3. MathIn DINO, explain why centering alone leads to a uniform teacher output and sharpening alone to a one-hot output that ignores the input, and why their combination can avoid both.
  4. MathWith a momentum $m=0.999$, roughly how many steps back does the momentum encoder Equation (26.2) “remember”? (Hint: an exponential moving average with factor $m$ has an effective window of about $1/(1-m)$.)
  5. CodingIn the implementation, remove the shift augmentation (keep erasing and noise) and repeat the shifted-test evaluation. Then add a horizontal-flip augmentation. Explain the results.
  6. CodingReplace InfoNCE in the implementation by a SimSiam-style loss (a predictor MLP on one branch, stop-gradient on the other, negative cosine similarity). Track the per-dimension standard deviation of the normalized embeddings during training with and without the stop-gradient.
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.