Machine Learning from First Principles

Chapter 25 · Part VI: Frontier Techniques

Generative Models

A generative model learns the distribution of its data well enough to produce new examples. This chapter follows the main families, autoregressive models, variational autoencoders, GANs, diffusion, and flow matching, to the systems that generate today’s images, video, and audio.

After this chapter you will be able to

  • Explain what a generative model learns and how such models are evaluated.
  • Derive the evidence lower bound (ELBO) of a variational autoencoder and the reparameterization trick.
  • Model images autoregressively with PixelRNN and PixelCNN, and explain the role of masked convolutions.
  • State the GAN objective and the optimal discriminator, and describe DCGAN, conditional GANs, and StyleGAN.
  • Derive the closed-form forward process of a diffusion model and its simplified noise-prediction loss.
  • Train and sample a flow-matching model, and explain why straight paths need fewer sampling steps.
  • Explain classifier-free guidance, latent diffusion, and the architecture of modern text-to-image systems.

25.1What Is a Generative Model?

A classifier learns $p(y\mid\vx)$. A generative model learns the distribution of the data itself, $p(\vx)$, well enough to draw new samples that look like the training data: new faces, new molecules, new sentences, new audio. Large language models (Chapter 24) are generative models of text that factor $p(\vx)$ autoregressively, one token at a time. The same approach works for images (Section 25.2), but it is slow, because images have no natural order and contain hundreds of thousands of values. This chapter covers the families of models developed for such data, culminating in the diffusion and flow-matching models behind today’s image, video, and audio generators.

All of them share one idea: start from a simple distribution that is easy to sample, such as a standard Gaussian, and learn a transformation that turns its samples into data. They differ in what the transformation is and how it is trained:

FamilyHow noise becomes data
Autoregressiveno noise: one value or token at a time, each conditioned on the previous ones
VAEone decoder pass; trained by maximizing a lower bound on likelihood
GANone generator pass; trained against a discriminator that tries to tell real from fake
Diffusionmany small denoising steps; trained to predict the noise added to data
Flow matchingintegrate a learned velocity field; trained to regress straight-line velocities

Evaluation. Measuring a generative model is hard. Likelihood-based models report the average negative log-likelihood of test data (in bits per dimension for images). Sample quality is often measured by comparing statistics of features of generated and real samples, extracted by a pretrained network (the Fr'echet Inception Distance, FID), and by human preference studies. No single number captures quality, diversity, and faithfulness to a prompt at once.

25.2Autoregressive Models of Images

Language models generate text one token at a time, and the same can be done for images. Order the pixels in raster order (row by row, and red, green, blue within a pixel) and apply the chain rule of probability:

$$p(\vx)=\prod_{i=1}^{n}p(x_i\mid x_1,\dots,x_{i-1}). \tag{25.1}$$

Each factor is a 256-way softmax over the possible intensities, predicted by a network that sees all previous pixels. The model is trained by maximum likelihood with teacher forcing (Section 18.6), and its log-likelihood can be evaluated exactly, which makes autoregressive models the best density estimators measured in bits per dimension.

PixelRNN (van den Oord et al. 2016) computed the conditionals with LSTMs running along rows and diagonals of the image. PixelCNN replaced the recurrence by masked convolutions: the kernel weights for positions below the current pixel, and to its right in the same row, are fixed at zero, so every output depends only on pixels that come earlier in the ordering, and stacking such layers preserves this property. Training is then fully parallel, since all conditionals are computed in one forward pass over a real image. Generation is not: a $32\times32$ color image needs 3,072 sequential network evaluations, one per value. Gated and conditional variants (van den Oord et al. 2016) fixed a blind spot of the original receptive field and conditioned generation on class labels.

The lasting idea is to combine an autoregressive model with a compressed, discrete representation. A VQ-VAE (van den Oord et al. 2017) is an autoencoder (Section 14.9) whose codes are snapped to the nearest entry of a learned codebook, turning an image into a short grid of discrete tokens; an autoregressive Transformer over those tokens then generates images as if they were sentences. Several text-to-image systems and today’s natively multimodal language models generate images this way.

25.3Variational Autoencoders

An ordinary autoencoder (Section 14.9) is not a generative model: its codes occupy an irregular region of the code space, so decoding a random code usually gives garbage. A variational autoencoder (VAE) (Kingma and Welling 2014) fixes this by making the codes follow a known distribution. It is a latent-variable model, like the Gaussian mixture of Section 13.5 but with a continuous latent variable and a neural network in place of the component means:

$$\vz\sim\Normal(\mathbf 0,\mI),\qquad \vx\mid\vz\sim\Normal\big(d_{\vtheta}(\vz),\sigma^2\mI\big).$$

To generate, sample $\vz$ and decode. To train, we would like to maximize $\log p_{\vtheta}(\vx)=\log\int p_{\vtheta}(\vx\mid\vz)p(\vz)\dd\vz$, but the integral over all $\vz$ is intractable.

The ELBO. As with EM (Section 13.6), introduce an approximate posterior $q_{\vphi}(\vz\mid\vx)$, here a Gaussian $\Normal(\vmu_{\vphi}(\vx),\diag(\bm\sigma^2_{\vphi}(\vx)))$ whose mean and variances are produced by an encoder network. Jensen’s inequality gives

$$\log p_{\vtheta}(\vx) \ \ge\ \underbrace{\E_{q_{\vphi}(\vz\mid\vx)}\big[\log p_{\vtheta}(\vx\mid\vz)\big]}_{\text{reconstruction}} - \underbrace{D_{\mathrm{KL}}\big(q_{\vphi}(\vz\mid\vx)\,\Vert\,p(\vz)\big)}_{\text{regularization}} = \mathrm{ELBO}, \tag{25.2}$$

with equality when $q_{\vphi}$ equals the true posterior. The first term rewards decoding $\vx$ accurately from codes drawn from $q$; with the Gaussian decoder it is a squared reconstruction error. The second pulls each $q(\vz\mid\vx)$ toward the prior $\Normal(\mathbf 0,\mI)$, so that the codes of all training examples fill the prior’s region without gaps, and random samples from the prior decode into plausible data. For Gaussians the KL term has a closed form:

$$D_{\mathrm{KL}}\big(\Normal(\vmu,\diag\bm\sigma^2)\,\Vert\,\Normal(\mathbf 0,\mI)\big) = \frac12\sum_{j}\big(\mu_j^2+\sigma_j^2-\log\sigma_j^2-1\big). \tag{25.3}$$

The reparameterization trick. Training maximizes the ELBO by gradient ascent on both $\vtheta$ and $\vphi$. The reconstruction term is an expectation over $\vz\sim q_{\vphi}$, and we cannot backpropagate through a random sampling step. The trick is to move the randomness outside the network: sample $\vepsilon\sim\Normal(\mathbf 0,\mI)$ and set

$$\vz = \vmu_{\vphi}(\vx) + \bm\sigma_{\vphi}(\vx)\odot\vepsilon . \tag{25.4}$$

Now $\vz$ is a deterministic, differentiable function of $\vphi$ and an independent noise variable, and ordinary backpropagation gives unbiased gradient estimates. VAEs train stably and have a meaningful latent space, but their samples tend to be blurry, because the Gaussian decoder averages over uncertainty. Today their most important role is as the compressor in latent diffusion models (Section 25.8).

25.4Generative Adversarial Networks

A generative adversarial network (GAN) (Goodfellow et al. 2014) trains a generator $G$ that maps noise $\vz$ to samples, and a discriminator $D$ that outputs the probability that its input is real. They play a minimax game:

$$\min_G\max_D\ \E_{\vx\sim p_{\text{data}}}\big[\log D(\vx)\big] + \E_{\vz\sim p(\vz)}\big[\log\big(1-D(G(\vz))\big)\big]. \tag{25.5}$$

The discriminator is a logistic-regression-style classifier trained with cross-entropy (Chapter 6); the generator is trained to fool it. For a fixed generator with sample density $p_g$, the optimal discriminator is $D^*(\vx)=p_{\text{data}}(\vx)/(p_{\text{data}}(\vx)+p_g(\vx))$, and plugging it in shows that the generator minimizes (up to constants) the Jensen–Shannon divergence between $p_{\text{data}}$ and $p_g$ (Exercise 25.4).

Training in practice. The two networks are updated alternately, one or a few discriminator steps per generator step. Early in training the discriminator rejects the generator’s samples confidently, $D(G(\vz))\approx0$, where $\log(1-D(G(\vz)))$ is flat and gives the generator almost no gradient. In practice the generator therefore maximizes $\log D(G(\vz))$ instead (the non-saturating loss), which has the same fixed point but strong gradients when the generator is losing.

Architectures. DCGAN (Radford et al. 2016) found a convolutional recipe that trained reliably: a generator made of transposed convolutions (Section 21.3) with batch normalization and ReLU, a discriminator made of strided convolutions with leaky ReLU, and no fully connected hidden layers. Its latent space had striking structure: vector arithmetic such as “smiling woman” $-$ “neutral woman” $+$ “neutral man” produced a smiling man. Conditional GANs (Mirza and Osindero 2014) feed a label or another input to both networks, which turns GANs into tools for image-to-image translation: pix2pix (Isola et al. 2017) learns maps from paired examples (sketches to photos, day to night) with a U-Net generator, and CycleGAN (Zhu et al. 2017) learns them from unpaired collections by requiring that translating to the other domain and back returns the original. StyleGAN (Karras et al. 2019) maps the noise through an MLP to an intermediate latent vector that modulates the generator at every resolution (“style”), adds per-pixel noise for stochastic detail, and produces photorealistic $1024\times1024$ faces whose coarse and fine attributes can be controlled separately.

Instability. Adversarial training is a game, not a minimization, and gradient descent on games can oscillate. GANs also suffer from mode collapse: the generator may cover only a few kinds of output that reliably fool the discriminator. Remedies such as the Wasserstein GAN (Arjovsky et al. 2017), which replaces the Jensen–Shannon divergence by an earth-mover distance estimated by a critic with bounded slope, and normalization of the discriminator’s weights, made training more reliable. GANs produce sharp samples in a single forward pass and dominated image generation from about 2016 to 2021. Diffusion models, trained with a simple regression loss and free of mode collapse, have largely replaced them, although adversarial losses survive in some fast samplers and in the decoders of latent diffusion models.

25.5Diffusion Models: The Forward Process

Diffusion models (Sohl-Dickstein et al. 2015; Ho et al. 2020) turn generation into many small denoising steps. They are defined by two processes: a fixed forward process that gradually destroys data by adding noise, and a learned reverse process that removes it.

The forward process starts from a data point $\vx_0$ and, for $t=1,\dots,T$ (typically $T=1000$), adds a little Gaussian noise while shrinking the signal:

$$\vx_t = \sqrt{1-\beta_t}\,\vx_{t-1} + \sqrt{\beta_t}\,\vepsilon_t,\qquad \vepsilon_t\sim\Normal(\mathbf 0,\mI), \tag{25.6}$$

with a small, increasing noise schedule $\beta_1<\dots<\beta_T$. The shrinking factor keeps the total variance bounded: if $\vx_{t-1}$ has unit variance, so does $\vx_t$.

Jumping to any step in closed form. Let $\alpha_t=1-\beta_t$ and $\bar\alpha_t=\prod_{s=1}^t\alpha_s$. Unrolling the recursion, $\vx_t$ is $\sqrt{\bar\alpha_t}\vx_0$ plus a sum of independent Gaussian noise terms, and a sum of independent Gaussians is Gaussian with the variances added. The variances telescope to $1-\bar\alpha_t$ (Exercise 25.1), so

$$\vx_t = \sqrt{\bar\alpha_t}\,\vx_0 + \sqrt{1-\bar\alpha_t}\,\vepsilon,\qquad \vepsilon\sim\Normal(\mathbf 0,\mI) \tag{25.7}$$

We can noise a clean example to any level in one step. As $t\to T$, $\bar\alpha_t\to0$ and $\vx_T$ is essentially pure noise, independent of the data.

25.6Learning to Denoise

The reverse process starts from $\vx_T\sim\Normal(\mathbf 0,\mI)$ and steps backward. When each $\beta_t$ is small, the true reverse step $p(\vx_{t-1}\mid\vx_t)$ is close to Gaussian, so we model it as one whose mean is computed by a neural network. A careful derivation of the ELBO for this model (Ho et al. 2020) shows that, up to weighting, learning the reverse means is equivalent to a strikingly simple task: given a noisy input, predict the noise that was added. Train a network $\vepsilon_{\vtheta}(\vx_t,t)$ with

$$\Loss(\vtheta) = \E_{\vx_0,\ t,\ \vepsilon}\Big[\big\Vert\vepsilon-\vepsilon_{\vtheta}\big(\sqrt{\bar\alpha_t}\vx_0+\sqrt{1-\bar\alpha_t}\vepsilon,\ t\big)\big\Vert^2\Big] \tag{25.8}$$

with $t$ drawn uniformly from $\{1,\dots,T\}$. This is plain least-squares regression (Chapter 4) with a neural network.

Algorithm 25.1  DDPM training and sampling
  1. Training: repeat
  2.   $\vx_0\sim$ data, $t\sim\mathrm{Uniform}\{1..T\}$, $\vepsilon\sim\Normal(\mathbf 0,\mI)$
  3.   take a gradient step on $\norm{\vepsilon-\vepsilon_{\vtheta}(\sqrt{\bar\alpha_t}\vx_0+\sqrt{1-\bar\alpha_t}\vepsilon,t)}^2$
  4. Sampling: $\vx_T\sim\Normal(\mathbf 0,\mI)$
  5. for $t=T,\dots,1$:
    1. $\vz\sim\Normal(\mathbf 0,\mI)$ if $t>1$, else $\vz=\mathbf 0$
    2. $\vx_{t-1}\gets\frac{1}{\sqrt{\alpha_t}}\Big(\vx_t-\frac{\beta_t}{\sqrt{1-\bar\alpha_t}}\vepsilon_{\vtheta}(\vx_t,t)\Big)+\sqrt{\beta_t}\,\vz$
  6. return $\vx_0$

Using $\sqrt{\beta_t}$ as the standard deviation of the added noise is one of two standard choices; the slightly smaller posterior value $\tilde\beta_t=\beta_t(1-\bar\alpha_{t-1})/(1-\bar\alpha_t)$, used in Project 7, gives somewhat sharper samples when few steps are used.

The score view. Why does predicting noise let us generate? The gradient of the log-density, $\nabla_{\vx}\log p_t(\vx)$, is called the score; it points toward regions of higher probability. A classic identity (Tweedie’s formula) shows that the best noise predictor is a rescaled score:

$$\vepsilon^*(\vx_t,t) = -\sqrt{1-\bar\alpha_t}\ \nabla_{\vx}\log p_t(\vx_t), \tag{25.9}$$

where $p_t$ is the distribution of noisy data at level $t$. Each sampling step therefore nudges the sample uphill on the noisy data density, then adds a little fresh noise; as the noise level decreases, the density sharpens toward the data. This connects diffusion to score matching and to a continuous-time description in which the forward process is a stochastic differential equation and sampling solves its time reversal (Song et al. 2021).

Fewer steps. The same continuous-time view yields a deterministic probability-flow ODE with the same marginal distributions, which can be integrated with large steps (DDIM; (Song et al. 2021)) using the same trained network: 25 to 50 steps instead of 1000.

25.7Flow Matching

Diffusion arrives at a deterministic ODE in a roundabout way. Flow matching (Lipman et al. 2023; Liu et al. 2023) learns an ODE directly. Take noise $\vx_0\sim\Normal(\mathbf 0,\mI)$ and a data point $\vx_1$ (note that the convention is reversed: here $t=0$ is noise and $t=1$ is data), and connect them with a straight line:

$$\vx_t = (1-t)\,\vx_0 + t\,\vx_1,\qquad t\in[0,1]. \tag{25.10}$$

A particle moving along this line has constant velocity $\vx_1-\vx_0$. Train a network $\vv_{\vtheta}(\vx,t)$ to predict that velocity from the particle’s current position and time:

$$\Loss(\vtheta) = \E_{t,\ \vx_0,\ \vx_1}\Big[\big\Vert\vv_{\vtheta}(\vx_t,t)-(\vx_1-\vx_0)\big\Vert^2\Big] \tag{25.11}$$

Many different lines pass through any given point $\vx_t$ (from different noise and data pairs), so the regression’s minimizer is the average velocity of all lines through that point, $\vv^*(\vx,t)=\E[\vx_1-\vx_0\mid\vx_t=\vx]$. The key theorem of flow matching is that following this averaged field from noise, by solving the ODE

$$\frac{\dd\vx}{\dd t}=\vv_{\vtheta}(\vx,t),\qquad \vx(0)\sim\Normal(\mathbf 0,\mI), \tag{25.12}$$

produces samples $\vx(1)$ from the data distribution: the averaged flow transports the noise distribution to the data distribution at every intermediate time, exactly as the individual straight lines do on average.

To sample, integrate the ODE with a simple solver, such as Euler’s method with $K$ steps, $\vx\gets\vx+\frac1K\vv_{\vtheta}(\vx,t)$. Because the training paths are straight, the learned trajectories are much straighter than those of diffusion, and good samples need fewer steps. Rectified flow straightens them further by retraining on pairs of noise and the samples they produced, approaching one-step generation. Flow matching with a Transformer backbone became the standard recipe for large image generators (Esser et al. 2024).

25.8Guidance and Conditioning

Conditioning. To generate an image of a given class or matching a text prompt $\mathbf c$, give the condition to the network as an extra input: $\vepsilon_{\vtheta}(\vx_t,t,\mathbf c)$ or $\vv_{\vtheta}(\vx,t,\mathbf c)$. Text prompts are first embedded by a pretrained text encoder, often a Transformer language model, and injected through cross-attention (Section 19.7) or by concatenating text and image tokens.

Classifier-free guidance. Conditioning alone often yields images that only loosely follow the prompt. Classifier-free guidance (Ho and Salimans 2022) trains one network both with the condition and, on a random fraction of examples (say 10%), with the condition dropped. At sampling time it extrapolates away from the unconditional prediction:

$$\tilde\vepsilon = \vepsilon_{\vtheta}(\vx_t,t,\varnothing) + w\big(\vepsilon_{\vtheta}(\vx_t,t,\mathbf c)-\vepsilon_{\vtheta}(\vx_t,t,\varnothing)\big), \tag{25.13}$$

with guidance weight $w>1$ (typical values 3 to 8). In score terms this sharpens the distribution toward samples for which $p(\mathbf c\mid\vx)$ is high, at the cost of diversity. Guidance is used by nearly every text-to-image system.

Latent diffusion. Running diffusion on raw pixels of a $1024\times1024$ image is expensive. Latent diffusion (Rombach et al. 2022) first trains an autoencoder (a VAE with perceptual and adversarial losses) that compresses images by a factor of about 8 in each spatial dimension into a latent tensor, then trains the diffusion or flow model in that latent space and decodes the result. Video models use autoencoders that also compress time.

Architectures. Early diffusion models used convolutional U-Nets. Diffusion Transformers (Peebles and Xie 2023) instead split the latent into patches, treat them as tokens, and process them with Transformer blocks, conditioning on the time step and the prompt; they scale more predictably and are now the standard backbone.

25.9Implementation

Both training objectives fit in a few lines of PyTorch. Here they are on 2D toy data; Project 7 adds training loops, samplers, and evaluation.

import torch, torch.nn as nn

class Net(nn.Module):                        # input: point (2) + time (1)
    def __init__(self, h=128):
        super().__init__()
        self.f = nn.Sequential(nn.Linear(3, h), nn.SiLU(), nn.Linear(h, h), nn.SiLU(),
                               nn.Linear(h, h), nn.SiLU(), nn.Linear(h, 2))
    def forward(self, x, t):
        return self.f(torch.cat([x, t[:, None]], dim=1))

T = 200
betas = torch.linspace(1e-4, 0.05, T)          # alpha_bar_T is about 0.006
alpha_bar = torch.cumprod(1 - betas, dim=0)

def ddpm_loss(net, x0):                      # Eq. (16.7)
    t = torch.randint(0, T, (len(x0),))
    eps = torch.randn_like(x0)
    ab = alpha_bar[t][:, None]
    xt = ab.sqrt() * x0 + (1 - ab).sqrt() * eps
    return ((net(xt, t / T) - eps)**2).mean()

def flow_matching_loss(net, x1):             # Eq. (16.10)
    x0 = torch.randn_like(x1)
    t = torch.rand(len(x1))
    xt = (1 - t[:, None]) * x0 + t[:, None] * x1
    return ((net(xt, t) - (x1 - x0))**2).mean()

@torch.no_grad()
def flow_sample(net, n, steps=50):
    x = torch.randn(n, 2)
    for k in range(steps):
        t = torch.full((n,), k / steps)
        x = x + net(x, t) / steps            # Euler step
    return x

25.10Where the Field Is Heading

As of late 2026, the dominant recipe for image generation is a flow-matching (or closely related diffusion) Transformer operating in the latent space of a learned autoencoder, conditioned on text through a large language-model encoder, and sampled with classifier-free guidance. Video generation extends the same recipe with spatio-temporal latents. Much current work targets cost and control: distilling models so that they sample in a handful of steps or even one (Salimans and Ho 2022; Song et al. 2023); fine-tuning generators with reinforcement learning against reward models of image quality and prompt faithfulness, adapting methods such as GRPO from language models (Liu et al. 2025); and unified models that generate text, images, and other modalities within one network, mixing autoregressive and diffusion-style objectives. Diffusion-style training is also used beyond media, for example to generate protein structures and molecules, and diffusion language models that generate many tokens in parallel are an active research direction.

25.11Summary

  • Generative models learn to turn simple noise into samples from the data distribution.
  • A VAE maximizes the ELBO, reconstruction minus a KL term, using the reparameterization trick; it is now mostly used as a latent compressor.
  • A GAN trains a generator against a discriminator; sharp but unstable and prone to mode collapse.
  • Diffusion noises data by $\vx_t=\sqrt{\bar\alpha_t}\vx_0+\sqrt{1-\bar\alpha_t}\vepsilon$ and trains a network to predict $\vepsilon$; the optimal noise predictor is a scaled score.
  • Flow matching regresses the straight-line velocity $\vx_1-\vx_0$ and samples by integrating an ODE in few steps.
  • Classifier-free guidance, latent spaces, and Transformer backbones make large text-to-image and video models practical.
  • Autoregressive models factor $p(\vx)$ by the chain rule over pixels (PixelRNN, PixelCNN with masked convolutions) or over VQ-VAE tokens: exact likelihood, slow sequential sampling.
  • GANs train with the non-saturating loss; DCGAN, conditional GANs (pix2pix, CycleGAN), and StyleGAN shaped the field before diffusion.

25.12Exercises

  1. MathProve Equation (25.7) by induction on $t$, using the fact that $a\vepsilon_1+b\vepsilon_2$ for independent standard Gaussians has the same distribution as $\sqrt{a^2+b^2}\,\vepsilon$.
  2. MathDerive the closed-form KL divergence between $\Normal(\mu,\sigma^2)$ and $\Normal(0,1)$ in one dimension.
  3. MathShow that the ELBO in Equation (25.2) equals $\log p(\vx)-D_{\mathrm{KL}}(q(\vz\mid\vx)\Vert p(\vz\mid\vx))$.
  4. MathFor a fixed generator, show that the discriminator maximizing the GAN objective is $D^*=p_{\text{data}}/(p_{\text{data}}+p_g)$, by maximizing $a\log D+b\log(1-D)$ pointwise.
  5. MathSuppose the data distribution is a single point, $\vx_1=\mathbf m$. Compute the optimal flow-matching velocity field $\vv^*(\vx,t)$ and show that integrating it from any starting point reaches $\mathbf m$ exactly at $t=1$.
  6. CodingTrain both models of the implementation section on a 2D two-moons data set (sklearn.datasets.make_moons). Compare samples from the flow model with 2, 5, 20, and 100 Euler steps, and from DDPM with its full 200 steps.
  7. CodingMake the flow model class-conditional on the two moons (append a one-hot label to the input, dropped 10% of the time) and implement classifier-free guidance. How do samples change as $w$ goes from 0 to 5?
  8. MathFor a $3\times3$ masked convolution in PixelCNN on a single-channel image, write the $0/1$ mask for the first layer (which must exclude the current pixel) and for later layers (which may include it). Show by induction that after $L$ layers every output still depends only on earlier pixels in raster order.
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.