Machine Learning from First Principles

Project 7 · Part VII: Projects

A Toy Diffusion Model

Generate a 2D spiral from Gaussian noise with a DDPM and with flow matching. Two dimensions keep every run under 30 seconds, yet the losses, samplers, and time-conditioned network are the same ones large image generators use.

After this project you will be able to

  • Implement the DDPM forward process, loss, and sampler.
  • Implement flow matching with an Euler sampler.
  • Evaluate generative models with MMD.
  • See how the number of sampling steps affects quality.

Code: projects/p7_toy_diffusion/. Background: Chapter 25.

P7.1The Problem

Learn to sample points on a 2D spiral starting from Gaussian noise.

P7.2The Data

A two-turn noisy spiral with roughly unit spread: 8,000 training points and 2,000 held-out points. Samples are compared with held-out data by the maximum mean discrepancy, $\mathrm{MMD}^2 = \E k(\vx,\vx') + \E k(\vy,\vy') - 2\E k(\vx,\vy)$ with Gaussian kernels: zero for identical distributions. Pure noise scores about 0.06.

P7.3Building It from Scratch

Both models share an MLP (three hidden layers of 128 SiLU units) whose input is the point plus sinusoidal features $\sin(2^j\pi t), \cos(2^j\pi t)$ of the time.

DDPM: $T = 200$, $\beta_t$ linear from $10^{-4}$ to 0.05 ($\bar\alpha_T\approx 0.006$), noise-prediction loss, ancestral sampling with the posterior standard deviation.

def q_sample(x0, t, eps):
    ab = ALPHA_BAR[t][:, None]
    return ab.sqrt() * x0 + (1 - ab).sqrt() * eps

def ddpm_loss(net, x0):
    t = torch.randint(0, T, (len(x0),))
    eps = torch.randn_like(x0)
    return ((net(q_sample(x0, t, eps), t.float() / T) - eps)**2).mean()

A first version with 100 large steps and $\sqrt{\beta_t}$ noise gave visibly blurred samples; 200 steps and the posterior variance cut the MMD by more than half. The schedule matters.

Flow matching:

def fm_loss(net, x1):
    x0 = torch.randn_like(x1)
    t = torch.rand(len(x1))
    xt = (1 - t[:, None]) * x0 + t[:, None] * x1
    return ((net(xt, t) - (x1 - x0))**2).mean()

@torch.no_grad()
def sample(net, n, steps=50):
    x = torch.randn(n, 2)
    for k in range(steps):
        x = x + net(x, torch.full((n,), k / steps)) / steps
    return x

P7.4Using Libraries

Only PyTorch was needed. For images, libraries such as Hugging Face Diffusers provide schedules, samplers, U-Net and Transformer backbones, and pretrained latent autoencoders.

P7.5Results and Discussion

SamplerNetwork evaluationsMMD² vs held-out data
Gaussian noise (no model)00.060
DDPM, ancestral2000.005
Flow matching, Euler20.226
Flow matching, Euler50.031
Flow matching, Euler200.006
Flow matching, Euler1000.005

Flow matching matches DDPM with a tenth of the network evaluations because its paths are nearly straight; with only 2 steps particles overshoot into a blurred cloud. The interactive figures in Chapter 25 were generated by this project's p7_export.py.

P7.6Extensions

  1. Implement DDIM sampling for the trained DDPM at 10, 20, and 50 steps.
  2. Add class conditioning and classifier-free guidance on two interleaved spirals.
  3. Rectify the flow by retraining on (noise, sample) pairs; re-measure 2-step MMD.
  4. Generate whole point clouds with a small Transformer.
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.