Project 6 · Part VII: Projects
A Tiny GPT
A decoder-only Transformer, built and trained on Shakespeare on a laptop CPU in about four minutes, then specialized with LoRA. Every piece is the one large language models use, just a million times smaller.
After this project you will be able to
- Implement causal multi-head attention in NumPy and verify it against PyTorch.
- Build and train a small character-level GPT.
- Use AdamW, warm-up with cosine decay, clipping, and weight tying.
- Generate with temperature and nucleus sampling.
- Fine-tune with LoRA and observe specialization and forgetting.
Code: projects/p6_tiny_gpt/. Background: Chapters 19 and 24.
P6.1The Problem
A GPT-style model small enough for a CPU. Characters instead of subwords: 65 symbols, so the model must learn spelling too. Metric: validation cross-entropy in nats per character; uniform guessing scores $\log 65\approx 4.17$.
P6.2The Data
About one megabyte of Shakespeare's plays (public domain) as speeches. Every ROMEO speech (about 24,000 characters) is held out for the fine-tuning experiment; the rest is split 90/10 into training and validation.
P6.3Building It from Scratch
Attention in NumPy matches PyTorch's scaled_dot_product_attention to $4\times10^{-16}$, and changing the last token leaves earlier outputs unchanged (causality).
The model: four pre-norm blocks, width 128, four heads, context 64, learned position embeddings, final LayerNorm, output layer tied to the embedding: 0.81 million parameters.
class TinyGPT(nn.Module):
def __init__(self, c):
super().__init__()
self.c = c
self.tok = nn.Embedding(c.vocab, c.d)
self.pos = nn.Embedding(c.ctx, c.d)
self.blocks = nn.ModuleList(Block(c) for _ in range(c.layers))
self.ln = nn.LayerNorm(c.d)
self.head = nn.Linear(c.d, c.vocab, bias=False)
self.head.weight = self.tok.weight # weight tying
def forward(self, idx, targets=None):
x = self.tok(idx) + self.pos(torch.arange(idx.shape[1]))
for b in self.blocks:
x = b(x)
logits = self.head(self.ln(x))
loss = None if targets is None else \
F.cross_entropy(logits.flatten(0, 1), targets.flatten())
return logits, loss
Training: random windows of 65 characters (input = first 64, target = shifted by one), batches of 32; AdamW (decay 0.1 on matrices), learning rate $2\times10^{-3}$ with 100 warm-up steps and cosine decay, clipping at 1, dropout 0.1; 2,000 steps. Sampling: temperature plus top-$p$.
P6.4Using Libraries
PyTorch supplies autograd, AdamW, and fused attention. At scale you would add subword tokenization, RoPE, mixed precision, and libraries such as Hugging Face Transformers and PEFT for pretrained models and LoRA. The ideas do not change.
P6.5Results and Discussion
Validation loss falls from 4.17 to 1.73 nats per character (perplexity 5.6). A sample at temperature 1.0:
JULIET:
Lord yet, give he sight them?
But had I their dother, it do my o'er!
QUEEN MARGARET:
Thou know your art, and thou art is alose.
Play format, real words, and Shakespearean vocabulary emerge from raw characters; meaning does not. The model has about a millionth of the parameters and training tokens of a real LLM.
LoRA: rank-8 adapters on every attention projection: 24,576 trainable parameters (3.0%), 150 steps on 80% of ROMEO's speeches.
| Model | Held-out ROMEO loss | General validation loss |
|---|---|---|
| Pretrained (ROMEO never seen) | 1.695 | 1.727 |
| LoRA on ROMEO only | 1.665 | 1.833 |
| LoRA on ROMEO + 50% general replay | 1.654 | 1.737 |
Training on ROMEO alone teaches the model that every speech is ROMEO's, and general loss jumps: forgetting. Mixing in general text (replay) gives a larger gain on ROMEO and almost no loss elsewhere: the specialization–retention trade-off seen at every scale.
P6.6Extensions
- Replace learned positions with RoPE.
- Add a KV cache and measure the generation speedup.
- Train on BPE tokens with the same compute and compare loss per character.
- Train widths 64, 96, 128 and plot loss against parameters on log–log axes.