Machine Learning from First Principles

Appendix A

Python and NumPy Crash Course

Exactly the Python needed for the code in this book. If you know another programming language, an afternoon with this page and an interpreter is enough. Type the examples; do not just read them.

A.1Setting Up

python -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate
pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install -r projects/requirements.txt

Run a script with python script.py, or experiment in python, IPython, or Jupyter. Run all project tests with python -m pytest -q from projects/.

A.2Python Essentials

x = 3                      # int
y = 2.5                    # float
name = "digits"            # str; f-strings format values:
print(f"{name}: {x * y:.2f}")

xs = [3, 1, 4, 1, 5]       # list: ordered, mutable
xs.append(9); xs[0] = 2
print(xs[1:3], xs[-1])     # slicing [start:stop), negative index from the end

point = (1.0, 2.0)         # tuple: immutable
a, b = point               # unpacking

counts = {"spam": 747, "ham": 4827}    # dict: key -> value
counts["total"] = counts["spam"] + counts["ham"]

squares = [k * k for k in range(5)]               # list comprehension
evens = {k: k * k for k in range(10) if k % 2 == 0}

def mse(pred, target):                            # function
    """Mean squared error of two equal-length lists."""
    return sum((p - t) ** 2 for p, t in zip(pred, target)) / len(pred)

class Counter:                                    # class with state
    def __init__(self):
        self.n = 0
    def step(self, k=1):
        self.n += k
        return self
print(Counter().step().step(5).n)                 # 6

Indentation defines blocks. Variables are names bound to objects: b = a does not copy a list. Use a.copy() when you need a copy.

A.3NumPy Arrays

import numpy as np

v = np.array([1.0, 2.0, 3.0])          # shape (3,)
M = np.arange(12).reshape(3, 4)        # shape (3, 4)
Z = np.zeros((2, 3)); I = np.eye(3)

M[1, 2]          # one element
M[1]             # row 1
M[:, 2]          # column 2
M[:2, 1:3]       # a 2x2 block
M[M > 5]         # boolean mask
M[[0, 2]]        # rows 0 and 2

M.sum(), M.sum(axis=0), M.mean(axis=1)      # reductions
M.T                                          # transpose
np.hstack([M, np.ones((3, 1))])              # append a column (the bias trick)

Axis convention: a data matrix has one row per example, X.shape == (n, d). X.mean(axis=0) gives one mean per feature.

A.4Broadcasting and Vectorization

NumPy compares shapes from the right and stretches dimensions of size 1:

X = np.random.default_rng(0).normal(size=(100, 3))
Xs = (X - X.mean(axis=0)) / X.std(axis=0)            # per-feature standardization

A, B = X[:5], X[5:9]
D2 = ((A[:, None, :] - B[None, :, :])**2).sum(-1)    # all pairwise squared distances, (5, 4)

If you write a Python loop over examples, look for a matrix formulation: predictions are X @ w, the squared-loss gradient is 2 / n * X.T @ (X @ w - y). Beware: shapes (n,) and (n, 1) broadcast to (n, n), a classic silent bug. Print shapes often.

A.5Linear Algebra and Random Numbers

A = np.array([[3.0, 1.0], [1.0, 2.0]])
b = np.array([9.0, 8.0])
x = np.linalg.solve(A, b)                     # never inv(A) @ b
w, *_ = np.linalg.lstsq(X, X @ [1., 2., 3.], rcond=None)
vals, vecs = np.linalg.eigh(A)                # symmetric eigendecomposition
U, s, Vt = np.linalg.svd(X, full_matrices=False)

rng = np.random.default_rng(42)               # reproducible generator
rng.normal(0, 1, size=(2, 3)), rng.uniform(size=5), rng.integers(10, size=4)
rng.permutation(10), rng.choice(10, size=3, replace=False)

A.6Plotting with Matplotlib

import matplotlib.pyplot as plt

xs = np.linspace(0, 3, 200)
fig, ax = plt.subplots(figsize=(6, 4))
ax.scatter(X[:, 0], X[:, 1], s=8, label="data")
ax.plot(xs, np.sin(2 * xs), color="black", label="sin 2x")
ax.set_xlabel("x"); ax.set_ylabel("y"); ax.legend()
fig.savefig("plot.png", dpi=150, bbox_inches="tight")

A.7PyTorch Tensors and Autograd

import torch

w = torch.zeros(3, requires_grad=True)
X = torch.randn(100, 3); y = X @ torch.tensor([1., 2., 3.])
loss = ((X @ w - y)**2).mean()
loss.backward()                               # fills w.grad = 2/n X^T (Xw - y)

with torch.no_grad():
    w -= 0.1 * w.grad
    w.grad.zero_()

a = torch.from_numpy(np.ones(3)); b = a.numpy()   # shares memory

Chapter 15 explains what backward computes; Chapter 16 shows a full training loop.

Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.