Machine Learning from First Principles

Project 5 · Part VII: Projects

Compressing Digits with PCA

How many numbers does a digit really need? Compress 64-pixel images with PCA and measure variance kept, reconstruction quality, and what a classifier can still do with the compressed codes.

After this project you will be able to

  • Implement PCA with the SVD as a reusable class.
  • Measure compression against reconstruction error and verify the eigenvalue formula.
  • Visualize high-dimensional data in 2D.
  • Use PCA in a pipeline with $k$ chosen by cross-validation.

Code: projects/p5_pca_digits/. Background: Chapter 14.

P5.1The Problem

The 64 pixels of a digit are strongly correlated. Compress with PCA and measure explained variance, reconstruction error, and downstream accuracy.

P5.2The Data

The 1,797 digits of Project 3 with raw values 0–16 (centered, not standardized, because pixels share units), split 70/30. PCA is fitted on training images only.

P5.3Building It from Scratch

class PCA:
    def fit(self, X):
        self.mean = X.mean(0)
        _, s, Vt = np.linalg.svd(X - self.mean, full_matrices=False)
        self.components, self.eigvals = Vt, s**2 / len(X)
        return self
    def transform(self, X, k):
        return (X - self.mean) @ self.components[:k].T
    def inverse_transform(self, Z):
        return Z @ self.components[:Z.shape[1]] + self.mean

Tests confirm to machine precision that training reconstruction error equals the sum of dropped eigenvalues, that projected coordinates are uncorrelated with variances $\lambda_j$, and that components match scikit-learn's up to sign.

P5.4Using Libraries

from sklearn.datasets import load_digits
from sklearn.decomposition import PCA
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import GridSearchCV, train_test_split
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_digits(return_X_y=True)
Xtr, Xte, ytr, yte = train_test_split(X, y, test_size=0.3, random_state=0, stratify=y)
pipe = make_pipeline(PCA(), StandardScaler(), LogisticRegression(max_iter=3000))
search = GridSearchCV(pipe, {"pca__n_components": [5, 10, 15, 20, 30, 40, 64]}, cv=5)
search.fit(Xtr, ytr)
print(search.best_params_, round(search.score(Xte, yte), 3))

P5.5Results and Discussion

$k$Explained varianceTest MSE per pixelNearest-centroid accuracy
228.4%13.560.0%
554.6%8.7081.3%
1073.9%5.0588.9%
2089.5%2.1090.2%
64100%090.4%

21 components keep 90% of the variance: threefold compression with legible digits. Nearest-centroid accuracy with 20 components equals that with all 64, since dropped directions are mostly noise. With logistic regression, CV chose 40 components (95.6% CV accuracy vs 94.5% with 64) and reached 96.5% on test: dropping low-variance directions regularized mildly.

P5.6Extensions

  1. Denoise noisy digits by projecting onto components from clean data.
  2. Compare with a PyTorch autoencoder with a 10-dimensional bottleneck.
  3. Run PCA per class: how many components does a "1" need versus an "8"?
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.