Machine Learning from First Principles

Project 1 · Part VII: Projects

Predicting House Prices

A complete regression project: baseline, leakage-free preprocessing, least squares and ridge from scratch, verification against scikit-learn, and a nonlinear model that shows what the linear one misses.

After this project you will be able to

  • Run a regression project end to end: baseline, split, preprocessing, model, evaluation.
  • Implement least squares, gradient descent, and ridge from scratch and verify them.
  • Choose hyperparameters on validation data or by cross-validation.
  • Compare linear and nonlinear models and interpret the difference.

Code: projects/p1_house_prices/ (linreg_scratch.py, linreg_sklearn.py, tests/). Background: Chapters 4, 5, 12.

P1.1The Problem

  • Task: predict the median house value of a California census block group (regression).
  • Metric: test RMSE in units of $100,000, plus $R^2$.
  • Baseline: predict the training mean everywhere.

P1.2The Data

California Housing: 20,640 block groups from the 1990 census, eight features (median income, house age, rooms, bedrooms, population, occupancy, latitude, longitude), target capped at 5.0. Shuffled once and split 60/20/20 into 12,384 training, 4,128 validation, and 4,128 test examples. The test set is used once.

P1.3Building It from Scratch

Preprocess without leakage: standardize with training-set statistics only, then append a bias column.

class Standardizer:
    def fit(self, X):
        self.mean, self.std = X.mean(0), X.std(0) + 1e-12
        return self
    def transform(self, X):
        return (X - self.mean) / self.std

def fit_least_squares(X, y):
    w, *_ = np.linalg.lstsq(X, y, rcond=None)   # never invert X^T X
    return w

def fit_gradient_descent(X, y, lr=0.1, steps=2000):
    w = np.zeros(X.shape[1])
    for _ in range(steps):
        w -= lr * (2 / len(y)) * X.T @ (X @ w - y)
    return w

def fit_ridge(X, y, lam):
    P = lam * np.eye(X.shape[1]); P[-1, -1] = 0.0          # bias unpenalized
    return np.linalg.solve(X.T @ X + P, X.T @ y)

On standardized features gradient descent agrees with lstsq to $10^{-12}$ after 3,000 steps. We then add $\log$ population, $\log$ occupancy, and income squared, fit ridge, choose $\lambda$ on the validation set, and refit on training plus validation data.

P1.4Using Libraries

models = {
    "linear": make_pipeline(StandardScaler(), LinearRegression()),
    "poly2 + ridge": make_pipeline(StandardScaler(), PolynomialFeatures(2), StandardScaler(),
                                   RidgeCV(alphas=np.logspace(-2, 3, 12))),
    "boosted trees": HistGradientBoostingRegressor(max_iter=400, learning_rate=0.08),
}

Pipelines refit the scaler inside every cross-validation fold.

P1.5Results and Discussion

ModelTest RMSETest $R^2$
Baseline: predict the mean1.1700.00
Least squares (8 features)0.7050.64
Engineered features + ridge0.6510.69
Degree-2 polynomial + ridge0.6430.70
Gradient-boosted trees0.4330.86
  • The baseline matters: linear regression cuts error by 40%.
  • Weights tell a story, carefully: latitude (−0.91), longitude (−0.89), and income (+0.81) dominate; together the location signs encode the expensive south-west coast. Correlated features share credit.
  • Regularization barely mattered with 8 features and 20,000 examples: bias, not variance, limits the linear model.
  • CV and one split can disagree: degree-2 features scored worse in CV (0.80 vs 0.73) but better on test, because they extrapolate wildly for a few extreme neighborhoods.
  • Trees win big ($R^2$ 0.86): prices depend on location nonlinearly (coasts, cities), which trees capture by carving the map into regions.

P1.6Extensions

  1. Add a cross-fitted $k$-NN feature on (latitude, longitude). How close does the linear model get to the trees?
  2. Plot residuals against each feature to see where the linear model fails.
  3. Study the effect of the 5.0 cap on each model.
  4. Try HuberRegressor for robustness to outliers.
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.