Project 1 · Part VII: Projects
Predicting House Prices
A complete regression project: baseline, leakage-free preprocessing, least squares and ridge from scratch, verification against scikit-learn, and a nonlinear model that shows what the linear one misses.
After this project you will be able to
- Run a regression project end to end: baseline, split, preprocessing, model, evaluation.
- Implement least squares, gradient descent, and ridge from scratch and verify them.
- Choose hyperparameters on validation data or by cross-validation.
- Compare linear and nonlinear models and interpret the difference.
Code: projects/p1_house_prices/ (linreg_scratch.py, linreg_sklearn.py, tests/). Background: Chapters 4, 5, 12.
P1.1The Problem
- Task: predict the median house value of a California census block group (regression).
- Metric: test RMSE in units of $100,000, plus $R^2$.
- Baseline: predict the training mean everywhere.
P1.2The Data
California Housing: 20,640 block groups from the 1990 census, eight features (median income, house age, rooms, bedrooms, population, occupancy, latitude, longitude), target capped at 5.0. Shuffled once and split 60/20/20 into 12,384 training, 4,128 validation, and 4,128 test examples. The test set is used once.
P1.3Building It from Scratch
Preprocess without leakage: standardize with training-set statistics only, then append a bias column.
class Standardizer:
def fit(self, X):
self.mean, self.std = X.mean(0), X.std(0) + 1e-12
return self
def transform(self, X):
return (X - self.mean) / self.std
def fit_least_squares(X, y):
w, *_ = np.linalg.lstsq(X, y, rcond=None) # never invert X^T X
return w
def fit_gradient_descent(X, y, lr=0.1, steps=2000):
w = np.zeros(X.shape[1])
for _ in range(steps):
w -= lr * (2 / len(y)) * X.T @ (X @ w - y)
return w
def fit_ridge(X, y, lam):
P = lam * np.eye(X.shape[1]); P[-1, -1] = 0.0 # bias unpenalized
return np.linalg.solve(X.T @ X + P, X.T @ y)
On standardized features gradient descent agrees with lstsq to $10^{-12}$ after 3,000 steps. We then add $\log$ population, $\log$ occupancy, and income squared, fit ridge, choose $\lambda$ on the validation set, and refit on training plus validation data.
P1.4Using Libraries
models = {
"linear": make_pipeline(StandardScaler(), LinearRegression()),
"poly2 + ridge": make_pipeline(StandardScaler(), PolynomialFeatures(2), StandardScaler(),
RidgeCV(alphas=np.logspace(-2, 3, 12))),
"boosted trees": HistGradientBoostingRegressor(max_iter=400, learning_rate=0.08),
}
Pipelines refit the scaler inside every cross-validation fold.
P1.5Results and Discussion
| Model | Test RMSE | Test $R^2$ |
|---|---|---|
| Baseline: predict the mean | 1.170 | 0.00 |
| Least squares (8 features) | 0.705 | 0.64 |
| Engineered features + ridge | 0.651 | 0.69 |
| Degree-2 polynomial + ridge | 0.643 | 0.70 |
| Gradient-boosted trees | 0.433 | 0.86 |
- The baseline matters: linear regression cuts error by 40%.
- Weights tell a story, carefully: latitude (−0.91), longitude (−0.89), and income (+0.81) dominate; together the location signs encode the expensive south-west coast. Correlated features share credit.
- Regularization barely mattered with 8 features and 20,000 examples: bias, not variance, limits the linear model.
- CV and one split can disagree: degree-2 features scored worse in CV (0.80 vs 0.73) but better on test, because they extrapolate wildly for a few extreme neighborhoods.
- Trees win big ($R^2$ 0.86): prices depend on location nonlinearly (coasts, cities), which trees capture by carving the map into regions.
P1.6Extensions
- Add a cross-fitted $k$-NN feature on (latitude, longitude). How close does the linear model get to the trees?
- Plot residuals against each feature to see where the linear model fails.
- Study the effect of the 5.0 cap on each model.
- Try
HuberRegressorfor robustness to outliers.