Machine Learning from First Principles

Chapter 12 · Part II: Supervised Learning

The Machine Learning Workflow

Most of the work in a real project, and most of the ways it fails, lie outside the algorithm: what to predict, how to prepare and split the data, how to evaluate honestly, and how to notice when the world changes.

After this chapter you will be able to

  • Frame a problem with a metric that matches the real goal, and build a baseline first.
  • Split data correctly for random, grouped, and time-ordered data.
  • Prepare features: missing values, scaling, categorical encoding.
  • Recognize and prevent data leakage.
  • Build pipelines and tune them with cross-validation.
  • Diagnose underfitting and overfitting with learning curves.
  • Make experiments reproducible and check for unfair or unstable behavior.

12.1Framing the Problem

  • What decision does the prediction support? If a retention team can call 200 customers a week, the right metric is precision among the top 200 predicted churners, not accuracy.
  • Choose the metric before the model, and write it down: MSE or MAE for regression; precision, recall, F₁, or AUC for classification; precision at $k$ for ranking.
  • Build a baseline: the mean, the majority class, last week's value, or a linear model. It checks the pipeline end to end and shows how much a sophisticated model really adds.

12.2Getting and Splitting Data

The test set must mimic deployment. Random splits are right only for i.i.d. data:

  • Stratified splits keep rare-class proportions equal across splits.
  • Group splits keep all examples of one patient, user, or document on one side (GroupKFold).
  • Time-based splits train on the past and test on the future (TimeSeriesSplit).

After deployment, watch for covariate shift ($p(\vx)$ changes), label shift (class frequencies change), and concept drift ($p(y\mid\vx)$ changes).

12.3Preparing Features

Missing values: impute (e.g. the median) and add a "was missing" indicator; modern tree libraries handle them natively. Scaling: standardize ($x' = (x - \hat\mu)/\hat\sigma$) for $k$-NN, $k$-means, penalized models, PCA, and gradient descent; log-transform heavy-tailed positives; trees need no scaling. With feature scales differing by 1000, the least-squares Hessian's eigenvalues differ by about $10^6$, and gradient descent crawls (Chapter 3).

Categorical features: one-hot encode; for thousands of categories, group rare ones, hash, target-encode inside CV folds, or learn embeddings. Feature engineering: ratios, differences, recent aggregates, and $\sin$/$\cos$ encodings of cyclic time.

12.4Data Leakage

Leakage is information unavailable at prediction time sneaking into training or evaluation:

  • Preprocessing: fitting scalers, imputers, PCA, or feature selection on the full data before splitting.
  • Target: features caused by the label ("collection calls" when predicting default).
  • Temporal: features computed with future data.
  • Duplicates: near-identical examples on both sides of a split.

Pitfall: if it looks too good, it probably is

A 99.5% AUC should trigger a leakage hunt. For each important feature, ask: would I know this value at the moment I need the prediction?

12.5Pipelines

Make preprocessing part of the model. A scikit-learn Pipeline fits every step on training data only; a ColumnTransformer routes columns to different preprocessing.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression

numeric = ["age", "income", "debt_ratio"]
categorical = ["city", "employment_type"]

preprocess = ColumnTransformer([
    ("num", Pipeline([("impute", SimpleImputer(strategy="median")),
                      ("scale", StandardScaler())]), numeric),
    ("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])
model = Pipeline([("prep", preprocess), ("clf", LogisticRegression(max_iter=1000))])

12.6Hyperparameter Search

Random search usually beats grid search: when only a few hyperparameters matter, every random trial tries a new value of each. Search learning rates and regularization strengths on a log scale.

import numpy as np
import pandas as pd
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform

# a synthetic loan data set with the columns used above, including missing incomes
rng = np.random.default_rng(0)
n = 1000
X_train = pd.DataFrame({
    "age": rng.integers(18, 80, n), "income": rng.lognormal(10, 0.5, n), "debt_ratio": rng.uniform(0, 1, n),
    "city": rng.choice(["A", "B", "C"], n), "employment_type": rng.choice(["salaried", "self", "none"], n)})
X_train.loc[rng.random(n) < 0.05, "income"] = np.nan
y_train = ((X_train["debt_ratio"] + 0.3 * (X_train["employment_type"] == "none")
            + rng.normal(0, 0.2, n)) > 0.7).astype(int)

search = RandomizedSearchCV(model, {"clf__C": loguniform(1e-3, 1e3)},
                            n_iter=30, cv=5, scoring="roc_auc", random_state=0)
search.fit(X_train, y_train)
print(search.best_params_, search.best_score_)

The best CV score among many configurations is optimistic; use nested CV or one final evaluation on a held-out test set. Bayesian optimization and successive halving help when training runs are expensive.

12.7Diagnosing Models with Learning Curves

  • High bias: both errors high, close, and flat. More data won't help; add capacity or features, or reduce regularization.
  • High variance: low training error, a large gap that narrows with more data. Get more data, regularize, simplify, or augment.

Error analysis: group the mistakes by source, class, input length, or time. One fixable cause (mislabeled data, a missing feature, a preprocessing bug) often explains many errors.

12.8Reproducibility and Responsible Practice

  • Reproducibility: fix seeds; record data, code, library versions, hyperparameters, and splits.
  • Fairness: report metrics per subgroup. Criteria such as equal error rates, equal positive rates, and within-group calibration can be mutually incompatible, so choosing among them is a value judgment to make explicitly with the people affected.
  • Documentation and monitoring: document purpose, data, evaluation, and limits; monitor inputs, predictions, and performance after deployment.

12.9Summary

  • Frame the decision, choose the metric, build a baseline.
  • Split the way the model will be used: stratified, grouped, or by time.
  • Impute, scale, and encode; scaling fixes the conditioning that slows GD.
  • Prevent leakage by fitting preprocessing inside pipelines.
  • Tune with random search and CV; keep an honest final estimate.
  • Learning curves say whether to add data, capacity, or regularization.
  • Track experiments, check subgroups, document, and monitor.

12.10Exercises

  1. ConceptualPropose a metric, baseline, and split for predicting 30-day hospital readmission. Name two leakage risks.
  2. MathFor two uncorrelated features with standard deviations $s_1, s_2$, show the MSE Hessian is about $2\diag(s_1^2, s_2^2)$ with condition number $(s_1/s_2)^2$.
  3. Math20 configurations all truly score 80%, with independent validation noise of s.d. 2%. Estimate the expected best observed score (the expected max of 20 standard normals is about 1.87).
  4. CodingWith 50 samples, 10,000 noise features, and random labels, select 20 features on all data and then cross-validate; repeat with selection inside a pipeline. Compare with the true 50%.
  5. CodingPlot learning_curve on load_digits for logistic regression and a random forest. Which benefits more from more data?
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.