Chapter 12 · Part II: Supervised Learning
The Machine Learning Workflow
Most of the work in a real project, and most of the ways it fails, lie outside the algorithm: what to predict, how to prepare and split the data, how to evaluate honestly, and how to notice when the world changes.
After this chapter you will be able to
- Frame a problem with a metric that matches the real goal, and build a baseline first.
- Split data correctly for random, grouped, and time-ordered data.
- Prepare features: missing values, scaling, categorical encoding.
- Recognize and prevent data leakage.
- Build pipelines and tune them with cross-validation.
- Diagnose underfitting and overfitting with learning curves.
- Make experiments reproducible and check for unfair or unstable behavior.
12.1Framing the Problem
- What decision does the prediction support? If a retention team can call 200 customers a week, the right metric is precision among the top 200 predicted churners, not accuracy.
- Choose the metric before the model, and write it down: MSE or MAE for regression; precision, recall, F₁, or AUC for classification; precision at $k$ for ranking.
- Build a baseline: the mean, the majority class, last week's value, or a linear model. It checks the pipeline end to end and shows how much a sophisticated model really adds.
12.2Getting and Splitting Data
The test set must mimic deployment. Random splits are right only for i.i.d. data:
- Stratified splits keep rare-class proportions equal across splits.
- Group splits keep all examples of one patient, user, or document on one side (
GroupKFold). - Time-based splits train on the past and test on the future (
TimeSeriesSplit).
After deployment, watch for covariate shift ($p(\vx)$ changes), label shift (class frequencies change), and concept drift ($p(y\mid\vx)$ changes).
12.3Preparing Features
Missing values: impute (e.g. the median) and add a "was missing" indicator; modern tree libraries handle them natively. Scaling: standardize ($x' = (x - \hat\mu)/\hat\sigma$) for $k$-NN, $k$-means, penalized models, PCA, and gradient descent; log-transform heavy-tailed positives; trees need no scaling. With feature scales differing by 1000, the least-squares Hessian's eigenvalues differ by about $10^6$, and gradient descent crawls (Chapter 3).
Categorical features: one-hot encode; for thousands of categories, group rare ones, hash, target-encode inside CV folds, or learn embeddings. Feature engineering: ratios, differences, recent aggregates, and $\sin$/$\cos$ encodings of cyclic time.
12.4Data Leakage
Leakage is information unavailable at prediction time sneaking into training or evaluation:
- Preprocessing: fitting scalers, imputers, PCA, or feature selection on the full data before splitting.
- Target: features caused by the label ("collection calls" when predicting default).
- Temporal: features computed with future data.
- Duplicates: near-identical examples on both sides of a split.
Pitfall: if it looks too good, it probably is
A 99.5% AUC should trigger a leakage hunt. For each important feature, ask: would I know this value at the moment I need the prediction?
12.5Pipelines
Make preprocessing part of the model. A scikit-learn Pipeline fits every step on training data only; a ColumnTransformer routes columns to different preprocessing.
from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler, OneHotEncoder
from sklearn.linear_model import LogisticRegression
numeric = ["age", "income", "debt_ratio"]
categorical = ["city", "employment_type"]
preprocess = ColumnTransformer([
("num", Pipeline([("impute", SimpleImputer(strategy="median")),
("scale", StandardScaler())]), numeric),
("cat", OneHotEncoder(handle_unknown="ignore"), categorical),
])
model = Pipeline([("prep", preprocess), ("clf", LogisticRegression(max_iter=1000))])
12.6Hyperparameter Search
Random search usually beats grid search: when only a few hyperparameters matter, every random trial tries a new value of each. Search learning rates and regularization strengths on a log scale.
import numpy as np
import pandas as pd
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import loguniform
# a synthetic loan data set with the columns used above, including missing incomes
rng = np.random.default_rng(0)
n = 1000
X_train = pd.DataFrame({
"age": rng.integers(18, 80, n), "income": rng.lognormal(10, 0.5, n), "debt_ratio": rng.uniform(0, 1, n),
"city": rng.choice(["A", "B", "C"], n), "employment_type": rng.choice(["salaried", "self", "none"], n)})
X_train.loc[rng.random(n) < 0.05, "income"] = np.nan
y_train = ((X_train["debt_ratio"] + 0.3 * (X_train["employment_type"] == "none")
+ rng.normal(0, 0.2, n)) > 0.7).astype(int)
search = RandomizedSearchCV(model, {"clf__C": loguniform(1e-3, 1e3)},
n_iter=30, cv=5, scoring="roc_auc", random_state=0)
search.fit(X_train, y_train)
print(search.best_params_, search.best_score_)
The best CV score among many configurations is optimistic; use nested CV or one final evaluation on a held-out test set. Bayesian optimization and successive halving help when training runs are expensive.
12.7Diagnosing Models with Learning Curves
- High bias: both errors high, close, and flat. More data won't help; add capacity or features, or reduce regularization.
- High variance: low training error, a large gap that narrows with more data. Get more data, regularize, simplify, or augment.
Error analysis: group the mistakes by source, class, input length, or time. One fixable cause (mislabeled data, a missing feature, a preprocessing bug) often explains many errors.
12.8Reproducibility and Responsible Practice
- Reproducibility: fix seeds; record data, code, library versions, hyperparameters, and splits.
- Fairness: report metrics per subgroup. Criteria such as equal error rates, equal positive rates, and within-group calibration can be mutually incompatible, so choosing among them is a value judgment to make explicitly with the people affected.
- Documentation and monitoring: document purpose, data, evaluation, and limits; monitor inputs, predictions, and performance after deployment.
12.9Summary
- Frame the decision, choose the metric, build a baseline.
- Split the way the model will be used: stratified, grouped, or by time.
- Impute, scale, and encode; scaling fixes the conditioning that slows GD.
- Prevent leakage by fitting preprocessing inside pipelines.
- Tune with random search and CV; keep an honest final estimate.
- Learning curves say whether to add data, capacity, or regularization.
- Track experiments, check subgroups, document, and monitor.
12.10Exercises
- ConceptualPropose a metric, baseline, and split for predicting 30-day hospital readmission. Name two leakage risks.
- MathFor two uncorrelated features with standard deviations $s_1, s_2$, show the MSE Hessian is about $2\diag(s_1^2, s_2^2)$ with condition number $(s_1/s_2)^2$.
- Math20 configurations all truly score 80%, with independent validation noise of s.d. 2%. Estimate the expected best observed score (the expected max of 20 standard normals is about 1.87).
- CodingWith 50 samples, 10,000 noise features, and random labels, select 20 features on all data and then cross-validate; repeat with selection inside a pipeline. Compare with the true 50%.
- CodingPlot
learning_curveonload_digitsfor logistic regression and a random forest. Which benefits more from more data?