Project 2 · Part VII: Projects
An SMS Spam Filter
From raw text to a working filter: tokenize, count words, fit Naive Bayes with a few lines of counting, then evaluate the way imbalanced problems demand.
After this project you will be able to
- Turn raw text into bag-of-words features.
- Implement multinomial Naive Bayes with sparse matrices.
- Evaluate with precision, recall, and F₁, and choose a threshold.
- Compare generative and discriminative classifiers.
Code: projects/p2_spam_filter/. Background: Chapters 6 and 8.
P2.1The Problem
Filter spam text messages. A legitimate message lost to the spam folder is worse than a missed spam, so precision matters. Metrics: precision, recall, F₁ for spam; false positives. Baseline: always "ham" scores 86.6% accuracy with zero recall: accuracy is meaningless here.
P2.2The Data
The UCI SMS Spam Collection (Almeida, Gómez Hidalgo & Yamakami, 2011; CC BY 4.0): 5,574 English messages, 13.4% spam, one per line as label<TAB>text. A stratified 80/20 split gives 4,460 training and 1,114 test messages.
P2.3Building It from Scratch
Tokenize into lower-case runs of letters, digits, currency signs, and apostrophes (spam is full of numbers and prices). Build the vocabulary from training messages only (7,912 words) and store counts in a sparse matrix.
class MultinomialNB:
def fit(self, X, y, alpha=1.0):
self.classes = np.unique(y)
counts = np.vstack([np.asarray(X[y == c].sum(0)).ravel() for c in self.classes])
self.log_theta = np.log(counts + alpha) - \
np.log(counts.sum(1, keepdims=True) + alpha * X.shape[1])
self.log_prior = np.log([np.mean(y == c) for c in self.classes])
return self
def decision(self, X):
return np.asarray(X @ self.log_theta.T) + self.log_prior
def predict(self, X):
return self.classes[self.decision(X).argmax(1)]
Given identical count features, these log-probabilities and predictions match scikit-learn's MultinomialNB exactly (checked by the tests).
P2.4Using Libraries
nb = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0))
lr = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True),
LogisticRegression(C=20, max_iter=2000, class_weight="balanced"))
P2.5Results and Discussion
| Model (test set) | Precision | Recall | F₁ | False positives |
|---|---|---|---|---|
| Always ham | – | 0.000 | – | 0 |
| Naive Bayes, from scratch | 0.964 | 0.906 | 0.934 | 5 |
| Naive Bayes, scikit-learn | 0.971 | 0.913 | 0.941 | 4 |
| Logistic regression, TF-IDF | 0.972 | 0.926 | 0.948 | 4 |
- Interpretable weights: the most spam-indicative words are claim, prize, won, 150p, tone, guaranteed, awarded, £1000, ringtone.
- Thresholds: requiring spam log-odds above 5 removes every false positive here while recall falls from 0.91 to 0.85. Choose the threshold on validation data, then report test results at that fixed value.
- Generative vs discriminative: logistic regression edges ahead with thousands of examples, but Naive Bayes trains in one counting pass with no tuning.
P2.6Extensions
- Plot precision–recall curves and pick a threshold allowing at most 1 false positive per 1,000 ham on validation data.
- Add message length, digit count, and URL features.
- Study the false negatives.
- Use pretrained sentence embeddings with a logistic-regression head.