Machine Learning from First Principles

Project 2 · Part VII: Projects

An SMS Spam Filter

From raw text to a working filter: tokenize, count words, fit Naive Bayes with a few lines of counting, then evaluate the way imbalanced problems demand.

After this project you will be able to

  • Turn raw text into bag-of-words features.
  • Implement multinomial Naive Bayes with sparse matrices.
  • Evaluate with precision, recall, and F₁, and choose a threshold.
  • Compare generative and discriminative classifiers.

Code: projects/p2_spam_filter/. Background: Chapters 6 and 8.

P2.1The Problem

Filter spam text messages. A legitimate message lost to the spam folder is worse than a missed spam, so precision matters. Metrics: precision, recall, F₁ for spam; false positives. Baseline: always "ham" scores 86.6% accuracy with zero recall: accuracy is meaningless here.

P2.2The Data

The UCI SMS Spam Collection (Almeida, Gómez Hidalgo & Yamakami, 2011; CC BY 4.0): 5,574 English messages, 13.4% spam, one per line as label<TAB>text. A stratified 80/20 split gives 4,460 training and 1,114 test messages.

P2.3Building It from Scratch

Tokenize into lower-case runs of letters, digits, currency signs, and apostrophes (spam is full of numbers and prices). Build the vocabulary from training messages only (7,912 words) and store counts in a sparse matrix.

class MultinomialNB:
    def fit(self, X, y, alpha=1.0):
        self.classes = np.unique(y)
        counts = np.vstack([np.asarray(X[y == c].sum(0)).ravel() for c in self.classes])
        self.log_theta = np.log(counts + alpha) - \
                         np.log(counts.sum(1, keepdims=True) + alpha * X.shape[1])
        self.log_prior = np.log([np.mean(y == c) for c in self.classes])
        return self
    def decision(self, X):
        return np.asarray(X @ self.log_theta.T) + self.log_prior
    def predict(self, X):
        return self.classes[self.decision(X).argmax(1)]

Given identical count features, these log-probabilities and predictions match scikit-learn's MultinomialNB exactly (checked by the tests).

P2.4Using Libraries

nb = make_pipeline(CountVectorizer(), MultinomialNB(alpha=1.0))
lr = make_pipeline(TfidfVectorizer(ngram_range=(1, 2), sublinear_tf=True),
                   LogisticRegression(C=20, max_iter=2000, class_weight="balanced"))

P2.5Results and Discussion

Model (test set)PrecisionRecallF₁False positives
Always ham–0.000–0
Naive Bayes, from scratch0.9640.9060.9345
Naive Bayes, scikit-learn0.9710.9130.9414
Logistic regression, TF-IDF0.9720.9260.9484
  • Interpretable weights: the most spam-indicative words are claim, prize, won, 150p, tone, guaranteed, awarded, £1000, ringtone.
  • Thresholds: requiring spam log-odds above 5 removes every false positive here while recall falls from 0.91 to 0.85. Choose the threshold on validation data, then report test results at that fixed value.
  • Generative vs discriminative: logistic regression edges ahead with thousands of examples, but Naive Bayes trains in one counting pass with no tuning.

P2.6Extensions

  1. Plot precision–recall curves and pick a threshold allowing at most 1 false positive per 1,000 ham on validation data.
  2. Add message length, digit count, and URL features.
  3. Study the false negatives.
  4. Use pretrained sentence embeddings with a logistic-regression head.
Machine Learning from First Principles · Edition 2.0 · Companion site to the PDF book.