Class imbalance: 99% accuracy and zero positives caught

The Friday demo: a fraud model, 99% accuracy, the room is pleased. On Monday someone asks how many of last month's frauds it caught. Zero. This page covers why accuracy flatters a model on a rare class, the two numbers to report instead, the three fixes, and the curve to read. The print-ready A4 PDF is at the bottom.

The demo

• Accuracy 99%: say "no" to everyone and you also get 99%.
• Positives caught: zero. The model learned the base rate.

With 1% fraud, a model that never flags anything is right 99 times out of 100. The loss function rewarded it for learning that lesson.

Measure it right

• Recall: of the real positives, how many did the model catch.
• Precision: of the rows it flagged, how many were real.
• The precision-recall curve, over ROC, when positives are rare.

Report both for the minority class. Accuracy can stay in the table, but it is not the headline.

Fix it

• class_weight='balanced': the rare class costs more in the loss.
• Resample: oversample the rare class, undersample the common one, on the training set only.
• Move the threshold off 0.5, and read it off the PR curve.

Weight the rare class, move the threshold

Two lines change. The model pays more for a missed positive, and the cut-off stops being 0.5 by default.

from sklearn.linear_model import LogisticRegression
from sklearn.metrics import classification_report

clf = LogisticRegression(class_weight="balanced",
                         max_iter=1000)
clf.fit(X_train, y_train)
proba = clf.predict_proba(X_test)[:, 1]
pred = (proba >= 0.3).astype(int)
print(classification_report(y_test, pred))

Report recall and precision for class 1, not accuracy. The 0.3 is a starting point: read it off the precision-recall curve with whoever pays for a missed case.

from sklearn.metrics import precision_recall_curve

precision, recall, thresholds = precision_recall_curve(y_test, proba)

Walk the thresholds with the business. At 0.2 the model catches 85% of frauds and one flag in three is real; at 0.5 it catches 40% and two in three are real. Which one is right depends on what a missed fraud costs against what a manual review costs. That is a decision, not a model parameter.

The trap: the metric that cannot fail

Base rate 1% fraud. The model that says no to everyone against a real one.

The say-no model wins on accuracy and catches nobody. With a 1% base rate, any model has to beat 99% before it has done anything at all.

Resampling, done on the right rows

Oversampling (duplicating or synthesising rare rows, SMOTE being the common tool) and undersampling (dropping common rows) both change the training set so the model sees the classes more evenly. Two rules:

• Resample the training split only. Resampling before the split leaks copies of the same row into the test set and the test score becomes fiction.
• Evaluate on the real distribution. The test set keeps its 1%, because that is what production looks like.

Class weights get you most of the same effect without touching the rows, which is why they are the first thing to try.

The full page is a free print-ready A4 PDF: recall, precision, the three fixes, the PR curve.