Data Scientist · #048 · September 9, 2026 · 1 min read
How should you handle missing data? 5 strategies beyond dropna()
How to measure the damage, the three kinds of missingness, when dropping rows is fine, what to fill with, and the two imputers that beat the mean: missing data on one page.
Get the free PDF
One page, print-ready, free to share. No signup needed.
dropna() is not a strategy. The holes in your data are information, and the way you handle them decides whether your model learns the dataset or your cleaning habits. One page. The print-ready A4 PDF is at the bottom.
First, look
df.isna().sum(): count per column.df.isna().mean(): share per column.- Sort by that: worst columns first.
The three kinds
- MCAR: truly random, rare.
- MAR: explained by other columns.
- MNAR: the value itself hides. High earners skip the income question.
Drop
dropna(): rows, if very few.dropna(axis=1): columns that are mostly empty.thresh=8: keep rows with at least 8 real values.
Fill
fillna(median): the skew-proof default.fillna(mode): categoricals.ffill(): time series only.
Smarter
# the two-line upgrade over plain fillna
df['age_missing'] = df['age'].isna().astype(int)
df['age'] = df['age'].fillna(df['age'].median())
# leakage-proof version: learn the median on train only
med = train['age'].median()
test['age'] = test['age'].fillna(med)
KNNImputer: neighbors vote.IterativeImputer: a model per column.was_missing = 1: flag it and keep it. The fact a value was missing often predicts better than the fill.
The trap: not-really-random
| Looks like | Is often | So |
|---|---|---|
| random NaNs | MAR: explained by another column | impute using that column |
| missing income | MNAR: high earners skip it | model it, add a flag |
| whole block empty | a broken join or export | fix upstream, not fillna |
Interview phrasing worth memorizing: dropping MNAR rows biases the sample toward people with nothing to hide.
Frequently asked questions
What are MCAR, MAR and MNAR?
The three kinds of missingness. MCAR (missing completely at random) means the holes have no pattern, which is rare. MAR (missing at random) means the missingness is explained by other columns you have, so you can impute using them. MNAR (missing not at random) means the value itself drives the hole, like high earners skipping the income question: the dangerous one, because dropping those rows biases the sample.
When is it okay to just drop rows with missing values?
When very few rows are affected and the missingness looks random: dropna() on 0.5% of rows is fine. Drop a column instead when it is mostly empty. For anything in between, use thresh= to keep partially complete rows, and never drop MNAR rows without saying so, because that silently biases the dataset.
What should you fill missing values with?
Median for skewed numeric columns (the mean fills every hole with outlier-flavored nonsense), mode for categoricals, forward-fill only for time series. Smarter options: KNNImputer lets similar rows vote, IterativeImputer models each column from the others, and in every case an extra was_missing flag column often predicts better than the filled value itself.
How do you avoid data leakage when imputing?
Learn the fill values on the training set only, then apply them to the test set: compute the median on train, fillna with it on test. Fitting an imputer on the full dataset lets test-set information leak into training, which inflates your scores. A sklearn Pipeline does the right thing automatically inside cross-validation.
Get the free PDF
One page, print-ready, free to share. No signup needed.