Why does your model ace the backtest and fail in production?

The backtest said 0.94. Production says 0.51. Nothing about the model changed, so the model was never the problem:

df = df.sort_values("date")
X_tr, X_te = train_test_split(X, shuffle=True)
# backtest 0.94, live 0.51 -- trained on the future

Shuffling a time series hands the model rows from the future of its own test set. The score was fiction from the start.

Three sets, not two

The third set exists because of a subtle failure: every time you change a hyperparameter after reading the test score, you are fitting yourself to the test set. After ten rounds the test number measures your persistence, not the model. Tune on validation. Open test once.

Split first. Always.

The order of operations is the part that silently goes wrong:

from sklearn.pipeline import Pipeline

# 1. split BEFORE anything touches the data
X_tr, X_te, y_tr, y_te = train_test_split(
    X, y, test_size=0.2, stratify=y, random_state=42)

# 2. the scaler learns from train rows only
pipe = Pipeline([("scale", StandardScaler()),
                 ("model", LogisticRegression())])
pipe.fit(X_tr, y_tr)

Fit a scaler on the full dataset and the test mean leaks into training. The symptom is a score that improves while the model worsens, which is the worst possible direction to be wrong in. The Pipeline makes the leak structurally impossible: transforms only ever see what fit received.

Pick the split that matches the data

The group split is the one most people meet too late. If the same customer appears in train and test, the model recognises that customer instead of generalising to new ones, and the validation score inflates for a reason that vanishes with real users.

The set you peeked at

A test set you have read ten times is not a test set. If the tuning loop already burned it, the honest options are to say so, or to collect a fresh holdout. There is no third option where the number becomes trustworthy again.

The takeaway

Three sets, split before any preprocessing, strategy matched to the data, and a test set that gets opened exactly once. The print-ready PDF above fits all of it on one page.