Data Scientist · #043 · September 4, 2026 · 2 min read
How do you run an A/B test without fooling yourself?
What to fix before launch, how to size and split a test, how to read a p-value honestly, and the peeking habit that quietly turns noise into winners.
Get the free PDF
One page, print-ready, free to share. No signup needed.
Most A/B tests do not die of bad math; they die of peeking. Here is the honest version: what to fix before launch, how big and how long, and how to read the result. The print-ready A4 PDF is at the bottom.
Before you start
- One hypothesis, written down first.
- One primary metric: decided, not found afterwards.
- MDE: the smallest lift you actually care about.
Size and duration
- Power = 80%: the convention.
- n per variant: computed from the MDE, upfront.
- Full weeks only: weekends behave differently.
Splitting
- Randomize by user, not by session.
hash(user_id): stable assignment across visits.- A/A test: two identical variants, checks the plumbing.
Reading it
p < 0.05: significant. Maybe real.- The CI of the lift: the honest answer to "how big?".
p = 0.07is not "almost significant". It is not significant.
The pre-launch checklist
- Hypothesis: one sentence, "we believe X will move Y because Z".
- Primary metric: one. Everything else is supporting evidence.
- MDE: the smallest lift worth shipping. It drives the sample size.
- Duration: computed from traffic and MDE, rounded to full weeks.
- Stop rule: the end date. Not the first day p dips under 0.05.
Everything on that list is decided before the first user is bucketed. That is the whole discipline.
Shipping
- Segment once, after the test: not to hunt for a win.
- Log the losers too: keep a test registry.
- Flat is an answer: ship the cheaper variant.
The trap: checking daily until it wins
| You did | What happens | Do instead |
|---|---|---|
| stopped when p hit 0.049 | noise crossed the line first | fixed end date, then read |
| scanned 20 metrics | one wins by pure luck | one primary, declared |
| shipped a day-3 spike | novelty effect fades | full weeks, minimum one |
A 5% false-positive rate is only 5% if you look once, at the end, at one metric. Peek at a flat test 20 times and you get a false winner more than half the time.
Frequently asked questions
What should I decide before launching an A/B test?
Five things, all before the first user is bucketed: one written hypothesis (we believe X will move Y because Z), one primary metric, the MDE (the smallest lift worth shipping), the duration computed from traffic and MDE and rounded to full weeks, and the stop rule. Deciding these upfront is the whole discipline.
How long should an A/B test run?
Long enough to reach the sample size your MDE and 80% power imply, rounded up to full weeks because weekend users behave differently from weekday users. Run at least one full week, and never stop on a day-3 spike: early lifts are often novelty effect, which fades.
Why is peeking at A/B test results a problem?
Because the 5% false-positive rate only holds if you look once, at the end, at one metric. Check a flat test daily and stop the first time p dips under 0.05, and noise will cross the line eventually: peek 20 times and you get a false winner more than half the time. Fix the end date first, then read.
Should I randomize an A/B test by user or by session?
By user, with a stable assignment like hash(user_id), so the same person sees the same variant on every visit. Randomizing by session lets one person land in both groups, which contaminates the comparison. An A/A test, two identical variants, is the cheap way to check the bucketing plumbing before the real test.
Get the free PDF
One page, print-ready, free to share. No signup needed.