Confidence intervals in plain English
What a 95% interval says and what it does not, how n sets the width (four times the users, half the width), why two overlapping intervals can still differ, and how to report a range instead of a point: one page.
Get the free PDF
One page, print-ready, free to share. No signup needed.
Readout, slide 9: conversion is 3.1%. Plus or minus what? The bracket is the part people argue with, and the part most slides leave off. This page covers what a 95% interval says and what it does not, how the sample size sets the width, why two overlapping brackets can still differ, and how to report a range instead of a point. The print-ready A4 PDF is at the bottom.
What 95% means
[2.7%, 3.5%]: the bracket from 248 conversions out of 8,000 visitors. The point is 3.1%.- 95%: of such brackets, computed the same way on repeated samples, catch the true rate.
- This bracket: holds or does not. There is no 95% about this one; the 95% belongs to the method.
Width
- n = 8,000: plus or minus 0.4 points, [2.7, 3.5].
- n = 500: plus or minus 1.5 points, [1.6, 4.6].
- Four times the n halves the width. Not a quarter: the width goes with the square root of n.
The bracket, six lines
Normal approximation. Fine when the successes and the failures are both above 10.
import math
conv, n = 248, 8_000
p = conv / n # 0.031
se = math.sqrt(p * (1 - p) / n) # 0.0019
lo, hi = p - 1.96 * se, p + 1.96 * se
print(f'{p:.1%} [{lo:.1%}, {hi:.1%}]')
# 3.1% [2.7%, 3.5%]
Report the point, the bracket and the n, in that order, every time: 3.1% [2.7, 3.5], n = 8,000. For small counts (under 10 either way) use the Wilson interval instead; statsmodels.stats.proportion.proportion_confint(conv, n, method='wilson') does it.
Two of them
- A and B overlap: they can still differ.
- B minus A: one bracket on the difference.
With A at 3.1% and B at 3.8%, both on 8,000 visitors:
pa, pb, n = 248 / 8_000, 304 / 8_000, 8_000
se = math.sqrt(pa * (1 - pa) / n + pb * (1 - pb) / n)
d = pb - pa
print(f'{d:+.1%} [{d - 1.96 * se:+.1%}, {d + 1.96 * se:+.1%}]')
# +0.7% [+0.1%, +1.3%]
The brackets [2.7, 3.5] and [3.4, 4.2] overlap. The difference excludes zero. B is ahead.
The trap: three readings of a bracket
| You see | You say | Say instead |
|---|---|---|
| A [2.7, 3.5], B [3.4, 4.2] | they overlap, no difference | B minus A: +0.7 [0.1, 1.3] |
| [2.7, 3.5] | 95% chance the rate is in there | the method is right 95 times in 100 |
| n = 500: [1.6, 4.6] | 3.1% conversion | 3.1%, between 1.6 and 4.6 |
Overlap is not a test. Compute the bracket on the difference: if it excludes zero, B is ahead, overlap or not.
What a p-value adds to this, and what it does not, is on the p-value sheet. This page is the bracket only.
Frequently asked questions
What does a 95% confidence interval mean?
Is there a 95% chance the true value is inside my interval?
How does the sample size change the width?
If two confidence intervals overlap, is there no difference?
Get the free PDF
One page, print-ready, free to share. No signup needed.