Simpson's paradox: the campaign that won every segment and lost overall
Campaign A beats B among new visitors and among returning visitors, and loses the total. The table with the numbers, the mix shift that does it, and the one question that tells you which number to report: one page.
Get the free PDF
One page, print-ready, free to share. No signup needed.
Same 2,400 visitors. Campaign A won among new visitors, A won among returning visitors, B won overall. All three statements are true, from one table. This page gives the table with numbers that actually reverse, the mix shift that does it, and the one question that tells you which of the two numbers to put on the slide. The print-ready A4 PDF is at the bottom.
The table
| Segment | Campaign A | Campaign B |
|---|---|---|
| new | 60 / 1,000 = 6.0% | 10 / 200 = 5.0% |
| returning | 50 / 200 = 25% | 220 / 1,000 = 22% |
| overall | 110 / 1,200 = 9.2% | 230 / 1,200 = 19.2% |
A wins new. A wins returning. B wins overall, by ten points.
Why
- The mix: B was shown to 1,000 returning visitors, A to 200. Same total, opposite split.
- Returning visitors convert about four times more than new ones, whatever the campaign. The segment sets the rate, not the ad.
- The overall row is a weighted average, and the weights are the segment sizes. B's 19.2% is mostly a count of returning visitors.
Print the n next to every percentage. A 25% on 200 next to a 22% on 1,000 is the warning sign.
Which number
- The segment was set before the ad ran (new vs returning, country, device): trust the split. A is the better campaign; the total is telling you about the routing.
- The segment was caused by the ad (clicked vs did not click, visited pricing vs did not): trust the total. Splitting on an effect of the campaign hides the campaign.
- Unknown: report both, with n, and say the mix differs.
Ask how visitors ended up in each campaign. If the answer is "returning customers get B", you have your answer.
The trap: which number to report
| The segment is | Report | Because |
|---|---|---|
| set before the ad: new vs returning | per segment: A | the mix is routing, not the ad |
| caused by the ad: clicked vs not | the total | splitting on an effect hides it |
| unknown | both, with n | and say the mix differs |
The same reversal shows up outside marketing: a hospital with a worse overall mortality rate because it takes the harder cases, a sales rep with a lower overall close rate because they get the enterprise leads. The question is always the same: was the group decided before the thing you are measuring, or by it?
The quiz
Which campaign is better, A or B?
segment A B
new 6.0% 5.0%
60/1000 10/200
returning 25% 22%
50/200 220/1000
overall 9.2% 19.2%
110/1200 230/1200
A. B's overall lead is a mix effect: B was shown to 1,000 returning visitors, who convert at 22 to 25% under either campaign, A to only 200. Within each segment A converts more. The split is the right read because new vs returning was decided before anyone saw an ad. If the segment were something the campaign caused, the total would be the number to trust.
Related: the percent-change trap has the other ways a comparison of two rates goes wrong.
Frequently asked questions
What is Simpson's paradox?
How can A win both segments and lose overall?
Which number should I report, the split or the total?
How do I spot it before it reaches a slide?
Get the free PDF
One page, print-ready, free to share. No signup needed.