Statistical significance, explained for marketers

What significance really means

Every test result contains some luck. Even if A and B were exactly the same, one would come out a bit ahead. Statistical significance answers one narrow question: is the difference we see bigger than what luck alone would usually produce?

When a tool says "95% significant", it means that if there were really no difference, you'd see a result this extreme less than 5% of the time. That's it.

What it doesn't mean

• It doesn't mean there's a 95% chance B is better. That's a different question, and only a Bayesian analysis answers it directly. I explain the difference in Bayesian vs frequentist testing.
• It doesn't tell you how big the effect is. A tiny uplift can be significant if you have huge traffic.
• It doesn't mean the result will last. Seasons, campaigns and novelty can all change things later.

Significant isn't the same as important

With enough visitors, a 0.5% uplift can be statistically significant and still not worth the effort to build and maintain. Always look at the size of the effect next to the significance, and translate it into money. My guide on the ROI of A/B testing shows how.

Look at the range, not just the number

A good report shows a confidence interval: for example, "uplift of 4%, likely somewhere between 1% and 7%". That range tells you far more than a single percentage. If it runs from minus 2% to plus 10%, you simply don't know yet.

Why you need enough traffic first

Significance only works if you collect the sample size you planned before launch. Too small a sample and real effects stay invisible. Check yours with the A/B test calculator.

Three traps that create fake winners

1. Peeking and stopping early. Check every day and stop the first time it says 95%, and your real error rate is much higher than 5%.
2. Too many metrics. Look at 20 metrics and one will probably look significant by chance. Pick one primary metric before launch.
3. Too many variants or segments. Every extra comparison is another chance for luck to look like a win.

How I write up a result

"Variant B increased add to cart rate by about 4% (likely between 1% and 7%). The test ran four full weeks and reached the planned sample size. We'll roll it out and expect roughly €X extra revenue a month."

One sentence on the effect and its range, one on whether the test was run properly, one on what you'll do. Anyone in the company can understand that, statistics degree or not.