Guide · A/B testing
A/B testing mistakes that waste your traffic
Traffic is the one thing you can't get back. Every badly run test uses up visitors that could have answered a better question. These are the mistakes I see most.
1. Testing without a hypothesis
"Let's try a green button" isn't a hypothesis. "Visitors don't see delivery times, so showing them near the price will increase add to cart" is. A hypothesis tells you what you'll learn, whatever the result.
2. Not having enough traffic
If the test needs three months to give an answer, it won't give a useful one. Check the numbers before you build anything, with the A/B test calculator.
3. Stopping the moment it looks good
Early results swing a lot. Decide the sample size and duration before launch and stick to it. I explain why in how long to run an A/B test.
4. Picking the winning metric afterwards
If the main metric is flat but "time on page" went up, it's tempting to call it a win. Choose one primary metric before launch, plus a few guardrail metrics to make sure nothing breaks.
5. Skipping QA
A variant that breaks on one browser or doesn't fire tracking will lose for the wrong reasons. Check every variant on the main devices, and check the data on day one.
6. Testing changes nobody will notice
Tiny tweaks need enormous traffic to measure. Most sites get more from bold changes based on research: a clearer offer, a simpler checkout, better product information.
7. Not running full weeks
Weekday and weekend visitors behave differently. Always run whole weeks, at least one, even when the maths says fewer days.
8. Drowning in segments
Slice a result into enough segments and you'll always find one where B "won". Use segments to generate new ideas, not to rescue a test that didn't win.
9. Forgetting to write it down
Six months later nobody remembers what was tested or why. Keep a simple log: hypothesis, screenshots, result, decision. It's how a team actually gets smarter.
10. Never shipping the winners
A winning test that sits in the backlog for months earns nothing. Plan the development time for rollouts as part of the programme, not as an afterthought.
Most of these come down to one thing: deciding the rules before the test starts. If you want help setting those rules up, have a look at my A/B testing programme.
Want help running tests that give clear answers?
In a free 30 minute call we look at your site and traffic together, and I tell you what I'd test first.
Book a free discovery call