How long should you run an A/B test?

Decide the duration before you start

The length of a test is not something you discover while it runs. You calculate it up front, from five things:

• Your current conversion rate on the page you're testing.
• The smallest uplift worth detecting (the minimum detectable effect).
• Your traffic: how many visitors enter the test per day.
• The number of variants: each extra variant needs its own share of traffic.
• Confidence and power: usually 95% and 80%.

Put these into the A/B test calculator and you get the visitors you need per variant and the number of weeks to run.

An example

A product page converts 3% of visitors to add-to-cart. You want to detect a 10% relative uplift (3.0% to 3.3%), at 95% confidence and 80% power. You need about 53,200 visitors per variant, so around 106,400 for a simple A/B test. With 5,000 visitors a day, that's 22 days, which rounds up to 4 full weeks.

The minimum detectable effect has the biggest influence. Same page, same settings:

Halving the effect you want to detect roughly quadruples the traffic you need. That's why small tweaks on low-traffic pages rarely give a clear answer.

Always run in full weeks

People buy differently on a Monday morning than on a Saturday evening. If a test runs from Tuesday to Friday, the result only describes weekday visitors. Run in full weeks, at least one, so every day of the week is in the data in equal measure.

Don't stop when it first looks significant

Checking results every day and stopping as soon as you see "95% significant" feels efficient, but it inflates false positives. Early in a test, results swing a lot. If you stop at the first good-looking moment, you'll often ship a "winner" that was just noise. Then the uplift disappears after launch.

Set the sample size and end date before launch, and judge the test only when it reaches them. Looking during the test is fine, to catch bugs. Deciding during the test is the problem.

Bayesian testing doesn't remove this. You still need a plan for when you'll decide and what result is good enough.

Too long is also a problem

Very long tests have their own issues. People clear cookies and get counted twice, campaigns and seasons change who visits, and the business waits weeks for an answer. As a rule of thumb, if a test needs more than six weeks, change the plan:

• Test a bolder change that could plausibly move the number more.
• Move the test to a page or step with more traffic.
• Measure a metric closer to the change (add-to-cart instead of purchase, for example), with revenue as a check.

Quick checklist

1. Calculate the sample size and duration before launch.
2. Round the duration up to full weeks.
3. Check that tracking works in every variant on day one.
4. Don't stop early, and don't extend the test just because it's "almost significant".
5. Write down the result, including when it's flat or negative.