How experimentation makes money, and how to prove it

Experimentation is one of the few marketing activities where you can measure the value almost directly. But it's also easy to oversell. I've seen dashboards that add up every winning test and end up claiming more extra revenue than the company actually made. So let's do it properly.

Three ways testing makes you money

1. Winners. A change that lifts conversion or order value keeps earning after you ship it.
2. Losses you avoided. Every idea that tests negative is a change you didn't roll out to everyone. A redesign that would have cost you 3% of sales is expensive, even if nobody notices it was stopped.
3. Better decisions. Teams stop arguing about opinions and spend their budget on what works. That's harder to put a number on, but it's often the biggest gain.

How I value a winning test

Start with the revenue that goes through the part of the site you tested. Then be conservative on purpose.

An example. Say €200,000 a month in revenue goes through your product pages, and a test shows a 4% uplift.

Why halve it? The uplift you measure in a test is an estimate with a range around it, and winning tests tend to look a bit better than they really are. You can also use the lower end of the confidence interval instead. Why six months? Effects fade as the site, the offer and the competition change. Counting forever is how ROI reports lose credibility.

How I value the losses you avoided

Same logic, the other way round. If a new checkout tested 3% worse on €200,000 a month, testing it saved you around €6,000 a month you would have lost by launching it blind. I report these separately from the winners, because they're real but they feel less tangible to most people.

ROI for the whole programme

Now put the costs next to it: people, tools, design and development time. A simple yearly example:

• 20 tests run, 5 of them clear winners (many tests won't win, and that's normal)
• those 5 winners together are worth €90,000, valued the conservative way above
• total cost of the programme: €40,000

ROI = (value minus cost) ÷ cost = (€90,000 minus €40,000) ÷ €40,000 = 125%. And that's before counting the losses you avoided.

Mistakes that make ROI look better than it is

• Adding up all the uplifts. Five tests of 5% each don't add up to 25%. Effects overlap and some cancel out.
• Counting the effect forever. Pick a period and stick to it.
• Using the measured peak. Take the conservative number, not the best day on the dashboard.
• Forgetting the cost of building it. Development time is part of the investment.
• Counting winners that never shipped. A winner sitting in the backlog earns nothing.

How to show it's real

The strongest proof I know is a holdout group: keep a small share of visitors, say 5%, on the old version of the site for a quarter. If the people who see all your winners convert better than the holdout, you've shown the combined effect in your own data. Also check that the trend shows up in business numbers, revenue and orders, not only in your testing tool.

Where to start if you want results

1. Fix your tracking first. You can't measure ROI on numbers you don't trust.
2. Test where the money is: pages with lots of traffic, close to the purchase.
3. Test bold ideas. Tiny tweaks rarely move revenue enough to see.
4. Prioritise with evidence, for example with ICE or PIE.
5. Ship winners fast, and keep a simple log of value and cost per test.

If you want help setting this up, that's exactly what my A/B testing programme is for.