ICE, PIE or RICE: how to prioritise A/B test ideas

Why bother scoring ideas?

Traffic is limited, so every test you run blocks another one. Without some kind of system, the loudest voice or the most senior person in the room decides what gets tested. A simple score doesn't make the decision for you, but it puts the reasoning on the table where everyone can argue with it.

ICE: Impact, Confidence, Ease

Give each idea a score from 1 to 10 for three questions. How big could the effect be? How sure are we it'll work? How easy is it to build? Then average them.

It's fast. You can get through a backlog of 30 ideas in an hour. The weak spot is Confidence, which quietly turns into gut feeling unless you tie it to evidence. More on that below.

PIE: Potential, Importance, Ease

Here you score how much room for improvement a page has, how valuable its traffic is and how easy the test would be. PIE was built to rank pages and areas rather than single ideas.

I find it most useful at the start of a project, when the question is "checkout, product page or homepage?". It doesn't look at how strong the evidence is behind an idea, though.

RICE: Reach, Impact, Confidence, Effort

RICE comes from product teams. The score is Reach × Impact × Confidence ÷ Effort, where reach is the number of people affected in a period and effort is measured in time.

Because it uses real units, it's handy when tests compete with other product work for the same developers. The catch is that there are more numbers to guess, so it takes longer and can feel more precise than it really is.

Side by side

How I make the scores less subjective

The framework matters less than how you fill it in. I give Confidence points for evidence, not for opinions:

• the problem shows up in analytics, as a clear drop off: plus 1
• customers mention it in surveys, interviews or reviews: plus 1
• you can see it in session recordings or usability tests: plus 1
• a similar test has won before, here or on a comparable site: plus 1

I do something similar for Impact: how much traffic will see the change, and how close the page is to the actual purchase. Once you score like this, meetings stop being about opinions and start being about evidence.

What I'd do

Start with ICE and evidence based confidence. Use PIE when you start on a new site, to choose where to focus. Move to RICE when tests have to compete with the product roadmap. And score your ideas again after every test, because each result changes what you know.

Picked a test? Check it has enough traffic in the A/B test calculator before you build it.