Bayesian vs frequentist A/B testing, in plain words

The difference in one sentence each

Frequentist: if there were really no difference between A and B, how surprising would this result be? You get a p value and a yes or no on "significant".

Bayesian: given what we've seen, how likely is it that B is better, and by how much? You get something like "92% chance B beats A".

Neither is more correct. In my experience the things that actually ruin tests are a sloppy setup, too little traffic and stopping too early. Those break both methods equally.

How the frequentist approach works

You decide the sample size up front (the calculator does this for you), you run the test until you hit it, and then you look at the p value. Below your threshold, usually 0.05, and you call it significant.

Here's the part almost everyone gets wrong. A p value is not the chance that B is better. A p value of 0.03 means that if A and B were identical, you'd see a gap this big only 3% of the time. Sounds similar. It isn't.

What I like about it: it's well understood, and the error rates are clear as long as you stick to the plan. What I don't: the results are hard to explain to a marketing manager, and the guarantees fall apart the moment someone stops the test early.

How the Bayesian approach works

You start with a prior, an assumption about which effects are likely (usually a neutral one), and update it as data comes in. At the end you get the chance that B beats A, the expected uplift, and the expected loss if you pick B and turn out to be wrong.

What I like: the output matches how people already think. "There's a 95% chance this is better" needs no explanation in a meeting. Expected loss is also really useful when you're making a business call rather than writing a paper. What I don't: the prior is a choice you have to justify, and a lot of people believe Bayesian means you can peek every day and stop whenever you like.

About that peeking myth

Looking at results is fine with either method. Deciding the moment the numbers look good is the problem. Stop a Bayesian test as soon as "chance to beat" crosses 95% and you'll end up with more false winners than that 95% suggests. So whatever tool you use, agree on the minimum duration, the sample size and the decision rule before you launch. I go into this in how long to run an A/B test.

So which should you use?

Honestly, the method matters less than the habits around it: one main metric, a sample size you set in advance, full weeks, and every result written down, the losers too.