The Beta-Bernoulli model in the context of A/B testing.
13 |The success probability distributions in test and control groups.
103 | 104 |Distribution of differences in success probability between test and control groups.
109 | 110 |Posterior probability that the difference lies below the value x.
115 |117 | The average difference between test and control is: 0. The probability that test performs better: 0.5 118 |
119 |This simple calculator uses the Beta-Bernoulli model (a binary outcome model, where the prior for the success probability is a Beta distribution) applied in the A/B testing context, where the goal of inference is understanding the probability that the test group performs better than the control group.
137 |Bayesian inference consists in first specifying a prior belief about what effects are likely, and then updating the prior with incoming data.
138 |For example, if our conversion rate is 5%, we may say that it's reasonably likely that a change we want to test could improve that by 5 percentage points—but that it is most likely that the change will have no effect, and that it is entirely unlikely that the conversion rate will shoot up to 30% (after all, we are only making a small change).
139 |As the data start coming in, we start updating our beliefs. If the incoming data points point to an improvement in the conversion rate, we start moving our estimate of the effect from the prior upwards; the more data we collect, the more confident we are in it and the further we can move away from our prior. The end result is what is called the posterior—a probability distribution describing the likely effect from our treatment.
140 |