Skip to content

Bayesian AB Test Calculator

BusinessOperations and HR
Variant A (control)
Variant B (challenger)
Chance B beats A —
Reading at 95%
—
Chance A beats B
—
Conversion rate A
—
Conversion rate B
—
Relative uplift of B
—
Chance B beats A against decision bands
The chart appears with your result

A Bayesian A/B test reports the probability that variant B converts better than A. With a flat prior, each variant's rate follows Beta(conversions + 1, failures + 1). For 100 of 1,000 visitors on A and 130 of 1,000 on B, the chance that B is better comes out at roughly 98%, above the common 95% decision line.

About this tool

Marketers, product managers and growth teams running landing page or checkout experiments often want a direct answer: how likely is it that the new version is actually better? A frequentist p-value does not say that; a Bayesian posterior does. Enter the visitors and conversions for the control and the challenger, and the calculator shows the chance that B beats A, the reverse chance, both observed conversion rates and the relative uplift, with a scale marking the 5% and 95% decision bands. The probability is computed with an exact sum over Beta functions, switching to a normal approximation only when B has more than about 20,000 conversions. One limit: it assumes a uniform prior and one binary conversion per visitor, so revenue per visitor or repeated purchases need a different model.

How to use it

  1. Fill in variant A

    Enter the visitors (or sessions) shown the control and how many of them converted.

  2. Fill in variant B

    Enter the same two counts for the challenger, using the same definition of a conversion.

  3. Read the probability

    The headline is the chance B has the higher true conversion rate; the scale shows whether it has crossed 95% or fallen under 5%.

  4. Check the uplift and working

    Compare both observed rates and the relative uplift, and read the posterior parameters in the steps.

Examples

50 of 1,000 in both arms

Visitors
1000
Conversions
50
Visitors
1000
Conversions
50

Result Chance B beats A: 50%
Reading at 95%: No clear winner yet
Chance A beats B: 50%
Conversion rate A: 5%
Conversion rate B: 5%
Relative uplift of B: 0%

  1. Posterior A = Beta(50 + 1, 950 + 1) = Beta(51, 951)
  2. Posterior B = Beta(50 + 1, 950 + 1) = Beta(51, 951)
  3. P(B > A) by exact Beta sum = 50%
  4. Observed rates: A 5%, B 5%

Identical data gives identical posteriors, so each variant has an even 50% chance of being the better one.

10% vs 13% on 1,000 visitors each

Visitors
1000
Conversions
100
Visitors
1000
Conversions
130

Result Chance B beats A: 98.2%
Reading at 95%: B is likely better
Chance A beats B: 1.8%
Conversion rate A: 10%
Conversion rate B: 13%
Relative uplift of B: 30%

  1. Posterior A = Beta(100 + 1, 900 + 1) = Beta(101, 901)
  2. Posterior B = Beta(130 + 1, 870 + 1) = Beta(131, 871)
  3. P(B > A) by exact Beta sum = 98.22%
  4. Observed rates: A 10%, B 13%

A 30% relative lift on a thousand visitors per arm puts the chance that B is better above the 95% line.

6 vs 9 conversions out of 200

Visitors
200
Conversions
6
Visitors
200
Conversions
9

Result Chance B beats A: 77.7%
Reading at 95%: No clear winner yet
Chance A beats B: 22.3%
Conversion rate A: 3%
Conversion rate B: 4.5%
Relative uplift of B: 50%

  1. Posterior A = Beta(6 + 1, 194 + 1) = Beta(7, 195)
  2. Posterior B = Beta(9 + 1, 191 + 1) = Beta(10, 192)
  3. P(B > A) by exact Beta sum = 77.74%
  4. Observed rates: A 3%, B 4.5%

B looks 50% better, but with only 15 conversions the posterior curves overlap heavily and no winner can be called.

How it is calculated

P(pB ≥ pA) = Σ from i = 0 to αB − 1 of B(αA + i, βA + βB) ÷ [(βB + i) · B(1 + i, βB) · B(αA, βA)]

αA, βA
Posterior parameters for A: conversions + 1 and non-converting visitors + 1
αB, βB
Posterior parameters for B, built the same way
B(x, y)
The Beta function, computed through log-gamma for stability
pA, pB
The unknown true conversion rates of each variant

Starting from a uniform Beta(1,1) prior, the Bernoulli likelihood updates each variant to a Beta posterior. Evan Miller's closed-form sum gives the exact probability that a draw from B's posterior exceeds a draw from A's. For very large counts the sum is replaced by a normal approximation using each posterior's mean and variance, which agrees to well under 0.1 percentage points at that scale.

When not to use it

  • Metrics that are not yes/no per visitor, such as revenue or time on page.
  • Tests with more than two variants compared at once.
  • Experiments where traffic was not randomly split between the arms.

Common mistakes

  • Stopping the test the first moment the probability touches 95% on a small sample.
  • Counting conversions from one variant by sessions and the other by users.
  • Reading 98% as a 98% uplift rather than a 98% chance that B is better at all.

Frequently asked questions

What prior does the calculator use?

A uniform Beta(1,1) prior for both variants, meaning every conversion rate from 0% to 100% is treated as equally plausible before data arrives. With a few hundred visitors the data dominates and the prior has almost no effect.

How is this different from a p-value?

A p-value is the chance of seeing data this extreme if the variants were identical. The Bayesian number is the chance that B's true rate exceeds A's given the data, which is the question most teams actually ask.

What probability should count as a win?

Many teams use 95%, which the scale marks. A cheap, reversible change might ship at 90%; a costly redesign might wait for 99%. The threshold is a business choice about how often you accept being wrong.

Can I stop the test as soon as B hits 95%?

Bayesian probabilities are less fragile under peeking than p-values, but stopping on the first crossing with small samples still overstates wins. Fix a minimum sample or run for full weekly cycles before deciding.

Why is the uplift shown as n/a?

Relative uplift divides by A's conversion rate. When A has zero conversions that rate is 0%, so the ratio is undefined; the probability is still computed correctly.

Does it need equal traffic in both arms?

No. Each arm's posterior uses its own visitor count, so a 70/30 split works. The smaller arm simply carries more uncertainty, which pulls the probability toward 50%.

Cite this tool

Using this tool in an assignment, article or report? Copy a citation, or a link for your website.

APA

Tools Directory Editorial Team. (2026, October 23). Bayesian AB Test Calculator. Tools Directory. Retrieved October 27, 2026, from https://tools.milunir.com/business/operations-and-hr/bayesian-ab-test-calculator/

MLA

Tools Directory Editorial Team. “Bayesian AB Test Calculator.” Tools Directory, 23 Oct. 2026, https://tools.milunir.com/business/operations-and-hr/bayesian-ab-test-calculator/. Accessed 27 Oct. 2026.

Chicago

Tools Directory Editorial Team. “Bayesian AB Test Calculator.” Tools Directory. Last modified October 23, 2026. Accessed October 27, 2026. https://tools.milunir.com/business/operations-and-hr/bayesian-ab-test-calculator/.

Link<a href="https://tools.milunir.com/business/operations-and-hr/bayesian-ab-test-calculator/">Bayesian AB Test Calculator – Tools Directory</a>