Skip to content

AB Test Significance Calculator

BusinessOperations and HR
Version A (control)
Version B (variant)
Test settings
Confidence level
Hypothesis
p-value —
Statistically significant?
—
Conversion rate A
—
Conversion rate B
—
Relative uplift of B
—
z-score
—

An A/B test is significant when the p-value of a two-proportion z-test falls below 1 minus your confidence level. With 100 of 1,000 visitors converting on A and 130 of 1,000 on B, z = 2.10 and the two-sided p-value is about 0.036, so B wins at 95% confidence but not at 99%.

About this tool

Marketers, product managers and growth analysts run split tests on landing pages, emails and checkout flows, then need to know whether the winner is real or luck. Enter the visitors and conversions for each version and the calculator returns the p-value, the z-score, both conversion rates and the relative uplift of B over A, with the working shown line by line. Choose a two-sided test to detect any difference or a one-sided test when you only care whether B beats A. The method is the classic normal approximation, so it assumes independent visitors and a fixed sample size decided before the test started; checking results daily and stopping at the first significant reading inflates false positives well beyond the stated level.

How to use it

  1. Enter version A

    Type the visitors and conversions recorded for the control.

  2. Enter version B

    Type the visitors and conversions for the variant.

  3. Pick confidence and hypothesis

    Choose 90%, 95% or 99%, and a two-sided or one-sided test.

  4. Read the verdict

    Compare the p-value with the threshold and review the uplift and z-score.

Examples

10% vs 13% on 1,000 visitors each

Visitors
1000
Conversions
100
Visitors
1000
Conversions
130
Confidence level
95%
Hypothesis
Two-sided

Result p-value: 0.0355
Statistically significant?: Yes, at 95% confidence
Conversion rate A: 10%
Conversion rate B: 13%
Relative uplift of B: 30%
z-score: 2.10

  1. Rate A = 100 ÷ 1000 = 10%; rate B = 130 ÷ 1000 = 13%
  2. Pooled rate p = (100 + 130) ÷ (1000 + 1000) = 0.115
  3. SE = √(p(1 − p)(1/1000 + 1/1000)) = 0.014267
  4. z = (0.13 − 0.1) ÷ 0.014267 = 2.103
  5. Two-sided p-value = 0.0355 < 0.05

A three-point lift on 1,000 visitors per arm gives z ≈ 2.10, enough to clear the 95% bar but not the 99% one.

10% vs 10.4% on 500 visitors each

Visitors
500
Conversions
50
Visitors
500
Conversions
52
Confidence level
95%
Hypothesis
Two-sided

Result p-value: 0.8345
Statistically significant?: No, not at 95% confidence
Conversion rate A: 10%
Conversion rate B: 10.4%
Relative uplift of B: 4%
z-score: 0.21

  1. Rate A = 50 ÷ 500 = 10%; rate B = 52 ÷ 500 = 10.4%
  2. Pooled rate p = (50 + 52) ÷ (500 + 500) = 0.102
  3. SE = √(p(1 − p)(1/500 + 1/500)) = 0.019141
  4. z = (0.104 − 0.1) ÷ 0.019141 = 0.209
  5. Two-sided p-value = 0.8345 ≥ 0.05

A 4% relative uplift on small samples is well within noise; the p-value sits far above 0.05.

How it is calculated

z = (pB − pA) ÷ √(p(1 − p)(1/nA + 1/nB)), where p = (cA + cB) ÷ (nA + nB)

pA, pB
Conversion rates of A and B (conversions ÷ visitors)
cA, cB
Conversions in each version
nA, nB
Visitors in each version
p
Pooled conversion rate under the null hypothesis of no difference
z
Standardised difference, read against the standard normal distribution

Under the null hypothesis both versions share one true rate, estimated by pooling all conversions. The difference in observed rates is divided by its pooled standard error to give z. The two-sided p-value is 2 × (1 − Φ(|z|)); the one-sided value is 1 − Φ(z). The result is significant when the p-value is below 1 minus the confidence level, for example 0.05 at 95%.

When not to use it

  • Revenue per visitor or order value needs a test on means, not proportions.
  • Tests with three or more variants need a correction for multiple comparisons.
  • Groups with fewer than about 10 conversions call for an exact test such as Fisher's.

Common mistakes

  • Stopping the test the moment the p-value dips below 0.05.
  • Counting sessions in one arm and unique users in the other.
  • Choosing a one-sided test after seeing which version is ahead.
  • Reading a non-significant result as proof that the versions perform the same.

Frequently asked questions

What does the p-value tell me in a split test?

It is the probability of seeing a difference at least this large if both versions truly convert at the same rate. A p-value of 0.03 means such a gap would arise by chance about 3 times in 100 identical tests.

Should I pick a one-sided or two-sided test?

Two-sided detects B being better or worse and is the safer default. One-sided only asks whether B beats A, halving the p-value for a positive lift, and is valid only if that direction was fixed before the data came in.

How many visitors do I need per version?

It depends on the baseline rate and the smallest lift worth detecting. Spotting a move from 10% to 11% at 95% confidence and 80% power takes roughly 15,000 visitors per arm; a jump to 13% needs under 2,000.

Why is my uplift large but not significant?

With few visitors the standard error is wide, so even a 20% relative uplift can fall inside random variation. Keep the test running to its planned sample size rather than judging early.

What is the z-score shown with the result?

It is the difference in conversion rates measured in standard errors. Values beyond ±1.96 correspond to two-sided significance at 95%, and beyond ±2.58 at 99%.

Can I use this for click-through rates on ads?

Yes, as long as each impression or visitor is counted once and the outcome is yes or no. Treat clicks as conversions and impressions as visitors for each ad variant.

Cite this tool

Using this tool in an assignment, article or report? Copy a citation, or a link for your website.

APA

Tools Directory Editorial Team. (2026, October 23). AB Test Significance Calculator. Tools Directory. Retrieved October 27, 2026, from https://tools.milunir.com/business/operations-and-hr/ab-test-significance-calculator/

MLA

Tools Directory Editorial Team. “AB Test Significance Calculator.” Tools Directory, 23 Oct. 2026, https://tools.milunir.com/business/operations-and-hr/ab-test-significance-calculator/. Accessed 27 Oct. 2026.

Chicago

Tools Directory Editorial Team. “AB Test Significance Calculator.” Tools Directory. Last modified October 23, 2026. Accessed October 27, 2026. https://tools.milunir.com/business/operations-and-hr/ab-test-significance-calculator/.

Link<a href="https://tools.milunir.com/business/operations-and-hr/ab-test-significance-calculator/">AB Test Significance Calculator – Tools Directory</a>