Skip to content

A/B Test Calculator

BusinessOperations and HR
Variant A (control)
Variant B (challenger)
Test settings
Confidence level
Two-sided p-value —
Result
—
Conversion rate A
—
Conversion rate B
—
Relative uplift of B
—
z-score
—
Observed confidence (1 − p)
—

An A/B test calculator checks whether the gap between two conversion rates is larger than chance would explain. Enter visitors and conversions for each variant. For example, 100 of 1,000 against 130 of 1,000 (10% vs 13%) gives z = 2.10 and a two-sided p-value of about 0.036, so B wins at 95% confidence but not at 99%.

About this tool

Marketers, product managers and growth teams run split tests on landing pages, emails, checkout flows and ad creatives, then have to decide whether variant B really beat the control or just had a lucky week. This calculator takes the visitor and conversion counts for both variants and runs a pooled two-proportion z-test. It reports the two-sided p-value, the z-score, each conversion rate, the relative uplift of B over A and a plain verdict at the confidence level you choose, along with the working behind each number. One limit: the test assumes you fixed the sample size in advance. Checking the result every day and stopping the moment it turns significant inflates false positives well beyond the nominal 5%.

How to use it

  1. Enter the control

    Type the visitors and conversions recorded for variant A, the version you already run.

  2. Enter the challenger

    Type the visitors and conversions for variant B over the same period.

  3. Pick a confidence level

    Choose 90%, 95% or 99%. Higher levels demand a larger z-score before declaring a winner.

  4. Read the verdict and working

    Check the p-value, uplift and the listed steps showing the pooled rate, standard error and z.

Examples

10% vs 13% on 1,000 visitors each

Visitors
1000
Conversions
100
Visitors
1000
Conversions
130
Confidence level
95%

Result Two-sided p-value: 0.0355
Result: B wins (significant)
Conversion rate A: 10%
Conversion rate B: 13%
Relative uplift of B: 30%
z-score: 2.103
Observed confidence (1 − p): 96.45%

  1. Rate A = 100 ÷ 1000 = 10%; Rate B = 130 ÷ 1000 = 13%
  2. Pooled rate = (100 + 130) ÷ (1000 + 1000) = 0.115
  3. SE = √(0.115 × 0.885 × (1/1000 + 1/1000)) = 0.014267
  4. z = (0.13 − 0.1) ÷ 0.014267 = 2.103
  5. Two-sided p = 2 × (1 − Φ(|z|)) = 0.0355; critical |z| at 95% = 1.96

A three-point lift on 1,000 visitors per arm gives z ≈ 2.10 and p ≈ 0.036, enough to clear the 95% bar.

5.0% vs 5.3% on 5,000 each

Visitors
5000
Conversions
250
Visitors
5000
Conversions
265
Confidence level
95%

Result Two-sided p-value: 0.4973
Result: Not significant
Conversion rate A: 5%
Conversion rate B: 5.3%
Relative uplift of B: 6%
z-score: 0.679
Observed confidence (1 − p): 50.27%

  1. Rate A = 250 ÷ 5000 = 5%; Rate B = 265 ÷ 5000 = 5.3%
  2. Pooled rate = (250 + 265) ÷ (5000 + 5000) = 0.0515
  3. SE = √(0.0515 × 0.9485 × (1/5000 + 1/5000)) = 0.00442
  4. z = (0.053 − 0.05) ÷ 0.00442 = 0.679
  5. Two-sided p = 2 × (1 − Φ(|z|)) = 0.4973; critical |z| at 95% = 1.96

A 6% relative uplift looks attractive, but with a p-value near 0.5 it is indistinguishable from noise at this sample size.

How it is calculated

z = (p_B − p_A) ÷ √(p̂ × (1 − p̂) × (1/n_A + 1/n_B)), p̂ = (c_A + c_B) ÷ (n_A + n_B), p-value = 2 × (1 − Φ(|z|))

p_A, p_B
Conversion rate of each variant, conversions ÷ visitors
c, n
Conversions and visitors per variant
p̂
Pooled conversion rate under the assumption that A and B are equal
Φ
Standard normal cumulative distribution function

Under the null hypothesis both variants share one true rate, so the standard error uses the pooled rate. The z-score counts how many standard errors separate the observed rates. The two-sided p-value is the chance of a gap at least that large if there were no real difference. The result is significant when |z| reaches 1.645, 1.960 or 2.576 for 90%, 95% or 99% confidence. Φ is evaluated with the Abramowitz and Stegun 7.1.26 approximation, accurate to about 1.5 × 10⁻⁷.

When not to use it

  • Metrics that are averages, such as revenue per visitor, need a t-test rather than a test on proportions.
  • Tests with three or more variants need a correction for multiple comparisons.
  • Very small samples with under five conversions per arm call for Fisher's exact test.

Common mistakes

  • Stopping the test the first day the p-value dips under 0.05 instead of at the planned sample size.
  • Counting the same user several times as separate visitors when the unit of randomisation is the user.
  • Reading a p-value of 0.03 as a 97% chance that B is better; it is not a probability of the hypothesis.

Frequently asked questions

What does the p-value mean in an A/B test?

It is the probability of seeing a difference at least as large as the one observed if A and B truly converted at the same rate. A p-value of 0.036 means a gap like 10% versus 13% on 1,000 visitors each would appear by chance in roughly 3.6% of identical experiments.

Why does a large uplift sometimes show as not significant?

Significance depends on sample size as well as the size of the lift. A 6% relative uplift from 5.0% to 5.3% on 5,000 visitors per arm gives z ≈ 0.68, far below 1.96. Roughly ten times the traffic would be needed for that gap to become convincing.

Which confidence level should a team pick?

95% is the common default. 90% suits cheap, easily reversed changes where speed matters more than certainty. 99% fits changes that are costly to roll back, such as pricing or checkout redesigns, because it cuts false positives to 1 in 100 at the price of needing more traffic.

Is this a one-sided or two-sided test?

Two-sided. It asks whether B differs from A in either direction, so a challenger that performs significantly worse is flagged as an A win. A one-sided p-value would be half the reported value, but it ignores the possibility of harm.

Do both variants need the same number of visitors?

No. The standard error uses 1/nA + 1/nB, so unequal splits such as 80/20 are handled. Equal splits are simply the most efficient: for a fixed total, a 50/50 allocation gives the smallest standard error.

Why use a pooled standard error?

The null hypothesis says both variants share one conversion rate, so the best estimate of that rate combines all conversions over all visitors. Using that pooled rate is the textbook form of the two-proportion z-test and matches what most testing platforms report.

Cite this tool

Using this tool in an assignment, article or report? Copy a citation, or a link for your website.

APA

Tools Directory Editorial Team. (2026, October 22). A/B Test Calculator. Tools Directory. Retrieved October 27, 2026, from https://tools.milunir.com/business/operations-and-hr/ab-test-calculator/

MLA

Tools Directory Editorial Team. “A/B Test Calculator.” Tools Directory, 22 Oct. 2026, https://tools.milunir.com/business/operations-and-hr/ab-test-calculator/. Accessed 27 Oct. 2026.

Chicago

Tools Directory Editorial Team. “A/B Test Calculator.” Tools Directory. Last modified October 22, 2026. Accessed October 27, 2026. https://tools.milunir.com/business/operations-and-hr/ab-test-calculator/.

Link<a href="https://tools.milunir.com/business/operations-and-hr/ab-test-calculator/">A/B Test Calculator – Tools Directory</a>