z = (rate B − rate A) ÷ √[pooled rate × (1 − pooled rate) × (1/nA + 1/nB)]How to use this calculator
- Use two independently randomized variants with a binary outcome per participant.
- Enter visitors and converted visitors for A and B from a preplanned analysis point.
- Read the absolute and relative change, two-sided p-value and per-variant intervals alongside the experiment’s assumptions.
Worked example
An illustrative A variant converts 250 of 10,000 visitors (2.5%); B converts 300 of 10,000 (3%). B’s observed rate is 0.5 percentage points higher, or 20% higher relative to A. The two-sided pooled z-test p-value is approximately 0.030623. This does not mean there is a 96.94% probability that B is better.
What statistical significance means here
The null model assumes equal underlying conversion rates. The two-sided p-value describes how unusual a difference at least as extreme as the observed one would be under that model and its assumptions. This tool compares it with a fixed 0.05 threshold; it does not estimate a posterior chance that one variant wins.
Absolute change versus relative uplift
Moving from 2.5% to 3% is a 0.5-percentage-point absolute change and a 20% relative increase. Both labels matter. When A’s observed conversion rate is zero, relative uplift is undefined, so the tool reports the absolute change without inventing an infinite improvement.
Plan the test before reading the result
Use a planned fixed analysis rather than repeatedly checking and stopping after a low p-value appears. The calculator does not adjust for sequential monitoring, many variants or many outcome metrics. It also cannot repair overlapping audiences, selection bias, seasonality or a non-random experiment.
Reference: NIST: comparing two independent proportions.
Common questions
Which statistical test does this calculator use?
A two-sided pooled two-proportion z-test without continuity correction, at a fixed alpha of 0.05. Each variant also has a separate 95% Wilson score interval. Those per-variant intervals are not a confidence interval for the difference or a second significance test.
Why is the p-value unavailable for small counts?
This implementation requires at least 10 observed and expected converters and non-converters in each group before using the normal approximation. For sparse data it shows descriptive rates and Wilson intervals, but withholds the z-test result. Use an appropriate exact method instead.
Can I enter revenue, repeated purchases or fractional conversions?
No. This model expects a whole-number visitor count and a binary converted/not-converted outcome for each participant. Revenue, repeat events, attributed fractional credits and dependent samples require a different analysis.
Does a non-significant result prove the variants are equal?
No. It means this analysis has not established a difference at the stated threshold. Equivalence and non-inferiority require different designs. A statistically detectable change also needs to be large enough to matter commercially.