Statistical Power & A/B Test Sample Size Calculator: Free Statistical Power & A/B Test Sample Size Calculator tool: calculate required sample sizes and test duration for A/B testing and hypothesis testing..
The Mathematics of Statistical Power & Sample Sizing
In online controlled experiments, clinical trials, and frequentist hypothesis testing, calculating the required sample size before launching a test is essential to prevent false positives and underpowered experiments. Running an A/B test without pre-calculating statistical power is like searching for a needle in a haystack with a flashlight whose battery is dying: even if an effect exists, the test lacks the statistical resolution to prove it beyond random chance.
Hypothesis testing weighs two competing explanations for observed conversion differences:
- Null Hypothesis ($H_0$): The variant has no real effect; any difference in conversion rate is purely due to random binomial sampling noise.
- Alternative Hypothesis ($H_1$): The variant induces a genuine structural change in user behavior equal to or greater than the Minimum Detectable Effect ($delta = p_2 - p_1$).
The Classical Evan Miller Proportion Sizing Formula
For a two-sample proportion Z-test with equal allocation (50/50 split), the required sample size $n$ per variation is governed by:
$n = frac{left(Z_{alpha/2} sqrt{2 bar{p}(1 - bar{p})} + Z_{beta} sqrt{p_1(1 - p_1) + p_2(1 - p_2)}right)^2}{(p_2 - p_1)^2}$
Where:
- $p_1$ = Baseline conversion proportion of control
- $p_2$ = Target conversion proportion of variant ($p_1 times (1 + text{MDE})$)
- $bar{p}$ = Pooled average conversion proportion: $frac{p_1 + p_2}{2}$
- $Z_{alpha/2}$ = Critical value for significance level $alpha$ (1.96 for $alpha = 0.05$ two-tailed)
- $Z_{beta}$ = Critical value for statistical power $1 - beta$ (0.8416 for 80% power, 1.2816 for 90% power)
Step-by-Step Worked Numerical Example
Suppose a growth team wants to test a redesigned pricing page:
- Baseline Conversion Rate ($p_1$): 5.0% = 0.050
- Desired Relative MDE: 10% uplift
- Variant Conversion Rate ($p_2$): $0.050 times 1.10 = 0.055$ (5.5%)
- Effect Size ($delta$): $0.055 - 0.050 = 0.005$
- Significance Level: $alpha = 0.05$ ($Z_{alpha/2} = 1.96$)
- Statistical Power: $1 - beta = 0.80$ ($Z_{beta} = 0.8416$)
- Pooled Proportion ($bar{p}$): $frac{0.050 + 0.055}{2} = 0.0525$
Step 1: Compute variance components:
Pooled Variance Term = $1.96 times sqrt{2 times 0.0525 times (1 - 0.0525)} = 1.96 times sqrt{0.09949} approx 1.96 times 0.3154 = mathbf{0.6182}$
Separate Variance Term = $0.8416 times sqrt{(0.05 times 0.95) + (0.055 times 0.945)} = 0.8416 times sqrt{0.0475 + 0.05198} approx 0.8416 times 0.3154 = mathbf{0.2654}$
Step 2: Combine and square the numerator:
Numerator = $(0.6182 + 0.2654)^2 = (0.8836)^2 approx mathbf{0.7808}$
Step 3: Divide by squared delta:
Denominator = $(0.005)^2 = mathbf{0.000025}$
$n = frac{0.7808}{0.000025} approx mathbf{31,232text{ users per variant}}$
Total Sample Size ($N_{text{total}}$) = $31,232 times 2 = mathbf{62,464text{ total visitors}}$
If the page receives 5,000 visitors per day, the test must run for exactly $62,464 / 5,000 = mathbf{12.5text{ days}}$ (approximately 2 full weeks to balance weekend vs weekday cycle effects).
The Pitfalls of Underpowered Tests: The Winner's Curse
When product teams run experiments with insufficient sample sizes (e.g., stopping after 500 conversions when 3,000 were required), statistical noise can produce apparent 30% uplifts with $p < 0.05$. This phenomenon is known as the Winner's Curse or Type M (Magnitude) Error.
Underpowered studies only achieve statistical significance when random sampling noise happens to align favorably with the variant. As a result, the observed uplift is dramatically exaggerated. When deployed to 100% of production traffic, the supposed 30% revenue gain vanishes completely or turns negative.