S Shopify Owners NewsletterDaily operational intelligence for Shopify store owners

A/B Test Significance Calculator

Updated .

Before you ship a winning variant, check whether the difference is real or just noise. This calculator runs a two-proportion z-test on your two variants and returns the p-value, the confidence achieved, and the sample size you'd need to detect the observed lift reliably.

Your numbers
visitors
orders
visitors
orders
%
Result
—
 

    Send me the test result

    We'll email the full statistics so you can attach them to the test write-up.

    How this is calculated

    This runs a two-proportion z-test, the standard method for comparing conversion rates between two groups. It asks a narrow question: if the two variants were truly identical, how often would you see a difference at least this large purely by chance? That probability is the p-value, and confidence is simply one minus it.

    The sample size figure works backwards from the effect you actually observed, at your chosen confidence and 80% statistical power. If it is far above the visitors you have so far, your test was underpowered from the start — the lift you are looking at may be real, but this test cannot tell you.

    One methodological warning that matters more than the arithmetic: checking results repeatedly and stopping as soon as significance appears inflates your false positive rate well beyond the threshold you set. Decide the sample size before you start, run to it, then look.

    p̄ = (conversions A + conversions B) ÷ (visitors A + visitors B) SE = √(p̄ × (1 − p̄) × (1/nA + 1/nB)) z = (rate B − rate A) ÷ SE p-value = 2 × (1 − Φ(|z|)) Confidence = 1 − p-value

    Questions merchants ask

    What does 95% confidence actually mean?

    That if the two variants performed identically, you would see a difference this large or larger less than 5% of the time by chance. It is not the probability that variant B is better, which is a common misreading.

    How long should I run a test?

    Until you reach the planned sample size, and for whole weeks. Traffic behaves differently on weekends and paydays, so a test that runs Tuesday to Friday measures the week as much as the variant.

    Can I stop as soon as it hits significance?

    No. Peeking and stopping on the first significant result substantially raises your false positive rate, because you are giving the test many chances to cross the line by chance. Fix the sample size in advance.

    Should I test conversion rate or revenue per visitor?

    Revenue per visitor is usually the better business metric, since a variant can lift conversion while lowering order value. This calculator tests conversion rate; treat a conversion win as a signal to check the revenue impact before rolling out.

    Estimates only. Enter your own processing rates and costs — nothing here is pulled from a platform price list, so it stays accurate as fees change.