AnyTool
Your files never leave your device. All processing happens locally in your browser.

How do I know if my A/B test result is statistically significant?

Enter the visitors and conversions for your control (A) and your variant (B) into AnyTool A/B Test Significance Calculator and pick a confidence level (90, 95 or 99%). It runs a frequentist two-proportion z-test: it computes each conversion rate, the pooled proportion, the standard error, the z-score and the p-value, then tells you in plain English whether the difference is significant — for example “Variant B is a statistically significant winner at 95%” or “Not significant yet”. You also get the absolute and relative lift, a confidence interval on the difference in rates and an observed-power note, plus a chart of both rates with confidence whiskers. Everything is computed in your browser, so nothing you type is uploaded.

  • Two-proportion z-test with p-value, z-score and a clear significance verdict
  • Absolute and relative lift plus a confidence interval on the difference
  • Choose 90 / 95 / 99% confidence and one- or two-tailed
  • Pure-SVG chart of both conversion rates with confidence whiskers
  • 100% in your browser — no upload, no signup, works offline

What is

A/B test significance calculator

An A/B test significance calculator decides whether the difference in conversion rate between two variants is likely real or just random noise. It uses a two-proportion z-test: it pools the two samples to estimate a common conversion rate, computes a standard error, divides the observed difference by that error to get a z-score, and converts the z-score to a p-value. If the p-value is below your chosen significance level (α = 1 − confidence), the result is called statistically significant. AnyTool runs the whole test in the browser and also reports lift, a confidence interval on the difference and observed power.

Calculators

Related terms

Two-proportion z-testP-valueConversion rateConfidence intervalStatistical power

Frequently Asked Questions

The calculator pools both groups to get a common rate, finds the standard error, divides the difference in rates by it to get a z-score, and converts that z-score to a p-value with the normal distribution.

AnyTool uses the two-proportion z-test. It first computes the pooled proportion (total conversions divided by total visitors across both groups) and the standard error √[p̂·(1−p̂)·(1/n₁ + 1/n₂)]. The z-score is the difference in the two conversion rates divided by that standard error. The p-value is then the tail probability of a standard normal distribution beyond that z-score — doubled for a two-tailed test, or single-tailed if you are only testing that the variant beats the control. A p-value below your significance level α (for example 0.05 at 95% confidence) means the observed difference is unlikely to be due to random chance.

Statistical significance only says the difference is probably real; business significance asks whether the lift is large enough to be worth shipping.

A result can be statistically significant yet trivial. With a very large sample, even a 0.1% lift can produce a tiny p-value, but that change may not be worth the engineering or risk. Business significance is about effect size: read the lift and especially the confidence interval, not just the p-value. AnyTool shows the absolute and relative lift and a confidence interval on the difference precisely so you can judge whether the real improvement is big enough to matter, rather than declaring victory on a statistically-significant-but-negligible change.

Repeatedly checking and stopping the moment p is below 0.05 — the peeking problem — sharply inflates your false-positive rate, so set a sample size in advance and wait for it.

A fixed-horizon z-test like this one assumes you decide on a sample size and test duration before you start and then look once. If you instead watch the result and stop the instant it crosses significance, you give random fluctuations many chances to cross the line, which can push the true false-positive rate far above the nominal 5%. This is the peeking problem. The honest approach is to compute the sample size you need up front, run the test until you reach it, and only then read the verdict. AnyTool is candid about this and reports observed power as a post-hoc figure for context only.

Yes — every calculation runs in your browser with no server call, and the page works offline after first load.

AnyTool A/B Test Significance Calculator computes the entire two-proportion z-test locally in JavaScript using a pure, unit-tested engine, so the visitor and conversion numbers you enter are never sent to a server, there is no account or tracking, and the tool keeps working without a connection once it has loaded. The normal-distribution p-value uses a numerically stable Abramowitz–Stegun approximation, and results use IEEE-754 double precision, rounded only for display.

Detailed Explanation

Methodology

How the A/B Test Significance Calculator Works

AnyTool A/B Test Significance Calculator runs a frequentist two-proportion z-test entirely in the browser. From the visitors and conversions of the control (A) and variant (B) it computes each conversion rate, then the pooled proportion p̂ = (x₁ + x₂) / (n₁ + n₂) and the standard error under the null hypothesis, √[p̂·(1 − p̂)·(1/n₁ + 1/n₂)]. The z-score is the difference in rates divided by that standard error, and the p-value is the corresponding tail probability of a standard normal distribution — doubled for a two-tailed test, or single-tailed when you are only testing that B beats A. The normal CDF uses a numerically stable Abramowitz–Stegun erf approximation. The result is compared against α = 1 − confidence/100 to decide significance, and a plain-English verdict (for example “Variant B is a statistically significant winner at 95%”) is generated. Everything is computed at full double precision and rounded only for display.

  • Two-proportion z-test with pooled standard error under the null
  • p-value from a stable Abramowitz–Stegun normal CDF approximation
  • One- or two-tailed, at 90 / 95 / 99% confidence
  • Significance decided by comparing the p-value to α = 1 − confidence
  • Pure, unit-testable engine — no DOM, no server
How It Works

Lift, Confidence Interval and Observed Power

Beyond the verdict, the tool reports the metrics that actually drive a decision. Absolute lift is the simple difference in conversion rates (B − A); relative lift is that difference divided by the control rate. A confidence interval on the difference uses the unpooled standard error √[p₁(1 − p₁)/n₁ + p₂(1 − p₂)/n₂] and the critical z for your confidence level; if the whole interval stays on one side of zero, you have a clear directional winner, and if it crosses zero you do not. A pure-SVG chart draws both conversion rates with confidence whiskers so overlap is visible at a glance, and an observed (post-hoc) power figure is shown for context. A small-sample warning appears when any n·p or n·(1 − p) falls below about 10, where the normal approximation becomes unreliable.

  • Absolute lift (B − A) and relative lift versus control
  • Confidence interval on the difference uses the unpooled standard error
  • Interval entirely above or below zero signals a directional winner
  • SVG rate chart with confidence whiskers
  • Small-sample warning when n·p or n·(1 − p) < ~10
Limitations

Significance Is Not Business Significance — and Other Honest Limits

A statistically significant result only means the difference is unlikely to be pure chance; it does not mean the lift is large enough to be worth shipping, so always read the confidence interval and effect size, not just the p-value. The biggest practical pitfall is the peeking problem: this is a fixed-horizon test, so repeatedly checking and stopping the instant p < 0.05 badly inflates the false-positive rate. Decide on a sample size and duration before you start and wait for it. The z-test also assumes random, independent samples and relies on the normal approximation, and it does not account for novelty or primacy effects, seasonality, multiple comparisons across many variants, or differences between user segments. The observed power shown is post-hoc and for context only — plan power up front instead.

  • Statistical significance ≠ business significance — judge effect size too
  • Peeking (stopping the moment p < 0.05) inflates false positives
  • Set the sample size and duration before the test begins
  • Assumes random, independent samples; ignores novelty and segments
  • Observed power is post-hoc context, not a planning substitute
Privacy & Security

Privacy and Offline Use

Because the entire two-proportion z-test runs in the browser, the visitor and conversion counts you enter are never sent to a server, there is no account or tracking, and the page keeps working offline after first load. The inputs and results live only in the page and disappear on reset or reload. The engine is shared and unit-testable, so its behaviour is consistent and verifiable.

A/B significance testing: in-browser (AnyTool) vs typical online calculators
CapabilityAnyToolTypical online calculators
Where it runs100% in your browserOften server-side
Test methodTwo-proportion z-test, statedVaries, rarely explained
Confidence levels90 / 95 / 99%, one- or two-tailedOften fixed at 95% two-tailed
Lift & intervalAbsolute + relative lift, CI on differenceOften p-value only
VisualsSVG rate chart with CI whiskersOften none
Honesty about limitsPeeking + significance≠business called outRarely mentioned
Works offlineYes (PWA)No
Cost / signupFree, no signupOften gated or ad-heavy

AnyTool runs the whole two-proportion z-test locally with a stable, unit-tested engine and uploads nothing.