Definition
Statistical Significance
The probability that an observed difference between test variants is not due to random chance. Typically requires 95% confidence before declaring a winner.
What does Statistical Significance mean?
Statistical significance is a threshold indicating how confident a team can be that a measured difference between test variants reflects a real effect rather than random sampling variation. It is most commonly expressed as a confidence level, with 95% being the conventional standard in conversion testing.
Why does Statistical Significance matter?
It matters because early or small-sample test results can look dramatic purely by chance, and acting on a result before reaching significance risks rolling out a change that has no real effect, or worse, reverses the intended outcome once exposed to full traffic.
How is Statistical Significance applied in practice?
It is calculated using statistical formulas built into A/B testing tools and calculators, based on sample size, conversion rate, and observed variance, and most testing programs set significance and minimum sample size as fixed rules before a test begins rather than deciding when to stop after seeing results.
Go deeper
Related terms
Where this fits
Statistical Significance sits in the conversion analytics part of the NotiProof resource network.
Read the Conversion Analytics guide