Statistical significance

Statistical significance means a test result meets a preselected rule for rejecting the null hypothesis under the test's assumptions. Error risk matters: NIST describes the significance level in terms of the probability of wrongly rejecting a true null hypothesis.[1] A significant result therefore does not eliminate the possibility of a poor business decision.

For a store, examine the effect's magnitude and uncertainty alongside the test result. With many observations, a small difference can be statistically detectable yet offer little value after implementation costs. Conversely, a non-significant result does not establish that variants are equivalent. Decide what improvement would matter practically in advance, and do not treat repeated interim testing or retrospectively selecting a favourable outcome as neutral analysis.