Sample size

Sample size is the number of observational units underlying an analysis or experiment. Those units could be customers, visitors or orders, so identify what is actually being studied. NIST stresses that a required size cannot be determined without assumptions: error risks, variability and the change to be detected all matter.[1] A universal minimum visitor count is therefore no substitute for an appropriate calculation.

For a checkout experiment, decide in advance what improvement would be practically worthwhile and which analysis method suits the outcome. Repeated visits by the same person are not automatically independent participants. Examine representativeness and timing as well as size: heavy promotional traffic may poorly represent normal trading. A large sample cannot repair incorrect assignment, missing purchases or an unsuitable success measure.