Randomized experiment

In science, randomized experiments are the experiments that allow the greatest reliability and validity of statistical estimates of treatment effects. Randomization-based inference is especially important in experimental design and in survey sampling.

Overview[edit]

In the statistical theory of design of experiments, randomization involves randomly allocating the experimental units across the treatment groups. For example, if an experiment compares a new drug against a standard drug, then the patients should be allocated to either the new drug or to the standard drug control using randomization.

Randomized experimentation is not haphazard. Randomization reduces bias by equalising other factors that have not been explicitly accounted for in the experimental design (according to the law of large numbers). Randomization also produces ignorable designs, which are valuable in model-based statistical inference, especially Bayesian or likelihood-based. In the design of experiments, the simplest design for comparing treatments is the "completely randomized design". Some "restriction on randomization" can occur with blocking and experiments that have hard-to-change factors; additional restrictions on randomization can occur when a full randomization is infeasible or when it is desirable to reduce the variance of estimators of selected effects.

Randomization of treatment in clinical trials pose ethical problems. In some cases, randomization reduces the therapeutic options for both physician and patient, and so randomization requires clinical equipoise regarding the treatments.

Logging: user interactions can be logged reliably.

Number of users: large sites, such as Amazon, Bing/Microsoft, and Google run experiments, each with over a million users.

Number of concurrent experiments: large sites run tens of overlapping, or concurrent, experiments.

[5]

Robots, whether from valid sources or malicious internet bots.

web crawlers

Ability to ramp-up experiments from low percentages to higher percentages.

Speed / performance has significant impact on key metrics.^[6]

[3]

Web sites can run randomized controlled experiments ^[2] to create a feedback loop.^[3] Key differences between offline experimentation and online experiments include:^[3]^[4]

Empirical evidence that randomization makes a difference[edit]

Empirically differences between randomized and non-randomized studies,^[14] and between adequately and inadequately randomized trials have been difficult to detect.^[15]^[16]

A/B testing

Allocation concealment

Random assignment

Randomized block design

Randomized controlled trial

Caliński, Tadeusz & Kageyama, Sanpei (2000). . Lecture Notes in Statistics. Vol. 150. New York: Springer-Verlag. ISBN 978-0-387-98578-7.

Block designs: A Randomization approach, Volume I: Analysis

Caliński, Tadeusz & Kageyama, Sanpei (2003). . Lecture Notes in Statistics. Vol. 170. New York: Springer-Verlag. ISBN 978-0-387-95470-7.

Block designs: A Randomization approach, Volume II: Design

(September 1988). "Telepathy: Origins of Randomization in Experimental Design". Isis. 79 (3): 427–451. doi:10.1086/354775. JSTOR 234674. MR 1013489. S2CID 52201011.

Hacking, Ian

Hinkelmann, Klaus; (2008). Design and Analysis of Experiments, Volume I: Introduction to Experimental Design (Second ed.). Wiley. ISBN 978-0-471-72756-9. MR 2363107.

Kempthorne, Oscar

(1992). "Intervention experiments, randomization and inference". In Malay Ghosh and Pramod K. Pathak (ed.). Current Issues in Statistical Inference—Essays in Honor of D. Basu. Institute of Mathematical Statistics Lecture Notes - Monograph Series. Hayward, CA: Institute for Mathematical Statistics. pp. 13–31. doi:10.1214/lnms/1215458836. ISBN 978-0-940600-24-9. MR 1194407.