Designing and Deploying Online Field Experiments

Abstract: Online experiments are widely used to compare specific design alternatives, but they can also be used to produce generalizable knowledge and inform strategic decision making. Doing so often requires sophisticated experimental designs, iterative refinement, and careful logging and analysis. Few tools exist that support these needs. We thus introduce a language for online field experiments called PlanOut. PlanOut separates experimental design from application code, allowing the experimenter to concisely describe experimental designs, whether common "A/B tests" and factorial designs, or more complex designs involving conditional logic or multiple experimental units. These latter designs are often useful for understanding causal mechanisms involved in user behaviors. We demonstrate how experiments from the literature can be implemented in PlanOut, and describe two large field experiments conducted on Facebook with PlanOut. For common scenarios in which experiments are run iteratively and in parallel, we introduce a namespaced management system that encourages sound experimental practice.

• The paper introduces PlanOut, a domain-specific language designed to separate online experiment design from application code, improving the deployment and analysis of field experiments.
• PlanOut supports various complex experimental designs beyond simple A/B tests, such as factorial and within-subjects setups, facilitating deeper understanding of causal mechanisms.
• The system manages random assignments and namespaces to ensure experimental integrity at scale across different platforms and concurrent experiments, promoting agile experimentation culture.

Designing and Deploying Online Field Experiments: An Overview

The proliferation of randomized field experiments as a methodology for decision-making in online services has highlighted the need for robust frameworks to streamline their design, deployment, and analysis. The paper "Designing and Deploying Online Field Experiments" addresses this need through the introduction of PlanOut, a language developed to separate the complexities of experimental design from application code, thereby facilitating the efficient execution of online experiments. By doing so, it empowers researchers and practitioners to derive more generalizable and scientifically rigorous insights from their experimentation efforts.

The PlanOut Language

Central to the paper is the introduction of PlanOut, a domain-specific language engineered to encapsulate experimental designs as scripts that succinctly describe random assignment procedures. PlanOut's significance lies in its ability to support both straightforward A/B tests and more intricate experimental frameworks such as factorial designs, within-subjects experiments, and conditional logic procedures. Such designs not only allow for the comparison of design alternatives but also enable a deeper understanding of causal mechanisms that influence user behavior.

Key Functionalities

PlanOut's utility is demonstrated through its implementation of experiments from both prior research and large-scale deployments at Facebook. For instance, assignments made through built-in operators such as uniformChoice and weightedChoice enable the precise manipulation of variables across user interfaces. Furthermore, the ability to conditionally assign variables and nest experimental units—such as pairing viewers with stories—illustrates PlanOut's flexibility in handling complex experimental needs.

Random Assignment and Infrastructure

The paper delineates how PlanOut manages random assignments using platform-independent hashing methods to ensure deterministic yet random mappings of units to conditions. This approach is pivotal for maintaining the integrity of experiments distributed across multilayered service architectures and different programming environments.

Additionally, the concept of namespaces is introduced to manage experiments that manipulate overlapping sets of parameters. By segmenting experimental populations and utilizing a centralized management system, PlanOut mitigates the risks of interference among concurrent experiments, thereby maintaining the statistical soundness of results.

Practical Implications and Future Directions

The practical implications of deploying PlanOut extend beyond individual experiments to broader organizational practices. The toolkit encourages the decomposition of large changes into smaller components that can be iteratively tested, fostering an agile experimentation culture. This approach not only augments the precision of experimental outcomes but also enhances the cumulative scientific knowledge generated from these endeavors.

The paper positions PlanOut as a catalyst for evolving the nature of online experimentation. It supports a transition from merely optimizing for immediate outcomes to conducting experiments that yield insights into underlying behavioral mechanisms. Such insights are critical for long-term strategic decision-making and fostering innovation in design principles.

Conclusion

In conclusion, the paper contributes significantly to the methodological toolkit available for online field experiments. By providing a structured and scalable approach to experimentation, PlanOut enhances both the robustness and applicability of experimental research in online environments. Future research could further explore the integration of PlanOut with emerging technologies and analytical frameworks, thus broadening the horizons of what can be accomplished through online experimentation. The systematic deployment of experiments at scale not only solidifies PlanOut as a pivotal tool for practitioners but also as a bridge connecting experimental rigor with practical application.