- The paper demonstrates that online labor markets, exemplified by MTurk, provide affordable, rapid access to diverse subjects for controlled social science experiments.
- The study replicates classic experiments on framing, social preferences, and priming, confirming MTurk’s capability to yield valid experimental outcomes.
- It addresses internal and external validity challenges by using randomization, unique subject accounts, and realistic work tasks, thereby informing future research methods.
The Online Laboratory: Using Internet-Based Labor Markets for Experimentation
In "The Online Laboratory: Using Internet-based Labor Markets for Experimentation," Horton, Rand, and Zeckhauser present a compelling case for the utilization of online labor markets as platforms for conducting experiments in social science research. The authors focus primarily on Amazon's Mechanical Turk (MTurk), highlighting its robust application programming interface (API) and advantageous features for experimental purposes. This paper outlines the benefits, replicates classic experiments, addresses validity concerns, and proposes future directions for the integration of online experiments within the social sciences.
Summary of Key Arguments
The article is structured systematically. The first section details the advantages of online labor markets. These platforms provide immediate access to a large and diverse subject pool, reduce costs, and overcome logistical hurdles associated with lab-based experiments. The ability to recruit and pay subjects easily and securely is emphasized as a significant advantage.
The authors conduct a series of experiments to demonstrate the feasibility and validity of online experimentation. They replicate three classic experiments:
- Framing Effect: Subjects' risk preferences change based on the framing of outcomes as gains versus losses, consistent with Tversky and Kahneman's findings.
- Social Preferences: A prisoner's dilemma game finds non-zero levels of cooperation, indicating pro-social preferences.
- Priming: Religious primes influence cooperation in the prisoner's dilemma game, but only among subjects who believe in God.
Furthermore, they conduct a labor supply experiment to demonstrate the applicability of online platforms. Subjects were asked to transcribe text at varying wage rates, showing that labor supply slopes upward, as economic theory predicts.
Internal Validity
The authors address several threats to internal validity in online experiments:
- Unique and Independent Observations: MTurk's system for creating unique individual accounts mitigates the concern of multiple entries from the same subject.
- Appropriate Assignment: Randomization and blocking techniques ensure that treatment and control groups are comparable.
- Attrition: The potential problem of differential dropout rates is managed by designing experiments that "hook" subjects into participating fully.
- SUTVA: Stability of unit treatments is preserved by designing experiments such that subjects are unaware or incapable of influencing each other’s outcomes due to the anonymous and asynchronous nature of online labor markets.
External Validity
The external validity of online experiments is discussed in terms of representativeness and realism. While acknowledging the limitations posed by the self-selection of participants into online labor markets, the authors argue that for many theory-driven social science phenomena, the diversity and broad base of MTurk subjects offer sufficient generalizability. They emphasize that online labor markets provide a more varied sample than university student samples common in lab-based research.
Experimental Designs and Ethical Considerations
The authors also suggest that the online platform is particularly suited for certain types of experimental designs, such as the experimenter-as-employer paradigm, where researchers can create realistic work tasks that subjects perform without suspecting they are in an experiment. They suggest that existing software tools, like those for survey administration, could be leveraged and enhanced to facilitate more complex experimental designs.
Ethical issues are addressed with a particular focus on maintaining transparency and avoiding deception, which can erode the trust crucial for both experimental and actual marketplace activities. They propose norms like data sharing and detailed documentation to foster reproducibility and collaboration within the research community.
Implications and Future Directions
The practical implications of this research are profound. By adopting online labor markets, social scientists can significantly reduce the cost and time needed to conduct experiments, enabling more iterative and extensive testing of hypotheses. Theoretically, this approach democratizes the experimental process, opening up possibilities for researchers without access to traditional lab resources.
Future developments may include the emergence of sophisticated software to support real-time, interactive games, and other complex experimental setups. Additionally, there is potential for integrating these methods into other fields that rely on experimental data.
The authors conclude by asserting the potential for the online laboratory to play a critical role in advancing social science research, aligning with the optimistic assessors of the NetLab workshop.
In summary, this paper methodically demonstrates that online labor markets offer a viable and, in many ways, superior alternative to traditional laboratory and field experiments. Their findings and propositions for future developments pave the way for exciting advancements in experimental social sciences.