Controlled Experiments with Student Participants in Software Engineering: Preliminary Results from a Systematic Mapping Study

Abstract: [Context] In software engineering research, emphasis is given to sound evaluations of new approaches. While industry surveys or industrial case studies are preferred to evaluate industrial applicability, controlled experiments with student participants are commonly used to determine measurements such as effectiveness and efficiency of a proposed approach. [Objectives] In this paper, we elaborate on the current state of the art of controlled experiments using student participants. As student participants are commonly only reluctantly accepted in scientific communities and threats regarding the generalizability are quite obvious, we want to determine how widespread controlled experiments with student participants are and in which settings they are used. [Methods] This paper reports on a systematic mapping study using high-quality journals and conferences from the software engineering field as data sources. We scanned all papers published between 2010 and 2014 and investigated all papers reporting student experiments in detail. [Results] From 2788 papers under investigation 175 report results from controlled experiments. 109 (62.29%) of these controlled experiments have been conducted with student participants. Most experiments used undergraduate student participants, recruited students on a voluntary basis, and set them tasks to measure their comprehension. However, many experiments lack information regarding the students' recruitment and other important factors. [Conclusions] In conclusion, student participation in software engineering experiments can be seen as a common evaluation approach. In contrast, there seems to be little knowledge about the threats to validity in student experiments, as major drivers such as the recruitment are not reported at all.