Sampling in Software Engineering Research: A Critical Review and Guidelines (2002.07764v6)

Abstract: Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis. To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.

• The paper reveals a critical lack of probability sampling in SE research, with only about 16% of studies employing it.
• It uncovers a prevalent reliance on non-probability methods and common misconceptions conflating randomness with true representativeness.
• The study offers actionable guidelines for enhancing sampling frameworks, transparency, and overall methodological rigor in empirical software research.

Sampling in Software Engineering Research: A Critical Review

The paper "Sampling in Software Engineering Research: A Critical Review and Guidelines" by Sebastian Baltes and Paul Ralph presents a thorough examination of sampling practices in software engineering (SE) research. The authors express concern over the apparent scarcity of representative sampling within the field, suggesting that it can contribute to a broader generalizability crisis. Their paper synthesizes an extensive review of recent empirical literature, delineating the prevalent sampling strategies, their justifications, and their implications for SE research methods.

Key Findings

Through their critical assessment, Baltes and Ralph highlight several important findings:

1. Limited Use of Probability Sampling: The paper reports that probability sampling is exceedingly rare in SE research. Non-probability sampling techniques, particularly purposive and convenience sampling, dominate, used in approximately 84% of the surveyed studies.
2. Misconceptions About Sampling: Many studies display misunderstandings concerning representativeness, with frequent conflation between randomness and representativeness. The paper underscores that while randomness in sampling is a conventional path to representativeness, it does not equate to it.
3. Sampling Strategy and Research Methodologies: Analysis shows an overwhelming preference for quantitative studies, with mixed-method approaches moderately represented. Over 60% of sampling efforts involve code-based artifacts, with 'experimental tool evaluation' being the most common category of empirical method.
4. Challenges with Sampling Frames: A significant barrier to representative sampling is the lack of adequate sampling frames. The absence of comprehensive, unbiased lists of SE phenomena, like software projects or developers, hinders random sampling efforts.

Implications

Baltes and Ralph propose that this lack in methodological rigor regarding sampling may erode the generalizability of SE research findings. They argue for a balance between qualitative and quantitative methods, emphasizing that representativeness is essential where statistical generalization is a research goal. The paper highlights the necessity for researchers to align their sampling approaches with their philosophical stances and paper objectives.

Recommendations for Improvement

The paper offers detailed guidelines to improve sampling practices and reporting:

• Clarify Research Philosophy: Researchers should state their philosophical stance and explicitly outline if statistical generalization is a paper objective. This contextual clarity aids understanding of the appropriateness of the sampling strategy.
• Representative Sampling Efforts: Future research should prioritize probability sampling where possible, simultaneously acknowledging practical constraints that might necessitate non-probability methods.
• Develop Better Sampling Frames: Collaborative efforts are needed to construct more comprehensive and unbiased sampling frames, which could support more representative, generalizable research.
• Transparent Reporting: The authors advocate for the provision of scripts or detailed algorithms used in sampling to facilitate reproducibility and transparency.

Future Directions

The paper emphasizes the need for methodological sophistication in sampling approaches within SE research. Future efforts could focus on developing curated corpora and improving sampling strategies to match known population parameters. Additionally, expanding the methodological toolkit to include more refined versions of established approaches, such as respondent-driven sampling, could mitigate inherent biases in convenience samples.

In conclusion, this paper identifies critical gaps and establishes a framework for enhancing the methodological foundation of SE research through improved sampling practices. It challenges researchers and reviewers alike to critically reassess assumptions and common practices around sampling, fostering a scientific culture that values transparency and reproducibility in empirical software engineering endeavors.