External validity of large-scale Pop music corpora

Establish the external validity of computational studies of Pop music culture by determining which generalizations are justified from music corpora assembled from hit charts or convenience samples and by addressing the sampling biases inherent in those corpus-construction practices.

Background

The survey identifies questionable external validity as a major open issue in computational research on Pop music culture. Existing corpora are commonly assembled either from commercially successful chart music, often reflecting the cultural background of the United States, or from convenience samples determined by what data researchers can access. These practices can systematically exclude music outside dominant markets, genres, periods, or cultural contexts.

Because large datasets make statistically significant findings more likely, unrecognized sampling biases may produce significant but non-generalizable results. The paper emphasizes the need to appraise and document such biases and to obtain sufficiently large and representative music corpora so that conclusions about Pop music culture can be appropriately generalized.