Range restriction, admissions criteria, and correlation studies of standardized tests

Abstract: Recent and influential critiques of standardized testing have noted the existence of non-trivial numbers of successful scientists who received low scores on the GRE. Here we use computer simulations to show that the prevalence of such examples is consistent with the reasonable hypothesis that academic performance depends on multiple variables. We examine the effects of admissions criteria on the observed predictive power of different variables, and show that observed correlations between student performance and student characteristics depend as much on the method of selecting students as on causal relationships detectable in the wider applicant pool. This is an example of the well-known statistical phenomenon of range restriction, and we offer relevant caveats and recommendations for further studies of admissions tests. We also show that the magnitude of range restriction effects depend on how student characteristics are weighted in admissions decisions.