Classification testing: A new framework for drawing qualitative conclusions from quantitative estimates

Abstract: Social scientists rely on hypothesis testing to support their research conclusions, but the standard tests are designed for testing one hypothesis rather than adjudicating between rival possibilities. We develop a new framework, "classification testing", as an alternative. Instead of selecting one hypothesis to test, a researcher conducting a classification test decides what qualitative distinctions ("classes") are most substantively relevant; the test either assigns the estimand to a class with error control similar to that of a conventional hypothesis test, or declares the result inconclusive. We argue that classification testing is superior to current practice not just when the objective is to adjudicate between rival possibilities but also when there is one research hypothesis to be tested, because classification testing exposes that hypothesis to refutation. We illustrate the framework by applying it to a well-known media experiment and offer an R package to aid in implementation.