The Bootstrapped Robustness Assessment for Qualitative Comparative Analysis

Abstract: Qualitative Comparative Analysis (QCA) has been increasingly used in recent years due to its purported construction of a middle path between case-oriented and variable-oriented methods. Despite its popularity, a key element of the method has been criticized for possibly not distinguishing random from real patterns in data, rendering its usefulness questionable. Critics of the method suggest a straightforward technique to test whether QCA will return a configuration when given random data. We adapt this technique to determine the probability that a given QCA application would return a random result. This assessment can be used as a hypothesis test for QCA, with an interpretation similar to a p-value. Using repeated applications of QCA to randomly-generated data, we first show that generally, the tendency for QCA to return spurious results is attenuated by using reasonable consistency score and configurational N thresholds; however, this varies considerably according to the basic structure of the data. Second, we suggest an application-specific assessment of QCA results, illustrated using the case of Tea Party rallies in Florida. This method, which we coin the Bootstrapped Robustness Assessment for QCA (baQCA), can provide researchers with recommendations for consistency score and configurational N thresholds.