Two fully specified Bayes factors for hypothesis testing and sensitivity analysis in process tracing

Abstract: Fairfield and Charman (2022) propose using a Bayes factor to summarize process tracing evidence, but they require researchers to specify the probability of evidence by hand, and this has drawn concern about bias (Zaks 2021). In this paper, we present a solution by deriving such probabilities directly from two fully specified generative models of observation tailored to process-tracing research designs. Our fully specified Bayes factors enable researchers to report how much observation bias a positive conclusion can absorb before flipping in favor of the rival, taking dependence on smoking gun weight into consideration as well. In practice, this means that final conclusions are driven by sensitivity tests more than by Bayes factors themselves. To show the usefulness of our approach we apply the framework to six recent process-tracing studies published in top political science journals.