The Bayes Factor Reversal Paradox

Abstract: In 1957, Lindley published "A statistical paradox" in Biometrika, revealing a fundamental conflict between frequentist and Bayesian inference as sample size approaches infinity. We present a new paradox of a different kind: a conflict within Bayesian inference itself. In the normal model with known variance, we prove that for any two-sided statistically significant result at the 0.05 level there exist prior variances such that the Bayes factor indicates evidence for the alternative with one choice while indicating evidence for the null with another. Thus, the same data, testing the same hypothesis, can yield opposite conclusions depending solely on prior choice. This answers Robert's 2016 call to investigate the impact of the prior scale on Bayes factors and formalises his concern that this choice involves arbitrariness to a high degree. Unlike the Jeffreys-Lindley paradox, which requires sample size approaching infinity, the paradox we identify occurs with realistic sample sizes.