Calibrating Bayesian Inference

Abstract: While Bayesian statistics is popular in psychological research for its intuitive uncertainty quantification and flexible decision-making, its performance in finite samples can be unreliable. In this paper, we demonstrate a key vulnerability: When analysts' chosen prior distribution mismatches the true parameter-generating process, Bayesian inference can be misleading in the long run. Given that this true process is rarely known in practice, we propose a safer alternative: calibrating Bayesian credible regions to achieve frequentist validity. This latter criterion is stronger and guarantees validity of Bayesian inference regardless of the underlying parameter-generating mechanism. To solve the calibration problem in practice, we propose a novel stochastic approximation algorithm. A Monte Carlo experiment is conducted and reported, in which we observe that uncalibrated Bayesian inference can be liberal under certain parameter-generating scenarios, whereas our calibrated solution is always able to maintain validity.