Estimating real-world probabilities: A forward-looking behavioral framework

Abstract: We show that disentangling sentiment-induced biases from fundamental expectations significantly improves the accuracy and consistency of probabilistic forecasts. Using data from 1994 to 2017, we analyze 15 stochastic models and risk-preference combinations and in all possible cases a simple behavioral transformation delivers substantial forecast gains. Our results are robust across different evaluation methods, risk-preference hypotheses and sentiment calibrations, demonstrating that behavioral effects can be effectively used to forecast asset prices. Further analyses confirm that our real-world densities outperform densities recalibrated to avoid past mistakes and improve predictive models where risk aversion is dynamically estimated from option prices.