The Illusion of Learning from Observational Data: An Empirical Bayes Perspective

Abstract: Randomized experiments have long been the gold standard for scientists seeking to learn about cause and effect. When randomized experiments are infeasible, scientists often resort to observational studies, which are widely available and often large but rely on untestable assumptions that, when violated, may result in biased estimates. Uncertainty about bias leads to a phenomenon known as the illusion of learning from observational research (Gerber, Green and Kaplan, 2004a): absent prior information about bias, observational results cannot meaningfully contribute to the estimation of a causal parameter. To shatter the illusion, we take an empirical Bayes perspective. We show that the distribution of observational biases can be learned from calibration studies-experiments that target a causal effect that is known a priori to be zero. Calibration identifies the distribution of observational bias and allows observational studies to inform the estimation of causal parameters via empirical Bayes shrinkage. We formalize the illusion phenomenon in an empirical Bayes setting and show that, with an increasing number of calibration and observation studies, both the bias distribution and the causal effect can be consistently recovered. We illustrate our method through a simulation study and a semi-synthetic application based on Ferraro and Miranda (2013)'s water-usage experiment.

• The paper shows that without prior bias knowledge, observational data do not reduce bias or uncertainty in causal effect estimates.
• It employs a hierarchical Gaussian model and negative control calibration to rigorously correct biases in observational studies.
• Simulations and semi-synthetic experiments confirm that the calibrated empirical Bayes estimator achieves vanishing risk with more studies.

The Empirical Bayes Illusion in Observational Causal Inference

Introduction and Problem Statement

The paper "The Illusion of Learning from Observational Data: An Empirical Bayes Perspective" (2604.08853) rigorously examines the integration of observational studies with experimental data for causal effect estimation, with a particular focus on empirical Bayes (EB) methodologies. The central claim is that, absent prior knowledge about the bias inherent in observational effect estimates, classical Bayesian or EB approaches can result in the illusion of learning: observational data appear to provide information, but in fact contribute neither bias reduction nor increased precision to target causal estimates. The work precisely formalizes this phenomenon and offers a calibrated empirical Bayes (CEB) solution leveraging negative-control (calibration) studies, thereby establishing conditions under which observational studies can indeed refine causal inference.

Formal Models and the Illusion Mechanism

The authors develop a hierarchical Gaussian model to describe the scenario in which a single experimental estimate and a collection of observational estimates are available for the same causal effect. Observational estimates are subject to study-specific biases, assumed unknown and potentially nonzero. The baseline Bayesian analysis with an improper prior on the effect and a flat prior on biases yields the result that the observational data are uninformative for the causal effect: the posterior mean and variance for $\theta$ are determined by the experimental data alone.

This is demonstrated as follows: with independent priors for the true effect $\theta$ and the biases $b_j$, the likelihood for the observed data leads to a marginalization over $b_j$ that, with a flat prior, simply integrates out the biases, leaving the experimental data as the sole information source for $\theta$. This phenomenon, previously discussed in the political science literature (cf. Gerber et al., 2004), is rigorously generalized with an explicit hierarchical model.

Empirical Bayes: Zero-Mean vs Estimable Mean Bias

The paper investigates the use of EB to estimate distributions of study-specific biases. When biases are modeled as i.i.d. from a zero-mean normal with unknown variance, EB estimation of the variance parameter with many observational studies enables shrinkage of the experimental estimate toward the aggregate observational estimate. The risk (mean squared error) of the resulting estimator vanishes as $J \to \infty$, demonstrating that observational data can, in this restricted setting, contribute meaningfully to inference and reduce uncertainty.

However, the authors stress that the zero-mean assumption is generally not credible, especially when all observational biases could be systematically positive or negative. Allowing both the mean and variance of the bias distribution to be empirical Bayes-estimated, the illusion returns: the marginal likelihood is optimized when the mean bias simply matches the offset between experiment and observation, rendering the observational data irrelevant. The posterior mean for $\theta$ defaults to the experimental value, but the posterior variance is artificially reduced, producing unfounded overconfidence.

Calibration Studies: Breaking the Illusion

To address this, the authors introduce calibration studies—observational analyses of "negative control" interventions with known null effects. By estimating the empirical distribution of bias from these calibration studies, it becomes possible to estimate both the mean and variance of the bias distribution robustly. This enables a calibrated empirical Bayes procedure for the target effect: observational studies are appropriately corrected for bias distribution learned from negative controls, and the combined estimator subsequently enjoys vanishing risk as both the number of observational and calibration studies increase.

The model with calibration is visualized as an expanded hierarchical graphical model capturing experimental, observational, and calibration components; its precise form immediately enables principled bias correction and uncertainty quantification.

Evaluations: Simulation and Real Data

Simulation Results

The paper presents simulations confirming theoretical expectations. The calibrated EB estimator achieves mean squared error that shrinks as $\mathcal{O}(1/J)$ with the number of observational studies, provided sufficient calibration studies. This behavior is robust to effect size and experimental/observational noise. Notably, the estimator's efficiency substantially exceeds that of the experiment-alone estimator for large $J$, provided bias is calibrated.

Figure 2: Top left: MSE of the calibrated empirical Bayes estimator $\hat \theta_{CEB}$ versus number of studies $\theta$0. MSE decays $\theta$1, validating vanishing risk claims.

Semi-Synthetic Real Data

A large-scale field experiment on water consumption [Ferraro et al., 2013] is repurposed to generate experimental, observational (confounded), and calibration (pseudo-treatment with known null) replicates in a semi-synthetic design. The alternative models are assessed as follows:

• Reduced experimental posterior: Wide and diffuse, high uncertainty.
• Illusion posterior: Artificially concentrated, centered at the reduced experiment, with poor credibility coverage.
• Calibrated EB posterior: Centered near the full experimental oracle, with credible intervals matching nominal confidence levels and greatly improved precision.

  Figure 1: Posterior distributions for $\theta$2 (causal effect) under three models. The Calibrated EB posterior (purple) aligns closely with the "oracle" estimate and is well-calibrated; the Illusion posterior is overconfident and fails to overlap.

Negative Controls and Internal Calibration

The framework generalizes to settings with "internal" calibration, i.e., paired negative control studies using the same observational design—critical in complex applied settings. The flexible hierarchical model (see Figure 4) formalizes how such internal calibration structures can be embedded into the EB framework, extending bias correction strategies to diverse applied domains.

Figure 5: Hierarchical model with internal calibration studies: calibration estimates are structurally paired with corresponding observational studies, enabling design-specific bias modeling.

Theoretical, Methodological, and Practical Implications

The analysis provides a precise answer to when observational studies can inform causal inference:

• Without any knowledge (or data-driven proxy) of the bias distribution, observational studies cannot improve causal effect inference beyond the experiment.
• Introducing empirically validated structured assumptions (e.g., calibration studies for bias estimation) enables principled combination—provided calibration studies share the same bias structure as the observational studies.

This undermines claims that the sheer volume of observational data compensates for their biased nature, unless supported by negative control evidence.

Methodologically, the CEB framework forms a recipe for the design and analysis of large-scale multi-source causal investigations in the absence of extensive experimental evidence. Practically, it provides rigorous statistical protection against spurious precision and unfounded effect estimation in high-dimensional observational settings.

Future Directions

Possible future research avenues include:

• Nonparametric EB estimation for bias distributions beyond Gaussian assumptions (cf. [Soloff et al., 2025]).
• Application of the framework to more complex outcome models; integration into semi-parametric and high-dimensional settings.
• Automated methods for constructing and validating appropriate calibration studies in domains with heterogeneous observational designs.
• Analysis of the trade-offs between experimental, observational, and calibration sample allocations.

Conclusion

This work delivers a mathematically rigorous foundation for understanding and overcoming the "illusion of learning" from observational data in causal inference via empirical Bayes. Introducing calibration studies as critical objects for learning the bias distribution renders observation-based causal estimation statistically valid and efficient. These results decisively clarify the circumstances under which observational data genuinely enhance inference about causal effects and provide an actionable blueprint for future empirical research and methodological development in both statistical and AI settings.