Application of Propensity Score Models and Causal Estimators in Observational Studies under Model Misspecification

Abstract: Propensity score (PS) methods are widely used in observational studies to reduce confounding and estimate causal treatment effects. However, the validity of PS-based causal estimators depends heavily on correct model specification, and model misspecification may lead to substantial bias and instability. In this study, we systematically evaluate the performance of commonly used causal estimators, including response surface modeling (RSM), inverse probability weighting (IPW), and augmented inverse probability weighting (AIPW), under varying levels of PS and outcome model misspecification. We compare classical logistic regression with several machine learning approaches for PS estimation, including random forests (RF), support vector machines (SVM), and linear discriminant analysis (LDA). Extensive simulation studies were conducted under multiple scenarios defined by combinations of correctly specified and misspecified PS and outcome models, varying sample sizes, and different covariate correlation structures. Estimator performance was assessed using bias, absolute bias, root mean squared error, empirical standard error, and confidence interval width. Results demonstrate that AIPW consistently provides robust and stable estimates across most scenarios due to its doubly robust property, whereas IPW is highly sensitive to PS misspecification and unstable PS estimates produced by flexible machine learning methods. RSM performs well only when the outcome model is correctly specified. Real-world applications using the ACTG175 clinical trial and the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset further illustrate the practical implications of estimator choice and PS modeling strategy. Overall, our findings highlight the importance of integrating flexible machine learning approaches within doubly robust frameworks to improve causal effect estimation in observational studies.

• The paper demonstrates that doubly robust AIPW yields lower bias and RMSE under model misspecification compared to IPW and RSM.
• It highlights IPW's vulnerability to high variance when using machine learning-based propensity score methods, especially under severe misspecification.
• Findings emphasize that larger sample sizes and stronger covariate correlations improve estimator reliability, though they do not fully overcome misspecification issues.

Performance of Propensity Score Models and Causal Estimators in Observational Studies under Model Misspecification

Introduction

This study systematically evaluates the robustness and reliability of causal effect estimators in observational studies under varying degrees of model misspecification, focusing specifically on the implications of different propensity score (PS) modeling strategies. The analysis encompasses classical parametric methods and several machine learning approaches for constructing the PS, scrutinizing their effectiveness when interfaced with key causal estimators such as inverse probability weighting (IPW), augmented inverse probability weighting (AIPW), and response surface modeling (RSM). The investigation is grounded in both simulated and real-world data settings, including analyses leveraging the ACTG175 clinical trial and Alzheimer's Disease Neuroimaging Initiative (ADNI) data. Emphasis is placed on estimator bias, variance, and robustness, with particular attention to the performance breakdowns induced by PS and outcome model misspecification.

Causal Estimation Framework and Misspecification

The estimation framework is rooted in the Rubin-Neyman potential outcomes paradigm. The estimand of interest is the average treatment effect (ATE), requiring identification assumptions: SUTVA, conditional exchangeability, and positivity. ATE estimation is encoded through RSM, IPW, and AIPW:

• RSM yields unbiased estimates only under correct outcome model specification.
• IPW consistency is contingent on correct PS model specification; susceptibility to high variance arises from instability in the estimated PS, especially under flexible, high-variance machine learning estimators.
• AIPW is doubly robust, offering consistency if either the PS or the outcome model is correctly specified. However, finite-sample instability and inflated variance may emerge in practical implementations.

Simulation Study: Design and Empirical Insights

Comprehensive Monte Carlo simulations were conducted to extensively characterize estimator behaviors across four model specification regimes: full specification, misspecified PS only, misspecified outcome only, and dual misspecification. Each regime was evaluated at two sample sizes ($n=200, 1000$) and two covariate correlation structures ($\rho=0.2, 0.7$). PS estimation was performed using logistic regression (LR), random forests (RF), linear discriminant analysis (LDA), and SVM. Confounder omission and interaction mis-modeling introduced misspecification.

Across all 16 scenarios, estimator performance was quantified by bias, absolute bias, RMSE, empirical SE, and confidence interval width. Notably, IPW demonstrates substantial sensitivity to PS model selection, amplifying bias when utilizing RF or SVM for PS estimation. In contrast, AIPW retains robust performance due to its doubly robust architecture, with low bias and RMSE observed as long as one nuisance model is correctly specified. RSM is reliable solely under correct outcome model specification.

When both nuisance models are misspecified, only AIPW’s bias and RMSE remain comparatively lower, underscoring the necessity of doubly robust approaches in realistic data environments prone to misspecification. Increasing $n$ and $\rho$ universally improves estimator stability and concentration around the true ATE.

Figure 1: Boxplots of treatment effect estimator distributions ($n=200$) indicate severe IPW instability under machine learning–driven PS estimation, while AIPW retains concentration near the true effect.

Figure 2: Estimator performance with larger $n$ ($n=1000$) shows reduction in variance and improved estimator reliability, particularly for doubly robust approaches.

Real-World Applications

Analysis of the ACTG175 Trial

The ACTG175 analysis serves as a calibration benchmark, exploiting randomized design to validate estimation strategies. Baseline comparability is confirmed, and the outcome is a change in CD4 count at 96 weeks. All methods—IPW, AIPW, and RSM—produce highly concordant, statistically significant treatment effects, demonstrating estimator validity when PS adjustment is unnecessary due to balance achieved by design.

Figure 3: All estimation strategies yield consistent, positive treatment effects on CD4 count change, with minimal variation across PS models—demonstrating estimator agreement under randomized allocation.

ADNI Observational Study

The ADNI data provide a more challenging setting. The exposure is a clinical diagnostic status (LMCI/AD vs. CN), and outcome is ADAS13 score change. Here, estimates diverge across strategies:

• IPW consistently indicates significant exposure effects, robust in sign but not necessarily in magnitude or inferential validity due to potential residual confounding and limited overlap.
• AIPW yields attenuated estimates with confidence intervals crossing zero, reflecting partial bias mitigation via outcome model information.
• RSM produces near-zero effects, showcasing high model dependence.

  Figure 4: IPW yields significant negative exposure effects on ADAS13 change, but AIPW and RSM produce attenuated, nonsignificant estimates, highlighting estimator sensitivity to model choice under strong confounding and limited overlap.

Practical and Theoretical Implications

The results explicitly highlight several critical implications:

• IPW’s susceptibility to PS misspecification by machine learning methods such as RF and SVM is pronounced, resulting in substantial estimator instability. Extreme PS values produce large weights, increasing both variance and bias.
• AIPW robustly integrates flexible, high-capacity nuisance modeling within a framework that tames the adverse impact of misspecification. This is consistent across all simulation and real-world settings.
• RSM, while performant under ideal conditions, is of limited practical utility given the pervasiveness of outcome model misspecification in high-dimensional or complex observational data.
• Larger sample sizes and higher covariate correlations enhance estimator performance, but cannot fully mitigate the consequences of severe misspecification.

In modern causal inference applications—especially in regulatory, clinical, and high-dimensional biomedical data—the imperative is to balance flexibility in PS construction with the robustness of the estimation architecture. Doubly robust estimators, especially when coupled with flexible machine learning for PS or outcome modeling, provide an empirically validated strategy for reliable effect estimation.

Directions for Future Research

The limitations outlined include moderate-dimensional settings, assumed absence of unmeasured confounding, and simplified outcome models. Extensions can include ultra-high dimensional covariate spaces, richer nonlinearities, or the use of variable selection and stability selection methods to pre-regularize nuisance function estimation. Integrating targeted learning and ensemble frameworks in even broader semiparametric architectures, and developing diagnostic and sensitivity tools for overlap and model dependence in practical settings, remain priority directions.

Conclusion

This study delivers detailed empirical evidence that, under PS or outcome model misspecification—especially when machine learning based PS estimation is employed—AIPW offers the most robust and reliable inference for causal effects in observational studies. Machine learning models for PS are advantageous only within doubly robust frameworks; direct insertion into IPW can severely degrade estimator properties. These findings should inform both practical analysis pipelines and methodological development for causal inference in complex, real-world datasets.