Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
140 tokens/sec
GPT-4o
8 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

IncomeSCM: From tabular data set to time-series simulator and causal estimation benchmark (2405.16069v3)

Published 25 May 2024 in cs.LG and stat.ME

Abstract: Evaluating observational estimators of causal effects demands information that is rarely available: unconfounded interventions and outcomes from the population of interest, created either by randomization or adjustment. As a result, it is customary to fall back on simulators when creating benchmark tasks. Simulators offer great control but are often too simplistic to make challenging tasks, either because they are hand-designed and lack the nuances of real-world data, or because they are fit to observational data without structural constraints. In this work, we propose a general, repeatable strategy for turning observational data into sequential structural causal models and challenging estimation tasks by following two simple principles: 1) fitting real-world data where possible, and 2) creating complexity by composing simple, hand-designed mechanisms. We implement these ideas in a highly configurable software package and apply it to the well-known Adult income data set to construct the IncomeSCM simulator. From this, we devise multiple estimation tasks and sample data sets to compare established estimators of causal effects. The tasks present a suitable challenge, with effect estimates varying greatly in quality between methods, despite similar performance in the modeling of factual outcomes, highlighting the need for dedicated causal estimators and model selection criteria.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. Adult. UCI Machine Learning Repository, 1996. DOI: https://doi.org/10.24432/C5XW20.
  2. Causal discovery for fairness. In Workshop on Algorithmic Fairness through the Lens of Causality and Privacy, pages 7–22. PMLR, 2023.
  3. The medkit-learn (ing) environment: Medical decision modelling through simulation. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2021.
  4. Causalml: Python package for causal machine learning. arXiv preprint arXiv:2002.11631, 2020.
  5. Vincent Dorie. Npci: Non-parametrics for causal inference, 2016. URL https://github. com/vdorie/npci, 2016.
  6. Automated versus do-it-yourself methods for causal inference: Lessons learned from a data analysis competition. Statistical Science, 34(1):43–68, 2019.
  7. The case for evaluating causal models using interventional measures and empirical data. Advances in Neural Information Processing Systems, 32, 2019.
  8. Emil Julius Gumbel. Statistical theory of extreme values and some practical applications: a series of lectures, volume 33. US Government Printing Office, 1954.
  9. Atlantic causal inference conference (acic) data analysis challenge 2017. arXiv preprint arXiv:1905.09515, 2019.
  10. Miguel A Hernán. Comment: Spherical cows in a vacuum: Data analysis competitions for causal inference. Statistical Science, 34(1):69–71, 2019.
  11. Estimating causal effects from epidemiological data. Journal of Epidemiology & Community Health, 60(7):578–586, 2006.
  12. A second chance to get causal inference right: a classification of data science tasks. Chance, 32(1):42–49, 2019.
  13. Target trial emulation: a framework for causal inference from observational data. Jama, 328(24):2446–2447, 2022.
  14. Jennifer L Hill. Bayesian nonparametric modeling for causal inference. Journal of Computational and Graphical Statistics, 20(1):217–240, 2011.
  15. Rubin causal model. In Microeconometrics, pages 229–241. Springer, 2010.
  16. Causal inference in statistics, social, and biomedical sciences. Cambridge university press, 2015.
  17. Edward H Kennedy. Towards optimal doubly robust estimation of heterogeneous causal effects. Electronic Journal of Statistics, 17(2):3008–3049, 2023.
  18. Adcb: An alzheimer’s disease simulator for benchmarking observational estimators of causal effects. In Conference on Health, Inference, and Learning, pages 103–118. PMLR, 2022.
  19. Metalearners for estimating heterogeneous treatment effects using machine learning. Proceedings of the national academy of sciences, 116(10):4156–4165, 2019.
  20. Robert J LaLonde. Evaluating the econometric evaluations of training programs with experimental data. The American economic review, pages 604–620, 1986.
  21. Causal decision trees. IEEE Transactions on Knowledge and Data Engineering, 29(2):257–271, 2016.
  22. Causal effect inference with deep latent-variable models. Advances in neural information processing systems, 30, 2017.
  23. Preserving causal constraints in counterfactual explanations for machine learning classifiers. arXiv preprint arXiv:1912.03277, 2019.
  24. Mathematica. 2022 ACIC Data Challenge. https://acic2022.mathematica.org/, 2022.
  25. Fair inference on outcomes. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
  26. Judea Pearl. Causality. Cambridge university press, 2009.
  27. Scikit-learn: Machine learning in python. the Journal of machine Learning research, 12:2825–2830, 2011.
  28. Estimating individual treatment effect: generalization bounds and algorithms. In International conference on machine learning, pages 3076–3085. PMLR, 2017.
  29. Dowhy: An end-to-end library for causal inference. arXiv preprint arXiv:2011.04216, 2020.
  30. On the fairness of causal algorithmic recourse. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pages 9584–9594, 2022.
  31. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228–1242, 2018.
  32. Ganite: Estimation of individualized treatment effects using generative adversarial nets. In International conference on learning representations, 2018.
  33. A causal framework for discovering and removing direct and indirect discrimination. arXiv preprint arXiv:1611.07509, 2016.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets