2000 character limit reached
Automated Statistical Model Discovery with Language Models (2402.17879v2)
Published 27 Feb 2024 in cs.LG and cs.CL
Abstract: Statistical model discovery is a challenging search over a vast space of models subject to domain-specific constraints. Efficiently searching over this space requires expertise in modeling and the problem domain. Motivated by the domain knowledge and programming capabilities of large LMs, we introduce a method for LLM driven automated statistical model discovery. We cast our automated procedure within the principled framework of Box's Loop: the LM iterates between proposing statistical models represented as probabilistic programs, acting as a modeler, and critiquing those models, acting as a domain expert. By leveraging LMs, we do not have to define a domain-specific language of models or design a handcrafted search procedure, which are key restrictions of previous systems. We evaluate our method in three settings in probabilistic modeling: searching within a restricted space of models, searching over an open-ended space, and improving expert models under natural language constraints (e.g., this model should be interpretable to an ecologist). Our method identifies models on par with human expert designed models and extends classic models in interpretable ways. Our results highlight the promise of LM-driven model discovery.
- PyMC: a modern, and comprehensive probabilistic programming framework in python. PeerJ Computer Science, 9, 2023.
- GPT-4 Technical Report. 2023.
- Blei, D. M. Build, compute, critique, repeat: Data analysis with latent variable models. Annual Review of Statistics and Its Application, 1(1):203–232, 2014. doi:10.1146/annurev-statistics-022513-115657.
- GPT Takes the Bar Exam. ArXiv, abs/2212.14402, 2022.
- Automated reverse engineering of nonlinear dynamical systems. Proceedings of the National Academy of Sciences, 104:9943 – 9948, 2007.
- A useful method for model-building. Technometrics, 4:301–318, 1962.
- JAX: composable transformations of Python+NumPy programs, 2018. URL http://github.com/google/jax.
- Language models are few-shot learners. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
- Evaluating large language models trained on code. ArXiv, abs/2107.03374, 2021.
- Neural ordinary differential equations. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, NIPS’18, pp. 6572–6583, Red Hook, NY, USA, 2018. Curran Associates Inc.
- Structure discovery in nonparametric regression through compositional kernel search. In Dasgupta, S. and McAllester, D. (eds.), Proceedings of the 30th International Conference on Machine Learning, volume 28 of Proceedings of Machine Learning Research, pp. 1166–1174, Atlanta, Georgia, USA, 17–19 Jun 2013. PMLR.
- PAL: program-aided language models. In Proceedings of the 40th International Conference on Machine Learning, ICML’23. JMLR.org, 2023.
- Inference from iterative simulation using multiple sequences. Statistical Science, 7(4):457–472, 1992. ISSN 08834237.
- Bayesian data analysis, third edition. 2013.
- Church: a language for generative models. In Conference on Uncertainty in Artificial Intelligence, 2008.
- Grosse, R. B. Model selection in compositional spaces. 2014.
- Kidger, P. On Neural Differential Equations. PhD thesis, University of Oxford, 2021.
- Large language models are zero-shot reasoners. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
- Benefits, Limits, and Risks of GPT-4 as an AI Chatbot for Medicine. New England Journal of Medicine, 388(13):1233–1239, 2023. doi:10.1056/NEJMsr2214184. PMID: 36988602.
- Automated model discovery for human brain using constitutive artificial neural networks. Acta Biomaterialia, 160:134–151, 2023. ISSN 1742-7061. doi:https://doi.org/10.1016/j.actbio.2023.01.055.
- Automatic construction and natural-language description of nonparametric regression models. In AAAI Conference on Artificial Intelligence, 2014.
- posteriordb: a set of posteriors for Bayesian inference and probabilistic programming, October 2023.
- Hybrid grammar-based approach to nonlinear dynamical system identification from biological time series. Phys. Rev. E, 73:021912, Feb 2006. doi:10.1103/PhysRevE.73.021912.
- Learning insulin-glucose dynamics in the wild. In Proceedings of the 5th Machine Learning for Healthcare Conference, volume 126 of Proceedings of Machine Learning Research, pp. 172–197. PMLR, 07–08 Aug 2020.
- N-beats: Neural basis expansion analysis for interpretable time series forecasting. ArXiv, abs/1905.10437, 2019.
- Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement. In The Twelfth International Conference on Learning Representations, 2024.
- Graphical representation and stability conditions of predator-prey interaction. American Naturalist, 97:209–223, 1963.
- Code Llama: Open Foundation Models for Code. ArXiv, abs/2308.12950, 2023.
- Sequential monte carlo learning for time series structure discovery. In International Conference on Machine Learning, 2023.
- Toolformer: Language models can teach themselves to use tools. ArXiv, abs/2302.04761, 2023.
- Distilling free-form natural laws from experimental data. Science, 324(5923):81–85, 2009. doi:10.1126/science.1165893.
- Reflexion: language agents with verbal reinforcement learning. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
- An introduction to probabilistic programming, 2021.
- Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing, 27(5):1413–1432, sep 2017. ISSN 0960-3174. doi:10.1007/s11222-016-9696-4.
- von Bertalanffy, L. Problems of organic growth. Nature, 163:156–158, 1949.
- Hypothesis search: Inductive reasoning with language models. In The Twelfth International Conference on Learning Representations, 2024.
- Chain of thought prompting elicits reasoning in large language models. In Oh, A. H., Agarwal, A., Belgrave, D., and Cho, K. (eds.), Advances in Neural Information Processing Systems, 2022.
- Gaussian process kernels for pattern discovery and extrapolation. In Proc. ICML, 2013.
- From Word Models to World Models: Translating from Natural Language to the Probabilistic Language of Thought. ArXiv, abs/2306.12672, 2023.
- A new approach to probabilistic programming inference. In Proceedings of the 17th International conference on Artificial Intelligence and Statistics, pp. 1024–1032, 2014.
- An Empirical Study on Challenging Math Problem Solving with GPT-4, 2023.
- Goal driven discovery of distributional differences via language descriptions. In Thirty-seventh Conference on Neural Information Processing Systems, 2023.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.