Implications of LLM integration for validation and calibration

Determine the implications of integrating Large Language Models (LLMs) into agent-based models for achieving rigorous validation and calibration, including whether and how generative agent-based models can attain operational validity across their intended domains.

Background

A central historical critique of agent-based models (ABMs) concerns difficulties in rigorous calibration and validation against empirical data. The paper argues that while LLMs may improve behavioral realism, it is uncertain whether they help resolve the longstanding validation and calibration challenges. Clarifying this is essential for assessing the scientific utility of generative ABMs.

This open problem is positioned as pivotal for the field’s future: if LLMs do not facilitate robust validation and calibration procedures, generative ABMs may fail to contribute meaningfully to social scientific theory or policy-relevant modeling.

References

However, while LLMs promise to address the first key challenge of ABMs by making agents more realistic, their implications for the second -- rigorous validation and calibration -- remain an open question that is central to the future potential of generative ABMs.

— Do Large Language Models Solve the Problems of Agent-Based Modeling? A Critical Review of Generative Social Simulations  (2504.03274 - Larooij et al., 4 Apr 2025) in Introduction (Section 1)

It is still not clear if two simulation contexts can be considered similar and how one can quantify this similarity.

— Total Simulated Survey Error: Designing and Diagnosing Survey Responses from Large Language Models  (2609.10280 - Sen et al., 9 Sep 2026) in Section 4, “Context Drift Fallacy”

However, according to a recent systematic literature review by Larooij and Törnberg, there is one common problem among these models; namely, verification has remained an unresolved problem for these works due to the reliance on subjective evaluation of believability and the failure to achieve empirical validation at multiple scales.

— Digital Twins for Opinion Dynamics: A Generative LLM Framework for Social Networks  (2609.19913 - Berjawi et al., 17 Sep 2026) in Section 2, Related Work

We distinguish representational adequacy from interpretability and alignment metrics, propose ways to integrate it into simulation research, and pose its measurement as an open problem.

— What People Almost Did: Evaluating LLM Social Simulations Beyond Behavioral Fit  (2609.20055 - Kim et al., 17 Sep 2026) in Abstract

That said, to the extent that LLMs are found to be good approximations for human behavior, which remains an open question, ABMs that rely on LLM agents instead of rules-based agents could prove to be useful.

— Competitive Market Behavior of LLMs  (2609.02580 - Struski et al., 2 Sep 2026) in Section 2, Related Work, paragraph “Agent-Based Models”