SOCIA-EVO: Automated Simulator Construction via Dual-Anchored Bi-Level Optimization

Abstract: Automated simulator construction requires distributional fidelity, distinguishing it from generic code generation. We identify two failure modes in long-horizon LLM agents: contextual drift and optimization instability arising from conflating structural and parametric errors. We propose SOCIA-EVO, a dual-anchored evolutionary framework. SOCIA-EVO introduces: (1) a static blueprint to enforce empirical constraints; (2) a bi-level optimization to decouple structural refinement from parameter calibration; and (3) a self-curating Strategy Playbook that manages remedial hypotheses via Bayesian-weighted retrieval. By falsifying ineffective strategies through execution feedback, SOCIA-EVO achieves robust convergence, generating simulators that are statistically consistent with observational data. The code and data of SOCIA-EVO are available here: https://github.com/cruiseresearchgroup/SOCIA/tree/evo.

• The paper introduces a dual-anchored bi-level optimization framework that decouples structural simulator refinement from parameter calibration.
• It employs a static Blueprint and a dynamic Strategy Playbook to enforce empirical constraints and mitigate optimization instability.
• Empirical evaluations show statistically significant improvements in simulation fidelity compared to both agentic and specialized simulation systems.

SOCIA-EVO: Automated Simulator Construction via Dual-Anchored Bi-Level Optimization

Introduction and Problem Characterization

The paper "SOCIA-EVO: Automated Simulator Construction via Dual-Anchored Bi-Level Optimization" (2604.17351) addresses the automated construction of high-fidelity simulators from observational data, with an emphasis on distributional fidelity over mere functional correctness. Existing approaches based on LLM-driven code generation are insufficient for this task due to two main long-horizon failure modes: contextual drift (loss of adherence to empirical constraints, leading to hallucinated or invalid simulation logic) and optimization instability (conflation of structural and parametric errors, resulting in oscillatory or ineffective remediation).

SOCIA-EVO introduces an agentic evolutionary framework explicitly designed to mitigate these issues. It does so via three key mechanisms:

1. A static Blueprint encoding immutable empirical and structural constraints derived from data.
2. A bi-level optimization engine that rigorously decouples structural code refinement (outer loop) from continuous parameter calibration (inner loop).
3. A self-curating Strategy Playbook, which accumulates, validates, and down-weights remedial strategies using metric-grounded execution feedback and Bayesian attribution.

   Figure 1: The SOCIA-EVO framework. The process is dual-anchored by a static Blueprint ($\mathcal{B}$) and a dynamic Strategy Playbook ($\mathcal{K}$). A bi-level optimization decouples structural refinement (Outer Loop) from parameter calibration (Inner Loop), leveraging metric-driven feedback to evolve strategies and prevent optimization instability.

Framework Overview

At its core, SOCIA-EVO formalizes simulator construction as a bi-level search in program space. Given an observational dataset $\mathcal{D}_{obs}$ and a task specification $\mathcal{I}$, the framework minimizes the distributional distance $$\mathrm{Dist}(\mathcal{D}_{sim}, \mathcal{D}_{obs})$$ by synthesizing a simulator $S$ parameterized by code structure $P$ and parameters $\theta$. The workflow involves six specialized agents acting in a closed iterative loop: Data Analysis, Code Generation, Simulation Execution, Feedback Generation, Playbook Management, and Iteration Control.

Dual Anchoring Mechanisms

• The Static Blueprint ($\mathcal{B}$): Functions as the immutable reference for simulation topology, agent schemas, exogenous signals, and evaluation metrics. It is established upfront with expert verification, ensuring the solution space respects empirical constraints and domain knowledge.
• The Strategy Playbook ($\mathcal{K}$): Maintains a dynamic set of remedial strategies (structured, evidence-linked hypotheses for repair) with empirical reliability scores (success/failure attributions) maintained via a Bayesian self-curation model. Strategies are prioritized and included in generation prompts via a 0-1 knapsack optimization over token budget, incorporating severity, backlog urgency, and posterior reliability.

Bi-Level Optimization

• Outer Loop (Structural Refinement): The LLM-based Code Generation Agent synthesizes program structure and calibration procedure, conditioned on the Blueprint and prioritized strategies from the Playbook. The agent does not select parameters directly but produces a calibrator that defines the search space.
• Inner Loop (Parameter Calibration): Calibrator executes numerical optimization (e.g., Bayesian optimization, random search) over the parameter space defined in the Blueprint, seeking to minimize a task-specific aggregate loss composed of pre-defined evaluation metrics.

This decoupling ensures that metric deviations attributable to parameter settings do not trigger unnecessary structural code rewrites, directly addressing the prevalent problem of "whack-a-mole" instability observed in prior work.

Evolutionary Loop: Diagnosis, Validation, and Self-Curation

The feedback agent restricts issue diagnosis to Blueprint-authoritative metrics, binding every defect to explicit quantitative evidence. Diagnostic records enforce symptom translation, mechanism localization, high-level remediation, and severity classification. The Playbook's stateful transition system allows only empirically validated strategies to persist; failures are down-weighted via Bayesian updating, and non-resolving or recurring strategies are effectively forgotten, mitigating memory overload and context drift.

Knapsack-based strategy retrieval prevents prompt bloat and enforces high-reliability, high-severity strategies at the boundary of LLM attention, optimizing for effective prompt engineering under strict context budgets.

Empirical Evaluation and Numerical Results

SOCIA-EVO was evaluated on three non-trivial simulation tasks: user modeling, mask adoption in social networks, and out-of-distribution personal mobility generation. Empirical results demonstrate strictly superior performance compared to both generalist agentic frameworks (Reflexion [shinn2023reflexion], Dynamic Cheatsheet, ACE) and specialized simulation systems (YuLan-OneSim, G-Sim).

Key numerical highlights:

• On personal mobility (out-of-distribution), SOCIA-EVO achieves the lowest JSD and Wasserstein Distance (WD), outperforming G-SIM-SBI and all generalist frameworks.
• On mask adoption, SOCIA-EVO achieves an RMSE of 0.07, compared to the next-best 0.11 (G-SIM-SBI) and 0.16/0.26 (YuLan-OneSim/Reflexion), establishing statistically significant improvements ($\mathcal{K}$0) across tasks.

Ablation studies confirm the necessity of each core component: removing the bi-level optimization (w/o inner) or the Blueprint (w/o $\mathcal{K}$1) causes the most severe regression, with oscillatory search and loss of context integrity.

Optimization Dynamics

Convergence analysis reveals a step-wise error reduction with early iterations correcting global structure and late iterations exhibiting metric trade-offs. The cumulative recurrent error (CRE) metric shows a 76% reduction in repeated mistakes by the fifth iteration, validating the efficacy of bayesian self-curation and Playbook memory management in suppressing repeated failures. Analysis of Issue Resolution Rate (IRR) identifies a transition from "rapid correction" of shallow errors to tackling deep structural bottlenecks.

Generalization and Backbone Portability

Replacing the proprietary GPT-5.1 backbone with open instruction-tuned models (Llama-3.3-70B, Qwen3-Next-80B) demonstrates that SOCIA-EVO's performance and error-suppression dynamics generalize to open-source LLMs. Qwen3 matches and, in some settings, outperforms GPT-5.1, with both models showing monotonic reduction in recurrent errors. The main variance lies in the quality of playbook strategies and the fidelity of diagnosis-to-repair mapping, not in the architectural backbone itself.

Structural Refinement over Pure Parameter Tuning

Qualitative code analysis on the Mobility task shows that iterative improvements involve concrete mechanism-level changes—such as elevating context features from initialization-only to persistent daily conditioning—rather than mere parameter retuning. This evidences that the bi-level outer loop is utilized to realize architectural refinements aligned with empirical feedback.

Theoretical and Practical Implications

SOCIA-EVO establishes that robust simulator construction from data requires more than prompt engineering or code completion by LLMs. Persistently dual-anchored memory (Blueprint + Playbook), rigorous decoupling of structure and calibration, and empirical falsification of repair hypotheses are essential to support distributional fidelity and optimization stability. Practically, the approach enables the construction of robust simulators for scientific, behavioral, and societal phenomena with minimal one-time expert input.

Theoretically, this demonstrates a translation of scientific modeling principles (evidence-grounded falsification, hypothesis validation, distributional calibration) into agentic LLM-based workflows, with implications for broader automated modeling, self-improving systems, and code synthesis under stochastic and underdetermined settings.

Limitations and Future Directions

The paper identifies several open challenges:

• Backbone dependence: Ultimate performance is still bounded by LLM capability; highly novel domains may yield weaker or semantically invalid simulators.
• Scope: The approach is best suited for tasks amenable to structural refinement plus bounded calibration; strong multi-agent strategic reasoning or real-time adaptation requires further advances in memory and causal modeling.
• Access and reproducibility: Strongest results rely on commercial LLMs, although open LLMs provide competitive results in some domains.

Further research directions may explore integration with more expressive causal modeling, richer multi-agent environments, and explicit handling of real-world noise and non-stationarity in the data-generating process.

Conclusion

SOCIA-EVO sets a new paradigm for automated simulator construction, architecting the search for executable simulation code as a dual-anchored, bi-level evolutionary process. By foregrounding data-grounded specification, persistent memory with empirical validation, and strict decoupling of logic and calibration, SOCIA-EVO consistently achieves higher distributional fidelity compared to existing agentic and domain-specialist baselines. The results provide strong evidence that robust automated modeling requires architectural mechanisms to manage memory, attribution, and feedback explicitly and quantitatively.