ProSim: Data-Driven Promptable Simulation

• ProSim is a framework that enables data-driven simulation of agent behavior in traffic and social contexts using modular, prompt-controlled architectures.
• It integrates scene encoding, prompt-conditioned policies, and closed-loop rollouts to achieve real-time control and precise modeling of emergent behaviors.
• Leveraging Bayesian optimization, ProSim efficiently discovers diverse, safety-critical scenarios and novel collision modes in autonomous driving simulations.

The ProSim framework refers to a family of data-driven, promptable simulation environments designed for high-fidelity modeling of agent behavior in both traffic and social contexts. Its purpose is to facilitate the realistic, controllable, and efficient simulation of agent interactions—either autonomous vehicles in traffic or LLM agents in synthetic societies—based on multimodal user prompts or policy interventions. ProSim’s modular design enables closed-loop simulation, fine-grained behavior manipulation, and statistical evaluation of emergent phenomena such as prosociality, fairness, safety-critical events, and norm diffusion (Tan et al., 2024, Zhou et al., 21 May 2025, Mondelli et al., 1 Jun 2025).

1. Formal Structure and System Components

ProSim’s architecture consistently incorporates three key modules across its traffic and social simulation deployments:

Module	Description	Implementation Contexts
Scene Encoder	Encodes static environment and agent history	Road networks, demographic/psych profiles
Prompt-Conditioned Policy	Fuses multimodal agent prompts into policy tokens	Traffic intent, prosocial traits, policy
Closed-Loop Rollout	Iteratively predicts agent states/actions in parallel	Trajectory, Likert action, unfairness

The traffic variant is transformer-based and integrates a heavy scene/prompt encoder with a lightweight per-agent rollout network, enabling real-time simulation of up to 128 agents on a single GPU. Prompt types include numerical goal points, route sketches, high-level categorical tags, and free-form natural language instructions (Tan et al., 2024). In the social simulation (prosociality studies), the environment is a dynamic small-world social network where agents interact, observe, and update behavior based on both their network neighbors and external interventions (Zhou et al., 21 May 2025).

2. Multimodal Promptability and Controllability

A core innovation of ProSim is its unified, multimodal prompt interface. Each agent’s behavior can be precisely directed using:

• Numerical Goal Points: Target positions with optional timing.
• Route Sketches: Sparse, perturbed future waypoint sequences.
• Categorical Action Tags: Discrete behavioral intentions, temporally encoded.
• Textual Instructions: Natural-language goals or constraints (e.g., “A1 yields to A0 at intersection”).

All prompts are embedded, aggregated, and incorporated into each agent’s policy query, combining static scenario encoding and dynamic context. This architectural design yields strong controllability, as measured by up to 69.7% improvements in trajectory realism relative to unconditional rollouts on real-world datasets (Tan et al., 2024).

In the LLM-based social domain, prompt generation enables the injection of complex trait vectors (e.g., Big Five, empathic concern) and scenario-specific instructions. Policy interventions (e.g., moral indoctrination, regulatory enforcement, economic incentives) are encoded as intervention prompts, which can be systematically varied per experiment (Zhou et al., 21 May 2025).

3. Closed-Loop Simulation and Learning Dynamics

ProSim operates entirely in a closed-loop regime, where agent states and environment evolve interactively—distinguishing it from prior open-loop simulation paradigms. For each iteration:

• The global scene and prompts are encoded into per-agent policy tokens.
• Parallel rollout networks predict $k$-step future motions.
• The environment is dynamically updated, and, if needed, new observations or neighbor actions are made available for the next timestep.

Learning is end-to-end and incorporates three principal loss terms:

• Imitation Loss ($\mathcal{L}_{IL}$): Penalizes deviation from ground truth trajectories or human benchmarks.
• Collision Loss ($\mathcal{L}_{coll}$): Penalizes predicted physical overlaps between agents.
• Offroad Loss ($\mathcal{L}_{off}$): Penalizes non-drivable state predictions.

In the social simulation, agent behavior updates are implemented as LLM queries whose input is a concatenation of demographic, trait, scenario, and social observation prompts. The network topology is a Watts–Strogatz small-world graph, with only a fraction of edges “activated” at each timestep to model incomplete social observation (Zhou et al., 21 May 2025).

4. Optimization and Scenario Generation

ProSim’s traffic testing deployment incorporates a Bayesian optimization (BO) wrapper to efficiently search the prompt space for safety-critical, rare, or diverse agent behaviors:

• Prompt Parameterization: Each agent’s goal position is treated as a latent uncertain parameter, forming a $2N$-dimensional prompt vector.
• Criticality Objective: $g(\mathbf{p}) = -\min_{t,i} \| x_t^E - x_t^i \|_2$, which identifies scenarios with the minimal ego–agent separation.
• BO Loop: A Gaussian Process with Matérn kernel models $g$, guiding prompt selection via the Upper Confidence Bound (UCB) acquisition.

Empirically, BO discovers both intuitive (e.g., rear-end, lane-cut) and novel collision modes more efficiently than random sampling, increasing the rate and diversity of safety-critical test scenarios (Mondelli et al., 1 Jun 2025).

5. Empirical Findings and Benchmarks

Traffic Domain (Tan et al., 2024, Mondelli et al., 1 Jun 2025):

• ProSim achieves an average displacement error (ADE) of 0.288 m when all prompt types are provided, with up to 69.7% controllability gain over unconditional baselines.
• It runs closed-loop rollouts of 64–128 agents in 40–55 ms on an A100 GPU.
• In AV planner stress-testing, BO+ProSim discovers higher collision rates and smaller minimum distances than random search, while preserving or improving trajectory diversity.

Social Domain (LLM Prosociality) (Zhou et al., 21 May 2025):

• LLM agents exhibit robust, human-like prosocial behaviors across diverse scenarios and interventions.
• GPT-4o and Qwen-2.5 align closely with human empirical patterns (Pearson $r\approx 0.95$), but some models display over/under-sensitivity to intervention types.
• Policy-induced reward asymmetry reduces prosocial acts by 25–32%; burden asymmetry by 19–23%.
• Unfairness propagates through the network, causing >50% of non-treated agents to reduce prosociality within 10 rounds—demonstrating norm erosion and behavioral contagion.

6. Limitations and Directions for Future Work

ProSim’s constraints include:

• Limitation to specified prompt modalities; it does not support arbitrary inter-agent relational constraints or vision-based agent instructions.
• Inability to robustly model out-of-distribution (irrational, panic-driven) or highly adversarial behaviors outside the training distribution.
• Static agent psychological traits in the social simulation; no adaptive or reinforcement learning over moral dispositions.
• Cultural and policy complexity is simplified; only two structural inequity forms and homogeneous language/culture are modeled in LLM studies.
• No integration of embodied or physically instantiated stakes in text-based simulations.

Future work is aimed at extending ProSim with hierarchical/nested prompt types, multi-agent or cross-lingual policies, richer dataset construction (such as ProSim-Instruct-520k), increased diversity via hybrid diffusion objectives, and hybrid simulation pipelines that combine data-driven, rule-based, and adversarial components (Tan et al., 2024, Zhou et al., 21 May 2025, Mondelli et al., 1 Jun 2025).

7. Broader Significance and Impact

ProSim defines a new paradigm for promptable and interactive agent-based simulation across both traffic and synthetic social environments. Its fusion of prompt-based control, transformer-derived policy networks, and closed-loop execution allows researchers to bridge algorithmic and behavioral sciences. In autonomous driving, ProSim sets a practical standard for safety-critical evaluation and realistic scenario generation. In social simulation, it offers a controlled laboratory for studying the emergence and decay of complex social norms among LLM societies, advancing both methodological rigor and theoretical understanding of policy effects and norm transmission in agent-driven systems (Tan et al., 2024, Zhou et al., 21 May 2025, Mondelli et al., 1 Jun 2025).