FactorSim: Automated Simulation Generation

• FactorSim is a framework for automatic generative simulation using factored POMDP representations to create diverse and runnable simulation environments.
• It employs an MVC pipeline with chain-of-thought decomposition to produce modular Python code, reducing LLM context and improving debugging.
• Evaluations in 2D games and robotics tasks demonstrate substantial gains in code correctness, pass rates, and human playability compared to existing methods.

FactorSim is a framework for automatic generative simulation from natural-language input, targeting agent training tasks in game environments and robotics. Unlike prior approaches that synthesize simulation components (reward functions, hyperparameters) or modify extant simulators, FactorSim directly generates complete, runnable simulation code from textual specifications, leveraging the modularity inherent to simulation structure via factored partially observable Markov decision process (POMDP) representations (Sun et al., 26 Sep 2024). This approach enables efficient creation of diverse agent training environments and facilitates downstream tasks such as zero-shot transfer in reinforcement learning (RL), with substantial gains in code correctness and generalization over existing methods.

1. Problem Statement and Motivation

Designing and coding simulations from language descriptions is an open challenge, demanding considerable manual effort and expertise. Existing approaches powered by LLMs often focus on fragmentary outputs—such as reward shaping or environmental parameterization—while struggling to meet end-to-end specification requirements in executable form. The ability to automate simulation synthesis from natural language would expedite the development of RL training regimes, lower the barrier for non-programmer users to define tasks, enable thorough inspection/debugging, and diversify the landscape of synthetic environments.

2. Factored POMDP Formalism

FactorSim employs a factored POMDP formalism, capturing simulation structure at a modular semantic level. The generative process targets the tuple: $\mathcal M = \langle S, A, O, T, \Omega, R\rangle$ with explicit factorization as follows:

• State: $S = S_1 \times S_2 \times \cdots \times S_n$
• Observation: $O = O_1 \times O_2 \times \cdots \times O_m$
• Transition: For discrete actions $a$,

$$T(s' \mid s, a) = \prod_{i=1}^m T^{(s)}_i(s'[i] \mid s[Z_i], a)$$

where each factor $T_i$ depends only on a subset $Z_i \subseteq \{1, \ldots, n\}$ of state variables.

• Observation function: $\Omega(o \mid s, a)$ factors over observed variables in analogous form.
• Reward decomposition:

$R(s,a) = \sum_{j=1}^\ell R_j(s[Z'_j], a)$

Factorization reduces context dependence by enabling focused updates on subsets $Z_k$ during simulation generation, which is critical for restricting LLM input context and improving code alignment.

3. Generation Pipeline and System Architecture

FactorSim orchestrates code synthesis through a Model–View–Controller (MVC) iterative pipeline, employing a five-prompt protocol for each task decomposition:

1. Chain-of-Thought Planning: Natural-language specification $Q_{\mathrm{text}}$ is decomposed into sub-tasks $(q_1,\ldots,q_K)$.
2. State Variable Identification: For each sub-task $q_k$, identify which state variables $Z_k$ to update.
3. Transition Code Generation: Controller prompt synthesizes action-dependent transition code $T^{(a)}$.
4. Action-Independent Model Update: Model prompt generates transition code $T^{(s)}$.
5. View Update: View prompt synthesizes observation (rendering) code $\Omega$.
6. Reward Update: Implicitly handled via a score variable tied to state updates.

Critically, each prompt references only the relevant code subset—functions and state definitions in $Z_k$—thereby reducing the LLM context window and mitigating incoherent code revisions. Python code modules/functions are generated by the LLM (including models such as GPT-4 and Llama-3) and composed into the final simulation codebase.

4. Training Objectives and Debugging Protocols

FactorSim does not utilize end-to-end differentiable training; objective functions rely on LLM likelihood estimates and code execution feedback:

• Decomposition Generation: The probability of generating a simulation, $\hat p(\mathcal M'|Q)$, is given by

$$\hat p(\mathcal M'|Q) = \frac{1}{N} \sum_i p(\mathcal M'|q^{(i)}_{1:K})$$

where samples $q^{(i)}$ are drawn from the decomposition distribution given $Q$.

• Self-Debugging Loop: Generated code is executed and, upon failure, error messages are fed back to the LLM for up to 10 iterations of correction.
• No RL-based Loss: Evaluation emphasizes system test pass/fail and downstream zero-shot RL transfer instead of agent learning gradients.

5. Generative Simulation Benchmark and Evaluation Metrics

A novel benchmark is introduced to quantify simulation code quality along three axes:

• Code Prompt Alignment (System Tests): Automated tests verify behavioral correspondence with the specification through key presses and assertions; the pass rate measures test fulfillment.
• Zero-Shot RL Transfer: A Proximal Policy Optimization (PPO) agent is trained for 10M steps on $N$ generated environments and evaluated on a held-out "ground-truth" environment. The normalized reward is calculated as:

$$\text{Zero-shot score} = \frac{R_{\mathrm{agent}} - R_{\mathrm{random}}}{R_{\mathrm{oracle}} - R_{\mathrm{random}}}$$

Aggregation spans eight 2D PLE games.

• Human Evaluation: Human subjects interact with the simulator and rate playability on a four-point ordinal scale.

Quantitative Highlights

Method	Pass Rate (GPT-4)	Zero-Shot Score	Playability (Human)
FactorSim	61%	~0.35	3.0 / 4
CoT+self-debug	48%	0.22	2.3 / 4
Llama-3+FactorSim	50%	[not stated]	[not stated]
Llama-3+CoT+debug	43%	[not stated]	[not stated]

FactorSim decreases token usage per context (≈30K) compared to CoT+self-debug (~50K).

6. Experimental Results and Ablation Studies

Empirical findings indicate superior performance over vanilla one-shot, self-debugging, CoT+self-debug, and AgentCoder baselines across eight 2D PLE games. Ablation studies show:

• Context Reduction: Removing factorized context selection halves system test pass rate.
• MVC Decomposition: Eliminating the MVC pipeline drops prompt alignment by ≈20%.
• Error Analysis: Errors in FactorSim-generated environments are more diffuse (rather than dominated by single failure points).
• Robotics Task Performance: Over 50 GenSim pick-and-place tasks, FactorSim achieved aggregate human-pass rates of 58% versus 15% for the best GenSim variant.

7. Contributions, Limitations, and Future Directions

Principal contributions are:

1. End-to-end framework for simulation code generation from natural language.
2. Introduction of factored POMDP and MVC pipeline, enabling principled LLM context reduction.
3. Proposal of a comprehensive simulation benchmark including system tests, zero-shot RL transfer, and human playability evaluation, with new 2D and robotics tasks.

Limitations include:

• Dependence on LLM quality and natural language specification clarity; nonsensical or contradictory inputs degrade performance.
• Restriction to single-agent, 2D and simple 3D tasks; does not address multi-agent or full-contact 3D physics at scale.
• No incorporation of RL training feedback into simulation refinement—open-loop simulation design only.

Planned extensions involve accommodating multi-agent interaction, integrating differentiable simulators suitable for policy gradient updates, and leveraging closed-loop RL feedback to iteratively refine simulation parameters. The exploration of controller parameterization suitable for real-world sim-to-real transfer is also indicated as a future direction (Sun et al., 26 Sep 2024).

This summary is grounded directly in the referenced works (Sun et al., 26 Sep 2024), and reflects only claims substantiated by the provided data.