FactorSim: Generative Simulation via Factorized Representation (2409.17652v2)

Abstract: Generating simulations to train intelligent agents in game-playing and robotics from natural language input, from user input or task documentation, remains an open-ended challenge. Existing approaches focus on parts of this challenge, such as generating reward functions or task hyperparameters. Unlike previous work, we introduce FACTORSIM that generates full simulations in code from language input that can be used to train agents. Exploiting the structural modularity specific to coded simulations, we propose to use a factored partially observable Markov decision process representation that allows us to reduce context dependence during each step of the generation. For evaluation, we introduce a generative simulation benchmark that assesses the generated simulation code's accuracy and effectiveness in facilitating zero-shot transfers in reinforcement learning settings. We show that FACTORSIM outperforms existing methods in generating simulations regarding prompt alignment (e.g., accuracy), zero-shot transfer abilities, and human evaluation. We also demonstrate its effectiveness in generating robotic tasks.

• The paper introduces FactorSim, which uses a factored POMDP to modularly generate simulation code that adheres closely to natural language prompts.
• The paper demonstrates superior performance across RL benchmark games and robotics tasks, achieving higher accuracy and better generalization than baseline methods.
• The method employs a chain-of-thought prompting strategy and MVC design to efficiently decompose complex task specifications into manageable steps.

FactorSim: Generative Simulation via Factorized Representation

Introduction

The paper presents FactorSim, an approach designed to generate fully coded simulations from natural language specifications. FactorSim addresses the problem of generating comprehensive simulations used to train intelligent agents for tasks such as video games and robotics. Unlike prior methodologies that focus on specific components like reward functions or task hyperparameters, FactorSim strategically decomposes the problem using a factored Partially Observable Markov Decision Process (POMDP) to reduce context dependence at each step.

Core Methodology

The key innovation in FactorSim lies in its use of a factored POMDP to enable modular generation of simulation code. This is a significant deviation from existing approaches that often fail to adhere to complex input prompts due to the limitations of LLMs when handling extensive context. FactorSim uses a Chain-of-Thought (CoT) prompting method to break down a text specification into a sequence of manageable steps. Each step is then tackled individually, leveraging a factored representation to minimize unnecessary context, thereby improving the accuracy and alignment of the generated code with the input specifications.

FactorSim employs a model-view-controller (MVC) design pattern to divide the simulation generation process into controller, model, and view components. This allows for clear separation and concise handling of user inputs, state transitions, and rendering, respectively.

Evaluation and Results

The paper introduces a new benchmark suite for generative simulation, comprising eight 2D Reinforcement Learning (RL) games. FactorSim is evaluated against several baseline methods, including Llama-3 and GPT-4 models, both with and without Chain-of-Thought and self-debugging mechanisms.

System Tests

Systematic testing shows that FactorSim outperforms all baselines across seven of the eight games in terms of passing rate for system tests. These tests programmatically verify the adherence of the generated simulation to the given prompt specifications. Notably, FactorSim achieves higher accuracy with fewer tokens compared to the best-performing baselines, indicating efficient computational resource utilization.

Zero-shot Transfer

In zero-shot transfer tasks, where RL agents are trained on FactorSim-generated environments and tested on unseen "ground-truth" environments, FactorSim demonstrates superior generalization capabilities. The performance metrics reveal that agents trained within FactorSim's environments exhibit better transfer performance compared to those trained in environments generated by baseline methods.

Human Evaluation

Further, a human paper is conducted to assess the playability and overall quality of the generated simulations. Participants rated FactorSim-generated games higher than those generated by other strong baselines such as GPT-4 with Chain-of-Thought and iterative self-debugging. This human evaluation underscores the utility of FactorSim in generating simulations that not only adhere to prompt specifications but are also playable and enjoyable.

Robotics Task Generation

The paper extends the scope of FactorSim to robotics task generation, evaluating its performance on GenSim's benchmark of 50 robotic tasks in the CLIPort framework. FactorSim shows superior performance across multiple metrics: syntax correctness, runtime verification, and human evaluation of task completion. The method's modular approach proves particularly effective in defining and generating complex spatial relationships between objects, thereby outperforming existing Chain-of-Thought-based methods.

Implications and Future Directions

FactorSim's methodological innovations have broad implications for both theoretical research and practical applications in AI. By leveraging a factored representation within a POMDP framework and employing a modular generation strategy, FactorSim addresses key limitations in scalability and adherence to detailed specifications seen in previous approaches. This method holds promise for enhancing the generalization capabilities of RL agents trained in simulation, as well as facilitating the generation of complex, real-world robotics tasks.

Future research could explore extending FactorSim to support multi-agent environments and more complex 3D simulations. Additionally, integrating feedback from the agent's training process to iteratively refine the simulation environment remains an interesting avenue for further investigation. In sum, FactorSim marks a significant step toward the automated generation of highly detailed and functional simulation environments, contributing to the advancement of generalized RL policy development.