Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents (2402.05746v3)

Abstract: Scene simulation in autonomous driving has gained significant attention because of its huge potential for generating customized data. However, existing editable scene simulation approaches face limitations in terms of user interaction efficiency, multi-camera photo-realistic rendering and external digital assets integration. To address these challenges, this paper introduces ChatSim, the first system that enables editable photo-realistic 3D driving scene simulations via natural language commands with external digital assets. To enable editing with high command flexibility,~ChatSim leverages a LLM agent collaboration framework. To generate photo-realistic outcomes, ChatSim employs a novel multi-camera neural radiance field method. Furthermore, to unleash the potential of extensive high-quality digital assets, ChatSim employs a novel multi-camera lighting estimation method to achieve scene-consistent assets' rendering. Our experiments on Waymo Open Dataset demonstrate that ChatSim can handle complex language commands and generate corresponding photo-realistic scene videos.

• The paper introduces ChatSim, a system that uses collaborative LLM-agents to enable editable, photo-realistic 3D scene simulation for autonomous driving.
• The paper details two novel rendering methods, McNeRF and McLight, which achieve consistent background alignment and realistic foreground integration.
• The paper demonstrates that using ChatSim for data augmentation leads to improved performance in autonomous driving models.

Exploring Editable Scene Simulation for Autonomous Driving using Collaborative LLM Agents

Introduction

The paper "Editable Scene Simulation for Autonomous Driving via Collaborative LLM-Agents" introduces ChatSim, a novel system designed to enhance autonomous driving technology through editable photo-realistic 3D driving scene simulations. Utilizing natural language commands and integrating external digital assets, ChatSim addresses the constraints present in existing editable scene simulation methods such as user interaction efficiency, multi-camera photo-realistic rendering, and the integration of external digital assets.

System Architecture

The core of ChatSim lies in its collaborative LLM agent framework, enabling high command flexibility through natural language processing. This framework encompasses several specialized agents, including a project manager agent and technology-specific agents for tasks like view adjustment, background rendering, vehicle deletion, 3D asset management, vehicle motion, and foreground rendering. Each agent uses a combination of LLM for interpreting language commands and role-specific functions to execute tasks, ensuring detailed and effective scene simulation.

Multi-Agent Collaboration

Collaborative LLM agents work together to decompose complex commands into actionable tasks, distributed among agents based on their specialties. This delegation allows for efficient and precise simulation adjustments, fulfilling the requirements outlined in user commands. The project manager agent plays a pivotal role in orchestrating this process, ensuring a coherent workflow and facilitating multi-round editing capabilities.

Novel Rendering Methods

To achieve photo-realistic rendering and accurate integration of digital assets, the paper introduces two novel methods: McNeRF and McLight.

McNeRF for Background Rendering

Addressing the challenge of misaligned poses and brightness inconsistency due to varied camera exposure times, McNeRF incorporates a multi-camera alignment strategy and brightness-consistent rendering. This method significantly improves the realism and consistency of rendered scenes by leveraging multi-camera inputs efficiently.

McLight for Foreground Rendering

For integrating external digital assets seamlessly into the scene, McLight estimates location-specific lighting conditions by blending skydome and surrounding lighting. This hybrid approach ensures that digital assets exhibit realistic textures, materials, and shadows, aligning perfectly with the scene's lighting conditions.

Experimental Results

Extensive experiments demonstrate that ChatSim can generate complex, photo-realistic scene videos based on a wide range of language commands. The system outperforms existing simulation methods in terms of photo-realism and command execution accuracy. Furthermore, autonomous driving models trained with data augmented by ChatSim exhibit improved performance, underscoring the practical benefits of this approach.

Implications and Future Directions

This research presents a significant advancement in editable scene simulation for autonomous driving, offering both theoretical contributions to the field and practical tools for data generation. The collaborative LLM-agent framework and novel rendering methods pave the way for more intuitive, efficient, and realistic scene simulations. Looking ahead, further exploration into dynamic scene elements and environmental conditions could expand the system's capabilities, enhancing autonomous vehicle training and testing methodologies.

Conclusion

The introduction of ChatSim represents a leap forward in autonomous driving simulation technology. By harnessing the power of collaborative LLM agents and pioneering rendering techniques, this system enables the generation of customizable and realistic driving scenes that can significantly aid the development and testing of autonomous vehicles.