DynamicCity: Large-Scale 4D Occupancy Generation from Dynamic Scenes (2410.18084v2)

Abstract: Urban scene generation has been developing rapidly recently. However, existing methods primarily focus on generating static and single-frame scenes, overlooking the inherently dynamic nature of real-world driving environments. In this work, we introduce DynamicCity, a novel 4D occupancy generation framework capable of generating large-scale, high-quality dynamic 4D scenes with semantics. DynamicCity mainly consists of two key models. 1) A VAE model for learning HexPlane as the compact 4D representation. Instead of using naive averaging operations, DynamicCity employs a novel Projection Module to effectively compress 4D features into six 2D feature maps for HexPlane construction, which significantly enhances HexPlane fitting quality (up to 12.56 mIoU gain). Furthermore, we utilize an Expansion & Squeeze Strategy to reconstruct 3D feature volumes in parallel, which improves both network training efficiency and reconstruction accuracy than naively querying each 3D point (up to 7.05 mIoU gain, 2.06x training speedup, and 70.84% memory reduction). 2) A DiT-based diffusion model for HexPlane generation. To make HexPlane feasible for DiT generation, a Padded Rollout Operation is proposed to reorganize all six feature planes of the HexPlane as a squared 2D feature map. In particular, various conditions could be introduced in the diffusion or sampling process, supporting versatile 4D generation applications, such as trajectory- and command-driven generation, inpainting, and layout-conditioned generation. Extensive experiments on the CarlaSC and Waymo datasets demonstrate that DynamicCity significantly outperforms existing state-of-the-art 4D occupancy generation methods across multiple metrics. The code and models have been released to facilitate future research.

• The paper introduces a novel framework that encodes dynamic 4D LiDAR scenes into a compact HexPlane representation using a Variational Autoencoder.
• It employs a Diffusion Transformer with a Padded Rollout Operation to capture complex spatial-temporal relationships, achieving state-of-the-art mIoU gains and training speed improvements.
• The approach enhances high-fidelity scene generation for autonomous driving and robotics, setting a new standard for modeling dynamic real-world environments.

Overview of "DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes"

The paper "DynamicCity: Large-Scale LiDAR Generation from Dynamic Scenes" introduces a novel framework for generating large-scale, high-quality 4D LiDAR scenes. This work primarily focuses on overcoming the limitations of existing models, which are often restricted to static or single-frame scenes, by capturing the dynamic nature and temporal evolution present in real-world driving environments.

Key Components

DynamicCity Framework: The core contributions of the DynamicCity framework are twofold:

1. Variational Autoencoder (VAE) for 4D Representation: The VAE is employed to encode dynamic LiDAR scenes into a compact 4D representation known as HexPlane, which consists of six 2D feature maps. This involves a Projection Module that compresses 4D features and an Expansion Squeeze Strategy for efficient reconstruction, resulting in substantial improvements in training speed, reconstruction accuracy, and memory efficiency.
2. Diffusion Transformer (DiT) for HexPlane Generation: For generating HexPlane, a DiT-based framework is utilized. The Padded Rollout Operation within this framework reorganizes the feature planes into a cohesive 2D structure, allowing the model to capture intricate spatial and temporal relationships, thus enhancing generation quality.

Numerical Results

The framework demonstrates significant advancements over state-of-the-art methods. Notably, the experiments conducted on CarlaSC and Waymo datasets illustrate that DynamicCity achieves superior 4D reconstruction and generation performance, evidenced by strong mIoU gains and training speedup metrics. The framework's capability is enhanced by integrating various conditions during the generation process, enabling diverse applications such as trajectory-guided generation and dynamic scene inpainting.

Implications and Future Research

From a practical perspective, DynamicCity has the potential to enhance applications in autonomous driving and robotic navigation by providing high-fidelity dynamic scenes that better reflect real-world conditions. Theoretical implications involve advancing understanding of how dynamic environments can be efficiently modeled and represented, paving the way for future research in high-dimensional data representation.

Looking forward, the framework's adaptability suggests its use could extend to other domains requiring dynamic spatial-temporal data generation. Future developments might focus on further improving the model's efficiency and exploring its integration with real-time data processing systems.

Conclusion

DynamicCity represents a significant advancement in the field of 4D LiDAR scene generation, offering a robust solution to the challenges of modeling dynamic environments. Through its innovative use of VAE and DiT, combined with HexPlane's efficient representation, DynamicCity sets a new standard for scene generation in complex driving scenarios. The open release of the code promises to facilitate further research and development, fostering continued innovation in the field.