EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision (2311.02077v1)

Abstract: We present EmerNeRF, a simple yet powerful approach for learning spatial-temporal representations of dynamic driving scenes. Grounded in neural fields, EmerNeRF simultaneously captures scene geometry, appearance, motion, and semantics via self-bootstrapping. EmerNeRF hinges upon two core components: First, it stratifies scenes into static and dynamic fields. This decomposition emerges purely from self-supervision, enabling our model to learn from general, in-the-wild data sources. Second, EmerNeRF parameterizes an induced flow field from the dynamic field and uses this flow field to further aggregate multi-frame features, amplifying the rendering precision of dynamic objects. Coupling these three fields (static, dynamic, and flow) enables EmerNeRF to represent highly-dynamic scenes self-sufficiently, without relying on ground truth object annotations or pre-trained models for dynamic object segmentation or optical flow estimation. Our method achieves state-of-the-art performance in sensor simulation, significantly outperforming previous methods when reconstructing static (+2.93 PSNR) and dynamic (+3.70 PSNR) scenes. In addition, to bolster EmerNeRF's semantic generalization, we lift 2D visual foundation model features into 4D space-time and address a general positional bias in modern Transformers, significantly boosting 3D perception performance (e.g., 37.50% relative improvement in occupancy prediction accuracy on average). Finally, we construct a diverse and challenging 120-sequence dataset to benchmark neural fields under extreme and highly-dynamic settings.

• The paper presents a self-supervised approach that decomposes scenes into static and dynamic components while achieving +2.93 PSNR improvement for static and +3.70 for dynamic reconstructions.
• It integrates an emergent flow field estimation to aggregate multi-frame features, which enhances novel view synthesis with a +2.91 PSNR boost in dynamic settings.
• The approach also lifts 2D features into 4D space-time to improve occupancy prediction accuracy by 37.50% and supports scalable, annotation-free autonomous driving applications.

EmerNeRF: Emergent Spatial-Temporal Scene Decomposition via Self-Supervision

The paper presents "EmerNeRF," a novel approach for learning spatial-temporal representations of dynamic driving scenes without relying on ground truth annotations or pre-trained models. This technique, built on the foundation of neural fields, handles scene decomposition into static and dynamic components purely from self-supervision. It further parameterizes a flow field to efficiently aggregate features from multiple frames, enhancing the reconstruction and rendering accuracy of dynamic elements in driving scenes.

Core Components and Contributions:

The EmerNeRF framework introduces two pivotal elements: self-supervised scene decomposition and emergent scene flow estimation. These elements allow the model to represent highly dynamic scenes by employing three distinct fields:

1. Static and Dynamic Fields: The method stratifies scenes into static and dynamic regions, driven by self-supervision that enables learning from diverse, unlabeled data. This decomposition allows the model to effectively differentiate between static elements (like buildings) and dynamic objects (such as vehicles and pedestrians).
2. Flow Field Integration: EmerNeRF computes an induced flow field from the dynamic component, enabling temporal aggregation of features. This integration both refines dynamic object rendering and naturally induces scene flow estimation capabilities without explicit supervision. By leveraging the temporal consistency in observed features, the model effectively estimates motion trajectories in dynamic scenes.

The paper delivers significant results in sensor simulation tasks, showcasing state-of-the-art performance in reconstructing both static and dynamic scenes. Notably, EmerNeRF outperforms previous methods in terms of Peak Signal-to-Noise Ratio (PSNR) by +2.93 for static and +3.70 for dynamic scenes. For novel view synthesis, the model also achieves a +2.91 PSNR improvement in dynamic settings.

Semantic Generalization and Benchmarking:

To enhance semantic generalization, EmerNeRF lifts 2D visual features from foundation models into 4D space-time, addressing positional biases inherent in modern Transformers. This adaptation significantly boosts 3D perception tasks, with a relative increase of 37.50% in occupancy prediction accuracy. The paper also introduces the NeRF On-The-Road (NOTR) benchmark, which includes 120 challenging sequences to assess the model's performance under diverse conditions.

Methodological Innovations:

The authors emphasize the model's ability to perform without pre-trained models or explicit annotations, which highlights its potential for scalable applications in autonomous driving. Key innovations include:

• Self-Supervised Static-Dynamic Decomposition: Allowing the model to learn from real-world data without labeled examples, reducing dependency on large-scale annotated datasets.
• Emergent Scene Flow Estimation: Achieving scene flow predictions as a byproduct of multi-frame feature aggregation, underscoring a novel emergent behavior within the neural network paradigm.
• Vision Transformer Augmentation: Addressing positional encoding artifacts in ViT models by introducing a shared learnable additive prior to enhance feature lifting from 2D to 4D spaces.

Implications and Future Directions:

This work suggests meaningful advancements in spatial-temporal representation learning, which can impact both theoretical and practical avenues. Theoretically, it aligns with the growing trend towards self-supervised, emergent behavior in machine learning systems, facilitating robust model performance with minimal human intervention. Practically, EmerNeRF's lack of dependency on ground truth data could inform the development of scalable, real-world applications in autonomous driving and robotics.

Potential future research directions could explore optimizations for rolling shutter effects, improve the balance between geometry and rendering quality, and enhance the training pipeline for learning-specific dynamic contexts. Additionally, expanding the architecture to handle multimodal inputs or adapting it to other domains (such as AR/VR) can broaden its utility and application scope. EmerNeRF's approach and results lay a solid foundation for such explorations, advocating for continued research into self-supervised spatial-temporal learning systems.