CARFF: Conditional Auto-encoded Radiance Field for 3D Scene Forecasting

Abstract: We propose CARFF, a method for predicting future 3D scenes given past observations. Our method maps 2D ego-centric images to a distribution over plausible 3D latent scene configurations and predicts the evolution of hypothesized scenes through time. Our latents condition a global Neural Radiance Field (NeRF) to represent a 3D scene model, enabling explainable predictions and straightforward downstream planning. This approach models the world as a POMDP and considers complex scenarios of uncertainty in environmental states and dynamics. Specifically, we employ a two-stage training of Pose-Conditional-VAE and NeRF to learn 3D representations, and auto-regressively predict latent scene representations utilizing a mixture density network. We demonstrate the utility of our method in scenarios using the CARLA driving simulator, where CARFF enables efficient trajectory and contingency planning in complex multi-agent autonomous driving scenarios involving occlusions.

• The paper presents a novel framework that probabilistically predicts future 3D scenes from past 2D ego-centric images using a two-stage training strategy.
• It employs a probabilistic encoder integrated with Neural Radiance Fields to capture complex latent scene configurations under uncertainty.
• The method is validated in CARLA simulations, demonstrating robust scene reconstruction and improved forecasting in occluded and dynamic driving scenarios.

Overview

The paper delineates upon Conditional Auto-encoded Radiance Field for 3D Scene Forecasting, abbreviated as CARFF, a pioneering method enabling the prediction of future 3D scenes based on previous 2D ego-centric images. Through the melding of a probabilistic encoder and the global representation capacity of Neural Radiance Fields (NeRF), CARFF offers an innovative way to forecast scene evolution, which is especially significant in environments where uncertainty and partial observations are prevalent.

Architecture

The architecture of CARFF is predicated on a two-stage training strategy encompassing Pose-Conditional-VAE (PC-VAE) followed by NeRF for three-dimensional representations. The significance of CARFF lies in its ability to propose complex, probabilistic scene predictions that go beyond the paradigm of deterministic models. What distinguishes CARFF is the employment of a probabilistic encoder that maps an image into a distribution over plausible 3D latent scene configurations, augmenting a traditional NeRF to infuse it with the capacity to model dynamics and environmental uncertainty.

Performance

The prowess of CARFF is demonstrated through simulation in the CARLA driving simulator. Numerically, CARFF showcased an average Peak Signal-to-Noise Ratio (PSNR) across trained data, and this metric was used to adjudge the reconstruction accuracy from various poses, delivering insight into the model's robustness. Additional benchmarks included the Support Vector Machine (SVM) based metric for assessing latent timestamp prediction. The model was capable of maintaining the stochastic characteristics of latent representations and ensured accurate reconstructions even under partial environmental observations.

Applications and Future Direction

CARFF’s utility was explored within autonomous driving applications, where intricate multi-agent scenarios involving visual occlusions and the need for dynamic contingency planning are commonplace. Escaping the confines of deterministic predictions, CARFF augments safety and adaptability by accurately predicting outcomes in occluded environs—bolstering decision-making protocols that sophisticate autonomy in navigation tasks.

Looking ahead, the paper recognizes limitations including CARFF's initial dependence on posed images and its applicability to specific environments like traffic intersections. While capable of handling state and dynamics uncertainty, incorporating extensive dynamics with a multitude of agents was noted as an area for future enhancements, hinting at ensuing advancements in the field of predictive modeling and planning in AI-driven systems.