Extrapolated Urban View Synthesis Benchmark (2412.05256v3)

Abstract: Photorealistic simulators are essential for the training and evaluation of vision-centric autonomous vehicles (AVs). At their core is Novel View Synthesis (NVS), a crucial capability that generates diverse unseen viewpoints to accommodate the broad and continuous pose distribution of AVs. Recent advances in radiance fields, such as 3D Gaussian Splatting, achieve photorealistic rendering at real-time speeds and have been widely used in modeling large-scale driving scenes. However, their performance is commonly evaluated using an interpolated setup with highly correlated training and test views. In contrast, extrapolation, where test views largely deviate from training views, remains underexplored, limiting progress in generalizable simulation technology. To address this gap, we leverage publicly available AV datasets with multiple traversals, multiple vehicles, and multiple cameras to build the first Extrapolated Urban View Synthesis (EUVS) benchmark. Meanwhile, we conduct both quantitative and qualitative evaluations of state-of-the-art NVS methods across different evaluation settings. Our results show that current NVS methods are prone to overfitting to training views. Besides, incorporating diffusion priors and improving geometry cannot fundamentally improve NVS under large view changes, highlighting the need for more robust approaches and large-scale training. We will release the data to help advance self-driving and urban robotics simulation technology.

• The paper presents the EUVS benchmark to evaluate NVS methods for extrapolated urban viewpoints in autonomous driving scenarios.
• It categorizes challenges into translation-only, rotation-only, and combined cases, revealing significant performance drops in metrics like PSNR, SSIM, and LPIPS.
• The findings advocate integrating diffusion models, planar optimizations, and depth priors to enhance generalization and robustness in novel view synthesis.

Extrapolated Urban View Synthesis Benchmark: A Synopsis

The paper "Extrapolated Urban View Synthesis Benchmark" presents a new benchmark designed to evaluate Novel View Synthesis (NVS) methods in urban environments, with a particular focus on extrapolated viewpoints. NVS is crucial for the development of vision-centric autonomous vehicles (AVs), providing the ability to generate unseen perspectives that are essential for simulating dynamic and realistic driving environments. Historically, NVS methodology has predominantly concentrated on interpolated setups, where testing views are closely aligned with the training views. This paper identifies the gap in handling extrapolated view synthesis, a scenario where the test viewpoints significantly deviate from training data, posing a challenge for the generalizability of NVS methods and subsequently developing autonomous driving technologies.

Benchmark Overview

The paper introduces the Extrapolated Urban View Synthesis (EUVS) benchmark, which is based on publicly available datasets such as NuPlan, MARS, and Argoverse 2. The datasets include multiple traversals, multiple agents, and multiple camera setups to ensure a comprehensive assessment of NVS techniques. The benchmark distinguishes itself by focusing on extrapolated scenarios, systematically categorized into three difficulty levels: translation-only, rotation-only, and translation + rotation. Each level represents varying complexities and aligns with real-world autonomous driving challenges like lane changes, directional shifts, and complex intersections.

Key Findings

Through rigorous qualitative and quantitative analysis, this research reveals that state-of-the-art methods, including 3D Gaussian Splatting and its variants, experience significant performance degradation when transitioning from interpolated to extrapolated test scenarios. The experimental results underscore that current NVS approaches often overfit to training views and struggle with unseen viewpoints, with notable decreases in performance metrics like PSNR, SSIM, and LPIPS. This performance discrepancy highlights the limitations in current NVS technologies and the need for methods that can robustly handle wide ranges of view shifts observed in realistic urban driving scenes.

Implications and Future Directions

The introduction of the EUVS benchmark represents a significant advancement in evaluating and improving novel view synthesis methods within the AV domain. By focusing on extrapolation, the benchmark challenges existing algorithms and paves the way for innovations that enhance the adaptability and robustness required for real-world applications. The paper suggests the potential benefits of integrating diffusion models, planar-based optimizations, and incorporating depth priors to address the gaps identified. Furthermore, the findings encourage future research to continue exploring hybrid representations and more comprehensive datasets to bolster the generalization capabilities of NVS systems.

Through the creation of this benchmark, the paper not only sets a new standard for NVS evaluation but also invites researchers to contribute to an evolving dataset that aims to refine view synthesis methodologies comprehensively. This will potentially catalyze significant advancements in the development of autonomous systems and photorealistic simulators, ultimately contributing to safer and more reliable autonomous driving technologies.