Know Your Neighbors: Improving Single-View Reconstruction via Spatial Vision-Language Reasoning (2404.03658v1)

Abstract: Recovering the 3D scene geometry from a single view is a fundamental yet ill-posed problem in computer vision. While classical depth estimation methods infer only a 2.5D scene representation limited to the image plane, recent approaches based on radiance fields reconstruct a full 3D representation. However, these methods still struggle with occluded regions since inferring geometry without visual observation requires (i) semantic knowledge of the surroundings, and (ii) reasoning about spatial context. We propose KYN, a novel method for single-view scene reconstruction that reasons about semantic and spatial context to predict each point's density. We introduce a vision-language modulation module to enrich point features with fine-grained semantic information. We aggregate point representations across the scene through a language-guided spatial attention mechanism to yield per-point density predictions aware of the 3D semantic context. We show that KYN improves 3D shape recovery compared to predicting density for each 3D point in isolation. We achieve state-of-the-art results in scene and object reconstruction on KITTI-360, and show improved zero-shot generalization compared to prior work. Project page: https://ruili3.github.io/kyn.

• The paper introduces KYN, a novel approach that integrates vision-language modulation and spatial reasoning to enhance reconstruction of occluded areas.
• It employs a guided spatial attention mechanism that aggregates enriched 3D point features for coherent and semantically informed density predictions.
• Experiments on KITTI-360 and DDAD datasets validate its state-of-the-art performance and robust zero-shot generalization in scene and object-level reconstructions.

Improving Single-View Reconstruction with Spatial Vision-Language Reasoning

Introduction

Single-view reconstruction aims to infer the 3D geometry of a scene from a single image, a task fundamental to various applications in computer vision. Despite advancements in depth estimation and radiance field methods, reconstructing occluded regions remains challenging. These methods often lack the semantic understanding and spatial reasoning necessary for accurate geometry inference in unobserved areas. This paper introduces Know Your Neighbors (KYN), a novel approach that leverages semantic knowledge and spatial context to improve the accuracy of single-view scene reconstruction. KYN features a vision-language (VL) modulation module and a language-guided spatial attention mechanism, enhancing point feature representations with semantic information and aggregating these across the scene for informed density predictions.

Vision-Language Modulation

The proposed method, KYN, begins by extracting fused visual features from both standard and VL image encoders, followed by enriching these features with semantic information obtained from category-wise text features. This enrichment is accomplished through a VL modulation module that iteratively augments 3D point features with semantic data, integrating both visual and textual information to provide a richer representation of each 3D point in space.

Spatial Attention with Vision-Language Guidance

Building upon enriched point-wise features, KYN utilizes a VL spatial attention mechanism to aggregate these features across the scene. This process ensures that the density predictions for each point consider the semantic context of neighboring points, facilitating a more coherent and plausible reconstruction of occluded regions. By guiding the attention mechanism with text-based category features, the model leverages both global and local semantic contexts, significantly improving the reconstruction accuracy over previous methods that treat points in isolation.

Experimental Validation

KYN's effectiveness is demonstrated through extensive experiments on the KITTI-360 dataset, where it achieves state-of-the-art performance in both scene and object-level reconstructions. Notably, KYN shows considerable improvement in accurately modeling occluded areas, mitigating trailing effects commonly observed in prior work. Furthermore, the application of KYN to the DDAD dataset illustrates its robust zero-shot generalization capability, underscoring the benefit of leveraging semantic and contextual knowledge in single-view reconstruction tasks.

Ablation Studies and Comparisons

A series of ablation studies highlight the individual contributions of the VL modulation and spatial attention mechanisms within KYN. These studies affirm the importance of integrating fine-grained semantic information and global-to-local spatial reasoning for improving single-view reconstruction outputs. Additionally, comparisons with existing semantic feature fusion techniques further validate the superiority of KYN's approach, combining VL features with spatial attention to achieve more accurate and semantically coherent reconstructions.

Conclusion and Future Directions

KYN represents a significant step forward in single-view scene reconstruction, addressing the limitations of existing methods by effectively incorporating semantic and spatial context into the reconstruction process. The introduction of VL features not only enhances single-view reconstruction accuracy but also presents exciting avenues for future research in open-vocabulary 3D scene understanding and modeling.