Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Correspondences of the Third Kind: Camera Pose Estimation from Object Reflection (2312.04527v2)

Published 7 Dec 2023 in cs.CV

Abstract: Computer vision has long relied on two kinds of correspondences: pixel correspondences in images and 3D correspondences on object surfaces. Is there another kind, and if there is, what can they do for us? In this paper, we introduce correspondences of the third kind we call reflection correspondences and show that they can help estimate camera pose by just looking at objects without relying on the background. Reflection correspondences are point correspondences in the reflected world, i.e., the scene reflected by the object surface. The object geometry and reflectance alters the scene geometrically and radiometrically, respectively, causing incorrect pixel correspondences. Geometry recovered from each image is also hampered by distortions, namely generalized bas-relief ambiguity, leading to erroneous 3D correspondences. We show that reflection correspondences can resolve the ambiguities arising from these distortions. We introduce a neural correspondence estimator and a RANSAC algorithm that fully leverages all three kinds of correspondences for robust and accurate joint camera pose and object shape estimation just from the object appearance. The method expands the horizon of numerous downstream tasks, including camera pose estimation for appearance modeling (e.g., NeRF) and motion estimation of reflective objects (e.g., cars on the road), to name a few, as it relieves the requirement of overlapping background.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (25)
  1. The Bas-Relief Ambiguity. IJCV, 35(1):33–44, 1999.
  2. Partially Calibrated Semi-Generalized Pose from Hybrid Point Correspondences. In Proc. WACV, 2023.
  3. SAMURAI: Shape And Material from Unconstrained Real-world Arbitrary Image collections. In Proc. NeurIPS, 2022.
  4. 3D Pose Estimation and Segmentation using Specular Cues. In Proc. CVPR, pages 1706–1713. IEEE Computer Society, 2009.
  5. Random Sample Consensus: A Paradigm for Model Fitting with Applications to Image Analysis and Automated Cartography. Commun. ACM, 24(6):381–395, 1981.
  6. Christopher G. Harris. Structure-from-Motion under Orthographic Projection. IJCV, 9(5):329–332, 1991.
  7. Berthold K. P. Horn and Robert W. Sjoberg. Calculating the Reflectance Map. Applied optics, 18(11):1770–1779, 1979.
  8. 3D Pose Refinement from Reflections. In Proc. CVPR. IEEE Computer Society, 2008.
  9. NeRO: Neural Geometry and BRDF Reconstruction of Reflective Objects from Multiview Images. ACM TOG, 2023.
  10. David G. Lowe. Distinctive Image Features from Scale-Invariant Keypoints. IJCV, 60(2):91–110, 2004.
  11. NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. In Proc. ECCV, 2020.
  12. Recognition Using Specular Highlights. TPAMI, 35(3):639–652, 2013.
  13. Differentiable Volumetric Rendering: Learning Implicit 3D Representations without 3D Supervision. In Proc. CVPR, 2020.
  14. Pix2Pose: Pixel-Wise Coordinate Regression of Objects for 6D Pose Estimation. In Proc. ICCV, 2019.
  15. Camera and Light Calibration from Reflections on a Sphere. CVIU, 117(10):1536–1547, 2013.
  16. Structure-from-Motion Revisited. In Proc. CVPR, 2016.
  17. Representation Learning with Contrastive Predictive Coding. arXiv preprint arXiv:1807.03748, 2018.
  18. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods, 17:261–272, 2020.
  19. NeuS: Learning Neural Implicit Surfaces by Volume Rendering for Multi-view Reconstruction. In Proc. NeurIPS, pages 27171–27183, 2021.
  20. nLMVS-Net: Deep Non-Lambertian Multi-View Stereo. In Proc. WACV, pages 3037–3046, 2023a.
  21. DeepShaRM: Multi-View Shape and Reflectance Map Recovery Under Unknown Lighting. arXiv preprint arXiv:2310.17632, 2023b.
  22. Multiview Neural Surface Reconstruction by Disentangling Geometry and Appearance. In Proc. NeurIPS, pages 2492–2502, 2020.
  23. PhySG: Inverse Rendering with Spherical Gaussians for Physics-based Material Editing and Relighting. In Proc. CVPR, pages 5453–5462, 2021a.
  24. NeRFactor: Neural Factorization of Shape and Reflectance Under an Unknown Illumination. ACM TOG, 40(6):237:1–237:18, 2021b.
  25. A Survey of Structure from Motion. Acta Numerica, 26:305–364, 2017.

Summary

  • The paper introduces reflection correspondences as a third modality that leverages neural networks and RANSAC to resolve camera pose ambiguities.
  • The paper employs reflectance maps to capture complex light interactions, enabling robust detection of both reflection and conventional correspondences.
  • The paper demonstrates enhanced camera calibration and object pose estimation, particularly for reflective, textureless objects where traditional methods falter.

Introduction

Computer vision methodologies commonly make use of two types of correspondences to comprehend three-dimensional space through images. Pixel correspondences across images and 3D correspondences on the surfaces of objects are well-established tools. A novel kind of correspondence, however, has recently been explored. Termed reflection correspondences, these are based on the idea that points on a reflective object's surface provide unique information due to the way they interact with light. This insight has been leveraged in new research to estimate camera pose without relying on a textured background or a known lighting environment.

Camera Pose from Reflections

Reflection correspondences are effectively the points on an object's glossy or reflective surface that reflect the same part of the environment in different images. These correspondences do not conform to traditional color consistency constraints because of specular highlights and other complex radiometric effects present on the object. By using reflection correspondences, along with conventional pixel and 3D correspondences, the ambiguity that typically plagues the estimation process can be resolved. This approach requires understanding how light interacts with the object's surface - a challenge that is addressed through the use of neural networks and geometric algorithms.

Methodology

To exploit reflection correspondences for camera pose estimation, the researchers introduced several innovations. A neural correspondence estimator aids in the detection of both 3D and reflection correspondences that are robust even when dealing with objects distorted by bas-relief ambiguity. Additionally, a RANSAC-based framework alternates between estimating camera positions and the object’s reflective geometry.

A key part of this process involves generating reflectance maps, which encode the surrounding environment modified by an object's surface reflectance. These maps are crucial for detecting reflection correspondences. The detected correspondences in turn help to resolve ambiguity from other types of correspondences which are traditionally used in isolation.

Applications and Implications

The method not only challenges previously held restrictions on image capture for camera pose estimation but also pushes the boundaries of how object appearances can be utilized. It opens up potential advancements in applications beyond camera pose and shape recovery. For instance, camera calibration and object pose estimation could significantly benefit from the proposed technique, particularly when classical correspondences are inadequate.

Concluding Remarks

This research introduces reflection correspondences as a third type of essential information for understanding the relationship between images, camera movement, and object geometry. It successfully addresses the challenge of estimating camera pose and object shape from reflective, textureless objects. This method stands to streamline processes in computer vision where current requirements for overlapping backgrounds or diffuse surface textures introduce limitations.

X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets