Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SD-MVS: Segmentation-Driven Deformation Multi-View Stereo with Spherical Refinement and EM optimization (2401.06385v1)

Published 12 Jan 2024 in cs.CV

Abstract: In this paper, we introduce Segmentation-Driven Deformation Multi-View Stereo (SD-MVS), a method that can effectively tackle challenges in 3D reconstruction of textureless areas. We are the first to adopt the Segment Anything Model (SAM) to distinguish semantic instances in scenes and further leverage these constraints for pixelwise patch deformation on both matching cost and propagation. Concurrently, we propose a unique refinement strategy that combines spherical coordinates and gradient descent on normals and pixelwise search interval on depths, significantly improving the completeness of reconstructed 3D model. Furthermore, we adopt the Expectation-Maximization (EM) algorithm to alternately optimize the aggregate matching cost and hyperparameters, effectively mitigating the problem of parameters being excessively dependent on empirical tuning. Evaluations on the ETH3D high-resolution multi-view stereo benchmark and the Tanks and Temples dataset demonstrate that our method can achieve state-of-the-art results with less time consumption.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. PatchMatch Stereo - Stereo Matching with Slanted Support Windows. In Proc. Brit. Mach. Vis. Conf. (BMVC), 14.1–14.11.
  2. Accurate 3-D Reconstruction Under IoT Environments and Its Applications to Augmented Reality. IEEE Trans. Ind. Inf., 17(3): 2090–2100.
  3. Multiview Stereo and Silhouette Consistency via Convex Functionals over Convex Domains. IEEE Trans. Pattern Anal. Mach. Intell., 33(6): 1161–1174.
  4. Massively Parallel Multiview Stereopsis by Surface Normal Diffusion. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 873–881.
  5. Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 2492–2501.
  6. Segment Anything. arXiv:2304.02643.
  7. Tanks and Temples: Benchmarking Large-Scale Scene Reconstruction. ACM Trans. Graph., 36(4).
  8. Plane Completion and Filtering for Multi-View Stereo Reconstruction. In Proc. DAGM German Conf. (GCPR), volume 11824, 18–32.
  9. PatchMatch-RL: Deep MVS with Pixelwise Depth, Normal, and Visibility. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 6138–6147.
  10. Dense Surface Reconstruction from Monocular Vision and LiDAR. In Proc. IEEE Conf. Robot. Automat. (ICRA), 6905–6911.
  11. Confidence-Based Large-Scale Dense Multi-View Stereo. IEEE Trans. on Image Process., 29: 7176–7191.
  12. EPP-MVSNet: Epipolar-assembling based Depth Prediction for Multi-view Stereo. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 5712–5720.
  13. Revisiting PatchMatch Multi-View Stereo for Urban 3D Reconstruction. In Proc. IEEE Intelligent Vehicles Symp. (IV), 190–196.
  14. A nonsmooth version of Newton’s method. Math. Program., 58: 353–367.
  15. TAPA-MVS: Textureless-Aware PAtchMatch Multi-View Stereo. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 10412–10421.
  16. A Multi-View Stereo Benchmark with High-Resolution Images and Multi-Camera Videos. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR).
  17. Pixelwise View Selection for Unstructured Multi-View Stereo. In Proc. Eur. Conf. Comput. Vis. (ECCV), volume 9907, 501–518.
  18. A Comparison and Evaluation of Multi-View Stereo Reconstruction Algorithms. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), volume 1, 519–528.
  19. Efficient Edge-Preserving Multi-View Stereo Network for Depth Estimation. In Proc. of the AAAI Conf. Artif. Intell. (AAAI), 2348–2356.
  20. Multiview Stereo via Volumetric Graph-Cuts and Occlusion Robust Photo-Consistency. IEEE Trans. Pattern Anal. Mach. Intell., 29(12): 2241–2246.
  21. IterMVS: Iterative Probability Estimation for Efficient Multi-View Stereo. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 8596–8605.
  22. PatchmatchNet: Learned Multi-View Patchmatch Stereo. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 14189–14198.
  23. MVSTER: Epipolar Transformer for Efficient Multi-view Stereo. In Proc. Eur. Conf. Comput. Vis. (ECCV), volume 13691, 573–591.
  24. AA-RMVSNet: Adaptive Aggregation Recurrent Multi-view Stereo Network. In Proc. IEEE/CVF Int. Conf. Comput. Vis. (ICCV), 6167–6176.
  25. Williams, L. 1983. Pyramidal Parametrics. In Proc. of the 10th Annu. Conf. on Comput. Graph. and Interact. Techn. (SIGGRAPH), 1–11.
  26. Multi-Scale Geometric Consistency Guided and Planar Prior Assisted Multi-View Stereo. IEEE Trans. Pattern Anal. Mach. Intell., 1–18.
  27. Multi-Scale Geometric Consistency Guided Multi-View Stereo. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 5478–5487.
  28. Planar Prior Assisted PatchMatch Multi-View Stereo. In Proc. of the AAAI Conf. Artif. Intell. (AAAI), volume 34, 12516–12523.
  29. MARMVS: Matching Ambiguity Reduced Multiple View Stereo for Efficient Large Scale Scene Reconstruction. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 5980–5989.
  30. Dense hybrid recurrent multi-view stereo net with dynamic consistency checking. In Proc. Eur. Conf. Comput. Vis. (ECCV), 674–689.
  31. MVSNet: Depth Inference for Unstructured Multi-view Stereo. In Proc. Eur. Conf. Comput. Vis. (ECCV), volume 11212, 785–801.
  32. Recurrent MVSNet for High-Resolution Multi-View Stereo Depth Inference. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 5520–5529.
  33. Yuesong Wang; Zhaojie Zeng; Tao Guan; Wei Yang; Zhuo Chen; Wenkai Liu; Luoyuan Xu; and Yawei Luo. 2023. Adaptive Patch Deformation for Textureless-Resilient Multi-View Stereo. In Proc. IEEE/CVF Conf. Comput. Vis. Pattern Recognit. (CVPR), 1621–1630.
Citations (8)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets