Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
166 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
42 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Single-view 3D reconstruction via inverse procedural modeling (2310.13373v1)

Published 20 Oct 2023 in cs.GR and cs.CV

Abstract: We propose an approach to 3D reconstruction via inverse procedural modeling and investigate two variants of this approach. The first option consists in the fitting set of input parameters using a genetic algorithm. We demonstrate the results of our work on tree models, complex objects, with the reconstruction of which most existing methods cannot handle. The second option allows us to significantly improve the precision by using gradients within memetic algorithm, differentiable rendering and also differentiable procedural generators. In our work we see 2 main contributions. First, we propose a method to join differentiable rendering and inverse procedural modeling. This gives us an opportunity to reconstruct 3D model more accurately than existing approaches when a small number of input images are available (even for single image). Second, we join both differentiable and non-differentiable procedural generators in a single framework which allow us to apply inverse procedural modeling to fairly complex generators: when gradient is available, reconstructions is precise, when gradient is not available, reconstruction is approximate, but always high quality without visual artifacts.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (66)
  1. Neural Fields in Visual Computing and Beyond. cgforum 2022, 41.
  2. Image2mesh: A learning framework for single image 3d reconstruction. In Proceedings of the Computer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part I 14. Springer, 2019, pp. 365–381.
  3. ZeroMesh: Zero-shot Single-view 3D Mesh Reconstruction. arXiv preprint arXiv:2208.02676 2022.
  4. NeRFMeshing: Distilling Neural Radiance Fields into Geometrically-Accurate 3D Meshes. arXiv preprint arXiv:2303.09431 2023.
  5. Delicate Textured Mesh Recovery from NeRF via Adaptive Surface Refinement. arXiv preprint arXiv:2303.02091 2023.
  6. Unified Shape and SVBRDF Recovery using Differentiable Monte Carlo Rendering. CoRR 2021, abs/2103.15208, [2103.15208].
  7. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. IEEE, 2019, pp. 165–174.
  8. 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics 2023, 42.
  9. Structure-from-motion revisited. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 4104–4113.
  10. Past, present, and future of simultaneous localization and mapping: Toward the robust-perception age. IEEE Transactions on robotics 2016, 32, 1309–1332.
  11. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the Proceedings of the European conference on computer vision (ECCV), 2018, pp. 52–67.
  12. Total3dunderstanding: Joint layout, object pose and mesh reconstruction for indoor scenes from a single image. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 55–64.
  13. Shelf-supervised mesh prediction in the wild. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8843–8852.
  14. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In Proceedings of the Computer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14. Springer, 2016, pp. 628–644.
  15. Corenet: Coherent 3d scene reconstruction from a single rgb image. In Proceedings of the Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16. Springer, 2020, pp. 366–383.
  16. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 605–613.
  17. Unsupervised learning of fine structure generation for 3d point clouds by 2d projections matching. In Proceedings of the Proceedings of the ieee/cvf international conference on computer vision, 2021, pp. 12466–12477.
  18. Learning to predict 3d objects with an interpolation-based differentiable renderer. Advances in neural information processing systems 2019, 32.
  19. What do single-view 3d reconstruction networks learn? In Proceedings of the Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 3405–3414.
  20. Learning to reconstruct shapes from unseen classes. Advances in neural information processing systems 2018, 31.
  21. Modular primitives for high-performance differentiable rendering. ACM Transactions on Graphics (TOG) 2020, 39, 1–14.
  22. Path-space differentiable rendering of participating media. ACM Transactions on Graphics (TOG) 2021, 40, 1–15.
  23. Reconstructing Translucent Objects Using Differentiable Rendering; Association for Computing Machinery: New York, NY, USA, 2022; SIGGRAPH ’22. https://doi.org/10.1145/3528233.3530714.
  24. Differentiable Rendering of Neural SDFs through Reparameterization. In Proceedings of the ACM SIGGRAPH Asia 2022 Conference Proceedings; Association for Computing Machinery: New York, NY, USA, 2022; SIGGRAPH Asia ’22. https://doi.org/10.1145/3550469.3555397.
  25. Deep Active Surface Models. In Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 11647–11656. https://doi.org/10.1109/CVPR46437.2021.01148.
  26. Fast quasi-harmonic weights for geometric data interpolation. ACM Transactions on Graphics (TOG) 2021, 40, 1–15.
  27. Large steps in inverse rendering of geometry. ACM Transactions on Graphics (TOG) 2021, 40, 1–13.
  28. Differentiable signed distance function rendering. ACM Transactions on Graphics (TOG) 2022, 41, 1–18.
  29. Oswald, M.R. Convex Variational Methods for Single-View and Space-Time Multi-View Reconstruction. Phd thesis, Technischen Universität München eingereicht und durch die Fakultät für Informatik, München, 2015. https://mediatum.ub.tum.de/doc/1232437/928830.pdf.
  30. Neural volumes: Learning dynamic renderable volumes from images. arXiv preprint arXiv:1906.07751 2019.
  31. A non-exponential transmittance model for volumetric scene representations. ACM Transactions on Graphics (TOG) 2021, 40, 1–16.
  32. Differentiable surface splatting for point-based geometry processing. ACM Transactions on Graphics (TOG) 2019, 38, 1–14.
  33. Adop: Approximate differentiable one-pixel point rendering. ACM Transactions on Graphics (TOG) 2022, 41, 1–14.
  34. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM 2021, 65, 99–106.
  35. Plenoxels: Radiance fields without neural networks. In Proceedings of the Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022, pp. 5501–5510.
  36. Instant neural graphics primitives with a multiresolution hash encoding. ACM Transactions on Graphics (ToG) 2022, 41, 1–15.
  37. Procedural content generation for games: A survey. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) 2013, 9, 1–22.
  38. A survey on the procedural generation of virtual worlds. Multimodal Technologies and Interaction 2017, 1, 27.
  39. Inc, S.I. SpeedTree, 2017. http://www.speedtree.com.
  40. The algorithmic beauty of plants; Springer Science & Business Media, 2012.
  41. Tree growth modelling constrained by growth equations. In Proceedings of the Computer Graphics Forum. Wiley Online Library, 2018, Vol. 37, pp. 239–253.
  42. Inverse procedural modelling of trees. In Proceedings of the Computer Graphics Forum. Wiley Online Library, 2014, Vol. 33, pp. 118–131.
  43. Proceduralization for Editing 3D Architectural Models. In Proceedings of the International Conference on 3D Vision (Proceedings 3DV 2016). 3DV’16, 2016, p. 9 pages.
  44. Inverse Procedural Modeling of 3D Models for Virtual Worlds. In Proceedings of the ACM SIGGRAPH 2016 Courses; ACM: New York, NY, USA, 2016; SIGGRAPH ’16, pp. 16:1–16:316. https://doi.org/10.1145/2897826.2927323.
  45. Inverse procedural modeling of branching structures by inferring L-systems. ACM Transactions on Graphics (TOG) 2020, 39, 1–13.
  46. Fitting Parameters for Procedural Plant Generation 2022.
  47. An inverse procedural modeling pipeline for svbrdf maps. ACM Transactions on Graphics (TOG) 2022, 41, 1–17.
  48. Inverse procedural modeling of facade layouts. arXiv preprint arXiv:1308.0419 2013.
  49. Fitting Procedural Yarn Models for Realistic Cloth Rendering 2016. 35. https://doi.org/10.1145/2897824.2925932.
  50. Neural inverse procedural modeling of knitting yarns from images. arXiv preprint arXiv:2303.00154 2023.
  51. Automatic differentiable procedural modeling. In Proceedings of the Computer Graphics Forum. Wiley Online Library, 2022, Vol. 41, pp. 289–307.
  52. Reconstructing Symmetric Curved Surfaces from a Single Image and Its Application. In Proceedings of the Interactive Technologies and Sociotechnical Systems; Zha, H.; Pan, Z.; Thwaites, H.; Addison, A.C.; Forte, M., Eds.; Springer Berlin Heidelberg: Berlin, Heidelberg, 2006; pp. 204–213.
  53. Single-View 3d Reconstruction of Surface of Revolution. Pattern Recognition Letters 2022, 1.
  54. 3D building reconstruction from single street view images using deep learning. International Journal of Applied Earth Observation and Geoinformation 2022, 112, 102859. https://doi.org/https://doi.org/10.1016/j.jag.2022.102859.
  55. Deep Textured 3D Reconstruction of Human Bodies. CoRR 2018, abs/1809.06547, [1809.06547].
  56. A review of 3D human body pose estimation and mesh recovery. Digital Signal Processing 2022, 128, 103628. https://doi.org/https://doi.org/10.1016/j.dsp.2022.103628.
  57. Mitsuba 3 renderer, 2022. https://mitsuba-renderer.org.
  58. Bell, B.M. CppAD: a package for C++ algorithmic differentiation. Computational Infrastructure for Operations Research 2012, 57.
  59. Mitchell, M. An introduction to genetic algorithms; MIT press, 1998.
  60. Memetic algorithms and memetic computing optimization: A literature review. Swarm and Evolutionary Computation 2012, 2, 1–14.
  61. Interactive modeling and authoring of climbing plants. In Proceedings of the Computer Graphics Forum. Wiley Online Library, 2017, Vol. 36, pp. 49–61.
  62. Light-Guided Tree Modeling of Diverse Biomorphs. In Proceedings of the PG (Short Papers), 2015, pp. 53–57.
  63. Creation and rendering of realistic trees. In Proceedings of the Proceedings of the 22nd annual conference on Computer graphics and interactive techniques, 1995, pp. 119–128.
  64. Learning to reconstruct botanical trees from single images. ACM Transactions on Graphics (TOG) 2021, 40, 1–15.
  65. Differentiable monte carlo ray tracing through edge sampling. ACM Transactions on Graphics (TOG) 2018, 37, 1–11.
  66. Differentiable Procedural Models for Single-view 3D Mesh Reconstruction. In Proceedings of the Computer Graphics and Visual Computing (CGVC); Vangorp, P.; Hunter, D., Eds. The Eurographics Association, 2023. https://doi.org/10.2312/cgvc.20231189.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now