Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
158 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

iFusion: Inverting Diffusion for Pose-Free Reconstruction from Sparse Views (2312.17250v1)

Published 28 Dec 2023 in cs.CV

Abstract: We present iFusion, a novel 3D object reconstruction framework that requires only two views with unknown camera poses. While single-view reconstruction yields visually appealing results, it can deviate significantly from the actual object, especially on unseen sides. Additional views improve reconstruction fidelity but necessitate known camera poses. However, assuming the availability of pose may be unrealistic, and existing pose estimators fail in sparse view scenarios. To address this, we harness a pre-trained novel view synthesis diffusion model, which embeds implicit knowledge about the geometry and appearance of diverse objects. Our strategy unfolds in three steps: (1) We invert the diffusion model for camera pose estimation instead of synthesizing novel views. (2) The diffusion model is fine-tuned using provided views and estimated poses, turned into a novel view synthesizer tailored for the target object. (3) Leveraging registered views and the fine-tuned diffusion model, we reconstruct the 3D object. Experiments demonstrate strong performance in both pose estimation and novel view synthesis. Moreover, iFusion seamlessly integrates with various reconstruction methods and enhances them.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (86)
  1. Renderdiffusion: Image diffusion for 3d reconstruction, inpainting and generation. In CVPR, 2023.
  2. Spatext: Spatio-textual representation for controllable image generation. In CVPR, 2023.
  3. GeNVS: Generative novel view synthesis with 3D-aware diffusion models. In ICCV, 2023.
  4. Shapenet: An information-rich 3d model repository. arXiv preprint arXiv:1512.03012, 2015.
  5. Mvsnerf: Fast generalizable radiance field reconstruction from multi-view stereo. In CVPR, 2021.
  6. Fantasia3d: Disentangling geometry and appearance for high-quality text-to-3d content creation. In ICCV, 2023.
  7. Category level object pose estimation via neural analysis-by-synthesis. In ECCV, 2020.
  8. Objaverse: A universe of annotated 3d objects. In CVPR, 2023.
  9. Google scanned objects: A high-quality dataset of 3d scanned household items. In ICRA, 2022.
  10. Frido: Feature pyramid diffusion for complex scene image synthesis. In AAAI, 2023.
  11. Make-a-scene: Scene-based text-to-image generation with human priors. In ECCV, 2022.
  12. An image is worth one word: Personalizing text-to-image generation using textual inversion. In ICLR, 2023.
  13. Nerfdiff: Single-image view synthesis with nerf-guided distillation from 3d-aware diffusion. In ICML, 2023.
  14. Denoising diffusion probabilistic models. In NeurIPS, 2020.
  15. LoRA: Low-rank adaptation of large language models. In ICLR, 2022.
  16. Dreamtime: An improved optimization strategy for text-to-3d content creation. arXiv preprint arXiv:2306.12422, 2023.
  17. Putting nerf on a diet: Semantically consistent few-shot view synthesis. In ICCV, 2021.
  18. Zero-shot text-guided object generation with dream fields. In CVPR, 2022.
  19. Few-view object reconstruction with unknown categories and camera poses. arXiv preprint arXiv:2212.04492, 2022.
  20. Leap: Liberate sparse-view 3d modeling from camera poses. arXiv preprint arXiv:2310.01410, 2023.
  21. Shap-e: Generating conditional 3d implicit functions. arXiv preprint arXiv:2305.02463, 2023.
  22. Holodiffusion: Training a 3d diffusion model using 2d images. In CVPR, 2023.
  23. 3d gaussian splatting for real-time radiance field rendering. ACM TOG, 2023.
  24. Infonerf: Ray entropy minimization for few-shot neural volume rendering. In CVPR, 2022.
  25. Adam: A method for stochastic optimization. In ICLR, 2015.
  26. Gligen: Open-set grounded text-to-image generation. In CVPR, 2023.
  27. Relpose++: Recovering 6d poses from sparse-view observations. arXiv preprint arXiv:2305.04926, 2023a.
  28. Barf: Bundle-adjusting neural radiance fields. In ICCV, 2021.
  29. Magic3d: High-resolution text-to-3d content creation. In CVPR, 2023b.
  30. One-2-3-45: Any single image to 3d mesh in 45 seconds without per-shape optimization. In NeurIPS, 2023a.
  31. Zero-1-to-3: Zero-shot one image to 3d object. In ICCV, 2023b.
  32. Syncdreamer: Learning to generate multiview-consistent images from a single-view image. arXiv preprint arXiv:2309.03453, 2023c.
  33. Sparseneus: Fast generalizable neural surface reconstruction from sparse views. In ECCV, 2022.
  34. Wonder3d: Single image to 3d using cross-domain diffusion. arXiv preprint arXiv:2310.15008, 2023.
  35. Diffusion probabilistic models for 3d point cloud generation. In CVPR, 2021.
  36. Realfusion: 360 reconstruction of any object from a single image. In CVPR, 2023.
  37. SDEdit: Guided image synthesis and editing with stochastic differential equations. In ICLR, 2022.
  38. Nerf: Representing scenes as neural radiance fields for view synthesis. In ECCV, 2020.
  39. Clip-mesh: Generating textured meshes from text using pretrained image-text models. In SIGGRAPH Asia, 2022.
  40. Diffrf: Rendering-guided 3d radiance field diffusion. In CVPR, 2023.
  41. Instant neural graphics primitives with a multiresolution hash encoding. ACM TOG, 2022.
  42. Point-e: A system for generating 3d point clouds from complex prompts. arXiv preprint arXiv:2212.08751, 2022.
  43. Regnerf: Regularizing neural radiance fields for view synthesis from sparse inputs. In CVPR, 2022.
  44. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193, 2023.
  45. Latentfusion: End-to-end differentiable reconstruction and rendering for unseen object pose estimation. In CVPR, 2020.
  46. Pytorch: An imperative style, high-performance deep learning library. In NeurIPS, 2019.
  47. Dreamfusion: Text-to-3d using 2d diffusion. In ICLR, 2023.
  48. Magic123: One image to high-quality 3d object generation using both 2d and 3d diffusion priors. arXiv preprint arXiv:2306.17843, 2023.
  49. Unicontrol: A unified diffusion model for controllable visual generation in the wild. In NeurIPS, 2023.
  50. Learning transferable visual models from natural language supervision. In ICML, 2021.
  51. Dreambooth3d: Subject-driven text-to-3d generation. In ICCV, 2023.
  52. High-resolution image synthesis with latent diffusion models. In CVPR, 2022.
  53. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In CVPR, 2023.
  54. Photorealistic text-to-image diffusion models with deep language understanding. In NeurIPS, 2022.
  55. Structure-from-motion revisited. In CVPR, 2016.
  56. H. Scudder. Probability of error of some adaptive pattern-recognition machines. IEEE Transactions on Information Theory, 11(3):363–371, 1965.
  57. 3d neural field generation using triplane diffusion. In CVPR, 2023.
  58. SparsePose: Sparse-view camera pose regression and refinement. In CVPR, 2023.
  59. Deep unsupervised learning using nonequilibrium thermodynamics. In ICML, 2015.
  60. A micro lie theory for state estimation in robotics. arXiv preprint arXiv:1812.01537, 2018.
  61. Generative modeling by estimating gradients of the data distribution. In NeurIPS, 2019.
  62. Test-time training with self-supervision for generalization under distribution shifts. In ICML, 2020.
  63. Viewset diffusion:(0-) image-conditioned 3d generative models from 2d data. In ICCV, 2023.
  64. Dreamgaussian: Generative gaussian splatting for efficient 3d content creation. arXiv preprint arXiv:2309.16653, 2023a.
  65. Make-it-3d: High-fidelity 3d creation from a single image with diffusion prior. In ICCV, 2023b.
  66. Tent: Fully test-time adaptation by entropy minimization. In ICLR, 2021a.
  67. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generation. In CVPR, 2023a.
  68. PoseDiffusion: Solving pose estimation via diffusion-aided bundle adjustment. In ICCV, 2023b.
  69. Neus: Learning neural implicit surfaces by volume rendering for multi-view reconstruction. In NeurIPS, 2021b.
  70. Ibrnet: Learning multi-view image-based rendering. In CVPR, 2021c.
  71. Prolificdreamer: High-fidelity and diverse text-to-3d generation with variational score distillation. In NeurIPS, 2023c.
  72. Novel view synthesis with diffusion models. In ICLR, 2023.
  73. Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation. In CVPR, 2023.
  74. Self-training with noisy student improves imagenet classification. In CVPR, 2020.
  75. Freestyle layout-to-image synthesis. In CVPR, 2023.
  76. Law-diffusion: Complex scene generation by diffusion with layouts. In ICCV, 2023a.
  77. Freenerf: Improving few-shot neural rendering with free frequency regularization. In CVPR, 2023b.
  78. iNeRF: Inverting neural radiance fields for pose estimation. In IROS, 2021.
  79. pixelNeRF: Neural radiance fields from one or few images. In CVPR, 2021.
  80. Lion: Latent point diffusion models for 3d shape generation. In NeurIPS, 2022.
  81. Ners: Neural reflectance surfaces for sparse-view 3d reconstruction in the wild. In NeurIPS, 2021.
  82. Relpose: Predicting probabilistic relative rotation for single objects in the wild. In ECCV, 2022.
  83. Adding conditional control to text-to-image diffusion models. In ICCV, 2023.
  84. Uni-controlnet: All-in-one control to text-to-image diffusion models. In NeurIPS, 2023.
  85. 3d shape generation and completion through point-voxel diffusion. In ICCV, 2021.
  86. Sparsefusion: Distilling view-conditioned diffusion for 3d reconstruction. In CVPR, 2023.
Citations (7)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now