Papers
Topics
Authors
Recent
2000 character limit reached

Pix2Repair: Implicit Shape Restoration from Images (2305.18273v3)

Published 29 May 2023 in cs.CV

Abstract: We present Pix2Repair, an automated shape repair approach that generates restoration shapes from images to repair fractured objects. Prior repair approaches require a high-resolution watertight 3D mesh of the fractured object as input. Input 3D meshes must be obtained using expensive 3D scanners, and scanned meshes require manual cleanup, limiting accessibility and scalability. Pix2Repair takes an image of the fractured object as input and automatically generates a 3D printable restoration shape. We contribute a novel shape function that deconstructs a latent code representing the fractured object into a complete shape and a break surface. We also introduce Fantastic Breaks Imaged, the first large-scale dataset of 11,653 real-world images of fractured objects for training and evaluating image-based shape repair approaches. Our dataset contains images of objects from Fantastic Breaks, complete with rich annotations. We show restorations for real fractures from our dataset, and for synthetic fractures from the Geometric Breaks and Breaking Bad datasets. Our approach outperforms shape completion approaches adapted for shape repair in terms of chamfer distance, normal consistency, and percent restorations generated.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (63)
  1. Combining 3d technologies in the field of cultural heritage: Three case studies. In VAST (Short and Project Papers), 2011.
  2. In-house manufacture of sterilizable, scaled, patient-specific 3d-printed models for rhinoplasty. Aesthetic Surgery Journal, 39(3):254–263, 2019.
  3. A system for high-volume acquisition and matching of fresco fragments: Reassembling theran wall paintings. ACM transactions on graphics (TOG), 27(3):1–9, 2008.
  4. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University — Princeton University — Toyota Technological Institute at Chicago, 2015.
  5. Neural shape mating: Self-supervised object assembly with adversarial shape priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12724–12733, 2022.
  6. Learning implicit fields for generative shape modeling. In Proc. CVPR, pages 5939–5948, Piscataway, NJ, 2019. IEEE.
  7. 3d-r2n2: A unified approach for single and multi-view 3d object reconstruction. In European conference on computer vision, pages 628–644. Springer, 2016.
  8. Sg-nn: Sparse generative neural networks for self-supervised scene completion of rgb-d scans. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 849–858, 2020.
  9. Google scanned objects: A high-quality dataset of 3d scanned household items. In 2022 International Conference on Robotics and Automation (ICRA), pages 2553–2560. IEEE, 2022.
  10. Learning how to match fresco fragments. Journal on Computing and Cultural Heritage (JOCCH), 4(2):1–13, 2011.
  11. Mesh r-cnn. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9785–9795, 2019.
  12. On the transfer of inductive bias from simulation to the real world: a new disentanglement dataset. Advances in Neural Information Processing Systems, 32, 2019.
  13. A papier-mâché approach to learning 3d surface generation. In Proc. CVPR, pages 216–224, Piscataway, NJ, 2018. IEEE.
  14. Theory of t-norms and fuzzy inference methods. Fuzzy sets and systems, 40(3):431–450, 1991.
  15. 3d reconstruction of incomplete archaeological objects using a generative adversarial network. In Proceedings of Computer Graphics International, pages 5–11. ACM, New York, NY, 2018.
  16. Potsac: A robust axis estimator for axially symmetric pot fragments. In ICCV Workshops, pages 1421–1428, 2019.
  17. Reassembling fractured objects by geometric matching. In ACM SIGGRAPH, pages 569–578, 2006.
  18. Planes vs. chairs: Category-guided 3d shape learning without any 3d cues. In European Conference on Computer Vision, pages 727–744. Springer, 2022.
  19. Neural template: Topology-aware reconstruction and disentangled generation of 3d meshes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18572–18582, 2022.
  20. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pages 448–456. pmlr, 2015.
  21. Adam: A method for stochastic optimization. In Proc. ICLR, pages 1–15, La Jolla, CA, 2014. International Conference on Representation Learning.
  22. Segment anything. arXiv preprint arXiv:2304.02643, 2023.
  23. Automated reconstruction of smoothly joining 3d printed restorations to fix broken objects. In Proc. SCF, pages 1–12, New York, NY, 2019. ACM.
  24. Deepjoin: Learning a joint occupancy, signed distance, and normal field function for shape repair. ACM Transactions on Graphics (TOG), 41(6):1–10, 2022a.
  25. Deepmend: Learning occupancy functions to represent shape for repair. In European Conference on Computer Vision, pages 433–450. Springer, 2022b.
  26. MendNet: Restoration of Fractured Shapes Using Learned Occupancy Functions. Computer Graphics Forum, 2022c.
  27. Fantastic breaks: A dataset of paired 3d scans of real-world broken objects and their complete counterparts. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2023.
  28. Symmetry based chinese ancient architecture reconstruction from incomplete point cloud. In 2014 5th International conference on digital home, pages 157–161. IEEE, 2014.
  29. D2im-net: Learning detail disentangled implicit fields from single images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10246–10255, 2021.
  30. Sdf-srn: Learning signed distance 3d object reconstruction from static images. arXiv preprint arXiv:2010.10505, 1(1):1–17, 2020.
  31. Marching cubes: A high resolution 3d surface construction algorithm. ACM SIGGRAPH Computer Graphics, 21(4):163–169, 1987.
  32. Segment anything in medical images. ArXiv, abs/2304.12306, 2023.
  33. 3d-lmnet: Latent embedding matching for accurate and diverse 3d point cloud reconstruction from a single image. arXiv preprint arXiv:1807.07796, 2018.
  34. Matthew Matl. Pyrender, 2023.
  35. Object completion using k-sparse optimization. In Computer Graphics Forum, pages 13–21. Wiley Online Library, 2015.
  36. Occupancy networks: Learning 3d reconstruction in function space. In Proc. CVPR, pages 4460–4470, Piscataway, NJ, 2019. IEEE.
  37. Partnet: A large-scale benchmark for fine-grained and hierarchical part-level 3d object understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 909–918, 2019.
  38. Atlas: End-to-end 3d scene reconstruction from posed images. In European conference on computer vision, pages 414–431. Springer, 2020.
  39. Deepsdf: Learning continuous signed distance functions for shape representation. In Proc. CVPR, pages 165–174, Piscataway, NJ, 2019. IEEE.
  40. Designing the next generation virtual museum: Making 3d artifacts available for viewing and download. In Making History Interactive: Proceedings of the 37th Annual International Conference on Computer Applications and Quantitative Methods in Archaeology (CAA), pages 1–6, 2009.
  41. Visual enhancement of single-view 3d point cloud reconstruction. Computers & Graphics, 102:112–119, 2022.
  42. Corenet: Coherent 3d scene reconstruction from a single rgb image. In European Conference on Computer Vision, pages 366–383. Springer, 2020.
  43. 3d printing based on imaging data: review of medical applications. International journal of computer assisted radiology and surgery, 5(4):335–341, 2010.
  44. Reviving the dinosaur: virtual reconstruction and three-dimensional printing of a dinosaur vertebra. Radiology, 270(3):864–871, 2014.
  45. 3d models for cultural heritage: Beyond plain visualization. Computer, 44(7):48–55, 2011.
  46. The use of rapid prototyping in the joining of fractured historical silver object. Rapid Prototyping Journal, 2018.
  47. Breaking bad: A dataset for geometric fracture and reassembly. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
  48. Geometric granularity aware pixel-to-mesh. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13097–13106, 2021.
  49. Ivan Sipiran. Completion of cultural heritage objects with rotational symmetry. In Proceedings of the 11th Eurographics Workshop on 3D Object Retrieval, pages 87–93, 2018.
  50. Pixel2mesh: Generating 3d mesh models from single rgb images. In Proceedings of the European conference on computer vision (ECCV), pages 52–67, 2018.
  51. 3dn: 3d deformation network. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1038–1046, 2019.
  52. Mdisn: Learning multiscale deformed implicit fields from single images. Visual Informatics, 2022.
  53. Marrnet: 3d shape reconstruction via 2.5 d sketches. Advances in neural information processing systems, 30, 2017.
  54. Learning shape priors for single-view 3d completion and reconstruction. In Proceedings of the European Conference on Computer Vision (ECCV), pages 646–662, 2018.
  55. Pq-net: A generative part seq2seq network for 3d shapes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 829–838, 2020.
  56. Density-aware chamfer distance as a comprehensive metric for point cloud completion. arXiv preprint arXiv:2111.12702, 2021.
  57. Voxurf: Voxel-based efficient and accurate neural surface reconstruction. arXiv preprint arXiv:2208.12697, 2022.
  58. Pix2vox: Context-aware 3d reconstruction from single and multi-view images. In Proceedings of the IEEE/CVF international conference on computer vision, pages 2690–2698, 2019.
  59. Pix2vox++: Multi-scale context-aware 3d object reconstruction from single and multiple images. International Journal of Computer Vision, 128(12):2919–2935, 2020.
  60. Ladybird: Quasi-monte carlo sampling for deep implicit field based 3d reconstruction with symmetry. In European Conference on Computer Vision, pages 248–263. Springer, 2020.
  61. Complete & label: A domain adaptation approach to semantic segmentation of lidar point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15363–15373, 2021.
  62. Part-wise atlasnet for 3d point cloud reconstruction from a single image. Knowledge-Based Systems, page 108395, 2022.
  63. Thingi10k: A dataset of 10,000 3d-printing models. arXiv preprint arXiv:1605.04797, 2016.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now

Whiteboard

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up