Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
97 tokens/sec
GPT-4o
53 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards 3D VR-Sketch to 3D Shape Retrieval (2209.10020v2)

Published 20 Sep 2022 in cs.CV

Abstract: Growing free online 3D shapes collections dictated research on 3D retrieval. Active debate has however been had on (i) what the best input modality is to trigger retrieval, and (ii) the ultimate usage scenario for such retrieval. In this paper, we offer a different perspective towards answering these questions -- we study the use of 3D sketches as an input modality and advocate a VR-scenario where retrieval is conducted. Thus, the ultimate vision is that users can freely retrieve a 3D model by air-doodling in a VR environment. As a first stab at this new 3D VR-sketch to 3D shape retrieval problem, we make four contributions. First, we code a VR utility to collect 3D VR-sketches and conduct retrieval. Second, we collect the first set of $167$ 3D VR-sketches on two shape categories from ModelNet. Third, we propose a novel approach to generate a synthetic dataset of human-like 3D sketches of different abstract levels to train deep networks. At last, we compare the common multi-view and volumetric approaches: We show that, in contrast to 3D shape to 3D shape retrieval, volumetric point-based approaches exhibit superior performance on 3D sketch to 3D shape retrieval due to the sparse and abstract nature of 3D VR-sketches. We believe these contributions will collectively serve as enablers for future attempts at this problem. The VR interface, code and datasets are available at https://tinyurl.com/3DSketch3DV.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (38)
  1. End-to-end cad model retrieval and 9dof alignment in 3d scans. In Proceedings of the IEEE International Conference on Computer Vision, pages 2551–2560, 2019.
  2. Generative and discriminative voxel modeling with convolutional neural networks. arXiv preprint arXiv:1608.04236, 2016.
  3. P. Bénard and A. Hertzmann. Line drawings from 3d models: A tutorial. Foundations and Trends® in Computer Graphics and Vision, 11(1-2):1–159, 2019.
  4. Repairing man-made meshes via visual driven global optimization with minimum intrusion. ACM Trans. Graph. (SIGGRAPH ASIA), 38(6):158:1–158:18, 2019.
  5. Joint embedding of 3d scan and cad objects. In Proceedings of the IEEE International Conference on Computer Vision, pages 8749–8758, 2019.
  6. A point set generation network for 3d object reconstruction from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 605–613, 2017.
  7. Gvcnn: Group-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 264–272, 2018.
  8. 3d sketching for interactive model retrieval in virtual reality. In Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering, pages 1–12, 2018.
  9. Flowrep: Descriptive curve networks for free-form design shapes. ACM Transactions on Graphics (TOG), 36(4):1–14, 2017.
  10. 3d pose estimation and 3d model retrieval for objects in the wild. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 3022–3031, 2018.
  11. Location field descriptors: Single image 3d model retrieval in the wild. In 2019 International Conference on 3D Vision (3DV), pages 583–593. IEEE, 2019.
  12. D. Ha and D. Eck. A neural representation of sketch drawings. In International Conference on Learning Representations (ICLR), 2018.
  13. View n-gram network for 3d object retrieval. In Proceedings of the IEEE International Conference on Computer Vision, pages 7515–7524, 2019.
  14. Triplet-center loss for multi-view 3d object retrieval. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1945–1954, 2018.
  15. Cross-domain image-based 3d shape retrieval by view sequence learning. In 2018 International Conference on 3D Vision (3DV), pages 258–266. IEEE, 2018.
  16. Shrec’16 track: 3d sketch-based 3d shape retrieval. 2016.
  17. 3d sketch-based 3d model retrieval. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, pages 555–558, 2015.
  18. Angular triplet-center loss for multi-view 3d shape retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 8682–8689, 2019.
  19. Learning a disentangled embedding for monocular 3d shape retrieval and pose estimation. arXiv preprint arXiv:1812.09899, 2018.
  20. Cross-domain 3d model retrieval via visual domain adaptation. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, pages 828–834. AAAI Press, 2018.
  21. Parametrization quantization with free boundaries for trimmed quad meshing. ACM Transactions on Graphics (TOG), 38(4):1–14, 2019.
  22. Boosting multi-view convolutional neural networks for 3d object recognition via view saliency. In Chinese Conference on Image and Graphics Technologies, pages 199–209. Springer, 2017.
  23. D. Maturana and S. Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In 2015 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 922–928. IEEE, 2015.
  24. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  25. Volumetric and multi-view cnns for object classification on 3d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5648–5656, 2016.
  26. Pointnet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in neural information processing systems, pages 5099–5108, 2017.
  27. Sketch-based image retrieval via siamese convolutional neural network. In 2016 IEEE International Conference on Image Processing (ICIP), pages 2460–2464. IEEE, 2016.
  28. N.-p. Ramer-Douglas-Peucker. Ramer-douglas-peucker algorithm. 1972.
  29. Deep learning with sets and point clouds. arXiv preprint arXiv:1611.04500, 2016.
  30. K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409.1556, 2014.
  31. Multi-view convolutional neural networks for 3d shape recognition. In Proceedings of the IEEE international conference on computer vision, pages 945–953, 2015.
  32. Deformation-aware 3d model embedding and retrieval. arXiv preprint arXiv:2004.01228, 2020.
  33. Sketch-based 3d shape retrieval using convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1875–1883, 2015.
  34. Learning fine-grained image similarity with deep ranking. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1386–1393, 2014.
  35. A discriminative feature learning approach for deep face recognition. In European conference on computer vision, pages 499–515. Springer, 2016.
  36. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1912–1920, 2015.
  37. 3d sketch-based 3d model retrieval with convolutional neural network. In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2936–2941. IEEE, 2016.
  38. Sketch me that shoe. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 799–807, 2016.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Ling Luo (32 papers)
  2. Yulia Gryaditskaya (11 papers)
  3. Yongxin Yang (73 papers)
  4. Tao Xiang (324 papers)
  5. Yi-Zhe Song (120 papers)
Citations (11)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now