Free-form Shape Modeling in XR: A Systematic Review (2401.00924v1)
Abstract: Shape modeling research in Computer Graphics has been an active area for decades. The ability to create and edit complex 3D shapes has been of key importance in Computer-Aided Design, Animation, Architecture, and Entertainment. With the growing popularity of Virtual and Augmented Reality, new applications and tools have been developed for artistic content creation; real-time interactive shape modeling has become increasingly important for a continuum of virtual and augmented reality environments (eXtended Reality (XR)). Shape modeling in XR opens new possibilities for intuitive design and shape modeling in an accessible way. AI approaches generating shape information from text prompts are set to change how artists create and edit 3D models. There has been a substantial body of research on interactive 3D shape modeling. However, there is no recent extensive review of the existing techniques and what AI shape generation means for shape modeling in interactive XR environments. In this state-of-the-art paper, we fill this research gap in the literature by surveying free-form shape modeling work in XR, with a focus on sculpting and 3D sketching, the most intuitive forms of free-form shape modeling. We classify and discuss these works across five dimensions: contribution of the articles, domain setting, interaction tool, auto-completion, and collaborative designing. The paper concludes by discussing the disconnect between interactive 3D sculpting and sketching and how this will likely evolve with the prevalence of AI shape-generation tools in the future.
- Stereoscan: Dense 3d reconstruction in real-time. In 2011 IEEE intelligent vehicles symposium (IV), pages 963–968. Ieee, 2011.
- Kinectfusion: real-time 3d reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology, pages 559–568, 2011.
- Real time localization and 3d reconstruction. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’06), volume 1, pages 363–370. IEEE, 2006.
- An information theoretic approach to statistical shape modelling. In BMVC, pages 1–10, 2001.
- CavePainting: A fully immersive 3D artistic medium and interactive experience. In Proceedings of the 2001 Symposium on Interactive 3D Graphics - SI3D ’01, pages 85–93. ACM Press, 2001.
- Sketch-based modeling: A survey. Computers & Graphics, 33(1):85–103, 2009.
- A survey of sketch based modeling systems. Frontiers of Computer Science, 10:985–999, 2016.
- Deep learning for free-hand sketch: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
- Ar/mr remote collaboration on physical tasks: A review. Robotics and Computer-Integrated Manufacturing, 72:102071, 2021.
- Prisma 2020 explanation and elaboration: updated guidance and exemplars for reporting systematic reviews. BMJ, 372, 2021.
- Sculpting: An interactive volumetric modeling technique. ACM SIGGRAPH Computer Graphics, 25(4):267–274, 1991.
- NURBS-based free-form deformations. IEEE Computer Graphics and Applications, 14(6):59–65, November 1994.
- Virtual 3d sculpting. The Journal of Visualization and Computer Animation, 11(3):155–166, 2000.
- Surface drawing: Creating organic 3D shapes with the hand and tangible tools. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’01, pages 261–268. Association for Computing Machinery, 2001.
- Virtual clay: A real-time sculpting system with haptic toolkits. In Proceedings of the 2001 Symposium on Interactive 3D Graphics - SI3D ’01, pages 179–190. ACM Press, 2001.
- Efficient 6DOF tools for free-form surface modelling. Vis Comput, 20(8):554–564, November 2004.
- An interface for virtual 3D sculpting via physical proxy. In Proceedings of the 4th International Conference on Computer Graphics and Interactive Techniques in Australasia and Southeast Asia - GRAPHITE ’06, page 213, Kuala Lumpur, Malaysia, 2006. ACM Press.
- AR Pottery: Experiencing Pottery Making in the Augmented Space. In Randall Shumaker, editor, Virtual Reality, Lecture Notes in Computer Science, pages 642–650. Springer, 2007.
- Hands on virtual clay. In 2008 IEEE International Conference on Shape Modeling and Applications, pages 267–268, 2008.
- Augmented foam sculpting for capturing 3D models. In 2010 IEEE Symposium on 3D User Interfaces (3DUI), pages 63–70, 2010.
- deForm: An interactive malleable surface for capturing 2.5D arbitrary objects, tools and touch. In Proceedings of the 24th Annual ACM Symposium on User Interface Software and Technology, UIST ’11, pages 527–536. Association for Computing Machinery, 2011.
- FreeD: A freehand digital sculpting tool. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’13, 2013.
- Sculpting multi-dimensional nested structures. Computers & graphics, 37(6):753–763, 2013.
- Mutable elastic models for sculpting structured shapes. Computer Graphics Forum, 32(2pt1):21–30, 2013.
- cSculpt: A system for collaborative sculpting. ACM Transactions on Graphics, 35(4):1–8, 2016.
- slayer: a system for multi-layered material sculpting. EGSR’17: Proceedings of the Eurographics Symposium on Rendering: Experimental Ideas & Implementations, 2017.
- Autocomplete 3D sculpting. ACM Transactions on Graphics, 37(4):132:1–132:15, 2018.
- DigiClay: An Interactive Installation for Virtual Pottery using Motion Sensing Technology. In Proceedings of the 4th International Conference on Virtual Reality, ICVR 2018, pages 126–132. Association for Computing Machinery, 2018.
- RealPot: An immersive virtual pottery system with handheld haptic devices. Multimedia Tools and Applications, 78(18):26569–26596, 2019.
- Mesh r-cnn. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9785–9795, 2019.
- Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 165–174, 2019.
- Development of AR Learning Assistance Tool for Clay-Sculpting 3D Model. In 2020 12th International Conference on Knowledge and Smart Technology (KST), pages 109–114, 2020.
- PotteryVR: Virtual reality pottery. The Visual Computer, 2022.
- Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint arXiv:2209.14988, 2022.
- Nerfshop: Interactive editing of neural radiance fields. Proceedings of the ACM on Computer Graphics and Interactive Techniques, 6(1), 2023.
- Mobi3DSketch: 3D Sketching in Mobile AR. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, CHI ’19, pages 1–11. Association for Computing Machinery, 2019.
- Experimental Evaluation of Sketching on Surfaces in VR. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, pages 5643–5654. ACM, 2017.
- Scientific Sketching for Collaborative VR Visualization Design. IEEE Transactions on Visualization and Computer Graphics, 14(4):835–847, 2008.
- Lift-Off: Using Reference Imagery and Freehand Sketching to Create 3D Models in VR. IEEE Transactions on Visualization and Computer Graphics, 22(4):1442–1451, 2016.
- Marching cubes: A high resolution 3d surface construction algorithm. In Seminal graphics: pioneering efforts that shaped the field, pages 347–353, 1998.
- Natural Tactile Interaction with Virtual Clay. In 2021 IEEE World Haptics Conference (WHC), pages 403–408, 2021.
- 3D Modeling by Scanning Physical Modifications. In 2015 28th SIBGRAPI Conference on Graphics, Patterns and Images, pages 25–32, 2015.
- PEP (3D Printed Electronic Papercrafts): An Integrated Approach for 3D Sculpting Paper-Based Electronic Devices. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, pages 1–12. ACM, 2018.
- Analytic drawing of 3D scaffolds. In ACM SIGGRAPH Asia 2009 Papers, SIGGRAPH Asia ’09, pages 1–10. Association for Computing Machinery, 2009.
- Michael F. Deering. HoloSketch: A virtual reality sketching/animation tool. ACM Transactions on Computer-Human Interaction, 2(3):220–238, 1995.
- An interface for sketching 3d curves. In Proceedings of the 1999 symposium on Interactive 3D graphics, pages 17–21, 1999.
- Teddy: A sketching interface for 3D freeform design. In ACM SIGGRAPH 2006 Courses, SIGGRAPH ’06, pages 11–es. Association for Computing Machinery, 2006.
- Free-form Sketch. In Proceedings of the 4th Eurographics Workshop on Sketch-based Interfaces and Modeling, SBIM ’07, pages 53–58. Association for Computing Machinery, 2007.
- Sketch-based shape retrieval. ACM Transactions on Graphics, 31(4):1–10, 2012. cites: eitzSketchbasedShapeRetrieval2012.
- A New User-Friendly Sketch-Based Modeling Method Using Convolution Surfaces. In 2016 XVIII Symposium on Virtual and Augmented Reality (SVR), pages 100–108, 2016.
- Sketch2mesh: Reconstructing and editing 3d shapes from sketches. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13023–13032, 2021.
- True2form: 3d curve networks from 2d sketches via selective regularization. ACM Transactions on Graphics, 33(4), 2014.
- Smoothsketch: 3d free-form shapes from complex sketches. In ACM SIGGRAPH 2006 Papers, pages 589–598. Association for Computing Machinery, 2006.
- DreamSketch: Early Stage 3D Design Explorations with Sketching and Generative Design. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, pages 401–414. ACM, 2017.
- Sketch2model: View-aware 3d modeling from single free-hand sketches. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6012–6021, 2021.
- Sketching reality: Realistic interpretation of architectural designs. ACM Transactions on Graphics, 27(2):1–15, 2008.
- 3D reconstruction of freeform shapes from 2D line drawings. In Proceedings of the 9th ACM SIGGRAPH Conference on Virtual-Reality Continuum and Its Applications in Industry, VRCAI ’10, pages 303–310. Association for Computing Machinery, 2010.
- 3D sketch-based 3D model retrieval with convolutional neural network ??? In 2016 23rd International Conference on Pattern Recognition (ICPR), pages 2936–2941, 2016.
- Structured annotations for 2D-to-3D modeling. In ACM SIGGRAPH Asia 2009 Papers, SIGGRAPH Asia ’09, pages 1–9. Association for Computing Machinery, 2009.
- FreeDrawer: A free-form sketching system on the responsive workbench. In Proceedings of the ACM Symposium on Virtual Reality Software and Technology, VRST ’01, pages 167–174, 2001.
- Drawing on Air: Input Techniques for Controlled 3D Line Illustration. IEEE Transactions on Visualization and Computer Graphics, 13(5):1067–1081, 2007.
- FiberMesh: Designing freeform surfaces with 3D curves. ACM Transactions on Graphics, 26(3):41, 2007.
- An interactive sketching modeling system: Sketch3d. In 2008 International Conference on Computer Science and Software Engineering, volume 2, pages 1049–1052. IEEE, 2008.
- ILoveSketch: As-natural-as-possible sketching system for creating 3d curve models. In Proceedings of the 21st Annual ACM Symposium on User Interface Software and Technology, UIST ’08, pages 151–160. Association for Computing Machinery, 2008.
- Napkin sketch: Handheld mixed reality 3D sketching. In Proceedings of the 2008 ACM Symposium on Virtual Reality Software and Technology, VRST ’08, pages 223–226. Association for Computing Machinery, 2008.
- EverybodyLovesSketch: 3D sketching for a broader audience. In Proceedings of the 22nd Annual ACM Symposium on User Interface Software and Technology, UIST ’09, pages 59–68. Association for Computing Machinery, 2009.
- 3D curve sketch: Flexible curve-based stereo reconstruction and calibration. In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 1538–1545, 2010.
- OverCoat: An implicit canvas for 3D painting. In ACM SIGGRAPH 2011 Papers, SIGGRAPH ’11, pages 1–10. Association for Computing Machinery, 2011.
- Single-view sketch based modeling. In Proceedings of the Eighth Eurographics Symposium on Sketch-Based Interfaces and Modeling, SBIM ’11, pages 133–140. Association for Computing Machinery, 2011.
- Just Draw It! A 3D Sketching System. Proceedings of the 9th Eurographics workshop on Sketch-based interfaces and modeling, page 16, 2012.
- SecondSkin: Sketch-based construction of layered 3D models. ACM Transactions on Graphics, 34(4):126:1–126:10, 2015.
- 3D Sketch-Based 3D Model Retrieval. In Proceedings of the 5th ACM on International Conference on Multimedia Retrieval, ICMR ’15, pages 555–558. Association for Computing Machinery, 2015.
- SmartCanvas: Context-inferred Interpretation of Sketches for Preparatory Design Studies. Computer Graphics Forum, pages 37–48, 2016.
- 3D Shape Reconstruction from Sketches via Multi-view Convolutional Networks. In 2017 International Conference on 3D Vision (3DV), pages 67–77, 2017.
- WireDraw: 3D Wire Sculpturing Guided with Mixed Reality. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, CHI ’17, pages 3693–3704. Association for Computing Machinery, 2017.
- SweepCanvas: Sketch-based 3D Prototyping on an RGB-D Image. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology, UIST ’17, pages 387–399. Association for Computing Machinery, 2017.
- SymbiosisSketch: Combining 2D & 3D Sketching for Designing Detailed 3D Objects in Situ. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, pages 1–15. Association for Computing Machinery, 2018.
- Model-Guided 3D Sketching. IEEE Transactions on Visualization and Computer Graphics, 25(10):2927–2939, 2019.
- VRSketchPen: Unconstrained Haptic Assistance for Sketching in Virtual 3D Environments. In 26th ACM Symposium on Virtual Reality Software and Technology, VRST ’20, pages 1–11. Association for Computing Machinery, 2020.
- VRSketchIn: Exploring the Design Space of Pen and Tablet Interaction for 3D Sketching in Virtual Reality. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, CHI ’20, pages 1–14. Association for Computing Machinery, 2020.
- HandPainter - 3D Sketching in VR with Hand-based Physical Proxy. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, CHI ’21, pages 1–13. Association for Computing Machinery, 2021.
- CASSIE: Curve and Surface Sketching in Immersive Environments. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–14. ACM, 2021.
- ScaffoldSketch: Accurate Industrial Design Drawing in VR. In The 34th Annual ACM Symposium on User Interface Software and Technology, UIST ’21, pages 372–384. Association for Computing Machinery, 2021.
- Mid-Air Drawing of Curves on 3D Surfaces in Virtual Reality. ACM Transactions on Graphics, 40(3):1–17, 2021-07-04.
- Piecewise-smooth surface fitting onto unstructured 3D sketches. ACM Transactions on Graphics, 41(4):1–16, 2022.
- Sked: Sketch-guided text-based 3d editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14607–14619, 2023.
- A 3d sketch-based modeling design system with paper-and-pencil interface. In 2013 International Conference on Computer Sciences and Applications, pages 446–453. IEEE, 2013.
- Smooth interpolation of curve networks with surface normals. In Eurographics 2016 Short Papers, pages 21–24. Eurographics Association, 2016.
- SKETCH: An interface for sketching 3D scenes. In ACM SIGGRAPH 2006 Courses, SIGGRAPH ’06, pages 9–es. Association for Computing Machinery, 2006.
- Sketch-based 3d shape retrieval using convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1875–1883, 2015.
- 3D sketching for interactive model retrieval in virtual reality. In Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering, pages 1–12. ACM, 2018.
- A suggestive interface for image guided 3d sketching. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pages 591–598, 2004.
- Tangible 3d: Hand gesture interaction for immersive 3d modeling. IPT/EGVE, 2005:191–9, 2005.
- Evaluating organic 3D sculpting using natural user interfaces with the Kinect. In Proceedings of the 25th Australian Computer-Human Interaction Conference: Augmentation, Application, Innovation, Collaboration, OzCHI ’13, pages 163–166. Association for Computing Machinery, 2013.
- SketchingWithHands: 3D Sketching Handheld Products with First-Person Hand Posture. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology, UIST ’16, pages 797–808. Association for Computing Machinery, 2016.
- HoloARt: Painting with Holograms in Mixed Reality. In Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, CHI EA ’17, pages 421–424. Association for Computing Machinery, 2017.
- Bare-Handed 3D Drawing in Augmented Reality. In Proceedings of the 2018 Designing Interactive Systems Conference, DIS ’18, pages 241–252. Association for Computing Machinery, 2018.
- A 3D Virtual Sketching System Using NURBS Surfaces and Leap Motion Controller. Computer-Aided Design and Applications, 17(1):167–177, 2019.
- Virtual Touch Screen for Mixed Reality. In Nicu Sebe, Michael Lew, and Thomas S. Huang, editors, Computer Vision in Human-Computer Interaction, Lecture Notes in Computer Science, pages 48–59. Springer, 2004.
- Martin Tosas and Li Bai. Virtual touch screen: a vision-based interactive surface. In 9th IASTED international conference on computer graphics and imaging, Innsbruck, Austria. ACTA Press, pages 81–86, 2007.
- v-glove: A 3d virtual touch interface. In 2011 XIII Symposium on Virtual Reality, pages 242–251. IEEE, 2011.
- A soft hand model for physically-based manipulation of virtual objects. In 2011 IEEE Virtual Reality Conference, pages 11–18, 2011.
- Agile 3D Sketching with Air Scaffolding. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, CHI ’18, pages 1–12. Association for Computing Machinery, 2018.
- Insitu: sketching architectural designs in context. ACM Trans. Graph., 30(6):182, 2011.
- Haptic editor: Creation and editing system for touchable 3D content. In SIGGRAPH Asia 2012 Technical Briefs, SA ’12, pages 1–4. Association for Computing Machinery, 2012.
- Thermalpen: Adding thermal haptic feedback to 3d sketching. In Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pages 1–4, 2023.
- Towards 3D VR-Sketch to 3D Shape Retrieval. In 2020 International Conference on 3D Vision (3DV), pages 81–90, 2020.
- Gesturesurface: Vr sketching through assembling scaffold surface with non-dominant hand. IEEE Transactions on Visualization and Computer Graphics, 29(5):2499–2507, 2023.
- Fluid Sketching—Immersive Sketching Based on Fluid Flow. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 475–482, 2018.
- Real-Time Re-Textured Geometry Modeling Using Microsoft HoloLens. In 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), pages 231–237, 2018.
- Spatial Motion Doodles: Sketching Animation in VR Using Hand Gestures and Laban Motion Analysis. In Motion, Interaction and Games, MIG ’19, pages 1–10. Association for Computing Machinery, 2019.
- Sketch-Up in the Virtual World. In 2008 International Conference on Cyberworlds, pages 109–116, 2008.
- In-Place Sketching for content authoring in Augmented Reality games. In 2010 IEEE Virtual Reality Conference (VR), pages 91–94, 2010.
- The drawing assistant: Automated drawing guidance and feedback from photographs. In ACM Symposium on User Interface Software and Technology (UIST). ACM, 2013.
- Autocomplete Animated Sculpting. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology, pages 760–777. Association for Computing Machinery, 2020.
- 3d-rcnn: Instance-level 3d object reconstruction via render-and-compare. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3559–3568, 2018.
- Deep local shapes: Learning local sdf priors for detailed 3d reconstruction. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIX 16, pages 608–625. Springer, 2020.
- Local deep implicit functions for 3d shape. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4857–4866, 2020.
- 3d neural sculpting (3dns): Editing neural signed distance functions. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 4521–4530, 2023.
- Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99–106, 2021.
- High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
- Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
- Mip-nerf 360: Unbounded anti-aliased neural radiance fields. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
- D-nerf: Neural radiance fields for dynamic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
- Plenoxels: Radiance fields without neural networks. arXiv preprint arXiv:2112.05131, 2021.
- Instant neural graphics primitives framework and its emerging applications in digitalization of dental care. In 2022 E-Health and Bioengineering Conference (EHB), pages 1–4. IEEE, 2022.
- Nerfacc: A general nerf acceleration toolbox. arXiv preprint arXiv:2210.04847, 2022.
- Realfusion: 360deg reconstruction of any object from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8446–8455, 2023.
- Magic3d: High-resolution text-to-3d content creation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 300–309, 2023.
- WeSketch: A 3D Real Time Collaborative Virtual Environment that Improves the GUI Sketching Task. In 2011 Eighth International Conference on Information Technology: New Generations, pages 163–168, 2011.
- MeshGit: Diffing and merging meshes for polygonal modeling. ACM Transactions on Graphics, 32(4):35:1–35:10, 2013.
- Shisha: Enabling shared perspective with face-to-face collaboration using redirected avatars in virtual reality. Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3):1–22, 2021.