Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
Gemini 2.5 Pro
GPT-5
GPT-4o
DeepSeek R1 via Azure
2000 character limit reached

RealityEffects: Augmenting 3D Volumetric Videos with Object-Centric Annotation and Dynamic Visual Effects (2405.17711v1)

Published 28 May 2024 in cs.HC

Abstract: This paper introduces RealityEffects, a desktop authoring interface designed for editing and augmenting 3D volumetric videos with object-centric annotations and visual effects. RealityEffects enhances volumetric capture by introducing a novel method for augmenting captured physical motion with embedded, responsive visual effects, referred to as object-centric augmentation. In RealityEffects, users can interactively attach various visual effects to physical objects within the captured 3D scene, enabling these effects to dynamically move and animate in sync with the corresponding physical motion and body movements. The primary contribution of this paper is the development of a taxonomy for such object-centric augmentations, which includes annotated labels, highlighted objects, ghost effects, and trajectory visualization. This taxonomy is informed by an analysis of 120 edited videos featuring object-centric visual effects. The findings from our user study confirm that our direct manipulation techniques lower the barriers to editing and annotating volumetric captures, thereby enhancing interactive and engaging viewing experiences of 3D volumetric videos.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (101)
  1. 4DViews. [n.d.]. 4Dfx. https://www.4dviews.com/volumetric-software.
  2. RemoteFusion: real time depth camera fusion for remote collaboration on physical tasks. In Proceedings of the 12th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry. 235–242.
  3. YouMove: enhancing movement training with an augmented reality mirror. In Proceedings of the 26th annual ACM symposium on User interface software and technology. 311–320.
  4. Dyadic projected spatial augmented reality. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 645–655.
  5. Eagleview: A video analysis tool for visualising and querying spatial interactions of people and devices. In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces. 61–72.
  6. Miria: A mixed reality toolkit for the in-situ visualization and analysis of spatio-temporal interaction data. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
  7. MobileTutAR: a Lightweight Augmented Reality Tutorial System using Spatially Situated Human Segmentation Videos. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–8.
  8. GhostAR: A time-space editor for embodied authoring of human-robot collaborative task with augmented reality. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 521–534.
  9. Augmenting sports videos with viscommentator. IEEE Transactions on Visualization and Computer Graphics 28, 1 (2021), 824–834.
  10. Vroamer: generating on-the-fly VR experiences while walking inside large, unknown real-world building environments. In 2019 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 359–366.
  11. Segment and track anything. arXiv preprint arXiv:2305.06558 (2023).
  12. Semanticadapt: Optimization-based adaptation of mixed reality layouts leveraging virtual-physical semantic connections. In The 34th Annual ACM Symposium on User Interface Software and Technology. 282–297.
  13. Towards Understanding Diminished Reality. In CHI Conference on Human Factors in Computing Systems. 1–16.
  14. Authoring illustrations of human movements by iterative physical demonstration. In Proceedings of the 29th Annual Symposium on User Interface Software and Technology. 809–820.
  15. Processar: An augmented reality-based tool to create in-situ procedural 2d/3d ar instructions. In Designing Interactive Systems Conference 2021. 234–249.
  16. Reactive video: adaptive video playback based on user motion for supporting physical activity. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 196–208.
  17. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE conference on computer vision and pattern recognition. 5828–5839.
  18. An immersive system for browsing and visualizing surveillance video. In Proceedings of the 18th ACM international conference on Multimedia. 371–380.
  19. EventAnchor: reducing human interactions in event annotation of racket sports videos. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–13.
  20. DepthKit. [n.d.]. DepthKit Studio. https://www.depthkit.tv/depthkit-studio.
  21. Fusion4d: Real-time performance capture of challenging scenes. ACM Transactions on Graphics (ToG) 35, 4 (2016), 1–13.
  22. Montage4d: Interactive seamless fusion of multiview video textures. (2018).
  23. DepthLab: Real-time 3D interaction with depth maps for mobile augmented reality. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 829–843.
  24. Optispace: automated placement of interactive 3D projection mapping content. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–11.
  25. Heatspace: Automatic placement of displays by empirical analysis of user behavior. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 611–621.
  26. Andreas Rene Fender and Christian Holz. 2022. Causality-preserving Asynchronous Reality. In CHI Conference on Human Factors in Computing Systems. 1–15.
  27. An oriented point-cloud view for MR remote collaboration. In SIGGRAPH ASIA 2016 Mobile Graphics and Interactive Applications. 1–4.
  28. In touch with the remote world: Remote collaboration with augmented reality drawings and virtual navigation. In Proceedings of the 20th ACM Symposium on Virtual Reality Software and Technology. 197–205.
  29. World-stabilized annotations and virtual scene navigation for remote collaboration. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 449–459.
  30. Video object annotation, navigation, and composition. In Proceedings of the 21st annual ACM symposium on User interface software and technology. 3–12.
  31. Holoboard: A large-format immersive teaching board based on pseudo holographics. In The 34th Annual ACM Symposium on User Interface Software and Technology. 441–456.
  32. The relightables: Volumetric performance capture of humans with realistic relighting. ACM Transactions on Graphics (ToG) 38, 6 (2019), 1–19.
  33. Augmented Chironomia for Presenting Data to Remote Audiences. arXiv preprint arXiv:2208.04451 (2022).
  34. My Tai-Chi coaches: an augmented-learning tool for practicing Tai-Chi Chuan. In Proceedings of the 8th Augmented Human International Conference. 1–4.
  35. Realitycheck: Blending virtual environments with situated physical reality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
  36. ReLive: Bridging In-Situ and Ex-Situ Visual Analytics for Analyzing Mixed Reality User Studies. In CHI Conference on Human Factors in Computing Systems. 1–20.
  37. Understanding newcomers to 3D printing: Motivations, workflows, and barriers of casual makers. In Proceedings of the 2016 CHI conference on human factors in computing systems. 384–396.
  38. Ke Huo and Karthik Ramani. 2017. Window-shaping: 3d design ideation by creating on, borrowing from, and looking at the physical world. In Proceedings of the Eleventh International Conference on Tangible, Embedded, and Embodied Interaction. 37–45.
  39. Volumedeform: Real-time volumetric non-rigid reconstruction. In European conference on computer vision. Springer, 362–379.
  40. KinectFusion: real-time 3D reconstruction and interaction using a moving depth camera. In Proceedings of the 24th annual ACM symposium on User interface software and technology. 559–568.
  41. Roomalive: Magical experiences enabled by scalable, adaptive projector-camera units. In Proceedings of the 27th annual ACM symposium on User interface software and technology. 637–644.
  42. IllumiRoom: peripheral projected illusions for interactive experiences. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 869–878.
  43. Sketched reality: Sketching bi-directional interactions between virtual and physical worlds with ar and actuated tangible ui. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–12.
  44. Virtualized reality: Constructing virtual worlds from real scenes. IEEE multimedia 4, 1 (1997), 34–47.
  45. TransforMR: Pose-aware object substitution for composing alternate mixed realities. In 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 69–79.
  46. Pocketdragon: a direct manipulation video navigation interface for mobile devices. In Proceedings of the 11th International Conference on Human-Computer Interaction with Mobile Devices and Services. 1–3.
  47. See, Feel, Move: player behaviour analysis through combined visualization of gaze, emotions, and movement. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.
  48. Immersive analysis of user motion in VR applications. The Visual Computer 36, 10 (2020), 1937–1949.
  49. Thomas H Kolbe. 2004. Augmented videos and panoramas for pedestrian navigation. In Proceedings of the 2nd Symposium on Location Based Services & TeleCartography 2004, 28-29th of January 2004 in Vienna.
  50. JackIn space: designing a seamless transition between first and third person view for effective telepresence collaborations. In Proceedings of the 8th Augmented Human International Conference. 1–9.
  51. Photoportals: shared references in space and time. In Proceedings of the 17th ACM conference on Computer supported cooperative work & social computing. 1388–1399.
  52. Project Starline: A high-fidelity telepresence system. (2021).
  53. Evaluation strategies for HCI toolkit research. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–17.
  54. Semantic human activity annotation tool using skeletonized surveillance videos. In Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. 312–315.
  55. Rapido: Prototyping Interactive AR Experiences through Programming by Demonstration. In The 34th Annual ACM Symposium on User Interface Software and Technology. 626–637.
  56. Pronto: Rapid augmented reality video prototyping using sketches and enaction. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–13.
  57. SweepCanvas: Sketch-based 3D prototyping on an RGB-D image. In Proceedings of the 30th Annual ACM Symposium on User Interface Software and Technology. 387–399.
  58. RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling. arXiv preprint arXiv:2208.06350 (2022).
  59. David Lindlbauer and Andy D Wilson. 2018. Remixed reality: Manipulating space and time in augmented reality. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. 1–13.
  60. Posetween: Pose-driven tween animation. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 791–804.
  61. Slicing-volume: Hybrid 3d/2d multi-target selection technique for dense virtual environments. In 2020 IEEE Conference on Virtual Reality and 3D User Interfaces (VR). IEEE, 53–62.
  62. Teachable reality: Prototyping tangible augmented reality with everyday objects by leveraging interactive machine teaching. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–15.
  63. A survey of diminished reality: Techniques for visually concealing, eliminating, and seeing through real objects. IPSJ Transactions on Computer Vision and Applications 9, 1 (2017), 1–14.
  64. Video summagator: An interface for video summarization and navigation. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 647–650.
  65. Direct manipulation video navigation in 3D. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1169–1172.
  66. Direct manipulation video navigation on touch screens. In Proceedings of the 16th international conference on Human-computer interaction with mobile devices & services. 273–282.
  67. Snaptoreality: Aligning augmented reality to the real world. In Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. 1233–1244.
  68. Holoportation: Virtual 3d teleportation in real-time. In Proceedings of the 29th annual symposium on user interface software and technology. 741–754.
  69. Room2room: Enabling life-size telepresence in a projected augmented reality environment. In Proceedings of the 19th ACM conference on computer-supported cooperative work & social computing. 1716–1725.
  70. Mini-me: An adaptive avatar for mixed reality remote collaboration. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–13.
  71. On the shoulder of the giant: A multi-scale mixed reality collaboration with 360 video sharing and tangible interaction. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–17.
  72. PolyCam. [n.d.]. Polycam. https://poly.cam/.
  73. Virtual makerspaces: merging AR/VR/MR to enable remote collaborations in physical maker activities. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–5.
  74. Mixed voxel reality: Presence and embodiment in low fidelity, visually coherent, mixed reality environments. In 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 90–99.
  75. AvatAR: An Immersive Analysis Environment for Human Motion Data Combining Interactive 3D Avatars and Trajectories. In CHI Conference on Human Factors in Computing Systems. 1–15.
  76. Virtual reality annotator: A tool to annotate dancers in a virtual environment. In Digital Cultural Heritage: Final Conference of the Marie Skłodowska-Curie Initial Training Network for Digital Cultural Heritage, ITN-DCH 2017, Olimje, Slovenia, May 23–25, 2017, Revised Selected Papers. Springer, 257–266.
  77. Direct space-time trajectory control for visual media editing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1149–1158.
  78. graphiti: Sketch-based Graph Analytics for Images and Videos. In CHI Conference on Human Factors in Computing Systems. 1–15.
  79. Interactive body-driven graphics for augmented video performance. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems. 1–12.
  80. Real-time annotation of video objects on tablet computers. In Proceedings of the 11th International Conference on Mobile and Ubiquitous Multimedia. 1–9.
  81. BeThere: 3D mobile collaboration with spatial input. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 179–188.
  82. Oasis: Procedurally generated social virtual spaces from 3d scanned real spaces. IEEE transactions on visualization and computer graphics 24, 12 (2017), 3174–3187.
  83. Procedurally generated virtual reality from 3D reconstructed physical space. In Proceedings of the 22nd ACM Conference on Virtual Reality Software and Technology. 191–200.
  84. Arcturus Studios. [n.d.]. HoloEdit. https://arcturus.studio/holoedit/.
  85. Browsing group first-person videos with 3d visualization. In Proceedings of the 2018 ACM International Conference on Interactive Surfaces and Spaces. 55–60.
  86. Augmented Reality and Robotics: A Survey and Taxonomy for AR-enhanced Human-Robot Interaction and Robotic Interfaces. In CHI Conference on Human Factors in Computing Systems. 1–33.
  87. Realitysketch: Embedding responsive graphics and visualizations in AR through dynamic sketching. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 166–181.
  88. Matthew Tait and Mark Billinghurst. 2015. The effect of view independence in a collaborative AR system. Computer Supported Cooperative Work (CSCW) 24, 6 (2015), 563–589.
  89. 3D helping hands: a gesture based MR system for remote collaboration. In Proceedings of the 11th ACM SIGGRAPH international conference on virtual-reality continuum and its applications in industry. 323–328.
  90. Mixed reality remote collaboration combining 360 video and 3d reconstruction. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–14.
  91. Loki: Facilitating remote instruction of physical tasks using bi-directional mixed-reality telepresence. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 161–174.
  92. Semanticpaint: Interactive 3d labeling and learning at your fingertips. ACM Transactions on Graphics (TOG) 34, 5 (2015), 1–17.
  93. Dragimation: direct manipulation keyframe timing for performance-based animation. In Proceedings of Graphics Interface 2012. 101–108.
  94. Slice of light: Transparent and integrative transition among realities in a multi-HMD-user environment. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 805–817.
  95. Distanciar: Authoring site-specific augmented reality experiences for remote environments. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–12.
  96. Embedded data representations. IEEE transactions on visualization and computer graphics 23, 1 (2016), 461–470.
  97. RealityCanvas: Augmented Reality Sketching for Embedded and Responsive Scribble Animation Effects. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–14.
  98. Track anything: Segment anything meets videos. arXiv preprint arXiv:2304.11968 (2023).
  99. Videodoodles: Hand-drawn animations on videos with scene-aware canvases. ACM Transactions on Graphics (TOG) 42, 4 (2023), 1–12.
  100. Perspective matters: Design implications for motion guidance in mixed reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 577–587.
  101. SceneCtrl: Mixed reality enhancement via efficient scene editing. In Proceedings of the 30th annual ACM symposium on user interface software and technology. 427–436.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Follow-up Questions

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets