Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
149 tokens/sec
GPT-4o
7 tokens/sec
Gemini 2.5 Pro Pro
45 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Video2MR: Automatically Generating Mixed Reality 3D Instructions by Augmenting Extracted Motion from 2D Videos (2405.18565v1)

Published 28 May 2024 in cs.HC

Abstract: This paper introduces Video2MR, a mixed reality system that automatically generates 3D sports and exercise instructions from 2D videos. Mixed reality instructions have great potential for physical training, but existing works require substantial time and cost to create these 3D experiences. Video2MR overcomes this limitation by transforming arbitrary instructional videos available online into MR 3D avatars with AI-enabled motion capture (DeepMotion). Then, it automatically enhances the avatar motion through the following augmentation techniques: 1) contrasting and highlighting differences between the user and avatar postures, 2) visualizing key trajectories and movements of specific body parts, 3) manipulation of time and speed using body motion, and 4) spatially repositioning avatars for different perspectives. Developed on Hololens 2 and Azure Kinect, we showcase various use cases, including yoga, dancing, soccer, tennis, and other physical exercises. The study results confirm that Video2MR provides more engaging and playful learning experiences, compared to existing 2D video instructions.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (58)
  1. Juggling in vr: Advantages of immersive virtual reality in juggling learning. In Proceedings of the 25th ACM Symposium on Virtual Reality Software and Technology. 1–5.
  2. ShowMe: A Remote Collaboration System That Supports Immersive Gestural Communication. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. 1343–1348.
  3. A user study on mixed reality remote collaboration with eye gaze and hand gesture sharing. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–13.
  4. Marion Buchenau and Jane Fulton Suri. 2000. Experience prototyping. In Proceedings of the 3rd conference on Designing interactive systems: processes, practices, methods, and techniques. 424–433.
  5. Reactive video: adaptive video playback based on user motion for supporting physical activity. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 196–208.
  6. Caleb Conner and Gene Michael Poor. 2016. Correcting exercise form using body tracking. In Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. 3028–3034.
  7. DeepMotion. 2023. DeepMotion. https://www.deepmotion.com/
  8. Pose tutor: an explainable system for pose correction in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 3540–3549.
  9. TutAR: augmented reality tutorials for hands-only procedures. In Proceedings of the 16th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry. 1–3.
  10. Understanding Perspectives for Single-and Multi-Limb Movement Guidance in Virtual 3D Environments. In Proceedings of the 28th ACM Symposium on Virtual Reality Software and Technology. 1–10.
  11. Rmpe: Regional multi-person pose estimation. In Proceedings of the IEEE international conference on computer vision. 2334–2343.
  12. ChameleonControl: Teleoperating Real Human Surrogates through Mixed Reality Gestural Guidance for Remote Hands-on Classrooms. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 13 pages.
  13. Aifit: Automatic 3d human-interpretable feedback models for fitness training. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 9919–9928.
  14. Assisting viewpoint to understand own posture as an avatar in-situation. In Proceedings of the 5th International ACM In-Cooperation HCI and UX Conference. 1–8.
  15. Natsuki Hamanishi and Jun Rekimoto. 2020. Poseasquery: Full-body interface for repeated observation of a person in a video with ambiguous pose indexes and performed poses. In Proceedings of the Augmented Humans International Conference. 1–11.
  16. Natsuki Hamanishi and Jun Rekimoto. 2021. Motion-specific browsing method by mapping to a circle for personal video Observation with Head-Mounted Displays. In Proceedings of the Augmented Humans International Conference 2021. 240–250.
  17. Ar-arm: Augmented visualization for guiding arm movement in the first-person perspective. In Proceedings of the 7th Augmented Human International Conference 2016. 1–4.
  18. My Tai-Chi coaches: an augmented-learning tool for practicing Tai-Chi Chuan. In Proceedings of the 8th Augmented Human International Conference. 1–4.
  19. Onebody: remote posture guidance system using first person view in virtual environment. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction. 1–10.
  20. Adaptutar: An adaptive tutoring system for machine tasks in augmented reality. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–15.
  21. ARrow: A Real-Time AR Rowing Coach. (2023).
  22. HoloBots: Augmenting Holographic Telepresence with Mobile Robots for Tangible Remote Collaboration in Mixed Reality. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–12.
  23. AR based Self-sports Learning System using Decayed Dynamic TimeWarping Algorithm.. In ICAT-EGVE. 171–174.
  24. Augmented Tai-Chi Chuan Practice Tool with Pose Evaluation. In 2021 IEEE 4th International Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, 35–41.
  25. Stylo and handifact: Modulating haptic perception through visualizations for posture training in augmented reality. In Proceedings of the 5th Symposium on Spatial User Interaction. 58–67.
  26. VirtualLadder: Using Interactive Projections for Agility Ladder Training. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems. 1–7.
  27. RealityTalk: Real-Time Speech-Driven Augmented Presentation for AR Live Storytelling. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–12.
  28. Towards an understanding of situated ar visualization for basketball free-throw training. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. 1–13.
  29. PoseCoach: A Customizable Analysis and Visualization System for Video-based Running Coaching. IEEE Transactions on Visualization and Computer Graphics (2022).
  30. PianoSyncAR: Enhancing Piano Learning through Visualizing Synchronized Hand Pose Discrepancies in Augmented Reality. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 859–868.
  31. Super Mirror: a kinect interface for ballet dancers. In CHI’12 Extended Abstracts on Human Factors in Computing Systems. 1619–1624.
  32. Skiing, Fast and Slow: Evaluation of Time Distortion for VR Ski Training. In Proceedings of the Augmented Humans International Conference 2022. 142–151.
  33. Augmented reality for older adults: exploring acceptability of virtual coaches for home-based balance training in an aging population. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–12.
  34. Mini-me: An adaptive avatar for mixed reality remote collaboration. In Proceedings of the 2018 CHI conference on human factors in computing systems. 1–13.
  35. On the shoulder of the giant: A multi-scale mixed reality collaboration with 360 video sharing and tangible interaction. In Proceedings of the 2019 CHI conference on human factors in computing systems. 1–17.
  36. Zelai Saenz-de Urturi and Begonya Garcia-Zapirain Soto. 2016. Kinect-based virtual game for the elderly that detects incorrect body postures in real time. Sensors 16, 5 (2016), 704.
  37. Multimodal motion guidance: techniques for adaptive and dynamic feedback. In Proceedings of the 14th ACM international conference on Multimodal interaction. 133–140.
  38. Yoones A Sekhavat and Mohammad S Namani. 2018. Projection-based AR: Effective visual feedback in gait rehabilitation. IEEE Transactions on Human-Machine Systems 48, 6 (2018), 626–636.
  39. AR Hero: Generating interactive augmented reality guitar tutorials. In 2022 IEEE Conference on Virtual Reality and 3D User Interfaces Abstracts and Workshops (VRW). IEEE, 395–401.
  40. LightGuide: projected visualizations for hand movement guidance. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 179–188.
  41. Realitysketch: Embedding responsive graphics and visualizations in AR through dynamic sketching. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 166–181.
  42. Gino. Aiki: Mixed Reality-based Physical Motor Skill Training in Aikido. In 2023 IEEE International Symposium on Mixed and Augmented Reality Adjunct (ISMAR-Adjunct). IEEE, 519–524.
  43. ObserVAR: Visualization system for observing virtual reality users using augmented reality. In 2019 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 258–268.
  44. Pose estimation for facilitating movement learning from online videos. In Proceedings of the International Conference on Advanced Visual Interfaces. 1–5.
  45. Loki: Facilitating remote instruction of physical tasks using bi-directional mixed-reality telepresence. In Proceedings of the 32nd Annual ACM Symposium on User Interface Software and Technology. 161–174.
  46. Enlightened yoga: Designing an augmented class with wearable lights to support instruction. In Proceedings of the 2019 on Designing Interactive Systems Conference. 1017–1031.
  47. Bodylights: Open-ended augmented feedback to support training towards a correct exercise execution. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems. 1–14.
  48. Motionma: motion modelling and analysis by demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1309–1318.
  49. VideoPoseVR: Authoring Virtual Reality Character Animations with Online Videos. Proceedings of the ACM on Human-Computer Interaction 6, ISS (2022), 448–467.
  50. AR-Enhanced Workouts: Exploring Visual Cues for At-Home Workout Videos in AR Environment. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–15.
  51. VoLearn: A Cross-Modal Operable Motion-Learning System Combined with Virtual Avatar and Auditory Feedback. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. (2022), 26 pages.
  52. RealityCanvas: Augmented Reality Sketching for Embedded and Responsive Scribble Animation Effects. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology. 1–14.
  53. Pose Flow: Efficient online pose tracking. arXiv preprint arXiv:1802.00977 (2018).
  54. Outsideme: Augmenting dancer’s external self-image by using a mixed reality system. In Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. 965–970.
  55. Shuttlespace: Exploring and analyzing movement trajectory in immersive visualization. IEEE transactions on visualization and computer graphics 27, 2 (2020), 860–869.
  56. Perspective matters: Design implications for motion guidance in mixed reality. In 2020 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). IEEE, 577–587.
  57. 3d pose based feedback for physical exercises. In Proceedings of the Asian Conference on Computer Vision. 1316–1332.
  58. Syncup: Vision-based practice support for synchronized dancing. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 3 (2021), 1–25.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets