SIM2VR: Towards Automated Biomechanical Testing in VR (2404.17695v2)
Abstract: Automated biomechanical testing has great potential for the development of VR applications, as initial insights into user behaviour can be gained in silico early in the design process. In particular, it allows prediction of user movements and ergonomic variables, such as fatigue, prior to conducting user studies. However, there is a fundamental disconnect between simulators hosting state-of-the-art biomechanical user models and simulators used to develop and run VR applications. Existing user simulators often struggle to capture the intricacies of real-world VR applications, reducing ecological validity of user predictions. In this paper, we introduce SIM2VR, a system that aligns user simulation with a given VR application by establishing a continuous closed loop between the two processes. This, for the first time, enables training simulated users directly in the same VR application that real users interact with. We demonstrate that SIM2VR can predict differences in user performance, ergonomics and strategies in a fast-paced, dynamic arcade game. In order to expand the scope of automated biomechanical testing beyond simple visuomotor tasks, advances in cognitive models and reward function design will be needed.
- A Biomechanical Model for the Upper Extremity using Optimization Techniques. [Human Factors] 16, 6 (1974), 585–594. https://doi.org/10.1177/001872087401600603 arXiv:https://doi.org/10.1177/001872087401600603 PMID: 4442903.
- [Modeling and Understanding Human Routine Behavior]. Association for Computing Machinery, New York, NY, USA, 248–260. https://doi.org/10.1145/2858036.2858557
- Automated testing of virtual reality application interfaces. In [Proceedings of the workshop on Virtual Environments 2003]. 107–114.
- Gunnar A Borg. 1982. Psychophysical bases of perceived exertion. [Medicine and science in sports and exercise] 14, 5 (1982), 377–381.
- MyoSuite–A contact-rich simulation suite for musculoskeletal motor control. [arXiv preprint arXiv:2205.13600] (2022).
- MyoChallenge: Learning contact-rich manipulation using a musculoskeletal hand. https://sites.google.com/view/myochallenge.
- [The psychology of human-computer interaction]. Crc Press.
- Predicting Mid-Air Interaction Movements and Fatigue Using Deep Reinforcement Learning. In [Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems] (Honolulu, HI, USA) [(CHI ’20)]. Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3313831.3376701
- The Emergence of Interactive Behaviour: A Model of Rational Menu Search. (2015).
- A cognitive model of how people make decisions through interaction with visual displays. In [Proceedings of the 2017 CHI conference on human factors in computing systems]. 1205–1216.
- Effort-Based Cost–Benefit Valuation and the Human Brain. [Journal of Neuroscience] 29, 14 (2009), 4531–4541. https://doi.org/10.1523/JNEUROSCI.4515-08.2009 arXiv:https://www.jneurosci.org/content/29/14/4531.full.pdf
- Analysis of musculoskeletal systems in the AnyBody Modeling System. [Simulation Modelling Practice and Theory] 14, 8 (2006), 1100–1111. https://doi.org/10.1016/j.simpat.2006.09.001 SIMS 2004.
- OpenSim: Open-Source Software to Create and Analyze Dynamic Simulations of Movement. [Biomedical Engineering, IEEE Transactions on] 54 (12 2007), 1940 – 1950. https://doi.org/10.1109/TBME.2007.901024
- OpenSim Moco: Musculoskeletal optimal control. [PLOS Computational Biology] 16, 12 (12 2021), 1–21. https://doi.org/10.1371/journal.pcbi.1008493
- A Brief Survey of Sim2Real Methods for Robot Learning. In [Advances in Service and Industrial Robotics], Andreas Müller and Mathias Brandstötter (Eds.). Springer International Publishing, Cham, 133–140.
- Reinforcement Learning Control of a Biomechanical Model of the Upper Extremity. [Scientific Reports] 11, 1 (2021), 1–15.
- Optimal Feedback Control for Modeling Human-Computer Interaction. [ACM Trans. Comput.-Hum. Interact.] 29, 6, Article 51 (april 2022), 70 pages. https://doi.org/10.1145/3524122
- Paul M Fitts. 1954. The information capacity of the human motor system in controlling the amplitude of movement. [Journal of experimental psychology] 47, 6 (1954), 381.
- Speech recognition with deep recurrent neural networks. In [2013 IEEE international conference on acoustics, speech and signal processing]. Ieee, 6645–6649.
- Patrick Harms. 2019. Automated usability evaluation of virtual reality applications. [ACM Transactions on Computer-Human Interaction (TOCHI)] 26, 3 (2019), 1–36.
- Deep residual learning for image recognition. In [Proceedings of the IEEE conference on computer vision and pattern recognition]. 770–778.
- Complex Interaction as Emergent Behaviour: Simulating Mid-Air Virtual Keyboard Typing using Reinforcement Learning. [IEEE Transactions on Visualization and Computer Graphics] (2021), 1–1. https://doi.org/10.1109/TVCG.2021.3106494
- Automatic testing of natural user interfaces. In [2014 IEEE Seventh International Conference on Software Testing, Verification and Validation]. IEEE, 123–132.
- Breathing Life Into Biomechanical User Models. In [Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology] (Bend, OR, USA) [(UIST ’22)]. Association for Computing Machinery, New York, NY, USA, Article 90, 14 pages. https://doi.org/10.1145/3526113.3545689
- Aleksi Ikkala and Perttu Hämäläinen. 2020. Converting Biomechanical Models from OpenSim to MuJoCo. , 277–281 pages. arXiv:2006.10618 [q-bio.QM] https://arxiv.org/abs/2006.10618
- Modeling Cumulative Arm Fatigue in Mid-Air Interaction Based on Perceived Exertion and Kinetics of Arm Motion. In [Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems] (Denver, Colorado, USA) [(CHI ’17)]. Association for Computing Machinery, New York, NY, USA, 3328–3339. https://doi.org/10.1145/3025453.3025523
- Touchscreen typing as optimal supervisory control. In [Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems]. 1–14.
- Inferring Cognitive Models from Data Using Approximate Bayesian Computation. In [Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems] (Denver, Colorado, USA) [(CHI ’17)]. Association for Computing Machinery, New York, NY, USA, 1295–1306. https://doi.org/10.1145/3025453.3025576
- Automated UX Evaluation for User-Centered Design of VR Interfaces. In [International Conference on Human-Centred Software Engineering]. Springer, 140–149.
- Simulating Interaction Movements via Model Predictive Control. [ACM Trans. Comput.-Hum. Interact.] 30, 3, Article 44 (jun 2023), 50 pages. https://doi.org/10.1145/3577016
- Eileen Kowler. 2011. Eye movements: The past 25 years. [Vision Research] 51, 13 (2011), 1457–1483. https://doi.org/10.1016/j.visres.2010.12.014 Vision Research 50th Anniversary Issue: Part 2.
- P. O. Kristensson and Th. Müllners. 2021. Design and Analysis of Intelligent Text Entry Systems with Function Structure Models and Envelope Analysis. In [Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems]. 1–12.
- The law relating the kinematic and figural aspects of drawing movements. [Acta Psychologica] 54, 1 (1983), 115 – 130.
- Simulation Intelligence: Towards a New Generation of Scientific Methods. [arXiv preprint arXiv:2112.03235] (2021).
- Leng-Feng Lee and Brian R Umberger. 2016. Generating optimal control simulations of musculoskeletal movement using OpenSim and MATLAB. [PeerJ] 4 (2016), e1638.
- Sub-policy adaptation for hierarchical reinforcement learning. [arXiv preprint arXiv:1906.05862] (2019).
- Challenges and Opportunities for Software Testing in Virtual Reality Application Development. [Graphics Interface 2023 -second deadline] (2023).
- Modification of a three-compartment muscle fatigue model to predict peak torque decline during intermittent tasks. [Journal of biomechanics] 77 (2018), 16–25.
- Intermittent control as a model of mouse movements. [ACM Transactions on Computer-Human Interaction (TOCHI)] 28, 5 (2021), 1–46.
- Human-level control through deep reinforcement learning. [nature] 518, 7540 (2015), 529–533.
- Control theoretic models of pointing. [ACM Transactions on Computer-Human Interaction (TOCHI)] 24, 4 (2017), 1–36.
- What simulation can do for HCI research. [Interactions] 29, 6 (2022), 48–53.
- Data-efficient hierarchical reinforcement learning. [Advances in neural information processing systems] 31 (2018).
- Deep Learning of Biomimetic Sensorimotor Control for Biomechanical Human Animation. [ACM Trans. Graph.] 37, 4, Article 56 (jul 2018), 15 pages. https://doi.org/10.1145/3197517.3201305
- Computational Rationality as a Theory of Interaction. In [Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems] [(CHI ’22)]. Association for Computing Machinery, New York, NY, USA, Article 359, 14 pages. https://doi.org/10.1145/3491102.3517739
- Melissa Quek. 2013. [The role of simulation in developing and designing applications for 2-class motor imagery brain-computer interfaces]. Ph. D. Dissertation. University of Glasgow.
- Simulating the feel of brain-computer interfaces for design, development and social interaction. [Proceedings of the 2011 Annual conference on Human factors in computing systems – CHI ’11] (2011), 25.
- Ehsan Rashedi and Maury A Nussbaum. 2015. Mathematical models of localized muscle fatigue: sensitivity analysis and assessment of two occupationally-relevant models. [PloS one] 10, 12 (2015), e0143872.
- Virtual Reality (VR) Automated Testing in the Wild: A Case Study on Unity-Based VR Applications. In [Proceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis] (Seattle, WA, USA) [(ISSTA 2023)]. Association for Computing Machinery, New York, NY, USA, 1269–1281. https://doi.org/10.1145/3597926.3598134
- Benchmarking of dynamic simulation predictions in two software platforms using an upper limb musculoskeletal model. [Computer methods in biomechanics and biomedical engineering] 5842, May 2016 (2014), 1–14. https://doi.org/10.1080/10255842.2014.916698
- DEP-RL: Embodied Exploration for Reinforcement Learning in Overactuated and Musculoskeletal Systems. In [The Eleventh International Conference on Learning Representations]. https://openreview.net/forum?id=C-xa_D3oTj6
- User Representations in Human-Computer Interaction. [Human–Computer Interaction] 0, 0 (2020), 1–39. https://doi.org/10.1080/07370024.2020.1724790 arXiv:https://doi.org/10.1080/07370024.2020.1724790
- OpenSim: a musculoskeletal modeling and simulation framework for in silico investigations and exchange. [Procedia IUTAM] 2 (2011), 212–232. https://doi.org/10.1016/j.piutam.2011.04.021 IUTAM Symposium on Human Body Dynamics.
- Peripheral vision and pattern recognition: A review. [Journal of Vision] 11, 5 (12 2011), 13–13. https://doi.org/10.1167/11.5.13 arXiv:https://arvojournals.org/arvo/content_public/journal/jov/933487/jov-11-5-13.pdf
- MuJoCo: A physics engine for model-based control. In [2012 IEEE/RSJ International Conference on Intelligent Robots and Systems]. 5026–5033. https://doi.org/10.1109/IROS.2012.6386109
- Emanuel Todorov and Michael I. Jordan. 2002. Optimal feedback control as a theory of motor coordination. [Nature Neuroscience] 5, 11 (2002), 1226–1235. https://doi.org/10.1038/nn963
- A Model of Human Muscle Energy Expenditure. [Computer Methods in Biomechanics and Biomedical Engineering] 6, 2 (2003), 99–111. https://doi.org/10.1080/1025584031000091678 arXiv:https://doi.org/10.1080/1025584031000091678 PMID: 12745424.
- MyoSim: Fast and physiologically realistic MuJoCo models for musculoskeletal and exoskeletal studies. In [2022 International Conference on Robotics and Automation (ICRA)]. IEEE, 8104–8111.
- Xiaoyin Wang. 2022. VRTest: an extensible framework for automatic testing of virtual reality scenes. In [Proceedings of the ACM/IEEE 44th International Conference on Software Engineering: Companion Proceedings]. 232–236.
- DA Winter. 1984. Biomechanics of human movement with applications to the study of human locomotion. [Critical reviews in biomedical engineering] 9, 4 (1984), 287—314. http://europepmc.org/abstract/MED/6368126
- Ting Xia and Laura A Frey Law. 2008. A theoretical approach for modeling peripheral muscle fatigue and recovery. [Journal of biomechanics] 41, 14 (2008), 3046–3052.
- Transfer learning in deep reinforcement learning: A survey. [arXiv preprint arXiv:2009.07888] (2020).