Papers
Topics
Authors
Recent
Search
2000 character limit reached

UbiPhysio: Support Daily Functioning, Fitness, and Rehabilitation with Action Understanding and Feedback in Natural Language

Published 21 Aug 2023 in cs.HC | (2308.10526v3)

Abstract: We introduce UbiPhysio, a milestone framework that delivers fine-grained action description and feedback in natural language to support people's daily functioning, fitness, and rehabilitation activities. This expert-like capability assists users in properly executing actions and maintaining engagement in remote fitness and rehabilitation programs. Specifically, the proposed UbiPhysio framework comprises a fine-grained action descriptor and a knowledge retrieval-enhanced feedback module. The action descriptor translates action data, represented by a set of biomechanical movement features we designed based on clinical priors, into textual descriptions of action types and potential movement patterns. Building on physiotherapeutic domain knowledge, the feedback module provides clear and engaging expert feedback. We evaluated UbiPhysio's performance through extensive experiments with data from 104 diverse participants, collected in a home-like setting during 25 types of everyday activities and exercises. We assessed the quality of the language output under different tuning strategies using standard benchmarks. We conducted a user study to gather insights from clinical physiotherapists and potential users about our framework. Our initial tests show promise for deploying UbiPhysio in real-life settings without specialized devices.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (72)
  1. Kinect. 2023. www.xbox.com/en-US/kinect
  2. Noitom. 2023. www.noitom.com/perception-neuron-series
  3. Body cues, not facial expressions, discriminate between intense positive and negative emotions. Science 338, 6111 (2012), 1225–1229.
  4. Context in emotion perception. Current directions in psychological science 20, 5 (2011), 286–290.
  5. Leveraging sound and wrist motion to detect activities of daily living with commodity smartwatches. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 2 (2022), 1–28.
  6. Clinical observation of standing trunk movements: What do the aberrant movement patterns tell us? journal of orthopaedic & sports physical therapy 44, 4 (2014), 262–272.
  7. Self-monitoring in weight loss: a systematic review of the literature. Journal of the American Dietetic Association 111, 1 (2011), 92–102.
  8. Jiangjie Chen and Yanghua Xiao. 2022. Harnessing Knowledge and Reasoning for Human-Like Natural Language Generation: A Brief Review. arXiv preprint arXiv:2212.03747 (2022).
  9. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416 (2022).
  10. Reactive video: adaptive video playback based on user motion for supporting physical activity. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology. 196–208.
  11. Ambient intelligence: Technologies, applications, and opportunities. Pervasive and mobile computing 5, 4 (2009), 277–298.
  12. CASAS: A smart home in a box. Computer 46, 7 (2012), 62–69.
  13. GLM: General Language Model Pretraining with Autoregressive Blank Infilling. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 320–335.
  14. Quantity and Quality of Exercise for Developing and Maintaining Cardiorespiratory, Musculoskeletal, and Neuromotor Fitness in Apparently Healthy Adults: Guidance for Prescribing Exercise. Medicine & Science in Sports & Exercise 43, 7 (July 2011), 1334–1359.
  15. Martin A Giese and Tomaso Poggio. 2003. Neural mechanisms for the recognition of biological movements. Nature Reviews Neuroscience 4, 3 (2003), 179–192.
  16. Pain neuroscience education and motor control exercises versus core stability exercises on pain, disability, and balance in women with chronic low back pain. International Journal of Environmental Research and Public Health 19, 5 (2022), 2694.
  17. Generating Diverse and Natural 3D Human Motions From Text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 5152–5161.
  18. TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Texts. In ECCV.
  19. Device-free personalized fitness assistant using WiFi. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 2, 4 (2018), 1–23.
  20. Effectiveness of supervised physiotherapy versus home exercise in subjects with rotator cuff disorders treated surgically: A systematic review and meta-analysis. Physiotherapy Research International 27, 2 (2022).
  21. Retrieval augmented language model pre-training. In International conference on machine learning. PMLR, 3929–3938.
  22. A randomized clinical trial comparing the McKenzie method and motor control exercises in people with chronic low back pain and a directional preference: 1-year follow-up. Physiotherapy 105, 4 (2019), 442–445.
  23. Remote monitoring of stroke patients’ rehabilitation using wearable accelerometers. In Proceedings of the 2019 ACM International Symposium on Wearable Computers. 72–77.
  24. Towards Learning Discrete Representations via Self-Supervision for Wearables-Based Human Activity Recognition. arXiv preprint arXiv:2306.01108 (2023).
  25. Bootstrapping Human Activity Recognition Systems for Smart Homes from Scratch. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 3 (2022), 1–27.
  26. Onebody: remote posture guidance system using first person view in virtual environment. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction. 1–10.
  27. Learned Motion Matching. ACM Trans. Graph. 39, 4, Article 53 (aug 2020), 13 pages.
  28. Phase-Functioned Neural Networks for Character Control. ACM Trans. Graph. 36, 4, Article 42 (jul 2017), 13 pages.
  29. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021).
  30. MotionGPT: Human Motion as a Foreign Language. arXiv preprint arXiv:2306.14795 (2023).
  31. FlowAR: How Different Augmented Reality Visualizations of Online Fitness Videos Support Flow for At-Home Yoga Exercises. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–17.
  32. A wearable motion capture suit and machine learning predict disease progression in Friedreich’s ataxia. Nature Medicine 29, 1 (2023), 86–94.
  33. Beomryong Kim and Jongeun Yim. 2020. Core stability and hip exercises improve physical function and activity in patients with non-specific low back pain: a randomized controlled trial. The Tohoku journal of experimental medicine 251, 3 (2020), 193–206.
  34. Learning Joint Representation of Human Motion and Language. arXiv:2210.15187
  35. Effects of functional tasks exercise on older adults with cognitive impairment at risk of Alzheimer’s disease: a randomised controlled trial. Age and ageing 43, 6 (2014), 813–820.
  36. A human-ai collaborative approach for clinical decision making on rehabilitation assessment. In Proceedings of the 2021 CHI conference on human factors in computing systems. 1–14.
  37. Enabling Voice-Accompanying Hand-to-Face Gesture Recognition with Cross-Device Sensing. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 313, 17 pages.
  38. A review of computational approaches for evaluation of rehabilitation exercises. Computers in biology and medicine 119 (2020), 103687.
  39. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81.
  40. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers). 61–68.
  41. SMPL: A Skinned Multi-Person Linear Model (1 ed.). Association for Computing Machinery, New York, NY, USA.
  42. Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017).
  43. Feasibility of a home-based interdisciplinary rehabilitation program for patients with Post-Intensive Care Syndrome: the REACH study. Critical Care 25, 1 (2021), 1–15.
  44. Interventions for enhancing adherence with physiotherapy: a systematic review. Manual therapy 15, 6 (2010), 514–521.
  45. Daily activity recognition with inertial ring and bracelet: An unsupervised approach. In 2017 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 3250–3255.
  46. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics. 311–318.
  47. The rise of consumer health wearables: promises and barriers. PLoS medicine 13, 2 (2016), e1001953.
  48. Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks. Robotics and Autonomous Systems (Nov 2018), 13–26.
  49. Activity recognition and healthier food preparation. Activity Recognition in Pervasive Intelligent Environments (2011), 313–329.
  50. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research 21, 140 (2020), 1–67.
  51. In-context retrieval-augmented language models. arXiv preprint arXiv:2302.00083 (2023).
  52. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems 32 (2019).
  53. Wearable full-body motion tracking of activities of daily living predicts disease trajectory in Duchenne muscular dystrophy. Nature medicine 29, 1 (2023), 95–103.
  54. Wearable movement-tracking data identify Parkinson’s disease years before clinical diagnosis. Nature Medicine (2023), 1–9.
  55. Kinect-enabled home-based rehabilitation system using Dynamic Time Warping and fuzzy logic. Applied Soft Computing 22 (2014), 652–666.
  56. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288 (2023).
  57. Neural discrete representation learning. Advances in neural information processing systems 30 (2017).
  58. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition. 4566–4575.
  59. Leveraging activity recognition to enable protective behavior detection in continuous data. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 5, 2 (2021), 1–27.
  60. Recurrent network based automatic detection of chronic pain protective behavior using mocap and semg data. In Proceedings of the 2019 ACM International Symposium on Wearable Computers. 225–230.
  61. Chronic pain protective behavior detection with deep learning. ACM Transactions on Computing for Healthcare 2, 3 (2021), 1–24.
  62. Hanchen David Wang and Meiyi Ma. 2023. PhysiQ: Off-site Quality Assessment of Exercise in Physical Therapy. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies 6, 4 (2023), 1–25.
  63. Recent developments in human motion analysis. Pattern recognition 36, 3 (2003), 585–601.
  64. HUMANISE: Language-conditioned Human Motion Generation in 3D Scenes. In Advances in Neural Information Processing Systems, S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (Eds.), Vol. 35. Curran Associates, Inc., 14959–14971.
  65. Towards on-demand virtual physical therapist: Machine learning-based patient action understanding, assessment and task recommendation. IEEE Transactions on Neural Systems and Rehabilitation Engineering 27, 9 (2019), 1824–1835.
  66. HearFit+: Personalized fitness monitoring via audio signals on smart speakers. IEEE Transactions on Mobile Computing (2021).
  67. Paired Recurrent Autoencoders for Bidirectional Translation Between Robot Actions and Linguistic Descriptions. IEEE Robotics and Automation Letters 3, 4 (2018), 3441–3448.
  68. Glm-130b: An open bilingual pre-trained model. arXiv preprint arXiv:2210.02414 (2022).
  69. T2M-GPT: Generating Human Motion from Textual Descriptions with Discrete Representations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
  70. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019).
  71. MotionGPT: Finetuned LLMs are General-Purpose Motion Generators. arXiv preprint arXiv:2306.10900 (2023).
  72. PoseFormerV2: Exploring Frequency Domain for Efficient and Robust 3D Human Pose Estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 8877–8886.
Citations (2)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up