Papers
Topics
Authors
Recent
Assistant
AI Research Assistant
Well-researched responses based on relevant abstracts and paper content.
Custom Instructions Pro
Preferences or requirements that you'd like Emergent Mind to consider when generating responses.
Gemini 2.5 Flash
Gemini 2.5 Flash 171 tok/s
Gemini 2.5 Pro 47 tok/s Pro
GPT-5 Medium 32 tok/s Pro
GPT-5 High 36 tok/s Pro
GPT-4o 60 tok/s Pro
Kimi K2 188 tok/s Pro
GPT OSS 120B 437 tok/s Pro
Claude Sonnet 4.5 36 tok/s Pro
2000 character limit reached

ImDy: Human Inverse Dynamics from Imitated Observations (2410.17610v3)

Published 23 Oct 2024 in cs.AI, cs.CV, cs.GR, and cs.RO

Abstract: Inverse dynamics (ID), which aims at reproducing the driven torques from human kinematic observations, has been a critical tool for gait analysis. However, it is hindered from wider application to general motion due to its limited scalability. Conventional optimization-based ID requires expensive laboratory setups, restricting its availability. To alleviate this problem, we propose to exploit the recently progressive human motion imitation algorithms to learn human inverse dynamics in a data-driven manner. The key insight is that the human ID knowledge is implicitly possessed by motion imitators, though not directly applicable. In light of this, we devise an efficient data collection pipeline with state-of-the-art motion imitation algorithms and physics simulators, resulting in a large-scale human inverse dynamics benchmark as Imitated Dynamics (ImDy). ImDy contains over 150 hours of motion with joint torque and full-body ground reaction force data. With ImDy, we train a data-driven human inverse dynamics solver ImDyS(olver) in a fully supervised manner, which conducts ID and ground reaction force estimation simultaneously. Experiments on ImDy and real-world data demonstrate the impressive competency of ImDyS in human inverse dynamics and ground reaction force estimation. Moreover, the potential of ImDy(-S) as a fundamental motion analysis tool is exhibited with downstream applications. The project page is https://foruck.github.io/ImDy/.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Drecon: data-driven responsive control of physics-based characters. ACM Transactions On Graphics (TOG), 38(6):1–11, 2019.
  2. Analysis of musculoskeletal systems in the anybody modeling system. Simulation Modelling Practice and Theory, 14(8):1100–1111, 2006.
  3. Opensim: open-source software to create and analyze dynamic simulations of movement. IEEE transactions on biomedical engineering, 54(11):1940–1950, 2007.
  4. A public dataset of overground and treadmill walking kinematics and kinetics in healthy individuals. PeerJ, 6:e4640, 2018.
  5. Differentiable dynamics for articulated 3d human motion reconstruction. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  13180–13190, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society. doi: 10.1109/CVPR52688.2022.01284. URL https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.01284.
  6. Generating diverse and natural 3d human motions from text. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  5142–5151, Los Alamitos, CA, USA, 2022. IEEE Computer Society. doi: 10.1109/CVPR52688.2022.00509.
  7. Trajectory optimization for physics-based reconstruction of 3d human pose from monocular video. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  13096–13105, Los Alamitos, CA, USA, 2022. IEEE Computer Society. doi: 10.1109/CVPR52688.2022.01276.
  8. Groundlink: A dataset unifying human body movement and ground reaction dynamics. In SIGGRAPH Asia 2023 Conference Papers, SA ’23, New York, NY, USA, 2023. Association for Computing Machinery. ISBN 9798400703157. doi: 10.1145/3610548.3618247. URL https://doi.org/10.1145/3610548.3618247.
  9. Neural mocon: Neural motion control for physically plausible human motion capture. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  6407–6416, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society. doi: 10.1109/CVPR52688.2022.00631. URL https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.00631.
  10. Efficient codes for inverse dynamics during walking. In Carla E. Brodley and Peter Stone (eds.), Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, pp.  343–349, Québec City, Québec, Canada, 2014. AAAI Press. doi: 10.1609/AAAI.V28I1.8747. URL https://doi.org/10.1609/aaai.v28i1.8747.
  11. From skin to skeleton: Towards biomechanically accurate 3d digital humans. ACM Transactions on Graphics (TOG), 42(6):1–12, 2023.
  12. Whole-body human inverse dynamics with distributed micro-accelerometers, gyros and force sensing. Sensors, 16(5), 2016. ISSN 1424-8220. doi: 10.3390/s16050727. URL https://www.mdpi.com/1424-8220/16/5/727.
  13. Simultaneous floating-base estimation of human kinematics and joint torques. Sensors, 19(12), 2019. ISSN 1424-8220. doi: 10.3390/s19122794. URL https://www.mdpi.com/1424-8220/19/12/2794.
  14. D &d: Learning human dynamics from dynamic camera. In Computer Vision – ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part V, pp.  479–496, Berlin, Heidelberg, 2022. Springer-Verlag. ISBN 978-3-031-20064-9. doi: 10.1007/978-3-031-20065-6˙28. URL https://doi.org/10.1007/978-3-031-20065-6_28.
  15. Smpl: a skinned multi-person linear model. In ACM Transactions on Graphics, volume 34, New York, NY, USA, oct 2015. Association for Computing Machinery. doi: 10.1145/2816795.2818013. URL https://doi.org/10.1145/2816795.2818013.
  16. Perpetual humanoid control for real-time simulated avatars. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp.  10861–10870, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society. doi: 10.1109/ICCV51070.2023.01000. URL https://doi.ieeecomputersociety.org/10.1109/ICCV51070.2023.01000.
  17. Dynamics-regulated kinematic policy for egocentric pose estimation. Advances in Neural Information Processing Systems, 34:25019–25032, 2021.
  18. Data-driven inverse dynamics for human motion. ACM Transactions on Graphics (TOG), 35(6):1–12, 2016.
  19. Amass: Archive of motion capture as surface shapes. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp.  5441–5450, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. doi: 10.1109/ICCV.2019.00554. URL https://doi.ieeecomputersociety.org/10.1109/ICCV.2019.00554.
  20. Isaac gym: High performance GPU based physics simulation for robot learning. In Joaquin Vanschoren and Sai-Kit Yeung (eds.), Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks, Virtual, 2021. Curran Associates Inc. URL https://datasets-benchmarks-proceedings.neurips.cc/paper/2021/hash/28dd2c7955ce926456240b2ff0100bde-Abstract-round2.html.
  21. Unifying representations and large-scale whole-body motion databases for studying human motion. IEEE Transactions on Robotics, 32(4):796–809, 2016.
  22. Artificial physics engine for real-time inverse dynamics of arm and hand movement. Plos one, 18(12):e0295750, 2023.
  23. Underpressure: Deep learning for foot contact detection, ground reaction force estimation and footskate cleanup. Computer Graphics Forum, 41(8):195–206, 2022. doi: https://doi.org/10.1111/cgf.14635. URL https://onlinelibrary.wiley.com/doi/abs/10.1111/cgf.14635.
  24. Amp: Adversarial motion priors for stylized physics-based character control. ACM Transactions on Graphics (ToG), 40(4):1–20, 2021.
  25. Ase: Large-scale reusable adversarial skill embeddings for physically simulated characters. ACM Transactions On Graphics (TOG), 41(4):1–17, 2022.
  26. Babel: Bodies, action and behavior with english labels. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  722–731, Los Alamitos, CA, USA, jun 2021. IEEE Computer Society. doi: 10.1109/CVPR46437.2021.00078. URL https://doi.ieeecomputersociety.org/10.1109/CVPR46437.2021.00078.
  27. Full-body musculoskeletal model for muscle-driven simulation of human gait. IEEE transactions on biomedical engineering, 63(10):2068–2079, 2016.
  28. Contact and human dynamics from monocular video. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V, pp.  71–87, Berlin, Heidelberg, 2020. Springer-Verlag. ISBN 978-3-030-58557-0. doi: 10.1007/978-3-030-58558-7˙5. URL https://doi.org/10.1007/978-3-030-58558-7_5.
  29. A multimodal dataset of human gait at different walking speeds established on injury-free adult participants. Scientific data, 6(1):111, 2019.
  30. Proximal policy optimization algorithms. preprint on webpage at arXiv:1707.06347, 2017.
  31. From image to stability: Learning dynamics from human pose. In Computer Vision – ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXIII, pp.  536–554, Berlin, Heidelberg, 2020. Springer-Verlag. ISBN 978-3-030-58591-4. doi: 10.1007/978-3-030-58592-1˙32. URL https://doi.org/10.1007/978-3-030-58592-1_32.
  32. Ntu rgb+d: A large scale dataset for 3d human activity analysis. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  1010–1019, Los Alamitos, CA, USA, jun 2016. IEEE Computer Society. doi: 10.1109/CVPR.2016.115. URL https://doi.ieeecomputersociety.org/10.1109/CVPR.2016.115.
  33. Neural monocular 3d human motion capture with physical awareness. ACM Transactions on Graphics (ToG), 40(4):1–15, 2021.
  34. Human motion diffusion model. In The Eleventh International Conference on Learning Representations ICLR 2023, Kigali, Rwanda, 2023. OpenReview.net. URL https://openreview.net/pdf?id=SJ1kSyO2jwu.
  35. Biomechanics of movement: the science of sports, robotics, and rehabilitation. Mit Press, 2021.
  36. Conclusion or illusion: Quantifying uncertainty in inverse analyses from marker-based motion capture due to errors in marker registration and model scaling. Frontiers in Bioengineering and Biotechnology, 10:874725, 2022.
  37. Learning human dynamics in autonomous driving scenarios. In 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp.  20739–20749, Los Alamitos, CA, USA, oct 2023. IEEE Computer Society. doi: 10.1109/ICCV51070.2023.01901. URL https://doi.ieeecomputersociety.org/10.1109/ICCV51070.2023.01901.
  38. Fast and Feature-Complete Differentiable Physics Engine for Articulated Rigid Bodies with Contact Constraints. In Robotics: Science and Systems XVII, Virtual Event, July 12-16, 2021, Virtual, July 2021. RSS Foundation. doi: 10.15607/RSS.2021.XVII.034. URL https://doi.org/10.15607/RSS.2021.XVII.034.
  39. Addbiomechanics: Automating model scaling, inverse kinematics, and inverse dynamics from human motion data through sequential optimization. Plos one, 18(11):e0295152, 2023.
  40. Addbiomechanics dataset: Capturing the physics of human motion at scale. arXiv preprint arXiv:2406.18537, 2024.
  41. Control strategies for physically simulated characters performing two-player competitive sports. ACM Transactions on Graphics (TOG), 40(4):1–11, 2021.
  42. Physics-based character controllers using conditional vaes. ACM Transactions on Graphics (TOG), 41(4):1–12, 2022.
  43. Intelligent prediction of human lower extremity joint moment: an artificial neural network approach. Ieee Access, 7:29973–29980, 2019.
  44. Physical inertial poser (pip): Physics-aware real-time human motion tracking from sparse inertial sensors. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  13157–13168, Los Alamitos, CA, USA, jun 2022. IEEE Computer Society. doi: 10.1109/CVPR52688.2022.01282. URL https://doi.ieeecomputersociety.org/10.1109/CVPR52688.2022.01282.
  45. Y. Yuan and K. Kitani. Ego-pose estimation and forecasting as real-time pd control. In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp.  10081–10091, Los Alamitos, CA, USA, nov 2019. IEEE Computer Society. doi: 10.1109/ICCV.2019.01018. URL https://doi.ieeecomputersociety.org/10.1109/ICCV.2019.01018.
  46. P. Zell and B. Rosenhahn. Learning-based inverse dynamics of human motion. In 2017 IEEE International Conference on Computer Vision Workshop (ICCVW), pp.  842–850, Los Alamitos, CA, USA, oct 2017. IEEE Computer Society. doi: 10.1109/ICCVW.2017.104. URL https://doi.ieeecomputersociety.org/10.1109/ICCVW.2017.104.
  47. Joint 3d human motion capture and physical analysis from monocular videos. In 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp.  17–26, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society. doi: 10.1109/CVPRW.2017.9. URL https://doi.ieeecomputersociety.org/10.1109/CVPRW.2017.9.
  48. A physics-based statistical model for human gait analysis. In Juergen Gall, Peter Gehler, and Bastian Leibe (eds.), Pattern Recognition, pp.  169–180, Cham, 2015. Springer International Publishing. ISBN 978-3-319-24947-6.
  49. Weakly-supervised learning of human dynamics. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm (eds.), Computer Vision – ECCV 2020, pp.  68–84, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58574-7.
  50. On the continuity of rotation representations in neural networks. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.  5738–5746, Los Alamitos, CA, USA, jun 2019. IEEE Computer Society. doi: 10.1109/CVPR.2019.00589. URL https://doi.ieeecomputersociety.org/10.1109/CVPR.2019.00589.

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up