Papers
Topics
Authors
Recent
Search
2000 character limit reached

Estimating Body and Hand Motion in an Ego-sensed World

Published 4 Oct 2024 in cs.CV and cs.AI | (2410.03665v3)

Abstract: We present EgoAllo, a system for human motion estimation from a head-mounted device. Using only egocentric SLAM poses and images, EgoAllo guides sampling from a conditional diffusion model to estimate 3D body pose, height, and hand parameters that capture a device wearer's actions in the allocentric coordinate frame of the scene. To achieve this, our key insight is in representation: we propose spatial and temporal invariance criteria for improving model performance, from which we derive a head motion conditioning parameterization that improves estimation by up to 18%. We also show how the bodies estimated by our system can improve hand estimation: the resulting kinematic and temporal constraints can reduce world-frame errors in single-frame estimates by 40%. Project page: https://egoallo.github.io/

Definition Search Book Streamline Icon: https://streamlinehq.com
References (112)
  1. Listen, denoise, action! audio-driven motion synthesis with diffusion models. ACM Transactions on Graphics (TOG), 42(4):1–20, 2023.
  2. Scape: shape completion and animation of people. In ACM SIGGRAPH 2005 Papers, pages 408–416. ACM New York, NY, USA, 2005.
  3. Multidiffusion: Fusing diffusion paths for controlled image generation. arXiv preprint arXiv:2302.08113, 2023.
  4. Seamless human motion composition with blended positional encodings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 457–469, 2024.
  5. Physics-based person tracking using the anthropomorphic walker. International journal of computer vision, 87(1-2):140–155, 2010.
  6. Bodiffusion: Diffusing sparse observations for full-body human motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4221–4231, 2023.
  7. Pointnet: Deep learning on point sets for 3d classification and segmentation. In 2017 IEEE conference on computer vision and pattern recognition (CVPR), pages 77–85. IEEE, 2017.
  8. Equivariant point network for 3d point cloud analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 14514–14523, 2021.
  9. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597–1607. PMLR, 2020.
  10. Algorithm 887: Cholmod, supernodal sparse cholesky factorization and update/downdate. ACM Transactions on Mathematical Software (TOMS), 35(3):1–14, 2008.
  11. Gfpose: Learning 3d human pose prior with gradient fields. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 4800–4810, 2023.
  12. Group equivariant convolutional networks. In International conference on machine learning, pages 2990–2999. PMLR, 2016.
  13. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780–8794, 2021.
  14. Generalizing neural human fitting to unseen poses with articulated se (3) equivariance. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7977–7988, 2023.
  15. Probabilistic character motion synthesis using a hierarchical deep latent variable model. In Computer Graphics Forum, pages 225–239. Wiley Online Library, 2020.
  16. Humans in 4D: Reconstructing and tracking humans with transformers. In Int. Conf. Comput. Vis., 2023.
  17. Ego-exo4d: Understanding skilled human activity from first-and third-person perspectives. arXiv preprint arXiv:2311.18259, 2023.
  18. Egolifter: Open-world 3d segmentation for egocentric perception. arXiv preprint arXiv:2403.18118, 2024.
  19. Estimating human shape and pose from a single image. In Int. Conf. Comput. Vis., pages 1381–1388. IEEE, 2009.
  20. Holopose: Holistic 3d human reconstruction in-the-wild. In IEEE Conf. Comput. Vis. Pattern Recog., pages 10884–10894, 2019.
  21. Hmd2: Environment-aware motion generation from single egocentric head-mounted device. arXiv preprint arXiv:2409.13426, 2024.
  22. Nemf: Neural motion fields for kinematic animation. In NeurIPS, 2022.
  23. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840–6851, 2020.
  24. Egolm: Multi-modal language model of egocentric motions. arXiv preprint arXiv:2409.18127, 2024.
  25. Bayesian reconstruction of 3d human motion from single-camera video. Advances in neural information processing systems, 12, 1999.
  26. Capturing and inferring dense full-body human-scene contact. In IEEE Conf. Comput. Vis. Pattern Recog., pages 13274–13285, 2022.
  27. Egocentric pose estimation from human vision span. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 10986–10994. IEEE, 2021.
  28. Avatarposer: Articulated full-body pose tracking from sparse motion sensing. In European conference on computer vision, pages 443–460. Springer, 2022.
  29. Egoposer: Robust real-time ego-body pose estimation in large scenes. arXiv preprint arXiv:2308.06493, 2023.
  30. Back to optimization: Diffusion-based zero-shot 3d human pose estimation. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6142–6152, 2024.
  31. Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation. In 2021 International Conference on 3D Vision (3DV), pages 42–52. IEEE, 2021.
  32. Locally scale-invariant convolutional neural networks. arXiv preprint arXiv:1412.5104, 2014.
  33. End-to-end recovery of human shape and pose. In IEEE Conf. Comput. Vis. Pattern Recog., pages 7122–7131, 2018.
  34. Learning 3d human dynamics from video. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5614–5623, 2019.
  35. Guided motion diffusion for controllable human motion synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2151–2162, 2023.
  36. Flame: Free-form language-based motion synthesis & editing. In Proceedings of the AAAI Conference on Artificial Intelligence, pages 8255–8263, 2023.
  37. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114, 2013.
  38. Vibe: Video inference for human body pose and shape estimation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5253–5263, 2020.
  39. Pare: Part attention regressor for 3d human body estimation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11127–11137, 2021.
  40. Pace: Human and motion estimation from in-the-wild videos. In 3DV, 2024.
  41. Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In Int. Conf. Comput. Vis., pages 2252–2261, 2019.
  42. Nifty: Neural object interaction fields for guided human motion synthesis. arXiv preprint arXiv:2307.07511, 2023.
  43. Unite the people: Closing the loop between 3d and 2d human representations. In IEEE Conf. Comput. Vis. Pattern Recog., pages 6050–6059, 2017.
  44. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 3361(10):1995, 1995.
  45. Mocap everyone everywhere: Lightweight motion capture with smartwatches and a head-mounted camera. arXiv preprint arXiv:2401.00847, 2024.
  46. Controllable human-object interaction synthesis. arXiv preprint arXiv:2312.03913, 2023a.
  47. Ego-body pose estimation via ego-head pose estimation. In IEEE Conf. Comput. Vis. Pattern Recog., pages 17142–17151, 2023b.
  48. Object motion guided human motion synthesis. ACM Transactions on Graphics (TOG), 42(6):1–11, 2023c.
  49. Estimating 3d motion and forces of person-object interactions from monocular video. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8640–8649, 2019.
  50. Character controllers using motion vaes. ACM Transactions on Graphics (TOG), 39(4):40–1, 2020.
  51. Smpl: A skinned multi-person linear model. In Seminal Graphics Papers: Pushing the Boundaries, Volume 2, pages 851–866, 2023.
  52. David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60:91–110, 2004.
  53. Dynamics-regulated kinematic policy for egocentric pose estimation. Advances in Neural Information Processing Systems, 34:25019–25032, 2021.
  54. Nymeria: A massive collection of multimodal egocentric daily motion in the wild. arXiv preprint arXiv:2406.09905, 2024.
  55. Amass: Archive of motion capture as surface shapes. In IEEE Conf. Comput. Vis. Pattern Recog., pages 5442–5451, 2019.
  56. A simple yet effective baseline for 3d human pose estimation. In Int. Conf. Comput. Vis., pages 2640–2649, 2017.
  57. You2me: Inferring body pose in egocentric video via first and second person interactions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9890–9900, 2020.
  58. Neural body fitting: Unifying deep learning and model based human pose and shape estimation. In 2018 international conference on 3D vision (3DV), pages 484–494. IEEE, 2018.
  59. Aria digital twin: A new benchmark dataset for egocentric 3d machine perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 20133–20143, 2023.
  60. Learning to estimate 3d human pose and shape from a single color image. In IEEE Conf. Comput. Vis. Pattern Recog., pages 459–468, 2018.
  61. Expressive body capture: 3D hands, face, and body from a single image. In IEEE Conf. Comput. Vis. Pattern Recog., pages 10975–10985, 2019.
  62. Human mesh recovery from multiple shots. In IEEE Conf. Comput. Vis. Pattern Recog., pages 1485–1495, 2022a.
  63. The one where they reconstructed 3d humans and environments in tv shows. In Eur. Conf. Comput. Vis., pages 732–749. Springer, 2022b.
  64. Reconstructing hands in 3D with transformers. In arxiv, 2023.
  65. Sfv: Reinforcement learning of physical skills from videos. ACM Transactions On Graphics (TOG), 37(6):1–14, 2018.
  66. Spatial cognition from egocentric video: Out of sight, not out of mind. arXiv preprint arXiv:2404.05072, 2024.
  67. State of the art on diffusion models for visual computing. arXiv preprint arXiv:2310.07204, 2023.
  68. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748–8763. PMLR, 2021.
  69. Tracking people by predicting 3d appearance, location and pose. In IEEE Conf. Comput. Vis. Pattern Recog., pages 2740–2749, 2022.
  70. Hierarchical text-conditional image generation with clip latents, 2022.
  71. Contact and human dynamics from monocular video. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part V 16, pages 71–87. Springer, 2020.
  72. Humor: 3d human motion model for robust pose estimation. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11488–11499, 2021.
  73. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning, pages 1278–1286. PMLR, 2014.
  74. Egocap: egocentric marker-less motion capture with two fisheye cameras. ACM Transactions on Graphics (TOG), 35(6):1–11, 2016.
  75. Lcr-net: Localization-classification-regression for human pose. In IEEE Conf. Comput. Vis. Pattern Recog., pages 3433–3441, 2017.
  76. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022.
  77. Embodied hands: Modeling and capturing hands and bodies together. arXiv preprint arXiv:2201.02610, 2022.
  78. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Information Processing Systems, 35:36479–36494, 2022.
  79. 3d generic object categorization, localization and pose estimation. In 2007 IEEE 11th international conference on computer vision, pages 1–8. IEEE, 2007.
  80. Human motion diffusion as a generative prior. arXiv preprint arXiv:2303.01418, 2023.
  81. Motion capture from body-mounted cameras. In ACM SIGGRAPH 2011 papers, pages 1–10. ACM New York, NY, USA, 2011.
  82. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256–2265. PMLR, 2015.
  83. Project aria: A new tool for egocentric multi-modal ai research. arXiv preprint arXiv:2308.13561, 2023.
  84. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020a.
  85. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
  86. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
  87. Droid-slam: Deep visual slam for monocular, stereo, and rgb-d cameras. Advances in neural information processing systems, 34:16558–16569, 2021.
  88. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219, 2018.
  89. xr-egopose: Egocentric 3d human pose from an hmd camera. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7728–7738, 2019.
  90. Selfpose: 3d egocentric pose estimation from a headset mounted camera. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
  91. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 448–458, 2023.
  92. Attention is all you need. Advances in neural information processing systems, 30, 2017.
  93. Estimating egocentric 3d human pose in global space. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11500–11509, 2021.
  94. Egocentric whole-body motion capture with fisheyevit and diffusion-based motion refinement. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 777–787, 2024.
  95. Slow feature analysis: Unsupervised learning of invariances. Neural computation, 14(4):715–770, 2002.
  96. Mo 2 cap 2: Real-time mobile 3d motion capture with a cap-mounted fisheye camera. IEEE transactions on visualization and computer graphics, 25(5):2093–2101, 2019.
  97. Vitpose+: Vision transformer foundation model for generic body pose estimation. arXiv preprint arXiv:2212.04246, 2022.
  98. Equivact: Sim(3)-equivariant visuomotor policies beyond rigid object manipulation, 2023.
  99. Decoupling human and camera motion from videos in the wild. In IEEE Conf. Comput. Vis. Pattern Recog., pages 21222–21232, 2023.
  100. Differentiable factor graph optimization for learning smoothers. In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2021a.
  101. Canonical factors for hybrid neural fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 3414–3426, 2023a.
  102. Transpose: Real-time 3d human translation and pose estimation with six inertial sensors. ACM Transactions on Graphics (TOG), 40(4):1–13, 2021b.
  103. Egolocate: Real-time motion capture, localization, and mapping with sparse body-mounted sensors. arXiv preprint arXiv:2305.01599, 2023b.
  104. 3d ego-pose estimation via imitation learning. In Proceedings of the European Conference on Computer Vision (ECCV), pages 735–750, 2018.
  105. Ego-pose estimation and forecasting as real-time pd control. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10082–10092, 2019.
  106. Glamr: Global occlusion-aware human mesh recovery with dynamic cameras. In IEEE Conf. Comput. Vis. Pattern Recog., 2022.
  107. Deep sets. Advances in neural information processing systems, 30, 2017.
  108. Egogaussian: Dynamic scene understanding from egocentric video with 3d gaussian splatting. arXiv preprint arXiv:2406.19811, 2024.
  109. Motiondiffuse: Text-driven human motion generation with diffusion model. arXiv preprint arXiv:2208.15001, 2022.
  110. Diffcollage: Parallel generation of large content with diffusion models. In 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10188–10198. IEEE, 2023a.
  111. Probabilistic human mesh recovery in 3d scenes from egocentric views. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7989–8000, 2023b.
  112. Tilt: Transform invariant low-rank textures. International journal of computer vision, 99:1–24, 2012.

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

GitHub

  1. EgoAllo 
  2. GitHub - brentyi/egoallo (1 star)  

Tweets

Sign up for free to view the 1 tweet with 1 like about this paper.

Sign Up for Free