Papers
Topics
Authors
Recent
Search
2000 character limit reached

Motion2Language, unsupervised learning of synchronized semantic motion segmentation

Published 16 Oct 2023 in cs.CV and cs.CL | (2310.10594v2)

Abstract: In this paper, we investigate building a sequence to sequence architecture for motion to language translation and synchronization. The aim is to translate motion capture inputs into English natural-language descriptions, such that the descriptions are generated synchronously with the actions performed, enabling semantic segmentation as a byproduct, but without requiring synchronized training data. We propose a new recurrent formulation of local attention that is suited for synchronous/live text generation, as well as an improved motion encoder architecture better suited to smaller data and for synchronous generation. We evaluate both contributions in individual experiments, using the standard BLEU4 metric, as well as a simple semantic equivalence measure, on the KIT motion language dataset. In a follow-up experiment, we assess the quality of the synchronization of generated text in our proposed approaches through multiple evaluation metrics. We find that both contributions to the attention mechanism and the encoder architecture additively improve the quality of generated text (BLEU and semantic equivalence), but also of synchronization. Our code is available at https://github.com/rd20karim/M2T-Segmentation/tree/main

Definition Search Book Streamline Icon: https://streamlinehq.com
References (26)
  1. Unifying representations and large-scale whole-body motion databases for studying human motion. IEEE Transactions on Robotics, 32:796–809, 8 2016. ISSN 15523098. doi:10.1109/TRO.2016.2572685.
  2. Generating diverse and natural 3d human motions from text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5152–5161, June 2022a.
  3. The KIT motion-language dataset. Big Data, 4(4):236–252, dec 2016. doi:10.1089/big.2016.0028. URL http://dx.doi.org/10.1089/big.2016.0028.
  4. Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks. Robotics and Autonomous Systems, 109:13–26, 5 2017. doi:10.1016/j.robot.2018.07.006.
  5. Generating animated videos of human activities from natural language descriptions, 2018.
  6. Synthesis of compositional animations from textual descriptions. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 1396–1406, October 2021.
  7. Temos: Generating diverse human motions from textual descriptions. In Shai Avidan, Gabriel Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, Computer Vision – ECCV 2022, pages 480–497, Cham, 2022. Springer Nature Switzerland. ISBN 978-3-031-20047-2.
  8. Linguistic descriptions of human motion with generative adversarial seq2seq learning. In 2021 IEEE International Conference on Robotics and Automation (ICRA), pages 4281–4287, 2021. doi:10.1109/ICRA48506.2021.9561519.
  9. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, 2022b.
  10. Jonathan Feng Shun Lin and Dana Kulic. Online segmentation of human motion for automated rehabilitation exercise analysis. IEEE Transactions on Neural Systems and Rehabilitation Engineering, 22:168–180, 2014. ISSN 15344320. doi:10.1109/TNSRE.2013.2259640.
  11. Online segmentation and clustering from continuous observation of whole body motions. IEEE Transactions on Robotics, 25:1158–1166, 2009. ISSN 15523098. doi:10.1109/TRO.2009.2026508.
  12. Arma-based segmentation of human limb motion sequences. Sensors, 21(16), 2021. ISSN 1424-8220. doi:10.3390/s21165577. URL https://www.mdpi.com/1424-8220/21/16/5577.
  13. Human motion segmentation using collaborative representations of 3d skeletal sequences. IET Computer Vision, 12:434–442, 2018. ISSN 17519640. doi:10.1049/IET-CVI.2016.0385.
  14. Aligned cluster analysis for temporal segmentation of human motion. In 2008 8th IEEE International Conference on Automatic Face and Gesture Recognition, pages 1–7, 2008. doi:10.1109/AFGR.2008.4813468.
  15. Hierarchical aligned cluster analysis for temporal clustering of human motion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 35(3):582–596, 2013. doi:10.1109/TPAMI.2012.137.
  16. Fine-grained unsupervised temporal action segmentation and distributed representation for skeleton-based human motion analysis. IEEE Transactions on Cybernetics, 2021. ISSN 21682275. doi:10.1109/TCYB.2021.3132016.
  17. Skeleton-based action segmentation with multi-stage spatial-temporal graph convolutional neural networks. IEEE Transactions on Emerging Topics in Computing, pages 1–11, 2022. doi:10.1109/tetc.2022.3230912. URL https://doi.org/10.1109%2Ftetc.2022.3230912.
  18. Sequence to sequence learning with neural networks. In Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2, NIPS’14, page 3104–3112, Cambridge, MA, USA, 2014. MIT Press.
  19. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, pages 1–15, 2015.
  20. Effective approaches to attention-based neural machine translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1412–1421, Lisbon, Portugal, 2015. Association for Computational Linguistics. doi:10.18653/v1/D15-1166. URL https://aclanthology.org/D15-1166.
  21. Automatic segmentation of sign language into subtitle-units. In Adrien Bartoli and Andrea Fusiello, editors, Computer Vision – ECCV 2020 Workshops, pages 186–198, Cham, 2020. Springer International Publishing. ISBN 978-3-030-66096-3.
  22. Aligning subtitles in sign language videos. ICCV, pages 11552–11561, 2021. URL https://www.robots.ox.ac.uk/.
  23. Tune: A research platform for distributed model selection and training. CoRR, abs/1807.05118, 2018.
  24. "sentence-BERT: Sentence embeddings using Siamese BERT-networks". In "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)", pages 3982–3992, Hong Kong, China, November 2019. Association for Computational Linguistics. doi:10.18653/v1/D19-1410. URL https://aclanthology.org/D19-1410.
  25. Movement primitive segmentation for human motion modeling: A framework for analysis. IEEE Transactions on Human-Machine Systems, 46(3):325–339, 2016. doi:10.1109/THMS.2015.2493536.
  26. Visual analysis of human motion: A survey on recent advances and applications. In 2018 IEEE Visual Communications and Image Processing (VCIP), pages 1–4, 2018. doi:10.1109/VCIP.2018.8698618.
Citations (4)
View on Semantic Scholar

Summary

No one has generated a summary of this paper yet.

Sign Up to Summarize

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Continue Learning

We haven't generated follow-up questions for this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up