Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
110 tokens/sec
GPT-4o
56 tokens/sec
Gemini 2.5 Pro Pro
44 tokens/sec
o3 Pro
6 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

PoseFix: Correcting 3D Human Poses with Natural Language (2309.08480v2)

Published 15 Sep 2023 in cs.CV

Abstract: Automatically producing instructions to modify one's posture could open the door to endless applications, such as personalized coaching and in-home physical therapy. Tackling the reverse problem (i.e., refining a 3D pose based on some natural language feedback) could help for assisted 3D character animation or robot teaching, for instance. Although a few recent works explore the connections between natural language and 3D human pose, none focus on describing 3D body pose differences. In this paper, we tackle the problem of correcting 3D human poses with natural language. To this end, we introduce the PoseFix dataset, which consists of several thousand paired 3D poses and their corresponding text feedback, that describe how the source pose needs to be modified to obtain the target pose. We demonstrate the potential of this dataset on two tasks: (1) text-based pose editing, that aims at generating corrected 3D body poses given a query pose and a text modifier; and (2) correctional text generation, where instructions are generated based on the differences between two body poses.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (67)
  1. Text2action: Generative adversarial synthesis from language to action. In ICRA, 2018.
  2. Language2pose: Natural language grounded pose forecasting. 3DV, 2019.
  3. Clipface: Text-guided editing of textured 3d morphable models. In SIGGRAPH, 2023.
  4. Vqa: Visual question answering. In ICCV, 2015.
  5. Conditioned and composed image retrieval combining and partially fine-tuning clip-based features. In CVPRW, 2022.
  6. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, 2005.
  7. Instructpix2pix: Learning to follow image editing instructions. In CVPR, 2023.
  8. Language models are few-shot learners. In NeurIPS, 2020.
  9. Learning phrase representations using rnn encoder-decoder for statistical machine translation. In EMNLP, 2014.
  10. Context-aware human motion prediction. In CVPR, 2020.
  11. Artemis: Attention-based retrieval with text-explicit matching and implicit similarity. In ICLR, 2022.
  12. PoseScript: 3D Human Poses from Natural Language. In ECCV, 2022.
  13. Pose tutor: An explainable system for pose correction in the wild. In CVPR, 2022.
  14. Action Modifiers: Learning from Adverbs in Instructional Videos. In CVPR, 2020.
  15. AIFit: Automatic 3D human-interpretable feedback models for fitness training. In CVPR, 2021.
  16. Stylegan-human: A data-centric odyssey of human generation. In ECCV, 2022.
  17. Synthesis of compositional animations from textual descriptions. In ICCV, 2021.
  18. Generating diverse and natural 3d human motions from text. In CVPR, 2022.
  19. Tm2t: Stochastic and tokenized modeling for the reciprocal generation of 3d human motions and texts. In ECCV, 2022.
  20. Action2motion: Conditioned generation of 3D human motions. In ACMMM, 2020.
  21. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415, 2016.
  22. Image captioning: Transforming objects into words. In NeurIPS, 2019.
  23. Prompt-to-prompt image editing with cross attention control. In ICLR, 2023.
  24. Avatarclip: Zero-shot text-driven generation and animation of 3d avatars. ACM TOG, 2022.
  25. Talk-to-edit: Fine-grained facial editing via dialog. In ICCV, 2021.
  26. FixMyPose: Pose correctional captioning and retrieval. In AAAI, 2021.
  27. Flame: Free-form language-based motion synthesis & editing. In AAAI, 2023.
  28. Adam: A method for stochastic optimization. In ICLR, 2015.
  29. Auto-encoding variational bayes. In ICLR, 2014.
  30. Dancing to music. In NeurIPS, 2019.
  31. Ai choreographer: Music conditioned 3d dance generation with aist++. In ICCV, 2021.
  32. Generating animated videos of human activities from natural language descriptions. In NeurIPS workshops, 2018.
  33. Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, 2004.
  34. Microsoft coco: Common objects in context. In ECCV, 2014.
  35. Posecoach: A customizable analysis and visualization system for video-based running coaching. IEEE trans. VCG, 2022.
  36. Image retrieval on real-life images with pre-trained vision-and-language models. In ICCV, 2021.
  37. SMPL: A skinned multi-person linear model. ACM TOG, 2015.
  38. AMASS: Archive of motion capture as surface shapes. In ICCV, 2019.
  39. Attributes as operators: factorizing unseen attribute-object compositions. In ECCV, 2018.
  40. Protores: Proto-residual network for pose authoring via learned inverse kinematics. In ICLR, 2021.
  41. Training language models to follow instructions with human feedback. arXiv preprint arXiv:2203.02155, 2022.
  42. Bleu: a method for automatic evaluation of machine translation. In ACL, 2002.
  43. Relative attributes. In ICCV, 2011.
  44. Expressive body capture: 3D hands, face, and body from a single image. In CVPR, 2019.
  45. Glove: Global vectors for word representation. In EMNLP, 2014.
  46. Action-conditioned 3D human motion synthesis with transformer VAE. In ICCV, 2021.
  47. Temos: Generating diverse human motions from textual descriptions. In ECCV, 2022.
  48. The kit motion-language dataset. Big data, 2016.
  49. BABEL: Bodies, action and behavior with english labels. In CVPR, 2021.
  50. Learning transferable visual models from natural language supervision. In ICML, 2021.
  51. Robust speech recognition via large-scale weak supervision. arXiv preprint arXiv:2212.04356, 2022.
  52. Embodied hands: Modeling and capturing hands and bodies together. In SIGGRAPH Asia, 2017.
  53. Simple and effective vae training with calibrated decoders. In ICML, 2021.
  54. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
  55. Generating text with recurrent neural networks. In ICML, 2011.
  56. Motionclip: Exposing human motion generation to clip space. In ECCV, 2022.
  57. Attention is all you need. In NeurIPS, 2017.
  58. Show and tell: A neural image caption generator. In CVPR, 2015.
  59. Composing text and image for image retrieval-an empirical odyssey. In CVPR, 2019.
  60. Fashion iq: A new dataset towards retrieving images by natural language feedback. In CVPR, 2021.
  61. Paired recurrent autoencoders for bidirectional translation between robot actions and linguistic descriptions. IEEE RAL, 2018.
  62. Clip-actor: Text-driven recommendation and stylization for animating human meshes. In ECCV, 2022.
  63. Dlow: Diversifying latent flows for diverse human motion prediction. In ECCV, 2020.
  64. We are more than our joints: Predicting how 3d bodies move. In CVPR, 2021.
  65. 3d pose based feedback for physical exercises. In ACCV, 2022.
  66. Text guided person image synthesis. In CVPR, 2019.
  67. On the continuity of rotation representations in neural networks. In CVPR, 2019.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Ginger Delmas (4 papers)
  2. Philippe Weinzaepfel (38 papers)
  3. Francesc Moreno-Noguer (68 papers)
  4. Grégory Rogez (17 papers)
Citations (15)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now