ChatPose: Chatting about 3D Human Pose (2311.18836v2)

Abstract: We introduce ChatPose, a framework employing LLMs to understand and reason about 3D human poses from images or textual descriptions. Our work is motivated by the human ability to intuitively understand postures from a single image or a brief description, a process that intertwines image interpretation, world knowledge, and an understanding of body language. Traditional human pose estimation and generation methods often operate in isolation, lacking semantic understanding and reasoning abilities. ChatPose addresses these limitations by embedding SMPL poses as distinct signal tokens within a multimodal LLM, enabling the direct generation of 3D body poses from both textual and visual inputs. Leveraging the powerful capabilities of multimodal LLMs, ChatPose unifies classical 3D human pose and generation tasks while offering user interactions. Additionally, ChatPose empowers LLMs to apply their extensive world knowledge in reasoning about human poses, leading to two advanced tasks: speculative pose generation and reasoning about pose estimation. These tasks involve reasoning about humans to generate 3D poses from subtle text queries, possibly accompanied by images. We establish benchmarks for these tasks, moving beyond traditional 3D pose generation and estimation methods. Our results show that ChatPose outperforms existing multimodal LLMs and task-specific methods on these newly proposed tasks. Furthermore, ChatPose's ability to understand and generate 3D human poses based on complex reasoning opens new directions in human pose analysis.

• The paper introduces PoseGPT, a framework that integrates SMPL projections into LLMs for effective 3D human pose generation.
• It establishes two novel tasks—Speculative Pose Generation and Reasoning-based Pose Estimation—that benchmark pose inference from indirect cues.
• The study sets the stage for advances in interactive AI, augmented reality, and HCI by unifying image and text modalities for pose estimation.

PoseGPT: Chatting about 3D Human Pose

The paper introduces PoseGPT, a multi-modal framework designed to extend the capabilities of LLMs into the domain of 3D human pose estimation and generation. Traditionally, there has been a disconnect between the nuanced understanding of human posture and the models employed to estimate pose from images or textual data. PoseGPT addresses these limitations by embedding SMPL (Skinned Multi-Person Linear Model) parameters directly into LLMs, facilitating the generation of 3D human poses from both text and visual inputs.

Methodology and Contributions

PoseGPT uses a specialized SMPL projection layer trained to convert language embeddings into 3D pose parameters. The approach leverages the inherent world knowledge of LLMs, allowing them to integrate contextual scene understanding with 3D pose estimation. The model processes inputs using both images and text, generating not only the SMPL pose parameters but also a 3D body mesh when details are requested.

The paper introduces two pivotal tasks: Speculative Pose Generation (SPG) and Reasoning-based Pose Estimation (RPE). SPG involves generating 3D poses from indirect textual descriptions, whereas RPE requires understanding and estimating poses within a scene, using descriptions of individuals within it.

Evaluation and Results

PoseGPT outperforms existing multimodal LLMs and traditional task-specific methods for the newly proposed tasks. On classical tasks such as text-to-pose and image-to-pose estimation, PoseGPT shows competitive performance, although its accuracy does not yet reach the heights of specialized methods. The newly introduced SPG and RPE benchmarks provide new evaluation metrics in this domain, emphasizing the ability to infer poses from nuanced, indirect text and complex scene contexts.

Implications and Future Directions

PoseGPT represents a significant step towards unified models capable of reasoning about human poses across various input modalities. The potential for LLMs to understand and generate 3D human poses extends into fields such as interactive AI, augmented reality, and human-computer interaction. Future research can build on this foundation by enhancing the robustness of pose predictions and expanding into motion sequences and shape estimations.

The paper paves the way for further exploration of LLMs' capabilities in understanding and interacting with the 3D human form, blending deep learning techniques with practical human pose applications. The availability of the framework's code and data will undoubtedly encourage further research and experimentation in this promising area of artificial intelligence.