PoseLLM: Language-Guided Pose Estimation

• PoseLLM is a language-guided human pose estimation framework that formulates keypoint localization as a multimodal reasoning task using text prompts.
• It replaces a linear projector with a nonlinear two-layer MLP, enhancing fine-grained localization and achieving 77.8 AP on the COCO validation set.
• Trained solely on COCO data, PoseLLM maintains zero-shot generalization on Human-Art and MPII, emphasizing improved vision-language alignment.

to=functions.shell 开号网址json content='{"command":"python - <<\'PY\'\nimport urllib.request\nurl=\'http://export.arxiv.org/api/query?search_query=all:PoseLLM&start=0&max_results=5\'\ntry:\n print(urllib.request.urlopen(url, timeout=10).read(500).decode())\nexcept Exception as e:\n print(\'ERR\', e)\nPY"}' ＿久久爱='false' code='json'
to=functions.shell qq彩票 content='{"command":"python - <<\'PY\'\nimport urllib.request\nurl=\'http://export.arxiv.org/api/query?search_query=all:PoseLLM&start=0&max_results=5\'\ntry:\n data=urllib.request.urlopen(url, timeout=10).read(500)\n print(data.decode(\'utf-8\',errors=\'ignore\'))\nexcept Exception as e:\n print(\'ERR\', e)\nPY"}' 娱乐彩票注册='false' code='json'
to=functions.shell 植物百科通 天天中彩票怎样json content='{"command":"python - <<\'PY\'\nimport urllib.request\nurl=\'http://export.arxiv.org/api/query?search_query=all:PoseLLM&start=0&max_results=5\'\ntry:\n data=urllib.request.urlopen(url, timeout=10).read(1000)\n print(data.decode(\'utf-8\',errors=\'ignore\'))\nexcept Exception as e:\n print(\'ERR\', e)\nPY"}'
to=functions.shell 天天中彩票投注 山大发json content='{"command":"python - <<\'PY\'\nimport urllib.request\nurl=\'http://export.arxiv.org/api/query?search_query=all:PoseLLM&start=0&max_results=5\'\ntry:\n data=urllib.request.urlopen(url, timeout=10).read(1000)\n print(data.decode(\'utf-8\',errors=\'ignore\'))\nexcept Exception as e:\n print(\'ERR\', e)\nPY"}'
to=functions.shell 大发快三的json content='{"command":"echo hello"}'
to=functions.shell 平台总代理 天天中彩票nbajson content='{"command":["bash","-lc","echo hello"]}'
to=functions.shell 天天爱彩票网站 +天天中彩票 content='echo hello'
to=functions.shell 无码不卡高清免费 天天中彩票公司json code='{"command":"echo hello"}'
PoseLLM is a language-guided human pose estimation framework that formulates human keypoint localization as a vision-language task: the model receives an image together with a text instruction describing a queried anatomical keypoint, and it outputs the 2D coordinate of that keypoint. It is presented as the first Large Language Model-based pose estimation framework that replaces the linear vision-language projector used in prior LocLLM-style systems with a nonlinear two-layer MLP connector. Trained exclusively on COCO Keypoint training data, PoseLLM reports 77.8 AP on the COCO validation set, outperforming LocLLM by +0.4 AP while maintaining zero-shot generalization on Human-Art and MPII [2507.09139].

1. Task formulation and conceptual shift

PoseLLM studies human keypoint localization as a multimodal reasoning problem rather than a purely geometric pose-estimation problem. The input consists of an image and a text instruction describing a specific keypoint query, and the output is the 2D coordinate of the queried anatomical keypoint. The prompt format matches LocLLM: each keypoint is described with anatomical and spatial language, followed by a question such as “Where is the left wrist?”.

This formulation differs from traditional heatmap or regression pose models in two stated ways. First, conventional pose estimators encode keypoint priors directly in the architecture or training targets; they are strong on in-distribution keypoints and datasets, but tend to generalize poorly to unseen poses, new datasets, or new keypoint formulations. Second, language-guided pose estimation uses natural-language descriptions of body parts and treats localization as a vision-language task, which enables zero-shot transfer because the model can leverage semantic prompts rather than dataset-specific keypoint heads.

Within this framing, PoseLLM does not replace the general LocLLM paradigm. Instead, it preserves the overall VLM/LLM structure and changes the alignment mechanism between visual tokens and the LLM input space. The central claim is therefore architectural rather than task-definitional: improved vision-language alignment yields better fine-grained localization without discarding the prompt-driven generalization properties of language-guided pose estimation.

2. System architecture and token interface

PoseLLM has three main components: a vision encoder, a vision-language connector, and an LLM decoder. The vision encoder is DINOv2 ViT-L/14. Given an input image, it produces patch-level visual tokens
[
\mathbf{I} \in \mathbb{R}^{B \times N \times 1024},
]
where (B) is the batch size, (N) is the number of visual tokens, and 1024 is the feature dimension of each token.

The text instruction contains a keypoint description and a question asking for the location. Tokenization is performed with the Vicuna-7B tokenizer, producing text embeddings
[
\mathbf{T} \in \mathbb{R}^{B \times L \times D},
]
where (L) is the number of text tokens and (D) is the LLM embedding dimension.

PoseLLM uses Vicuna-7B as the language model backbone. The transformed visual sequence and the text embeddings are concatenated as ([\mathbf{V}; \mathbf{T}]) and fed to the LLM, which autoregressively generates the coordinate answer. Training uses teacher forcing with ground-truth coordinate tokens as targets, and the optimization objective is cross-entropy loss computed only over the answer tokens.

This organization preserves the LLM-style generative interface of LocLLM while making the connector the principal site of methodological intervention. A plausible implication is that PoseLLM treats localization not as direct coordinate regression from a vision head, but as coordinate-token generation conditioned on aligned visual patches and text prompts.

3. Nonlinear MLP alignment and hierarchical cross-modal transformation

The defining modification in PoseLLM is the replacement of LocLLM’s linear projector with a nonlinear two-layer MLP with GELU activation. The connector is defined as
[
\mathbf{Z} = \text{GELU}(\mathbf{I} \cdot \mathbf{W}_1 + \mathbf{b}_1) \tag{3}
]
[
\mathbf{V} = \mathbf{Z} \cdot \mathbf{W}_2 + \mathbf{b}_2 \tag{4}
]
with parameters
[
\mathbf{W}_1 \in \mathbb{R}^{1024 \times 4096}, \quad
\mathbf{b}_1 \in \mathbb{R}^{4096},
]
[
\mathbf{W}_2 \in \mathbb{R}^{4096 \times 4096}, \quad
\mathbf{b}_2 \in \mathbb{R}^{4096}.
]
The output is
[
\mathbf{V} \in \mathbb{R}^{B \times N \times 4096}.
]

In shape terms, the connector maps input visual features of size (B \times N \times 1024) to a hidden MLP layer of size (B \times N \times 4096), and then to output visual tokens of size (B \times N \times 4096). The module therefore both expands representational capacity and aligns visual features to the LLM token space.

The paper describes this as a hierarchical cross-modal feature transformation. The first linear layer expands the visual feature representation, GELU introduces nonlinearity and selective gating of features, and the second linear layer refines the transformed representation into the final LLM-compatible space. The stated motivation is that keypoint localization requires fine-grained spatial reasoning, interactions between local visual patches and text cues, and nonlinear cross-modal transformations. GELU is said to help model subtle distinctions relevant to localization, including left/right symmetry, body-part adjacency, relative geometry, and interactions between coarse body context and local keypoint evidence.

The contrast with a purely linear projector is explicit. A linear projector can only perform an affine transformation, whereas PoseLLM argues that pose estimation benefits from a nonlinear mapping that can better modulate patch-level information and better match the semantics of the keypoint prompt. This suggests that the connector is not merely a dimensionality adapter but an alignment module with task-specific importance for spatial-textual grounding.

4. Training regime, parameter updates, and evaluation protocol

PoseLLM is trained only on COCO Keypoint training data, comprising around 57K images, around 150K human instances, and 17 keypoints. No extra pose datasets are used for training. After training on COCO, the model is evaluated without finetuning on Human-Art and MPII to test cross-dataset and zero-shot generalization.

The reported optimization setup uses 12 epochs, 2 NVIDIA A6000 GPUs, AdamW, a learning rate of (5 \times 10^{-4}), batch size 32 via gradient accumulation, weight decay 0.05, and image size (224 \times 224). The fine-tuning strategy is parameter-efficient: the vision encoder DINOv2 ViT-L/14 is fine-tuned with LoRA, the Vicuna-7B LLM is fine-tuned with LoRA, and the MLP connector is fully updated [2507.09139].

The evaluation protocols are dataset-specific. On COCO, the reported metrics are AP, AP50, AP75, APM, APL, and AR; evaluation is performed with ground-truth bounding boxes and without flip test. Human-Art uses the same COCO-style metrics. MPII uses [email protected] and [email protected], together with per-joint scores for shoulder, elbow, hip, and knee.

A common misunderstanding is that the reported gains could be attributed to broader training data. The reported setup rules this out directly: training is restricted to COCO, and cross-dataset performance is measured without finetuning. The empirical question is therefore whether the connector change alone improves precision while preserving transfer.

5. Quantitative performance and cross-dataset behavior

The main reported results are summarized below.

Dataset / setting	PoseLLM	Comparison to LocLLM
COCO validation	77.8 AP	+0.4 AP over 77.4 AP
Human-Art zero-shot	67.9 AP	+0.1 AP over 67.8 AP
MPII zero-shot	Mean 98.1; Mean0.1 80.7	Mean matched at 98.1; +0.1 on Mean0.1

On COCO Keypoint validation, PoseLLM reports AP50 94.4, AP75 85.4, APM 74.7, APL 82.9, and AR 80.8. It is reported as better than the CLIP baseline at 73.1 AP, better than LocLLM at 77.4 AP, and above ViTPose at 77.4 AP in the listed COCO comparison. On Human-Art, the remaining reported zero-shot metrics are AP50 87.7, AP75 74.2, APM 41.7, APL 71.1, and AR 72.0. On MPII, PoseLLM reports per-joint scores of 98.9 for shoulder, 98.1 for elbow, 99.3 for hip, and 98.4 for knee [2507.09139].

The evidence for the connector hypothesis is mainly comparative rather than based on a large ablation suite. The support highlighted for the claim consists of three elements: the COCO validation improvement of +0.4 AP over LocLLM, cross-dataset stability on Human-Art and MPII, and qualitative examples showing accurate keypoint prediction across diverse body parts. The paper’s stated conclusion is that the nonlinear MLP improves localization accuracy while keeping the generalization behavior of LocLLM intact.

The quantitative pattern is narrow but consistent. Improvements are largest on COCO, smaller on Human-Art, and effectively neutral-to-slightly-positive on MPII. This suggests that the nonlinear connector primarily improves fine localization accuracy while not degrading the semantic transfer properties associated with prompt-driven keypoint localization.

6. Position within pose-language research and stated significance

PoseLLM belongs to a broader line of work that links pose representations with language. An important precursor is PoseScript, which introduced the first dataset for mapping single 3D human poses to free-form natural language descriptions and paired 3D poses from AMASS with both human-written and automatically generated captions. PoseScript-H contains human-written captions for about 6,283 poses, while PoseScript-A provides automatic synthetic captions for 100,000 poses using a posecode-based captioning pipeline. It also established text-to-pose retrieval, text-conditioned pose generation, and pose description generation as multimodal tasks over pose-text supervision [2210.11795].

PoseScript is not itself an LLM-based pose estimator, but it is highly relevant as a data and representation precursor. Its posecodes formalize low-level semantic units extracted from normalized 3D poses, including angles, distances, relative positions, pitch and roll, and ground contact. The synthetic-caption pipeline shows that pose-language supervision can be scaled, and the reported pretraining results indicate that automatic captions can improve downstream pose-language modeling after finetuning on human captions. This suggests a broader research trajectory in which PoseLLM occupies the 2D keypoint-localization end of a pose-language continuum rather than emerging in isolation.

The significance attributed to PoseLLM is correspondingly specific. The reported takeaway is that replacing a linear vision-language projector with a two-layer GELU MLP is sufficient to improve language-guided human pose estimation, preserve zero-shot transfer to Human-Art and MPII, and strengthen the baseline for LLM-based keypoint localization [2507.09139]. The summary further frames the result as evidence that nonlinear alignment matters for fine-grained spatial reasoning in multimodal pose estimation, and that better multimodal alignment, rather than simply larger language models, is a key factor in precise language-guided pose estimation.

At the same time, the paper’s own evidentiary profile is limited in a recognizable way. The analysis emphasizes comparative gains and qualitative examples rather than an extensive ablation suite. For that reason, the strongest supported claim is not that PoseLLM resolves language-guided pose estimation in general, but that a lightweight nonlinear connector can improve accuracy within the LocLLM-style framework without sacrificing the generalization properties that motivate language-guided pose estimation in the first place.