KptLLM++: Unified Multimodal Keypoint Model

• KptLLM++ is a unified multimodal LLM designed for fine-grained, pixel-level keypoint comprehension using an identify-then-detect chain-of-thought.
• It integrates a DINOv2 visual encoder and Vicuna-7B LLM to process images and textual prompts, enabling detailed semantic and spatial outputs.
• The model achieves state-of-the-art performance across benchmarks through extensive, diverse training data and a unified cross-entropy language modeling loss.

KptLLM++ is a unified multimodal LLM designed for generic keypoint comprehension in images, addressing the limitations of prior Multimodal LLMs (MLLMs) in fine-grained, pixel-level semantic understanding. This model integrates visual and textual modalities through user-defined instructions and supports both visual and textual prompts to identify and localize keypoints in a wide variety of object classes and settings. KptLLM++ leverages an identify-then-detect (ItD) chain-of-thought paradigm in conjunction with large-scale, diverse training data to achieve state-of-the-art accuracy and generalization, providing an advanced human-AI collaboration interface (Yang et al., 15 Jul 2025).

1. Architecture and Model Design

KptLLM++ processes a “query” image $I_q$, optionally a “support” image $I_s$ with a keypoint prompt $x$ (a 2D pixel coordinate), and free-form textual instructions $t$. The model outputs both a natural language description and the precise 2D location of the target keypoint. The architecture consists of five principal modules:

• Visual Encoder $\mathcal{V}(\cdot)$: Utilizes a DINOv2 ViT-Large backbone, mapping $I \in \mathbb{R}^{H \times W \times 3}$ to feature maps $F \in \mathbb{R}^{h \times w \times d}$.
• Prompt Encoder: Applies an MLP over sine-cosine positional encoding of $x$ to obtain a prompt embedding $F_p \in \mathbb{R}^d$.
• Prompt Feature Extractor: Employs two-layer cross-attention to attend $F_p$ over the support feature map $I_s$0, yielding a keypoint-specific visual token $I_s$1.
• Visual-to-Language Projection: Projects the query feature map $I_s$2 through a linear head to aligned tokens $I_s$3.
• Pre-trained LLM: Vicuna-7B is used, with transformer layers fusing $I_s$4, generating both semantic text and numeric keypoint coordinates.

The forward pass is defined as: $I_s$5 where $I_s$6 denotes the tokenized instructions. The overall model function is

$I_s$7

with $I_s$8 the semantic description and $I_s$9 the coordinate string.

2. Identify-then-Detect Chain-of-Thought

KptLLM++ departs from traditional coordinate regression, employing a two-stage “Identify-then-Detect” (ItD) chain-of-thought:

• Stage 1 (Identify): The LLM generates a natural language description $x$0 (e.g. “the left front wheel center”) for the support keypoint.
• Stage 2 (Detect): Conditioned on $x$1, the model emits normalized coordinates (e.g. “0.532,0.187”) localizing the keypoint in $x$2.

The inference pseudocode is: $x$9 A single cross-entropy language modeling loss is used: $x$3 where $x$4 includes the concatenated text and coordinate tokens. This approach enables structured multimodal reasoning and increases interpretability and robustness by explicitly decoupling semantic identification and localization.

3. Dataset Construction and Training Protocol

KptLLM++ is trained on over 500,000 annotated images aggregated from eight datasets: COCO (150K human instances, 17 keypoints), Human-Art, AP-10K (10K animal images, 23 families), APT-36K, MacaquePose, AnimalWeb, Animal Kingdom, and CarFusion. This collection spans a wide array of object classes (human, animal, vehicle, furniture, clothing), depiction styles (photography, painting, cartoon), and occlusion scenarios.

Key training details:

• Images are randomly cropped to ground-truth bounding boxes and resized to $x$5 pixels, with random horizontal flipping. No specialized keypoint data augmentation is used beyond these steps.
• The LLM is fine-tuned using LoRA (rank 128, $x$6); visual encoder and prompt feature modules are fully fine-tuned.
• AdamW optimizer: learning rate $x$7, zero weight decay, batch size $x$8 GPUs.
• Unified cross-entropy language modeling loss as above.

This extensive dataset scaling is critical to the model’s generalization and accuracy across domains (Yang et al., 15 Jul 2025).

4. Quantitative Performance and Benchmarking

KptLLM++ achieves state-of-the-art performance across several benchmarks for keypoint detection and semantic comprehension.

Semantic Understanding (MP-100, Split-1):

Method	Accuracy (%)
LLaVA (zero-shot)	3
LLaVA* (LoRA-tuned)	72
KptLLM	83

Visual Prompt-based Keypoint Detection (MP-100, 1-shot; PCK@0.2):

Method	Mean PCK (%)
ProtoNet	44.78
MAML	61.50
Finetune	63.58
POMNet	79.70
CapeFormer	85.31
KptLLM	87.10

Generic Keypoint Detection (COCO val):

Method	AP	AP₅₀	AP₇₅	AP_M	AP_L	AR
ViTPose	77.4	93.6	84.8	74.7	81.9	80.2
X-Pose	76.8	91.9	83.8	71.6	84.8	–
LocLLM	77.4	94.4	85.2	74.5	81.8	80.6
KptLL