GKDT: General Keypoint Detection Transformer

Abstract: With the emergence of various pre-trained vision and LLMs, computer vision is shifting from narrow-domain to open-domain recognition. The construction of a more powerful yet general keypoint detection (GKD) model to support diverse tasks has become increasingly important in the field. To this end, we firstly present a large-scale unified keypoint dataset called MegaKPT. The dataset is composed of over 1.3 million diverse object instances from twenty-nine existing datasets, and enjoys high-quality unified annotations with keypoint text descriptions. Based on MegaKPT, we develop GKDT, a simple, flexible and powerful DINOv3 based Transformer model for General Keypoint Detection. Our GKDT supports visual prompts, text prompts, or both. To enhance model training, we also propose a suite of useful strategies such as mix-modal prompted training and dynamic importance sampling. By testing over 22 test sets with seen or unseen objects, our single GKDT model shows strong performance and generality in detecting keypoints on broad categories, with most categories over 90\% [email protected] accuracy, offering high practical applicability to real-world problems. The dataset, models, and codes will be released at https://github.com/AlanLuSun/General-Keypoint-Detection.

• The paper introduces GKDT, a prompt-driven transformer that unifies keypoint detection across diverse domains by leveraging the large-scale MegaKPT dataset.
• It employs a DINOv3 visual backbone and a specialized KG Transformer for prompt-derived prototype refinement, achieving over 90% [email protected] accuracy on varied benchmarks.
• The study demonstrates effective mix-modal prompting and dynamic importance sampling to handle long-tail data distributions and enable robust zero- and few-shot transfer.

GKDT: A General Transformer for Keypoint Detection Across Domains

Introduction

This work introduces the General Keypoint Detection Transformer (GKDT), a method for keypoint localization that unifies and extends the scope of detection tasks across diverse object categories and application domains. The key premise is that prior methods—whether classic or deep learning-based—were limited by close-set detection, lack of prompt-based generalizability, and insufficient data diversity. The authors tackle these through two main contributions: (1) MegaKPT, a large-scale, high-quality, unified keypoint dataset covering $1{,}348{,}228$ instances and $740$ unique keypoint types across $1{,}587$ categories, and (2) GKDT, a DINOv3-anchored, prompt-driven transformer enabling robust generalization under visual, text, or multimodal prompting. Empirical evidence across 22 test sets, including previously unseen categories, substantiates the effectiveness of this approach.

Figure 1: A glance of the MegaKPT dataset illustrating its diversity in both object categories and annotated modalities (keypoints and texts).

MegaKPT: Dataset Construction and Statistical Properties

Past datasets, such as MP-100 and UniKPT, were constrained by either limited volume or inconsistent annotation quality. MegaKPT is constructed by consolidating 29 diverse datasets, standardizing all keypoint labels and textual descriptions, and assigning explicit super-category divisions and clean indices. It encompasses human and animal poses, faces, limbs, vehicles, furniture, and medical scans. The unified annotation procedure is rigorous, including egocentric text labels, curation of ambiguous locations, leveraging LLMs for name normalization, and manual validation for quality assurance.

Figure 2: MegaKPT, compared to predecessors, offers orders of magnitude more categories, keypoint types, images, and annotated instances; instance distribution reflects a pronounced long-tail, motivating dynamic sampling for effective model optimization.

The dataset notably supports zero/few-shot transfer. Medical landmark categories, like cephalometrics and hand X-ray images, are provided with expert-validated text prompts, enabling zero-shot landmark localization.

GKDT Architecture and Prompt Modalities

GKDT’s architecture is predicated on the use of a DINOv3 visual backbone, fully tuned to maximize extraction of dense semantic feature representations, and a KG Transformer module for converting prompt-derived prototypes into actionable convolution kernels.

Figure 3: The GKDT pipeline flexibly accommodates text, visual, or mixed prompts, encoding both images and texts, creating keypoint prototypes, and synthesizing detection kernels for non-parametric heatmap regression.

The transformer-based kernel generator enables (1) prototype feature refinement (across modalities), (2) flexible adaptation to variable numbers of keypoints and object anatomy, and (3) efficient, non-parametric detection via convolution with query image features. Cross-attention refines prompt representations using image context, while prompt modality indicators allow seamless fusion of outputs during inference.

Training Strategies: Mix-Modal Prompting and Dynamic Importance Sampling

Real-world generalization requires consistency between training and deployment prompt modalities. To this end, mix-modal prompting randomizes episode prompting (visual, text, or both) during training, aligning train/test behavior. The data imbalance—long-tail distribution of categories in MegaKPT—motivates dynamic importance sampling. Unlike naive uniform or head-class-skewed schemes, the proposed strategy adaptively balances removal and replacement prioritization, dynamically reducing bias toward head classes as sampling progresses.

Figure 4: Dynamic importance sampling counteracts head-tail imbalance by progressively balancing super-category sampling proportions as training advances.

Experimental Validation and Ablations

Single-Object and Multi-Object Detection

Testing on 22 varied datasets, GKDT consistently outperforms baselines (DINOv3, OpenKD, X-Pose, CapeFormer), achieving $\text{PCK}@0.1$ above $90\%$ on the majority of benchmarks, including those with substantial domain and anatomy shift (e.g., animals, vehicles, medical images). Generalization to unseen test domains (e.g., unseen bird species, unseen animal faces) is robust, with only minimal performance degradation.

Figure 5: GKDT’s visual and text prompt flexibility is reflected in accurate multimodal keypoint localization, with high overlap between predicted heatmaps and ground truth, across domains.

In multi-object settings, GKDT paired with open-vocabulary detectors (e.g., Grounding DINO) surpasses prior general approaches and achieves results on par with state-of-the-art specialized models.

Figure 6: GKDT deployed for zero-shot keypoint detection in multi-object scenarios, evidencing capacity for transfer and localization on in-the-wild images across disparate domains.

Mix-Modal Prompting and Data Efficiency

Ablations reveal that mix-modal prompted training is required for optimal cross-modal performance at test time. Even with 10% of Cephalometric data, GKDT surpasses $90\%$ accuracy, underscoring the efficacy of DINOv3-based transfer and representation power.

Figure 7: Model studies demonstrate DINOv3’s pre-training advantage, the necessity of mix-modal training for consistent prompt generality, and data efficiency in medical domains.

Component Analysis and Further Visualizations

KG transformer, when combined with a tuned visual backbone, boosts both visual and text prompted detection. Inclusion of self- and cross-attention in the KG transformer further improves prototype refinement, especially for weak modalities.

Further, the model is shown to effectively transfer between human and non-human animal domains (e.g., from human to macaque), and can robustly handle head-tail shifts with dynamic sampling.

Figure 8: GKDT generalizes to keypoint detection in highly diverse categories—human and animal poses, faces, insect morphologies, vehicles, furniture—using a single unified model.

Figure 9: Zero-shot video keypoint detection highlights GKDT’s robustness and practical applicability to unconstrained temporal sequences with only text prompts.

Implications and Future Developments

The GKDT framework, enabled by MegaKPT and DINOv3-based learning, marks a substantial progression in prompt-driven, open-set keypoint localization. Its flexible architecture and training regimen allow high generality, robust zero- and few-shot transfer, and practical applicability to medical imaging, video, and uncommon object categories. The inclusion of textual keypoint definitions allows seamless integration with higher-level reasoning (e.g., LLM-based task logic or instruction following).

In future systems, such models could facilitate:

• Fully automated pipeline integration for robotics, activity analysis, or behavioral science (where annotation cost and data diversity are constraints)
• Medical applications where expert-verified text prompts enable robust zero-shot anatomical landmarking
• Open-vocabulary fine-grained categorization pipelines, composable with language-only supervision.

The theoretical implications extend to demonstrating the power of self-supervised vision representation and cross-modal transformer refinement for compositional generalization.

Conclusion

MegaKPT and GKDT together demonstrate that unified high-quality data and prompt-based transformer architectures can achieve state-of-the-art general keypoint detection across highly diverse categories and prompt modalities. This paradigm provides a practical and extensible foundation for domain-agnostic keypoint detection, supports robust zero/few-shot generalization, and opens avenues for compositional, multimodal reasoning in downstream vision-language tasks.

(2607.00752)