MK-Pose: Multimodal Keypoint Learning

• The paper presents a novel multimodal keypoint learning approach that integrates RGB, point clouds, and text to predict category-level 6D object pose and size.
• It employs self-supervised keypoint detection with soft heatmap matching and graph-based relational reasoning to capture local and global geometric features.
• Evaluation on CAMERA25, REAL275, and HouseCat6D demonstrates improved IoU and average precision without shape priors, showcasing robust generalization.

Searching arXiv for the primary MK-Pose paper and closely related keypoint-learning work to ground the article in current literature. {"query":"arXiv (Yang et al., 9 Jul 2025) MK-Pose Category-Level Object Pose Estimation via Multimodal-Based Keypoint Learning", "max_results": 5} Searching for complementary papers on multimodal keypoint learning, graph-based keypoint reasoning, and self-supervised keypoint correspondence that are directly relevant to MK-Pose. {"query":"multimodal keypoint learning graph pose estimation arXiv category-level object pose", "max_results": 10} MK-Pose is a category-level object pose estimation framework that predicts an object’s 6D pose—its 3D rotation and 3D translation—and its 3D size for objects belonging to a known semantic category, without access to the exact CAD model or instance template of the test object. It is designed around multimodal-based keypoint learning, integrating RGB images, point clouds, and category-level textual descriptions, and combines a self-supervised keypoint detection module, attention-based query generation, soft heatmap matching, graph-based relational modeling, and graph-enhanced local-global feature fusion. The framework is evaluated on CAMERA25 and REAL275, and is further tested for cross-dataset capability on HouseCat6D; it is reported to outperform existing state-of-the-art methods in both IoU and average precision without shape priors (Yang et al., 9 Jul 2025).

1. Problem formulation and motivation

Category-level object pose estimation differs from instance-level pose estimation in that the object category is known, but the specific test instance is not. In this setting, the model must handle variation in shape, aspect ratio, topology, texture, symmetry, and occlusion across instances within a category such as mug, bottle, or laptop. MK-Pose is motivated by three difficulties stated explicitly for this regime: occlusion and clutter, instance variation and generalization, and symmetry together with domain and category robustness (Yang et al., 9 Jul 2025).

The framework is positioned against several limitations of earlier approaches. RGB-only methods can exploit texture and visual semantics, but they remain vulnerable to weak texture, lighting change, background clutter, heavy occlusion, and geometric ambiguity. Point-cloud-only methods capture 3D geometry more directly, yet can still degrade under incomplete visibility, sparse or noisy depth, and insufficient category semantics. Shape-prior-based methods can improve performance, but introduce dependence on canonical templates or deformation priors that may overfit the seen training distribution and become less suitable for zero-shot or cross-dataset settings. Manually annotated keypoint methods, in turn, inherit the cost and rigidity of predefined semantic landmarks and may not transfer well across diverse instances (Yang et al., 9 Jul 2025).

MK-Pose addresses these issues through a multimodal formulation in which RGB contributes appearance and contour information, point clouds contribute metric geometry and spatial structure, and text contributes category-level semantic and structural priors. A central claim of the method is that textual category descriptions remain stable under occlusion and can act as a semantic bridge between visual appearance and 3D geometry. This suggests that multimodality is not treated as simple sensor concatenation, but as a mechanism for category-consistent structural reasoning under incomplete evidence (Yang et al., 9 Jul 2025).

2. Multimodal representation and overall architecture

The framework begins from three modality-specific feature streams. RGB images are encoded with DINOv2-ViT-S/14, point clouds are encoded with PointNet++, and the category label is expanded into a detailed shape description using GPT-4o and then encoded with the CLIP text encoder. These streams are fused by cross-attention to form an object-level representation (Yang et al., 9 Jul 2025).

$$\mathbf{F}_{\text{RGB}} = \mathcal{F}_{\text{RGB}}(\mathbf{I}) \in \mathbb{R}^{N \times d_1}$$

$$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$

$$\mathbf{F}_{\text{text}} = \mathcal{F}_{\text{text}}(\mathrm{LLM}(c)) \in \mathbb{R}^{N \times d}$$

The multimodal fusion stage uses the concatenated RGB and point-cloud features as queries and the text features as keys and values:

$$\begin{aligned} \mathbf{Q} &= W_Q\, \mathrm{cat}(\mathbf{F}_{\text{RGB}}, \mathbf{F}_{\text{PC}}), \ \mathbf{K} &= W_K \mathbf{F}_{\text{text}}, \ \mathbf{V} &= W_V \mathbf{F}_{\text{text}}, \ \mathbf{F}_{\text{obj}} &= \mathrm{softmax}\left(\frac{\mathbf{Q}\mathbf{K}^\top}{\sqrt{d}}\right)\mathbf{V}. \end{aligned}$$

This formulation assigns the text stream the role of semantic memory. The RGB and point-cloud streams define what should be queried from that memory, while the attention output yields an object representation aligned with category-level structural semantics rather than raw appearance alone (Yang et al., 9 Jul 2025).

Component	Input	Role
RGB branch	RGB image	Appearance and contour cues
Point-cloud branch	Point cloud from depth	3D geometry and spatial structure
Text branch	GPT-4o category description	Category-level semantic prior
Cross-attention fusion	RGB + point cloud + text	Object representation aligned with category semantics

Architecturally, MK-Pose is organized into three stages: multimodal self-supervised keypoint detection, graph-enhanced local-global feature fusion, and pose-size estimation. This structure is key to the framework’s identity: keypoints are not an auxiliary output but the central representation through which multimodal fusion is converted into category-level pose estimation (Yang et al., 9 Jul 2025).

3. Self-supervised keypoint detection

The self-supervised keypoint detector augments the fused object representation with point-coordinate positional encoding and a global pooled feature:

$$\mathbf{F}_{\text{obj}}' = \mathrm{MLP}\left( \left[ \mathbf{F}_{\text{obj}} + \mathrm{MLP}(\mathbf{P}), \frac{1}{N}\sum_{i=1}^{N}(\mathbf{F}_{\text{obj}}+\mathrm{MLP}(\mathbf{P}))_{i,:} \right] \right).$$

This operation explicitly injects both local position and whole-object context. The paper notes an ablation denoted “w/o global,” which indicates that global information contributes to stable keypoint prediction. A plausible implication is that the detector benefits from knowing not only local geometry but also the object-scale arrangement within which each candidate keypoint must remain consistent (Yang et al., 9 Jul 2025).

Keypoint proposals are generated through learnable keypoint queries derived from category embeddings. Letting the queries attend to the object representation produces query-conditioned keypoint features:

$$\mathbf{F}_{\text{kpt}}^{\text{out}} = \mathrm{Attn}(\mathbf{F}_{\text{kpt}}^{\text{in}}, \mathbf{F}_{\text{obj}}').$$

This is the mechanism through which MK-Pose avoids manually defined semantic keypoints. The queries are learned latent anchors rather than annotated landmarks, and the training objective encourages them to become category-aware and pose-useful. The method is therefore self-supervised at the keypoint level: the network is not given ground-truth keypoint coordinates, but learns keypoint structure as an internal representation optimized for downstream pose prediction (Yang et al., 9 Jul 2025).

Localization is performed through soft heatmap matching. Each keypoint feature is compared with all object-point features via cosine similarity, followed by a temperature-scaled softmax:

$$S_{ij} = \frac{ \mathbf{F}_{\text{kpt},i}\cdot \mathbf{F}_{\text{obj},j} }{ \|\mathbf{F}_{\text{kpt},i}\|_2 \|\mathbf{F}_{\text{obj},j}\|_2 + \epsilon }, \qquad \mathbf{H} = \mathrm{Softmax}\left(\frac{S}{\tau}\right).$$

Keypoint coordinates and features are then obtained as soft weighted sums over the object points:

$$\mathbf{P}_{\text{kpt}} = \mathbf{H}\mathbf{P}, \qquad \mathbf{F}_{\text{kpt}} = \mathbf{H}\mathbf{F}_{\text{obj}}.$$

This soft assignment avoids hard selection of a single support point and instead treats each keypoint as a distribution over object points. In practice, that design is intended to improve robustness to point sparsity, occlusion, and intra-category deformation, because the model can preserve uncertainty or multimodal evidence rather than collapsing early to a brittle discrete match (Yang et al., 9 Jul 2025).

The keypoint set is then refined with a fully connected Graph Attention Network. Treating keypoints as graph nodes, the update rule is

$$h_i' = \sigma \left( \sum_{j\in \mathbf{N}_i \cup \{i\}} \alpha_{ij}\mathbf{W}h_j \right).$$

This graph stage makes the detector relational rather than pointwise. A keypoint’s meaning depends not only on its own local evidence but also on the configuration of the other learned keypoints, which helps encode category-consistent structure (Yang et al., 9 Jul 2025).

4. Graph-enhanced local-global feature fusion and pose-size estimation

After keypoint detection, MK-Pose refines the keypoint descriptors through a graph-enhanced local-global feature fusion module. For each keypoint $i$, a K-nearest-neighbor set is constructed in the original object point cloud:

$$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$0

The associated local coordinates and features are

$$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$1

Local geometry is encoded with relative positional features. Keypoints are represented relative to the object center, and neighbors are represented relative to the keypoint:

$$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$2

These are injected into the feature streams through MLPs:

$$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$3

Cross-attention then uses the keypoints as queries and the neighborhood features as keys and values:

$$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$4

This local fusion stage integrates fine geometric evidence around each keypoint. The model then adds absolute positional encoding,

$$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$5

and aggregates a global keypoint descriptor through mean pooling,

$$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$6

According to the method description, this global feature is broadcast back to all keypoints, fused through an MLP, and further refined by self-attention. The resulting representation combines local surface geometry and global object context. This design suggests a two-scale structural prior: local neighborhoods stabilize each keypoint under geometric incompleteness, while global aggregation preserves category-consistent arrangement when local evidence is ambiguous (Yang et al., 9 Jul 2025).

The final stage predicts NOCS-like keypoint coordinates and regresses pose and size, optimized with a symmetry-aware pose-size loss. The details block does not print the explicit regression equation for this stage, but it identifies the outputs as the rotation matrix $$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$7, translation vector $$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$8, and size vector $$\mathbf{F}_{\text{PC}} = \mathcal{F}_{\text{PC}}(\mathbf{P}) \in \mathbb{R}^{N \times d_2}$$9, and states that symmetry is handled at the loss level. That is a significant design choice for category-level estimation, since mugs, bottles, and related categories frequently contain finite or approximate symmetries that make naive pointwise supervision unstable (Yang et al., 9 Jul 2025).

5. Evaluation, generalization, and empirical profile

MK-Pose is evaluated on CAMERA25 and REAL275 and further tested for cross-dataset capability on HouseCat6D. The reported outcome is that it outperforms existing state-of-the-art methods in both IoU and average precision without shape priors (Yang et al., 9 Jul 2025). The absence of shape priors is a substantive part of the claim: the method is presented not as a category-template deformation model, but as a multimodal keypoint-learning framework whose category knowledge is carried by learned keypoint structure and textual semantics.

The method is also described as achieving strong zero-shot performance on unseen objects. This suggests that the textual branch is not merely auxiliary regularization. Because the category name is expanded into a detailed shape description and encoded through CLIP, textual information acts as a category-level prior that is not tied to any specific object instance. In turn, this makes the framework especially relevant to domains where instance-level CAD models are unavailable but coarse semantic categories are known, such as warehouse automation and manufacturing (Yang et al., 9 Jul 2025).

Several misconceptions are addressed implicitly by the architecture. First, MK-Pose is not a manually annotated keypoint system: its keypoint detector is explicitly described as self-supervised. Second, its multimodality is not limited to RGB-D fusion; it includes category-level textual descriptions as a third information stream. Third, despite using category descriptions, the framework is explicitly reported without shape priors, so the text branch should not be conflated with classical category-template models. A plausible implication is that MK-Pose’s generalization strategy is semantic and relational rather than template-based (Yang et al., 9 Jul 2025).

6. Position within the broader keypoint-learning literature

MK-Pose sits at the intersection of several strands of keypoint research. Relative to RGB-D 6D pose systems, it is closest in spirit to methods that combine multimodal feature fusion with graph-structured keypoint reasoning. For example, KGpose also combines RGB and point cloud features, predicts 3D keypoints, converts them into a keypoint-graph, and regresses 6D pose from graph embeddings; however, it is an instance-level, single-view framework tied to object-model keypoints, whereas MK-Pose is category-level and explicitly incorporates textual category descriptions (Jeong, 2024).

From the perspective of keypoint design itself, KeyGNet shows that keypoint selection can be learned rather than fixed heuristically, using a graph network supervised by a combined Wasserstein and dispersion loss; it improves multiple 6DoF pose estimators and largely removes the SISO–MIMO gap on Occlusion LINEMOD (Wu et al., 2023). MK-Pose does not learn keypoints through that objective, but it shares the broader position that keypoints should be optimized as internal latent anchors rather than inherited from manual annotation or fixed geometric heuristics.

In relation to self-supervision, CorrTrack demonstrates that keypoint correspondences can be learned self-supervised from static image pose annotations through synthetic geometric transforms, without manual video correspondences or identity labels (Rafi et al., 2020). MK-Pose extends the self-supervised theme into category-level object pose by coupling latent keypoint discovery with multimodal fusion and relational modeling. A plausible interpretation is that both methods exemplify a larger shift away from dense manual correspondence supervision toward structured synthetic or latent objectives.

Finally, multimodal large-language-model work such as KptLLM and KptLLM++ frames keypoint tasks as semantic and promptable, using an identify-then-detect strategy and combining images, prompts, and language to support generic keypoint comprehension (Yang et al., 2024, Yang et al., 15 Jul 2025). MK-Pose is not an instruction-following landmark model in that sense: its text input is category description rather than user-defined prompt. Even so, both lines of work treat language as a source of structural prior rather than mere metadata. That commonality places MK-Pose within a broader redefinition of keypoint learning as a multimodal reasoning problem rather than a purely geometric regression problem.