SCAPE: Category-Agnostic Pose Estimator

• SCAPE is a category-agnostic pose estimator that localizes semantic keypoints using few-shot support exemplars and transformer attention.
• It streamlines the estimation process by eliminating explicit similarity maps, using modules like Global Keypoint Feature Perceptor and Keypoint Attention Refiner.
• Evaluated on MP-100, SCAPE shows improved accuracy and efficiency in 1-shot and 5-shot settings compared to prior CAPE approaches.

SCAPE, short for Simple and strong Category-Agnostic Pose Estimator, is a model for category-agnostic pose estimation (CAPE) that localizes semantic keypoints on an object of arbitrary category from a small number of annotated exemplars in an in-context few-shot setting (Liang et al., 2024). Its central claim is that CAPE is fundamentally a feature-matching problem already expressible inside transformer attention, so explicit similarity-map heads, two-stage decoding, and heatmap supervision are unnecessary. SCAPE therefore uses a streamlined architecture built from a shared feature extractor, pure self-attention over support and query tokens, a Global Keypoint Feature Perceptor (GKP), a Keypoint Attention Refiner (KAR), and a lightweight MLP coordinate regressor, yielding improvements in both accuracy and efficiency on MP-100 under 1-shot and 5-shot evaluation (Liang et al., 2024).

1. Problem formulation and task setting

Category-Agnostic Pose Estimation addresses pose localization for arbitrary object categories without category-specific training (Liang et al., 2024). In the in-context few-shot setting, the model receives a small number of annotated support exemplars from the same object instance or category at test time and must localize the same semantic keypoints in a query image. In 1-shot CAPE, one annotated support image is given; in 5-shot CAPE, five supports are given. The support input consists of each support image together with its annotated keypoints represented as 2D coordinates, from which support keypoint tokens are formed using the support feature map and keypoint heatmaps. The query input is the target image in which the corresponding keypoints must be localized, and the output is the set of 2D coordinates for all target keypoints in the query image (Liang et al., 2024).

The primary evaluation metric is Probability of Correct Keypoint. On MP-100, following POMNet and CapeFormer, evaluation uses a normalized distance threshold of $0.2$ on resized crops. If $K$ is the number of keypoints, $p_i,\hat{p}_i\in\mathbb{R}^2$ are predicted and ground-truth coordinates, and $S$ is the normalization scale, then

$$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$

Because preprocessing crops objects by bounding box and resizes images to $256\times256$, $S$ can be taken as $\max(H,W)=256$. The paper also reports AUC and NME as complementary localization metrics, but training and primary comparisons focus on PCK (Liang et al., 2024).

SCAPE’s interpretation of this task is that CAPE ultimately reduces to matching sparse support keypoint descriptors to locations in the query image. Transformer attention already computes similarities through the $\mathrm{softmax}(QK^\top)$ operator, and the method argues that these implicit attention maps are empirically closer to ground truth than explicitly supervised similarity maps. This motivates an architecture in which attention itself performs the matching and the model directly regresses coordinates, rather than generating and supervising intermediate heatmaps (Liang et al., 2024).

2. Tokenization, architecture, and inference path

SCAPE uses a shared backbone for support and query images. The reported backbones are ResNet-50, ViT, and Swin, with average pooling added for transformer backbones to harmonize output resolution and computational load (Liang et al., 2024). From the support feature map $F_{\text{sup}}\in\mathbb{R}^{N\times C}$, support keypoint tokens $K$0 are built by weighted pooling with Gaussian keypoint heatmaps:

$K$1

A learnable support keypoint identifier embedding $K$2 is then added so that different semantic keypoints remain distinguishable:

$K$3

For the query image, the token set is the query feature map with positional embeddings:

$K$4

The end-to-end pipeline contains four stages. First, GKP enriches each support keypoint token with global support-image context. Second, the globally enriched support tokens are concatenated with the query tokens and passed through a stack of pure self-attention encoder layers. Third, KAR refines attention inside each self-attention block by denoising and reweighting inter-keypoint relations. Fourth, an MLP regression head directly outputs $K$5 two-dimensional coordinates from the refined support tokens, in a DETR-style token-to-output formulation and without heatmaps (Liang et al., 2024).

A critical implementation detail is the use of unshared linear projections for support and query streams when forming $K$6 and $K$7. Because support tokens carry identifier embeddings while query tokens carry positional embeddings, SCAPE treats them as distinct encodings and avoids forcing them through shared projections. With globally enriched support tokens $K$8 and query tokens $K$9, the model forms

$p_i,\hat{p}_i\in\mathbb{R}^2$0

$p_i,\hat{p}_i\in\mathbb{R}^2$1

$p_i,\hat{p}_i\in\mathbb{R}^2$2

and then applies standard self-attention

$p_i,\hat{p}_i\in\mathbb{R}^2$3

The default configuration uses 2 GKP cross-attention blocks followed by 4 self-attention encoder blocks with KAR, then the MLP head. A lighter variant, Lite-SCAPE, uses 1 GKP block and 2 self-attention blocks (Liang et al., 2024).

3. Global Keypoint Feature Perceptor and Keypoint Attention Refiner

The Global Keypoint Feature Perceptor injects support-image semantics into local support keypoint tokens through cross-attention. Using support keypoints as queries and support-image tokens as keys and values,

$p_i,\hat{p}_i\in\mathbb{R}^2$4

$p_i,\hat{p}_i\in\mathbb{R}^2$5

and the resulting globally enriched support representation is

$p_i,\hat{p}_i\in\mathbb{R}^2$6

In SCAPE this cross-attention can be repeated for two blocks. Its stated role is to fuse global support-image context into support keypoint tokens, thereby encoding relative structure and disambiguating symmetric points (Liang et al., 2024).

The Keypoint Attention Refiner operates in attention space rather than feature space. Let $p_i,\hat{p}_i\in\mathbb{R}^2$7 denote the attention matrix computed inside a self-attention block. KAR applies multiple attention filters $p_i,\hat{p}_i\in\mathbb{R}^2$8 to reshape $p_i,\hat{p}_i\in\mathbb{R}^2$9 and combines them through learned assignment weights derived from the support keypoints:

$S$0

$S$1

$S$2

The refined attention is then obtained by residual adjustment followed by softmax:

$S$3

The paper emphasizes the keypoint-to-keypoint slice because CAPE relies on inter-node relations, although the refinement is applied to the full attention where implemented. Empirically, multiple filters, exemplified by $S$4 with modest hidden dimensions, outperform a single large filter and improve generalization across categories (Liang et al., 2024).

These two modules define the method’s main architectural additions over a pure attention baseline. GKP injects global semantics before joint support-query interaction, whereas KAR regularizes the attention process during that interaction. The model’s final prediction is direct coordinate regression:

$S$5

Training uses only an $S$6 coordinate loss,

$S$7

with no similarity-map or offset losses (Liang et al., 2024).

4. Training protocol, dataset, and efficiency profile

SCAPE is evaluated on MP-100, a benchmark with 100 object categories and more than 20k instances (Liang et al., 2024). The dataset is split into five folds, each containing 70 training categories, 10 validation categories, and 20 test categories, with no category overlap across train, validation, and test. Preprocessing crops objects by bounding box and resizes them to $S$8. Reported augmentation consists of random scaling from $S$9 to $$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$0 and random rotation from $$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$1 to $$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$2 (Liang et al., 2024).

Optimization uses Adam with batch size 16 for 180 epochs. The initial learning rate is $$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$3 and is decayed by $$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$4 at epochs 140 and 170; cosine annealing is reported to yield similar results. The loss is $$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$5 only. Reproducibility-sensitive choices include unshared $$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$6 projections, the use of positional embeddings for query tokens and identifier embeddings for support keypoints, and the default 2 GKP + 4 self-attention configuration (Liang et al., 2024).

From a complexity standpoint, if $$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$7 is the number of support keypoints and $$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$8 is the number of query tokens, self-attention over concatenated tokens costs $$\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.$$9 per layer. SCAPE mitigates this through pure attention blocks without heavy decoders or explicit matching heads, a lightweight MLP regressor instead of heatmap prediction, and GKP as cross-attention replacing early self-attention while improving convergence (Liang et al., 2024).

The reported efficiency measurements, all with a ResNet-50 backbone, $256\times256$0 input, RTX 3090, and the 1-shot setting, are as follows.

Method	GFLOPs / FPS	Mean PCK / ΔParams
POMNet	38.01 / 6.80	79.70 / +1.19M
CapeFormer	23.68 / 26.09	85.31 / +7.63M
Lite-SCAPE	22.20 / 36.89	86.13 / +1.95M
SCAPE	22.81 / 29.43	87.33 / +3.88M

Excluding backbones, SCAPE uses about 51% of CapeFormer’s parameters while being faster (Liang et al., 2024). This efficiency profile is central to the method’s identity: it is presented not as a larger or more elaborate pose estimator, but as a simplification that improves attention quality and removes ancillary prediction heads.

5. Empirical performance and ablation findings

On MP-100 with a ResNet-50 backbone, SCAPE reports a 1-shot mean PCK of 87.55, compared with 85.31 for CapeFormer, and a 5-shot mean PCK of 90.66, compared with 89.30 for CapeFormer (Liang et al., 2024). The gains are therefore +2.24 in the 1-shot setting and +1.36 in the 5-shot setting. Per split, SCAPE exceeds CapeFormer on all five splits for both 1-shot and 5-shot evaluation (Liang et al., 2024).

The method also shows strong compatibility with transformer backbones. Reported results are 88.19 and 91.53 for SCAPE ViT-B trained with ImageNet-1K pretraining under 1-shot and 5-shot, 87.93 and 92.01 for Swin-S, 90.74 and 92.65 for ViT-S with DINOv2, and 91.95 and 93.98 for ViT-B with DINOv2 (Liang et al., 2024). Relative to CapeFormer ViT-B with DINOv2, SCAPE improves by +4.4 in 1-shot and +3.4 in 5-shot, which the paper interprets as strong compatibility with modern self-supervised ViT features (Liang et al., 2024).

Cross super-category generalization in the 1-shot setting is reported as 84.24 for Human body, 85.98 for Human face, 45.61 for Vehicle, and 54.13 for Furniture (Liang et al., 2024). The paper states that SCAPE provides the best performance across super-categories relative to prior CAPE baselines. This suggests that the method’s attention-centric design is not merely category-agnostic in training protocol, but also operationally robust across broad structural regimes.

The ablation study isolates several effects. Replacing explicit similarity-map heads and supervising only coordinates improves performance, with direct coordinate regression achieving 89.1 on split1 over map-based variants. Replacing CapeFormer’s DETR-style decoder with a full self-attention encoder over concatenated tokens improves PCK by +0.5. Using unshared $256\times256$1 projections for support and query improves PCK by +0.7 independently of parameter count. Adding GKP yields +1.0 on split1, while KAR yields +1.4 on split1; multiple small attention filters outperform a single large filter. The best reported ResNet-50 configuration is 2 GKP plus 4 self-attention interactor layers, reaching 91.6 on split1 (Liang et al., 2024).

A recurring conclusion of these experiments is that implicit attention-based matching is sufficient and, in the reported comparisons, more robust than explicitly supervised similarity maps. The paper further states that attention maps generated during feature interaction are cleaner and closer to ground truth than explicit similarity maps, especially after the addition of GKP and KAR (Liang et al., 2024).

6. Relation to prior CAPE models, limitations, and terminological ambiguity

SCAPE is explicitly positioned against CAPE systems such as POMNet and CapeFormer (Liang et al., 2024). POMNet constructs and supervises explicit similarity maps, while CapeFormer uses two-stage matching and iterative refinement with additional losses. SCAPE removes these components and replaces them with attention-only feature interaction plus direct coordinate regression. The resulting contrast is architectural as much as empirical: SCAPE frames CAPE as an attention-quality problem rather than as a specialized pipeline requiring explicit matching heads and heatmap supervision (Liang et al., 2024).

The paper also states several limitations. The current CAPE formulation assumes one-to-one keypoint correspondences, so multi-instance scenarios and one-to-many correspondences remain challenging. Severe occlusions and large appearance gaps can still reduce accuracy, although KAR helps by reasoning from visible to occluded keypoints (Liang et al., 2024). These limitations are consistent with the method’s design: it models relations among a fixed support keypoint set and predicts a corresponding fixed query keypoint set, which is effective under the evaluated protocol but does not address more open-ended correspondence structures.

A potential source of confusion is the acronym itself. Within the broader literature, “SCAPE” is not unique to category-agnostic pose estimation. The acronym also appears in unrelated contexts, including the human-body dataset Shape Completion and Animation of PEople and its associated statistical body models (Pishchulin et al., 2015), geometry-processing benchmarks that evaluate on SCAPE meshes (Shtern et al., 2013, Nagar et al., 2018), a robotics method for Stiffness Control from Augmented Position control Experiences (Kim et al., 2021), an architectural design system for Searching Conceptual Architecture Prompts using Evolution (Lim et al., 2024), a distributed optimizer for LLM training (Zheng et al., 2 Jul 2026), and a personalized headline generation framework (Lian et al., 21 Jan 2025). In the ParFormer literature, the phrase “SCAPE” is also associated with a patch-embedding module, but the paper itself names that module CAPE rather than a sparse selection mechanism (Setyawan et al., 2024). In contemporary computer vision, however, the SCAPE considered here denotes the category-agnostic pose estimator introduced in 2024 (Liang et al., 2024).

SCAPE’s place in CAPE research is therefore unusually clear despite the acronymic overlap: it is a simplified, attention-centric estimator whose main contribution is to show that category-agnostic keypoint localization can be handled effectively by pure attention, augmented only by global support fusion and attention refinement, and trained with a single coordinate regression objective (Liang et al., 2024).