---
title: 'SCAPE: Category-Agnostic Pose Estimator'
url: https://www.emergentmind.com/topics/scape
type: topic
---

# SCAPE: Category-Agnostic Pose Estimator

SCAPE, short for *Simple and strong Category-Agnostic Pose Estimator*, is a model for category-agnostic pose estimation (CAPE) that localizes semantic keypoints on an object of arbitrary category from a small number of annotated exemplars in an in-context few-shot setting [2407.13483]. Its central claim is that CAPE is fundamentally a feature-matching problem already expressible inside transformer attention, so explicit similarity-map heads, two-stage decoding, and heatmap supervision are unnecessary. SCAPE therefore uses a streamlined architecture built from a shared feature extractor, pure self-attention over support and query tokens, a Global Keypoint Feature Perceptor (GKP), a Keypoint Attention Refiner (KAR), and a lightweight MLP coordinate regressor, yielding improvements in both accuracy and efficiency on MP-100 under 1-shot and 5-shot evaluation [2407.13483].

## 1. Problem formulation and task setting

Category-Agnostic Pose Estimation addresses pose localization for arbitrary object categories without category-specific training [2407.13483]. In the in-context few-shot setting, the model receives a small number of annotated support exemplars from the same object instance or category at test time and must localize the same semantic keypoints in a query image. In 1-shot CAPE, one annotated support image is given; in 5-shot CAPE, five supports are given. The support input consists of each support image together with its annotated keypoints represented as 2D coordinates, from which support keypoint tokens are formed using the support feature map and keypoint heatmaps. The query input is the target image in which the corresponding keypoints must be localized, and the output is the set of 2D coordinates for all target keypoints in the query image [2407.13483].

The primary evaluation metric is Probability of Correct Keypoint. On MP-100, following POMNet and CapeFormer, evaluation uses a normalized distance threshold of $0.2$ on resized crops. If $K$ is the number of keypoints, $p_i,\hat{p}_i\in\mathbb{R}^2$ are predicted and ground-truth coordinates, and $S$ is the normalization scale, then
$$
\mathrm{PCK}_\alpha = \frac{1}{K}\sum_{i=1}^{K}\mathbf{1}\big(\lVert p_i-\hat{p}_i\rVert_2 \le \alpha\cdot S\big),\qquad \alpha=0.2.
$$
Because preprocessing crops objects by bounding box and resizes images to $256\times256$, $S$ can be taken as $\max(H,W)=256$. The paper also reports AUC and NME as complementary localization metrics, but training and primary comparisons focus on PCK [2407.13483].

SCAPE’s interpretation of this task is that CAPE ultimately reduces to matching sparse support keypoint descriptors to locations in the query image. Transformer attention already computes similarities through the $\mathrm{softmax}(QK^\top)$ operator, and the method argues that these implicit attention maps are empirically closer to ground truth than explicitly supervised similarity maps. This motivates an architecture in which attention itself performs the matching and the model directly regresses coordinates, rather than generating and supervising intermediate heatmaps [2407.13483].

## 2. Tokenization, architecture, and inference path

SCAPE uses a shared backbone for support and query images. The reported backbones are ResNet-50, ViT, and Swin, with average pooling added for transformer backbones to harmonize output resolution and computational load [2407.13483]. From the support feature map $F_{\text{sup}}\in\mathbb{R}^{N\times C}$, support keypoint tokens $F_s\in\mathbb{R}^{K\times C}$ are built by weighted pooling with Gaussian keypoint heatmaps:
$$
F_s(i)=\sum_x H_i(x)\cdot F_{\text{sup}}(x).
$$
A learnable support keypoint identifier embedding $I_s$ is then added so that different semantic keypoints remain distinguishable:
$$
F_s \leftarrow F_s + I_s.
$$
For the query image, the token set is the query feature map with positional embeddings:
$$
F_q \leftarrow F_q + PE.
$$

The end-to-end pipeline contains four stages. First, GKP enriches each support keypoint token with global support-image context. Second, the globally enriched support tokens are concatenated with the query tokens and passed through a stack of pure self-attention encoder layers. Third, KAR refines attention inside each self-attention block by denoising and reweighting inter-keypoint relations. Fourth, an MLP regression head directly outputs $K$ two-dimensional coordinates from the refined support tokens, in a DETR-style token-to-output formulation and without heatmaps [2407.13483].

A critical implementation detail is the use of unshared linear projections for support and query streams when forming $Q$ and $K$. Because support tokens carry identifier embeddings while query tokens carry positional embeddings, SCAPE treats them as distinct encodings and avoids forcing them through shared projections. With globally enriched support tokens $F_s^{\text{global}}$ and query tokens $F_q$, the model forms
$$
K_s = W_{K1}F_s^{\text{global}},\qquad K_q = W_{K2}F_q,\qquad K=\mathrm{concat}(K_s,K_q),
$$
$$
Q_s = W_{Q1}F_s^{\text{global}},\qquad Q_q = W_{Q2}F_q,\qquad Q=\mathrm{concat}(Q_s,Q_q),
$$
$$
V=\mathrm{concat}(F_s^{\text{global}},F_q),
$$
and then applies standard self-attention
$$
\mathrm{Attn}(Q,K,V)=\mathrm{softmax}\!\left(\frac{QK^\top}{\sqrt d}\right)V.
$$
The default configuration uses 2 GKP cross-attention blocks followed by 4 self-attention encoder blocks with KAR, then the MLP head. A lighter variant, Lite-SCAPE, uses 1 GKP block and 2 self-attention blocks [2407.13483].

## 3. Global Keypoint Feature Perceptor and Keypoint Attention Refiner

The Global Keypoint Feature Perceptor injects support-image semantics into local support keypoint tokens through cross-attention. Using support keypoints as queries and support-image tokens as keys and values,
$$
Q_s = W_Q^sF_s,\qquad K_{\text{sup}} = W_K^{img}F_{\text{sup}},\qquad V_{\text{sup}} = W_V^{img}F_{\text{sup}},
$$
$$
SA(Q_s,K_{\text{sup}},V_{\text{sup}})=\mathrm{softmax}\!\left(\frac{Q_sK_{\text{sup}}^\top}{\sqrt d}\right)V_{\text{sup}},
$$
and the resulting globally enriched support representation is
$$
F_s^{\text{global}} = LN(F_s + SA(Q_s,K_{\text{sup}},V_{\text{sup}})).
$$
In SCAPE this cross-attention can be repeated for two blocks. Its stated role is to fuse global support-image context into support keypoint tokens, thereby encoding relative structure and disambiguating symmetric points [2407.13483].

The Keypoint Attention Refiner operates in attention space rather than feature space. Let $\mathcal{A}$ denote the attention matrix computed inside a self-attention block. KAR applies multiple attention filters $AF_i$ to reshape $\mathcal{A}$ and combines them through learned assignment weights derived from the support keypoints:
$$
AF_i(\mathcal{A}) = MLP_i(\mathrm{ReLU}(\mathcal{A})),
$$
$$
Assign_i(F_s)=\mathrm{softmax}(\mathrm{layernorm}(\mathrm{dropout}(F_sW_{\text{assign}}))),
$$
$$
KAR(\mathcal{A})=\sum_{i=1}^{n} Assign_i(F_s)\cdot AF_i(\mathcal{A}).
$$
The refined attention is then obtained by residual adjustment followed by softmax:
$$
\mathcal{A}_{\text{refined}}=\mathrm{softmax}(\mathcal{A}+KAR(\mathcal{A})).
$$
The paper emphasizes the keypoint-to-keypoint slice because CAPE relies on inter-node relations, although the refinement is applied to the full attention where implemented. Empirically, multiple filters, exemplified by $n=4$ with modest hidden dimensions, outperform a single large filter and improve generalization across categories [2407.13483].

These two modules define the method’s main architectural additions over a pure attention baseline. GKP injects global semantics before joint support-query interaction, whereas KAR regularizes the attention process during that interaction. The model’s final prediction is direct coordinate regression:
$$
P = MLP(F_s^{\text{refined}}),\qquad P\in\mathbb{R}^{K\times 2}.
$$
Training uses only an $\ell_1$ coordinate loss,
$$
\mathcal{L}_{coord}=\frac{1}{K}\sum_{i=1}^{K}\lVert P_i-\hat{P}_i\rVert_1,
$$
with no similarity-map or offset losses [2407.13483].

## 4. Training protocol, dataset, and efficiency profile

SCAPE is evaluated on MP-100, a benchmark with 100 object categories and more than 20k instances [2407.13483]. The dataset is split into five folds, each containing 70 training categories, 10 validation categories, and 20 test categories, with no category overlap across train, validation, and test. Preprocessing crops objects by bounding box and resizes them to $256\times256$. Reported augmentation consists of random scaling from $-15\%$ to $+15\%$ and random rotation from $-15^\circ$ to $+15^\circ$ [2407.13483].

Optimization uses Adam with batch size 16 for 180 epochs. The initial learning rate is $2\times10^{-4}$ and is decayed by $0.1$ at epochs 140 and 170; cosine annealing is reported to yield similar results. The loss is $\mathcal{L}_{coord}$ only. Reproducibility-sensitive choices include unshared $Q/K$ projections, the use of positional embeddings for query tokens and identifier embeddings for support keypoints, and the default 2 GKP + 4 self-attention configuration [2407.13483].

From a complexity standpoint, if $k$ is the number of support keypoints and $n$ is the number of query tokens, self-attention over concatenated tokens costs $O((k+n)^2\cdot d)$ per layer. SCAPE mitigates this through pure attention blocks without heavy decoders or explicit matching heads, a lightweight MLP regressor instead of heatmap prediction, and GKP as cross-attention replacing early self-attention while improving convergence [2407.13483].

The reported efficiency measurements, all with a ResNet-50 backbone, $256\times256$ input, RTX 3090, and the 1-shot setting, are as follows.

| Method | GFLOPs / FPS | Mean PCK / ΔParams |
|---|---:|---:|
| POMNet | 38.01 / 6.80 | 79.70 / +1.19M |
| CapeFormer | 23.68 / 26.09 | 85.31 / +7.63M |
| Lite-SCAPE | 22.20 / 36.89 | 86.13 / +1.95M |
| SCAPE | 22.81 / 29.43 | 87.33 / +3.88M |

Excluding backbones, SCAPE uses about 51% of CapeFormer’s parameters while being faster [2407.13483]. This efficiency profile is central to the method’s identity: it is presented not as a larger or more elaborate pose estimator, but as a simplification that improves attention quality and removes ancillary prediction heads.

## 5. Empirical performance and ablation findings

On MP-100 with a ResNet-50 backbone, SCAPE reports a 1-shot mean PCK of 87.55, compared with 85.31 for CapeFormer, and a 5-shot mean PCK of 90.66, compared with 89.30 for CapeFormer [2407.13483]. The gains are therefore +2.24 in the 1-shot setting and +1.36 in the 5-shot setting. Per split, SCAPE exceeds CapeFormer on all five splits for both 1-shot and 5-shot evaluation [2407.13483].

The method also shows strong compatibility with transformer backbones. Reported results are 88.19 and 91.53 for SCAPE ViT-B trained with ImageNet-1K pretraining under 1-shot and 5-shot, 87.93 and 92.01 for Swin-S, 90.74 and 92.65 for ViT-S with DINOv2, and 91.95 and 93.98 for ViT-B with DINOv2 [2407.13483]. Relative to CapeFormer ViT-B with DINOv2, SCAPE improves by +4.4 in 1-shot and +3.4 in 5-shot, which the paper interprets as strong compatibility with modern self-supervised ViT features [2407.13483].

Cross super-category generalization in the 1-shot setting is reported as 84.24 for Human body, 85.98 for Human face, 45.61 for Vehicle, and 54.13 for Furniture [2407.13483]. The paper states that SCAPE provides the best performance across super-categories relative to prior CAPE baselines. This suggests that the method’s attention-centric design is not merely category-agnostic in training protocol, but also operationally robust across broad structural regimes.

The ablation study isolates several effects. Replacing explicit similarity-map heads and supervising only coordinates improves performance, with direct coordinate regression achieving 89.1 on split1 over map-based variants. Replacing CapeFormer’s DETR-style decoder with a full self-attention encoder over concatenated tokens improves PCK by +0.5. Using unshared $Q/K$ projections for support and query improves PCK by +0.7 independently of parameter count. Adding GKP yields +1.0 on split1, while KAR yields +1.4 on split1; multiple small attention filters outperform a single large filter. The best reported ResNet-50 configuration is 2 GKP plus 4 self-attention interactor layers, reaching 91.6 on split1 [2407.13483].

A recurring conclusion of these experiments is that implicit attention-based matching is sufficient and, in the reported comparisons, more robust than explicitly supervised similarity maps. The paper further states that attention maps generated during feature interaction are cleaner and closer to ground truth than explicit similarity maps, especially after the addition of GKP and KAR [2407.13483].

## 6. Relation to prior CAPE models, limitations, and terminological ambiguity

SCAPE is explicitly positioned against CAPE systems such as POMNet and CapeFormer [2407.13483]. POMNet constructs and supervises explicit similarity maps, while CapeFormer uses two-stage matching and iterative refinement with additional losses. SCAPE removes these components and replaces them with attention-only feature interaction plus direct coordinate regression. The resulting contrast is architectural as much as empirical: SCAPE frames CAPE as an attention-quality problem rather than as a specialized pipeline requiring explicit matching heads and heatmap supervision [2407.13483].

The paper also states several limitations. The current CAPE formulation assumes one-to-one keypoint correspondences, so multi-instance scenarios and one-to-many correspondences remain challenging. Severe occlusions and large appearance gaps can still reduce accuracy, although KAR helps by reasoning from visible to occluded keypoints [2407.13483]. These limitations are consistent with the method’s design: it models relations among a fixed support keypoint set and predicts a corresponding fixed query keypoint set, which is effective under the evaluated protocol but does not address more open-ended correspondence structures.

A potential source of confusion is the acronym itself. Within the broader literature, “SCAPE” is not unique to category-agnostic pose estimation. The acronym also appears in unrelated contexts, including the human-body dataset *Shape Completion and Animation of PEople* and its associated statistical body models [1503.05860], geometry-processing benchmarks that evaluate on SCAPE meshes [1311.5595], [1807.10162], a robotics method for *Stiffness Control from Augmented Position control Experiences* [2102.08442], an architectural design system for *Searching Conceptual Architecture Prompts using Evolution* [2402.00089], a distributed optimizer for LLM training [2607.01678], and a personalized headline generation framework [2501.11900]. In the ParFormer literature, the phrase “SCAPE” is also associated with a patch-embedding module, but the paper itself names that module CAPE rather than a sparse selection mechanism [2403.15004]. In contemporary computer vision, however, the SCAPE considered here denotes the category-agnostic pose estimator introduced in 2024 [2407.13483].

SCAPE’s place in CAPE research is therefore unusually clear despite the acronymic overlap: it is a simplified, attention-centric estimator whose main contribution is to show that category-agnostic keypoint localization can be handled effectively by pure attention, augmented only by global support fusion and attention refinement, and trained with a single coordinate regression objective [2407.13483].

Source: https://www.emergentmind.com/topics/scape