Pose Masked Transformer for Efficient Pose Estimation

• Pose Masked Transformer is a neural architecture that uses adaptive masking to focus computation on human-relevant areas, reducing redundancy in images and videos.
• It employs attention-based token pruning to selectively retain informative regions, lowering computational complexity while preserving pose accuracy.
• The approach supports multi-view fusion and spatio-temporal masking, demonstrating efficiency and robustness in both 2D and 3D pose estimation tasks.

A Pose Masked Transformer is a transformer-based neural architecture that leverages adaptive masking and token selection strategies for efficient and accurate human pose estimation in images and videos, both in monocular and multi-view settings. The central concept is to restrict computational attention to the most informative—typically human-occupied—regions of the visual input, dramatically reducing the quadratic complexity inherent in global self-attention while maintaining or improving pose accuracy. Key variants include dynamic pruning based on attention aggregation, as well as related masked autoencoding and spatio-temporal masking schemes for robust training and pretext tasks.

1. Core Architecture and Principles

Pose Masked Transformers, exemplified by the token-Pruned Pose Transformer (PPT), are built atop a standard transformer stack that operates on visual tokens derived from image patches and, optionally, dedicated keypoint tokens. The typical processing flow involves:

• Extraction of a dense feature map $$F = \mathcal{B}(I) \in \mathbb{R}^{H \times W \times C}$$ from the input image $I$ via a lightweight CNN backbone.
• Partitioning $F$ into non-overlapping patches, each of which is mapped to a $D$-dimensional “visual token” $X_v \in \mathbb{R}^{N_v \times D}$.
• Introduction of $J$ learnable “keypoint tokens” $X_k \in \mathbb{R}^{J \times D}$, corresponding to joint hypotheses.
• The transformer operates on the concatenated input $X^0 = [X_k; X_v]$.

Within the transformer, standard multi-head self-attention (MHSA) is computed: $$SA(X) = \mathrm{Softmax}\left(\frac{QK^T}{\sqrt{D}}\right) V$$ where $Q,K,V$ are the query, key, and value projections of $I$0 at each layer. Output keypoint tokens are ultimately mapped to heatmaps or pose parameters.

2. Adaptive Token Pruning via Attention-Based Masking

Unlike global transformers, the Pose Masked Transformer prunes visual tokens that are likely to correspond to background regions, retaining only those with significant relevance to pose estimation:

• At selected transformer layers, an attention aggregation process computes per-token relevance $I$1 by summing the attention scores from all keypoint tokens to each visual token:

$I$2

where $I$3 is the $I$4th keypoint token's query vector.

• A top-$I$5 selection is performed by thresholding $I$6 (with $I$7 for a specified keep ratio $I$8). Visual tokens with $I$9 (where $F$0 is the $F$1th largest value) are retained.
• The selected tokens $F$2 are concatenated with $F$3 for input to subsequent layers.

This mechanism, referred to as Human Token Identification (HTI), does not require external segmentation masks or labels, as the mask emerges from the keypoint-to-visual attention.

3. Multi-View and Cross-View Fusion Strategies

The Pose Masked Transformer framework extends naturally to multi-view 2D and 3D pose estimation:

• Each camera view is processed independently to extract keypoint tokens and pruned human-area tokens.
• For cross-view fusion, human-area tokens and keypoint tokens from all views are concatenated and fed into a joint transformer equipped with 3D positional encodings, enabling intra- and inter-view attention.
• In contrast with global fusion (all-pixels, all-views), this “human area fusion” reduces spatial redundancy: the complexity drops from $F$4 for $F$5 views to $F$6, where $F$7 is the number of kept tokens per view.

4. Computational Efficiency and Experimental Results

The token selection methodology provides significant efficiency gains with negligible or improved pose accuracy. On the COCO validation set, for single-view 2D pose estimation (TokenPose-S backbone):

Model	Params	GFLOPs	AP
TokenPose-S	6.6 M	2.2	72.5
PPT-S (r=0.7)	6.6 M	1.6	72.2

For multi-view Human3.6M 3D pose estimation, multi-view PPT achieves a mean per-joint position error (MPJPE) of 24.4 mm, outperforming other fusion approaches. Transformer FLOPs are reduced by up to 38% through token pruning, and multi-view fusion MACs are reduced by approximately 80% (Ma et al., 2022).

Additionally, on the Ski-Pose dataset, PPT improves 2D joint detection rate to 96.3% and reduces 3D MPJPE to 34.1 mm, surpassing existing baselines.

5. Variations and Related Masking Strategies

Several related transformer variants for pose tasks employ masking schemes for robustness, efficiency, or self-supervised learning:

• Spatio-temporal Masked Transformers for sequence data use high spatio-temporal masking ratios (e.g., 75%) for masked autoencoding pre-training, leading to improved performance on ambiguous or noisy pose signals (e.g., pressure maps) (Davoodnia et al., 2023).
• Occlusion-Aware Masked Transformers implement target-focused masking, enforcing structured masking over probable interaction or occlusion regions (e.g., masked autoencoding over hand–object contacts), promoting inpainting and robust reasoning under occlusion (Yang et al., 12 Jun 2025).

These masking approaches, although architecturally distinct, contribute to learning representations that capture global dependencies while mitigating the computation/overfitting burden posed by redundant background input.

6. Limitations and Future Perspectives

Despite the practical benefits, human-area masking via internal attention aggregation is susceptible to failure modes:

• The HTI mask may be inaccurate in scenes with heavy occlusion, partial visibility, or crowded/multi-person scenarios.
• The heuristic top-$F$8 selection operates uniformly across samples; adaptive or learned pruning rates may potentially yield greater robustness.
• Current schemes do not explicitly integrate temporal or hierarchical token merging, although the integration of such mechanisms is a noted direction for future research (Ma et al., 2022).

Further advancements may involve combining hierarchical masking, explicit mask heads (e.g., U-Nets), and fusion strategies that couple spatial, temporal, and cross-modal cues.

7. Significance and Impact

Pose Masked Transformers represent a class of architectures successfully reconciling the tradeoff between expressivity and computational tractability for pose estimation. By focusing attention and computation on dynamically identified human-relevant regions, these models achieve state-of-the-art pose accuracy—both for monocular and multi-view input—while enabling real-time, large-scale deployment scenarios. Their design principles are widely applicable, with demonstrated benefit in RGB, depth, tactile, and occlusion-heavy domains (Ma et al., 2022, Davoodnia et al., 2023, Yang et al., 12 Jun 2025). A plausible implication is the extension of token pruning and structured masking to broader tasks in vision and spatio-temporal reasoning—including video, multimodal fusion, and non-human pose estimation.