RTMPose Framework

• RTMPose is a real-time, multi-person pose estimation framework that uses a top-down approach with CSPNeXt backbones and SimCC coordinate classification.
• The framework achieves a strong speed–accuracy trade-off, delivering competitive accuracy on benchmarks like COCO while running efficiently on CPUs, GPUs, and mobile devices.
• RTMPose integrates seamlessly with the MMPose ecosystem, enabling modular configurations and serving as a foundation for whole-body and 3D pose estimation extensions.

RTMPose is a real-time, multi-person pose estimation framework implementing a top-down pipeline with a focus on high throughput, accuracy, and deployment efficiency. Developed as part of the MMPose toolbox, RTMPose advances the speed-accuracy trade-off for 2D human pose estimation on both commodity hardware and edge devices, and serves as the foundation for several subsequent frameworks targeting whole-body and 3D pose estimation.

1. Core Paradigm and System Design

RTMPose applies a top-down pose estimation pipeline: an off-the-shelf real-time detector (e.g., RTMDet, YOLOv3, Faster-RCNN) first predicts person bounding boxes. Each box is then cropped and processed by an efficient pose estimation network. Key system goals include:

• Real-time inference on CPUs, GPUs, and mobile devices
• Competitive accuracy on benchmarks such as COCO and COCO-WholeBody
• Minimal parameters and FLOPs for deployability
• Seamless integration with export toolchains (PyTorch→ONNX→TensorRT/ncnn/MMDeploy)

Empirical analysis revealed that real-time detectors are no longer a computational bottleneck on commodity systems. Instead, architectural and representation choices in the pose estimation head—specifically the use of CSPNeXt backbones and SimCC coordinate classification—are critical to optimizing both latency and accuracy (Jiang et al., 2023).

2. Model Architectures and Coordinate Representation

Each RTMPose variant (t, s, m, l, x) shares a unified high-level architecture:

• Backbone: CSPNeXt-{t,s,m,l,x} for varying model sizes, prioritizing a speed–accuracy–deployment trade-off superior to ResNet or HRNet.
• Neck: 1×1 conv projects final feature maps into two 1D sequences representing horizontal and vertical keypoint positions.
• Keypoint Representation Expansion: Each bin’s embedding is expanded via a fully connected layer to D=256.
• Self-attention Refinement: One or more Gated Attention Units (GAU), formulated as:

$$U = \phi_u(XW_u),~~V = \phi_v(XW_v),~~O = (U \odot A V) W_o,$$

where $A = (1/n)\,\text{relu}^2(Q(X)K(X)^{T}/\sqrt{s})$.

• Output: Two classification heads (7×7 convs) per axis, followed by softmax binning.

Major differences between variants are in backbone depth and the number of GAUs:

• RTMPose-s: CSPNeXt-s backbone, 2 GAUs, ~0.68M params, 0.68 GFLOPs (pose head only)
• RTMPose-m: CSPNeXt-m, 3 GAUs, ~1.93M params, 1.93 GFLOPs
• RTMPose-l: CSPNeXt-l, 4 GAUs, ~4.16M params, 4.16 GFLOPs

The SimCC coordinate classification paradigm replaces regression or dense heatmaps with axis-wise soft classification, eliminating the upsampling bottleneck and allowing lightweight output heads (Jiang et al., 2023).

3. Training Strategies and Optimization

RTMPose utilizes a two-stage training protocol:

• Pre-training via UDP-based Gaussian heatmaps establishes a strong initialization.
• Main Optimization:
  • Loss: Soft-label cross-entropy per axis, using unnormalized Gaussian labels,

  $$y_i = \frac{\exp(\phi(r_t, r_i)/\tau)}{\sum_{k}\exp(\phi(r_t, r_k)/\tau)},$$

  with axis-wise variance $\sigma$ and temperature $\tau=0.1$. - Optimizer: AdamW (β₁=0.9, β₂=0.999), base LR 0.004, weight decay schedule by model size - LR schedule: flat-then-cosine, with 1K warmup iterations - Training epochs: 210 (pre-train), up to 420 (fine-tune, multi-dataset training) - Augmentation: Two-stage regime with strong (scale∈[0.6,1.4], rotation∈[–80°,80°], Cutout=1.0) and weak (minor rotation, Cutout=0.5) policies - Additional datasets (AI-Challenger) and longer training improve COCO AP by up to 2 points (Jiang et al., 2023).

4. Deployment, Hardware Efficiency, and Software Integration

Key features for deployment and integration include:

• Exporters: ONNX, TensorRT, and ncnn via MMDeploy, supporting FP32 and FP16 precision.

• No architectural pruning or distillation; efficiency derives from base model design.

• Cross-platform runtime:

  • Snapdragon 865 (mobile): RTMPose-s achieves 72.8 AP at ~72 FPS (FP16)
  • Intel i7-11700 (CPU): RTMPose-m achieves 75.3 AP at ~90 FPS; RTMPose-l achieves 76.3 AP at ~53 FPS
  • GTX 1660 Ti (GPU): RTMPose-m achieves ~430 FPS, RTMPose-l ~290 FPS

RTMPose is implemented as a project in MMPose, with configuration files, models, and demo scripts supporting reproducibility and end-to-end deployment (Jiang et al., 2023).

5. Benchmark Performance and Trade-off Analysis

RTMPose establishes strong speed–accuracy trade-offs across multiple datasets.

COCO val2017 (single-frame, no flip):

Variant	Params (M)	GFLOPs	AP	FPS (GPU)	FPS (CPU)	FPS (Mobile)
RTMPose-t	0.36	0.36	67.1	>200	–	>300
RTMPose-s	0.68	0.68	71.1	~430	–	72
RTMPose-m	1.93	1.93	75.3	~430	90	35
RTMPose-l	4.16	4.16	76.3	~290	53	22

COCO-WholeBody V1.0: RTMPose-x achieves 65.3 AP at 18.1 GFLOPs, competitive with state-of-the-art models. On specialized benchmarks such as AP-10K, CrowdPose, and MPII, RTMPose-m with extended pre-training surpasses prior SOTA by approximately 3 AP (Jiang et al., 2023).

Best-practice recommendations:

• RTMPose-s: optimal for embedded/mobile,
• RTMPose-m: high-throughput and accuracy on CPU,
• RTMPose-l/x: maximum accuracy on GPUs or for batch inference.

6. MMPose Ecosystem and Extensions

RTMPose forms the basis for subsequent frameworks such as RTMW (Real-Time Multi-person Whole-body models) (Jiang et al., 2024). RTMW extends RTMPose by integrating FPN-based feature fusion and a Hierarchical Encoding Module (HEM), supporting fine-grained multi-scale keypoint localization for hands, face, and feet. RTMW-l achieves 70.2 mAP on COCO-WholeBody at ~45 ms/frame (CPU), setting new open-source records. The RTMO framework (Lu et al., 2023) adapts RTMPose’s coordinate classification head for one-stage architectures, employing dual 1D heatmaps and a dynamic bin encoding scheme to achieve 74.8 AP on COCO at ~141 FPS on a V100 GPU.

Within the MMPose repository, RTMPose and its derivatives are available with modular configuration files, backbone and head implementations, and deployment utilities (Jiang et al., 2023, Jiang et al., 2024, Lu et al., 2023).

7. Limitations and Future Directions

While RTMPose’s top-down design achieves superior accuracy and deployment flexibility, the inference time increases linearly with the number of detected persons, unlike one-stage approaches such as RTMO where latency is input-size dependent. Model variants tuned for specific hardware provide flexibility, but the absence of explicit pruning/distillation means extreme resource-constrained scenarios may require further compression.

Open research challenges include extending top-down coordinate-classification paradigms to temporal sequences, multi-modal (e.g., radar or LiDAR) settings as in RT-Pose (Ho et al., 2024), and generic dense prediction tasks. The inclusion of spatio-temporal attention, semi/self-supervised pre-training, and distillation for ultra-lightweight deployment are promising future directions suggested by the current literature.

---

References:

• "RTMPose: Real-Time Multi-Person Pose Estimation based on MMPose" (Jiang et al., 2023)
• "RTMW: Real-Time Multi-Person 2D and 3D Whole-body Pose Estimation" (Jiang et al., 2024)
• "RTMO: Towards High-Performance One-Stage Real-Time Multi-Person Pose Estimation" (Lu et al., 2023)
• "RT-Pose: A 4D Radar Tensor-based 3D Human Pose Estimation and Localization Benchmark" (Ho et al., 2024)