---
title: End-to-End Multi-Person 2D Pose Estimation
url: https://www.emergentmind.com/topics/end-to-end-multi-person-2d-pose-estimation
type: topic
---

# End-to-End Multi-Person 2D Pose Estimation

End-to-end multi-person 2D pose estimation (MPPE) refers to the family of methods that, given a single image (or a sequence, in the case of video), directly and simultaneously infer the number of people, their spatial extents, and the precise 2D locations of their anatomical keypoints—typically body joints—without reliance on non-differentiable post-processing, heuristic grouping, or explicit person detection as a separate stage. Modern approaches unify keypoint detection and instance association into a single trainable model such that all parameters, losses, and prediction dependencies are co-optimized in a fully differentiable computation graph. This paradigm has subsumed classic top-down and bottom-up separation, yielding a spectrum of architectures that exploit CNNs, RNNs, and, more recently, transformer-based set prediction mechanisms.

## 1. Architectural Taxonomy

End-to-end MPPE systems employ various architectural principles to jointly localize and associate person keypoints:

- **Recurrent Pose Extraction**: The two-stage pipeline introduced by Newell et al. uses a Stacked Hourglass CNN to produce dense joint heatmaps and associative embedding features, which are then consumed by a recurrent ConvLSTM module. At each step, the ConvLSTM predicts disjoint joint heatmaps for a single person and a learned confidence signal for stopping, enabling the network to sequentially “extract” person-wise poses without explicit clustering [1904.11864].

- **Explicit Box Detection**: ED-Pose introduces cascaded explicit box detection, formulating both global person localization and per-joint regression as parameterized axis-aligned box predictions within a unified transformer decoder. The architecture alternates between human detection and keypoint box regression, using L1 and OKS-based losses to supervise both stages in an end-to-end differentiable manner [2302.01593].

- **Centroid-Guided Grouping**: Bottom-up approaches, such as the DS-Hourglass model, predict per-joint heatmaps, centroid heatmaps, and offset fields from each joint to its parent (often a person centroid). A lightweight greedy assignment maps joints to centroids, assembling full poses with minimal overhead and constant memory footprint, preserving end-to-end trainability up to the grouping stage [2012.03316].

- **Transformer-based Set Prediction**: Modern transformer models (e.g., Group Pose, POET, DETRPose) treat the entire scene as a direct set prediction problem. These models are built around a CNN or multi-scale encoder whose output tokens seed a transformer decoder tasked with predicting, for each of N queries, the full set of keypoints for a hypothesized person. Bipartite Hungarian matching and global set-based losses ensure that instance partitioning and keypoint regression are jointly optimized [2103.12115, 2308.07313, 2506.13027].

- **End-to-End Video Pose Estimation**: PAVE-Net demonstrates an extension to the video domain, utilizing a pose-aware spatiotemporal transformer. The model directly aggregates evidence for pose queries across neighboring frames and integrates joint-level refinement, achieving fully end-to-end operation and state-of-the-art accuracy on multi-person video pose benchmarks [2511.13208].

## 2. Loss Functions and Optimization

End-to-end MPPE systems leverage composite multi-task losses designed to supervise joint localization, instance association, and, in some cases, auxiliary cues:

- **Heatmap-Based Detection**: Mean squared error (MSE) between predicted and ground-truth Gaussian heatmaps is standard for dense, per-joint detection [1904.11864, 2012.03316, 1911.10529].

- **Embedding Losses for Association**: Pull-and-push terms penalize embeddings of joints from distinct persons being too similar and encourage proximity for joints of the same person [1904.11864].

- **Bipartite Matching for Set Prediction**: Hungarian assignment is used to align predicted person-wise pose sets with ground-truth instances, with composite costs involving L1/OKS losses on joint coordinates and cross-entropy on presence/no-object classification [2103.12115, 2302.01593, 2308.07313, 2506.13027].

- **Association Losses**: Some methods use spatial offsets and object keypoint similarity (OKS) metrics to align predictions with the evaluation standard (e.g., YOLO-Pose) [2204.06806].

- **Recurrent/Stopping Loss**: RNN-based extractors include a binary cross-entropy loss on stopping confidence, ensuring the network predicts as many poses as there are persons in the scene [1904.11864].

- **Joint Visibility Losses**: Transformers such as POET and DETRPose include joint-specific visibility terms to improve robustness in occlusion [2103.12115, 2506.13027].

## 3. Grouping and Association Strategies

Association of detected keypoints into coherent person-level poses is a central challenge. End-to-end MPPE eliminates heuristic grouping:

- **Recurrent Extraction**: Associations are implicit in the sequence of extracted poses by the ConvLSTM, which conditions on previous predictions and the current spatial context [1904.11864].

- **Keypoint and Instance Queries**: Query-based transformers such as Group Pose and DETRPose assign each query block the role of either a specific joint or an instance, leveraging grouped self-attention to restrict interaction within and across instances and joint types [2308.07313, 2506.13027].

- **Direct Regression with Assignment**: Anchor-free methods like DirectPose regress K keypoints at each location and use non-maximum suppression on composite bounding regions, avoiding explicit posthoc grouping [1911.07451].

- **Box or Centroid-based Schemes**: Some architectures regress boxes for persons and keypoints, aligning outputs using nearest centroid or Hungarian assignment; for example, ED-Pose's explicit box regression jointly optimizes for person and joint localization [2302.01593, 2012.03316].

- **Set Prediction**: POET and related DETR-style models construct the final set of person-wise poses by direct set regression, with the matching process ensuring unique assignments per person [2103.12115, 2506.13027].

## 4. Training and Evaluation Protocols

End-to-end MPPE systems employ rigorous, dataset-driven training regimens:

- **Data Augmentation**: Random cropping, scaling, flipping, and mosaic augmentation are widespread to improve generalization across scales and poses [1904.11864, 2302.01593, 2204.06806, 2506.13027].

- **Batch Size and Memory**: Memory constraints can limit RNN unrolling length or batch size (e.g., ConvLSTM models train with batch=1 and unroll up to 6 steps on MSCOCO due to GPU limits) [1904.11864].

- **Multi-Scale Testing**: Single and multi-scale inference—such as averaging output heatmaps at several image scales—is common for benchmarking; single-shot regressors typically avoid this [1904.11864, 2012.03316, 2204.06806].

- **Optimization Algorithms**: Adam, AdamW, and SGD (with learning-rate schedules and decays) are standard; self-attention models often employ longer training schedules (up to 250 epochs) and per-module learning rates [2308.07313, 2103.12115, 2506.13027].

- **Evaluation Metrics**: Average Precision (AP) at multiple OKS thresholds (e.g., AP, AP^50, AP^75) and average recall (AR) are the principal metrics, with modern methods optimizing directly for OKS [2302.01593, 2204.06806, 2506.13027]. Video models report mAP on benchmarks such as PoseTrack2017 [2511.13208].

## 5. Quantitative Performance and Comparison

Recent end-to-end MPPE methods report strong performance and favorable speed/accuracy trade-offs:

| Method         | Backbone    | COCO AP | CrowdPose AP | FPS/Inference Time      | Reference        |
| -------------- | ----------- | ------- | ------------ | ---------------------- | ---------------- |
| ED-Pose        | RN50/Swin-L | 71.6/75.8 | 69.9/76.6    | 50 ms/image            | [2302.01593]     |
| Group Pose     | RN50/Swin-L | 72.0/74.8 | 74.1         | 68.6 FPS (480×800)     | [2308.07313]     |
| DETRPose-L     | HGNetv2-B4  | 71.2    | 73.3         | 32.5 ms/image          | [2506.13027]     |
| YOLO-Pose      | YOLOv5l6    | 69.4    | —            | Constant                | [2204.06806]     |
| POET           | RN50        | 53.6    | —            | 33 FPS (512² @ batch 1) | [2103.12115]     |
| MultiPoseNet   | RN101       | 69.6    | —            | 23 FPS                  | [1807.04067]     |
| Simple Pose    | Stacked HG  | 68.1    | —            | 38.5 FPS (GPU)          | [1911.10529]     |
| PAVE-Net       | HRNet-W48   | —       | —            | 153 ms/video frame      | [2511.13208]     |

Improvements are tangible in dense, occluded, or crowded scenes, with transformer set-prediction models demonstrating robust grouping and state-of-the-art accuracy without the compute overhead of per-person cropping or sliding windows.

## 6. Strengths, Limitations, and Open Directions

**Strengths:**
- Unified, fully-differentiable learning of keypoint localization and person association [1904.11864, 2302.01593].
- Elimination of heuristic grouping, non-maximum suppression, or external detectors [2302.01593, 2308.07313, 2103.12115].
- Inference runtime is constant with respect to person count in most transformer-based and single-shot models [2204.06806, 2506.13027].
- State-of-the-art accuracy in real-time or near-real-time regimes [2506.13027, 2308.07313].

**Limitations:**
- Memory and compute constraints in recurrent (ConvLSTM) or attention-based systems can restrict batch size and sequence length during training [1904.11864].
- Performance on highly occluded or fine-scale joints can lag explicit high-resolution/heatmap-based systems [2302.01593, 2204.06806].
- Direct set prediction approaches require careful design of losses and matching to avoid duplicate or missing detections [2103.12115, 2506.13027].
- Most approaches focus on single-frame estimation; explicit cross-frame tracking in video remains an active area [2511.13208].

Open directions highlighted include integration of spatial/temporal attention modules, lightweight refinement heads for challenging joints, hybrid representations (box and mask), exploration of memory-efficient backbones for larger batch or sequence sizes, and extension to combined detection-tracking or 3D pose estimation [1904.11864, 2302.01593, 2506.13027, 2511.13208].

## 7. Historical Perspective and Impact

End-to-end MPPE methods have redefined the canonical pose estimation pipeline, transcending the two-stage detect-then-pose paradigm. Early approaches such as DeepCut demonstrated the potential of joint subset partitioning with integer programming, albeit with prohibitive computational cost [1511.06645]. The advent of associative embedding, centroid-based grouping, and supervised recurrent inference models signaled progress toward differentiable, jointly trainable networks [1904.11864, 2012.03316, 1911.07451]. The transformer revolution introduced a direct set prediction perspective, now dominant in the field due to its effectiveness in resolving the ambiguities of joint-to-person association and its scalability to video and whole-body keypoints [2103.12115, 2302.01593, 2308.07313, 2506.13027, 2511.13208]. This paradigm shift has accelerated both the practical deployment and the research frontier of large-scale, real-time, and highly accurate multi-person 2D pose estimation.

Source: https://www.emergentmind.com/topics/end-to-end-multi-person-2d-pose-estimation