MultiPoseNet: Unified Multi-Person Pose Estimation

• MultiPoseNet is a unified bottom-up system that integrates person detection, keypoint detection, segmentation, and pose assembly via a multi-task backbone and a novel PRN module.
• It leverages ResNet with FPN and separate subnets to achieve state-of-the-art accuracy (69.6 mAP on COCO) while maintaining real-time processing speeds (23 FPS).
• Ablation studies demonstrate PRN's effectiveness in crowded scenes and suggest future enhancements with stronger backbones and temporal consistency.

MultiPoseNet is a bottom-up, multi-person 2D pose estimation architecture designed to perform person detection, keypoint detection, person segmentation, and pose assembly in a unified, end-to-end framework. It introduces a novel assignment approach—the Pose Residual Network (PRN)—which assigns detected keypoints to person instances by refining heatmap predictions, delivering state-of-the-art accuracy and real-time inference speeds. MultiPoseNet jointly leverages a multi-task backbone with separate subnets for different tasks and couples them with the PRN for robust pose grouping, especially in crowded scenes (Kocabas et al., 2018).

1. Architectural Overview

MultiPoseNet utilizes a shared convolutional backbone with feature pyramid networks (FPN) to extract rich multiscale features, processed in parallel by two subnets: one dedicated to keypoint detection and person segmentation, and the other to person detection. The outputs from the keypoint and detection subnets are further handled by the Pose Residual Network, which resolves the assignment of detected joints to individual person boxes.

Main architectural components:

• Backbone: ResNet-50 or ResNet-101, with FPN attached to C₂–C₅ layers (with strides 4, 8, 16, 32, respectively).
• Keypoint & Segmentation Subnet: Receives features from its FPN; processes them through cascaded convolutional layers and outputs (K + 1) heatmaps, where K is the number of keypoints and 1 is for person segmentation.
• Person Detection Subnet: Utilizes a RetinaNet one-stage detector, taking features from its copy of FPN.
• Pose Residual Network: For each detected bounding box, crops and resizes heatmaps, and refines them to produce accurate assignment of keypoints to persons using a residual MLP structure.

The pipeline is illustrated as:

Input image ↓ ResNet + FPN ──► Keypoint & Segmentation subnet ──► Heatmaps │ └──► Person‑detection subnet ──► Boxes ↓ ↓ Crop & resize heatmap RoIs per box ↓ Pose Residual Network ↓ Final poses (joint coordinates per person)

2. Multi-Task Learning Paradigm

MultiPoseNet’s multi-task backbone is designed to jointly address:

• Person detection: Predicts bounding boxes for each person with class scores and bounding box regressions, supervised via a combination of focal loss (classification) and smooth-L₁ loss (bounding box regression).
• Keypoint heatmap estimation: Predicts per-pixel Gaussian heatmaps for K keypoints; supervised using weighted mean square error.
• Person segmentation: Generates a binary mask channel; supervised in the same loss calculation as keypoints.

Losses are aggregated:

$$L_{total} = \lambda_{det} L_{det} + \lambda_{kp} L_{kp}$$

In practice, separate sequential training phases are adopted, avoiding explicit λ weighting.

Intermediate supervision is applied on each feature map prior to their concatenation in the keypoint & segmentation subnet, which empirically improves accuracy.

3. Pose Residual Network (PRN) and Keypoint Assignment

Traditional bottom-up methods for pose grouping, such as Part Affinity Fields or Associative Embeddings, employ pairwise or unary relationships. PRN generalizes these by learning a full $K$-order correction. For each detected person bounding box, the PRN:

• Crops the corresponding $(K+1)$ heatmap channels to a fixed spatial size $(W, H) = (36, 56)$.
• For input tensor $$X = \{x_1, ..., x_K\} \in \mathbb{R}^{K \times W \times H}$$, a fully-connected layer (1024 units, ReLU, dropout) processes the flattened input vector.
• Residual output: For each keypoint channel,

$y_k = \phi_k(X) + x_k$

where $\phi_k$ denotes the MLP’s correction for channel $k$.

• Outputs pass through a spatial softmax, enforcing each output channel (joint type) to have a unique peak.
• Binary cross-entropy per-pixel loss against one-hot ground truth labels supervises PRN:

$$L_{prn} = -\sum_{k=1}^K \sum_{i=1}^W \sum_{j=1}^H y_k^*(i,j) \log \hat{y}_k(i,j)$$

By learning global pose configurations for all joints of a person, PRN resolves challenging assignment issues in crowded and overlapping scenarios, yielding higher localization accuracy than lower-order grouping approaches.

4. Training Regimen and Augmentation

MultiPoseNet is trained on the COCO 2017 keypoints dataset ($64\text{K}$ train, $2.7\text{K}$ val, $20\text{K}$ test-dev) without extra data. Training incorporates:

• Augmentation: Random rotations (±40°), scale jitter (0.8–1.2), horizontal flipping ($p=0.3$).
• Optimization protocol:
  • Keypoint subnet: Adam, $lr = 1\times10^{-4}$ with decay on plateau.
  • Person subnet: Backbone frozen, Adam, $lr = 1\times10^{-5}$.
  • PRN: Cropped heatmaps and ground truth boxes, Adam, $lr = 1\times10^{-4}$, convergence in ~1.5h.
• The two-phase training regime (first keypoint subnet, then person subnet) avoids complications due to disparate convergence rates and scale of different losses.

5. Empirical Performance and Comparative Analysis

MultiPoseNet demonstrates significant advances in accuracy and speed across multiple benchmarks:

Method	Modality	COCO mAP	Speed (FPS)
CMU-Pose (Cao)	bottom-up	61.8	10
Assoc. Embeddings	bottom-up	65.5	6
MultiPoseNet	bottom-up	69.6	23
Mask R-CNN	top-down	69.2	5
Megvii	top-down	73.0	-

MultiPoseNet outperforms prior bottom-up approaches by at least +4 mAP on COCO, matches leading top-down methods in accuracy, and achieves a real-time speed of 23 FPS on average (ResNet-50, 384×576 input, 3 people/image). The PRN module contributes substantially to grouping performance: On ground truth keypoints and boxes, the residual MLP yields 89.4 AP, while the end-to-end system achieves 69.6 AP. Person segmentation and detection performance are also competitive, with 87.8 IoU on PASCAL VOC '12 and 52.5 AP on COCO, respectively.

6. Design Ablations and Insights

A suite of ablation studies elucidates the contribution of different architectural choices:

• Backbone depth: ResNet-101 and dilated variants provide modest AP gains over ResNet-50.
• Keypoint architecture: Removing intermediate supervision or multi-scale concatenation degrades AP by up to 2.6.
• PRN assignment: Removing the residual MLP structure or using lower-order grouping (UCR, Max-selection) substantially worsens grouping quality.
• Speed: PRN inference is efficient (~2 ms/person), and the overall model retains real-time throughput.

Notably, keypoint localization remains a bottleneck; PRN performance on ground truth inputs is substantially higher than end-to-end results, indicating upstream heatmap localization errors as a limiting factor.

7. Limitations and Prospective Extensions

Current limitations include reduced robustness in extreme occlusion or dense crowding scenarios, and potential missed joint optimization benefits due to the two-phase training schedule. Future directions proposed include:

• Strengthening the backbone (e.g., ResNeXt, EfficientNet), deeper or convolutional PRN architectures to further leverage spatial context.
• Incorporation of temporal consistency for videos.
• Extending the paradigm to 3D pose estimation, action recognition, AR/VR, and robotics.

A plausible implication is that design choices in learned, high-order grouping (e.g., PRN) will continue to drive advances in robust bottom-up multi-person pose estimation, particularly as upstream features and assignment mechanisms are further co-optimized (Kocabas et al., 2018).