Detection, Pose Estimation and Segmentation for Multiple Bodies: Closing the Virtuous Circle (2412.01562v2)

Abstract: Human pose estimation methods work well on isolated people but struggle with multiple-bodies-in-proximity scenarios. Previous work has addressed this problem by conditioning pose estimation by detected bounding boxes or keypoints, but overlooked instance masks. We propose to iteratively enforce mutual consistency of bounding boxes, instance masks, and poses. The introduced BBox-Mask-Pose (BMP) method uses three specialized models that improve each other's output in a closed loop. All models are adapted for mutual conditioning, which improves robustness in multi-body scenes. MaskPose, a new mask-conditioned pose estimation model, is the best among top-down approaches on OCHuman. BBox-Mask-Pose pushes SOTA on OCHuman dataset in all three tasks - detection, instance segmentation, and pose estimation. It also achieves SOTA performance on COCO pose estimation. The method is especially good in scenes with large instances overlap, where it improves detection by 39% over the baseline detector. With small specialized models and faster runtime, BMP is an effective alternative to large human-centered foundational models. Code and models are available on https://MiraPurkrabek.github.io/BBox-Mask-Pose.

• The paper presents the BMP method that combines detection, segmentation, and pose estimation into a unified feedback loop for enhanced performance.
• It leverages mask-based conditioning to accurately separate overlapping instances, significantly improving pose accuracy in crowded scenes.
• BMP outperforms state-of-the-art methods on OCHuman and COCO, achieving notable gains in bounding box and mask average precision.

Detection, Pose Estimation, and Segmentation for Multi-Body Scenarios

The paper "Detection, Pose Estimation and Segmentation for Multiple Bodies: Closing the Virtuous Circle" by Miroslav Purkrabek and Jiri Matas presents a significant contribution to the field of human pose estimation (HPE), particularly in the context of multi-body scenarios. The authors address the inherent challenges associated with overlapping people in images, where traditional methods often struggle with merged bounding boxes or collapsed poses.

At the core of this research is the development of the BBox-Mask-Pose (BMP) method, which synergistically combines detection, segmentation, and pose estimation within a feedback loop that incrementally improves the accuracy of each component. This integration leverages the strengths of both top-down and detector-free methods.

Key Contributions and Methodology

1. Mask-Based Conditioning: A primary innovation lies in conditioning pose estimation by segmentation masks rather than bounding boxes. This change enhances instance separation in dense scenes and leads to more robust pose estimation.
2. BMP Feedback Loop: The authors propose a feedback loop that integrates three specialized models - an enhanced RTMDet for detection, a new MaskPose model for pose estimation, and SAM2 for segmentation. The loop operates such that:
   • The detector identifies and ignores masked instances.
   • MaskPose estimates poses using segmentation masks, which improves robustness in crowded scenes.
   • SAM2 refines segmentation using pose keypoints, enhancing overall segmentation performance.
3. Performance and Evaluation: On the OCHuman and COCO datasets, BMP demonstrated superior performance compared to state-of-the-art methods. For instance, with only moderately-sized models, BMP matches or surpasses the effectiveness of top-down methods on the COCO dataset and detector-free methods on the OCHuman dataset.

Numerical Results

The paper highlights strong numerical outcomes, particularly noting a significant improvement in handling multi-body scenarios. For example, the detection and segmentation on OCHuman test data achieved a bounding box AP of 31.3 and a mask AP of 32.4 with BMP after two iterations, compared to the baseline RTMDet-l model.

Challenges and Limitations

Despite its advancements, BMP encounters certain limitations, particularly in cases of over-segmentation where the model might segment only skin, missing the clothing, due to incorrect keypoint prompts. This issue often arises in scenarios where the attire or the position leads to misinterpretations by SAM2, emphasizing the need for enhanced SAM prompting strategies.

Future Directions

The research offers a platform for various future improvements in AI:

• Enhanced Automatic Prompting: Developing better automatic prompting techniques for SAM could further fine-tune segmentation outputs, especially in ambiguous visual contexts.
• Incorporation of Larger Models: Leveraging larger models could potentially boost BMP's performance, particularly in extremely crowded scenarios or complex interactions.
• Integration with Other HPE Methods: BMP's modular structure allows for integration with other pose refinement techniques, like BUCTD, to achieve even finer results.

In conclusion, the BMP method represents an incremental yet significant advance in the successful integration of detection, pose estimation, and segmentation for multi-body scenarios, providing a valuable tool for applications ranging from action recognition to medical imaging. As research progresses, further refinements and adaptations could enhance BMP's applicability and robustness across a wider range of complex, real-world environments.