Combining detection and tracking for human pose estimation in videos (2003.13743v1)

Abstract: We propose a novel top-down approach that tackles the problem of multi-person human pose estimation and tracking in videos. In contrast to existing top-down approaches, our method is not limited by the performance of its person detector and can predict the poses of person instances not localized. It achieves this capability by propagating known person locations forward and backward in time and searching for poses in those regions. Our approach consists of three components: (i) a Clip Tracking Network that performs body joint detection and tracking simultaneously on small video clips; (ii) a Video Tracking Pipeline that merges the fixed-length tracklets produced by the Clip Tracking Network to arbitrary length tracks; and (iii) a Spatial-Temporal Merging procedure that refines the joint locations based on spatial and temporal smoothing terms. Thanks to the precision of our Clip Tracking Network and our merging procedure, our approach produces very accurate joint predictions and can fix common mistakes on hard scenarios like heavily entangled people. Our approach achieves state-of-the-art results on both joint detection and tracking, on both the PoseTrack 2017 and 2018 datasets, and against all top-down and bottom-down approaches.

• The paper demonstrates that integrating detection and tracking in a top-down framework significantly improves pose estimation accuracy.
• It introduces a Clip Tracking Network using temporal convolutions to predict body joints even with weak detections in occluded or complex poses.
• A spatial-temporal merging procedure refines joint predictions for consistent and robust performance across video sequences.

Combining Detection and Tracking for Human Pose Estimation in Videos

This paper presents a novel top-down approach that addresses the challenges of multi-person human pose estimation and tracking in video sequences. The approach decisively shifts away from the conventional limitations imposed by relying solely on frame-based person detections. Instead, it leverages temporal information to enhance the robustness and accuracy of pose estimation, explicitly addressing cases where traditional person detectors may struggle, such as heavily occluded or closely interacting individuals.

The proposed method is constituted by three main components: the Clip Tracking Network, the Video Tracking Pipeline, and a Spatial-Temporal Merging procedure. The Clip Tracking Network is designed to perform joint detection and tracking tasks concurrently within small video clips, employing a high-capability detection network based on an extended version of the High-Resolution Network (HRNet). This novel architecture utilizes temporal convolutions to enhance temporal reasoning, allowing the network to predict body joint locations even in frames where the person is not explicitly detected. The effectiveness of this predictive model is particularly evident in scenarios characterized by occlusions and atypical poses, where spatial cues alone are necessary but insufficient.

The Video Tracking Pipeline operates by merging the fixed-length tracklets output by the Clip Tracking Network into tracks covering arbitrary video lengths. This system bridges tracklets when overlapping frames provide sufficient spatial cues to suggest these segments belong to the same individual, effectively creating continuous tracks over the video sequence. The capability to extend tracks and compensate for missed detections enables a flexible, resilient response to dynamic changes and complications like motion blur and varying viewpoints.

The Spatial-Temporal Merging procedure refines the joint predictions by leveraging spatial and temporal smoothing techniques. This step attributes a high priority to consistency, ensuring that joint locations are not only accurate frame-wise but also maintain coherence through the video timeline. This procedure enhances precision, particularly resolving ambiguities arising in crowded scenes where interactions between individuals increase the difficulty of reliably determining individual poses.

On validation and test sets of the PoseTrack 2017 and 2018 datasets, this approach achieves state-of-the-art results against both top-down and bottom-up methods. It boasts a reduction in joint detection errors by 28% and boosts tracking performance by 9% compared to the leading competing methodologies. Such robust improvements highlight the effectiveness of integrating detection and tracking within a cohesive system that capitalizes on smooth transitions enabled by temporal information.

The paper also presents a comprehensive ablation paper to validate the contributions of each component and the interplay of key hyperparameters. Analyzing the impact of the Clip Tracking Network on video-length track construction and delineating the role of temporal information integration was pivotal to establishing a clear performance metric across scenarios characterized by traditionally high error rates and challenges.

The implications of this research extend into both practical applications and theoretical advancements within computer vision. Practically, the integration of robust detection with thoughtful temporal association holds potential for development in video surveillance, activity recognition, and augmented reality systems. Theoretically, it challenges the paradigm of static frame analysis by underscoring the critical role of temporal dynamics in achieving higher-order understanding and persistence across scenes. Future advancements in this direction may explore more granular temporal models and devise even richer associative algorithms to further expand the depth and breadth of pose estimation in increasingly complex scenarios.