SmoothNet: A Plug-and-Play Network for Refining Human Poses in Videos

Abstract: When analyzing human motion videos, the output jitters from existing pose estimators are highly-unbalanced with varied estimation errors across frames. Most frames in a video are relatively easy to estimate and only suffer from slight jitters. In contrast, for rarely seen or occluded actions, the estimated positions of multiple joints largely deviate from the ground truth values for a consecutive sequence of frames, rendering significant jitters on them. To tackle this problem, we propose to attach a dedicated temporal-only refinement network to existing pose estimators for jitter mitigation, named SmoothNet. Unlike existing learning-based solutions that employ spatio-temporal models to co-optimize per-frame precision and temporal smoothness at all the joints, SmoothNet models the natural smoothness characteristics in body movements by learning the long-range temporal relations of every joint without considering the noisy correlations among joints. With a simple yet effective motion-aware fully-connected network, SmoothNet improves the temporal smoothness of existing pose estimators significantly and enhances the estimation accuracy of those challenging frames as a side-effect. Moreover, as a temporal-only model, a unique advantage of SmoothNet is its strong transferability across various types of estimators and datasets. Comprehensive experiments on five datasets with eleven popular backbone networks across 2D and 3D pose estimation and body recovery tasks demonstrate the efficacy of the proposed solution. Code is available at https://github.com/cure-lab/SmoothNet.

• The paper introduces SmoothNet, a temporal-only refinement network that cuts acceleration error by 86.88% and improves mean per joint position error by 8.82% in video pose estimation.
• SmoothNet uses a motion-aware fully-connected network to independently model joint position, velocity, and acceleration, effectively reducing jitter without relying on spatial features.
• Its plug-and-play design offers robust transferability across multiple estimators and datasets, enhancing video analysis in fields such as motion capture and human-computer interaction.

SmoothNet: Refining Human Poses in Video Analysis

Introduction

The advancement in human pose estimation has spurred its application in several domains, such as motion analysis and human-computer interaction. However, traditional pose estimators often encounter significant jitters, particularly in frames with occluded or rarely seen poses, leading to untrustworthy motions. This paper introduces "SmoothNet," a novel temporal-only refinement network that mitigates such jitters, improving the reliability and accuracy of pose estimations.

Technical Approach

SmoothNet distinguishes itself from existing solutions by employing a temporal-only refinement approach. Traditional methods often leverage spatio-temporal models, which can be suboptimal due to the inherent difficulty of jointly optimizing for precision and temporal smoothness. SmoothNet, however, learns long-range temporal relations for each joint independently, avoiding interference from noisy spatial correlations that usually accompany jittery frames.

The architecture of SmoothNet is based on a motion-aware fully-connected network. Three key motion components—position, velocity, and acceleration—are modeled explicitly, facilitating the network’s ability to learn smoothness characteristics in body movements effectively. This design choice contrasts with convolutional or transformer-based approaches, ensuring focus solely on temporal continuity, thus providing better handling of long-term and significant jitters.

Experimental Analysis

SmoothNet’s efficacy is validated through comprehensive experiments across five datasets and eleven popular backbone networks, spanning both 2D and 3D pose estimation as well as human mesh recovery tasks. The results demonstrate a significant reduction in jitter, with an impressive 86.88% decrease in Accel (acceleration error) and an 8.82% improvement in MPJPE (mean per joint position error) compared to the original outputs of evaluated pose estimators.

A remarkable advantage of SmoothNet is its plug-and-play nature, demonstrating strong transferability across different estimators and datasets, a benefit stemming from its temporal-only design.

Key Findings

1. Jitter Mitigation: SmoothNet substantially outperforms several traditional low-pass filters and state-of-the-art temporal refinement networks, indicating its superior capability in handling massive and persistent jitters without spatial feature dependency.
2. Transferability and Generalization: As a temporal-only model, it excels in maintaining performance across various backbones and datasets, demonstrating its robust generalization capabilities.
3. Enhancement of Challenging Frames: By improving temporal smoothness, SmoothNet also indirectly enhances frame estimation accuracy, particularly in sequences characterized by occlusions or rare poses.

Implications and Future Work

The refined estimation results offered by SmoothNet have substantial implications for improving system reliability in applications involving complex human motions. Its plug-and-play nature facilitates its integration into existing systems, providing immediate benefits without significant architectural overhauls.

Future research avenues could explore the adaptation of SmoothNet for real-time systems, addressing the current limitation of being a non-causal, sliding-window-based approach. Additionally, extending SmoothNet's principles to other related tasks like pose tracking and multi-object tracking could further solidify its utility in broader AI applications for motion capture and analysis.

Conclusion

SmoothNet presents a significant contribution to the field of human pose estimation by addressing pervasive issues with pose jitter. Through a targeted temporal refinement approach, this work enhances both the smoothness and precision of pose estimates, setting a new standard for robust video-based human motion analysis.