Dual-stream Spatio-Temporal GCN-Transformer Network for 3D Human Pose Estimation

Abstract: 3D human pose estimation is a classic and important research direction in the field of computer vision. In recent years, Transformer-based methods have made significant progress in lifting 2D to 3D human pose estimation. However, these methods primarily focus on modeling global temporal and spatial relationships, neglecting local skeletal relationships and the information interaction between different channels. Therefore, we have proposed a novel method,the Dual-stream Spatio-temporal GCN-Transformer Network (MixTGFormer). This method models the spatial and temporal relationships of human skeletons simultaneously through two parallel channels, achieving effective fusion of global and local features. The core of MixTGFormer is composed of stacked Mixformers. Specifically, the Mixformer includes the Mixformer Block and the Squeeze-and-Excitation Layer ( SE Layer). It first extracts and fuses various information of human skeletons through two parallel Mixformer Blocks with different modes. Then, it further supplements the fused information through the SE Layer. The Mixformer Block integrates Graph Convolutional Networks (GCN) into the Transformer, enhancing both local and global information utilization. Additionally, we further implement its temporal and spatial forms to extract both spatial and temporal relationships. We extensively evaluated our model on two benchmark datasets (Human3.6M and MPI-INF-3DHP). The experimental results showed that, compared to other methods, our MixTGFormer achieved state-of-the-art results, with P1 errors of 37.6mm and 15.7mm on these datasets, respectively.

• The paper introduces MixTGFormer, a novel dual-stream Spatio-Temporal Graph Convolutional Network (GCN)-Transformer architecture that proficiently captures both local skeletal dependencies and global spatio-temporal relationships for 3D human pose estimation.
• MixTGFormer employs specialized Spatial and Temporal Mixformer Blocks, integrating GCNs and Multi-Head Self-Attention in distinct streams, which are then adaptively fused with a Squeeze-and-Excitation Layer for robust feature refinement and improved accuracy.
• Achieving state-of-the-art results on Human3.6M (P1 error: 37.6mm) and MPI-INF-3DHP datasets (AUC: 85.4%, MPJPE: 16.5mm), MixTGFormer demonstrates significant advancements over existing methods, validating its hybrid modeling approach for 2D-to-3D pose lifting.
• follow_up_questions
• How does MixTGFormer's dual-stream architecture specifically balance computational efficiency with accuracy compared to single-stream models?
• What are the limitations of MixTGFormer when dealing with severe occlusions or highly ambiguous 2D input poses?
• Could the Squeeze-and-Excitation Layer be replaced or augmented with other channel attention mechanisms to further enhance performance, and if so, which ones?
• How well does MixTGFormer generalize to real-world, unconstrained environments outside of the benchmark datasets, and what adaptations would be necessary?
• Find recent papers about efficient 3D human pose estimation.

Dual-stream Spatio-Temporal GCN-Transformer Network for 3D Human Pose Estimation

The paper "Dual-stream Spatio-Temporal GCN-Transformer Network for 3D Human Pose Estimation" (2604.17688) introduces MixTGFormer, a novel architecture for 3D human pose estimation that addresses the limitations of existing Transformer-based and Graph Convolutional Network (GCN)-based methods. Traditional Transformer models excel at capturing global spatio-temporal relationships but often neglect local skeletal dependencies and inter-channel information interaction. Conversely, GCNs are proficient in modeling local graph structures but have restricted capabilities for perceiving long-range and global information. MixTGFormer is designed to synergistically combine the strengths of both paradigms by integrating GCNs into a Transformer framework, augmented with a Squeeze-and-Excitation (SE) Layer for enhanced channel-wise feature refinement.

Architectural Innovations

The core of MixTGFormer is the proposed Mixformer module, which comprises stacked Mixformer Blocks and an SE Layer. The Mixformer Block is a hybrid architecture that adaptively fuses features from both Graph Convolutional Networks and Multi-Head Self-Attention (MHSA) mechanisms. This integration is crucial for simultaneously capturing local joint relationships, akin to how GCNs operate on graph-structured data, and global spatio-temporal dependencies, which are the hallmark of Transformer's self-attention.

The Mixformer Block is instantiated in two specialized forms: the Spatial Mixformer Block and the Temporal Mixformer Block. The Spatial Mixformer Block employs Spatial MHSA (S-MHSA) and Spatial GCN (S-GCN) to model relationships between joints within a single frame, treating individual joints as tokens. S-MHSA captures global spatial correlations, while S-GCN specifically learns local skeletal adjacency patterns. The Temporal Mixformer Block, conversely, utilizes Temporal MHSA (T-MHSA) and Temporal GCN (T-GCN) to model relationships between consecutive frames in a sequence, with each frame considered a token. T-MHSA captures long-range temporal dependencies, and T-GCN focuses on local temporal transitions within the pose sequence.

A key design choice in MixTGFormer is its dual-stream architecture. Two parallel computational branches are formed by stacking Mixformer Blocks in reverse spatio-temporal order. This configuration ensures comprehensive modeling, with each branch emphasizing different spatio-temporal aspects, facilitating a richer fusion of information. The features extracted from these two streams are then adaptively fused using learnable weights, allowing the model to dynamically balance the contributions of different spatio-temporal modeling emphases.

Following the adaptive fusion, a Squeeze-and-Excitation Layer is integrated. This layer functions as a channel-wise self-attention mechanism by adaptively recalibrating channel-wise feature responses. It globally pools feature maps to compute channel statistics, then uses a small Multi-Layer Perceptron (MLP) to generate channel-specific weights, which are subsequently applied to the input features. This process enhances the model's ability to selectively emphasize salient features across channels, compensating for potential information loss or disregard in earlier stages, especially concerning intricate local dependencies or global contextual cues.

The overall architecture takes a 2D input sequence of keypoints with confidence scores, projects it to a high-dimensional feature space, adds learnable spatial position encoding, processes it through stacked Mixformers, and finally regresses to 3D human poses. The training objective incorporates both a 3D position loss ($L_{3D}$) and an acceleration loss ($L_{AA}$) to ensure accurate joint localization and motion smoothness, alongside a 2D loss ($L_{2D}$) derived from the 2D pose detector.

Experimental Validation and Performance

MixTGFormer's efficacy was rigorously evaluated on two benchmark datasets for 3D human pose estimation: Human3.6M and MPI-INF-3DHP. The model's performance was measured using common metrics such as Mean Per Joint Position Error (MPJPE, P1), Procrustes-aligned MPJPE (P-MPJPE, P2), Percentage of Correct Keypoints (PCK), and Area Under the Curve (AUC).

On the Human3.6M dataset, MixTGFormer achieved state-of-the-art results. The MixTGFormer model reported a P1 error of 37.6mm and a P2 error of 31.8mm. This represents a notable improvement over prior competitive methods, such as DSTFormer (2604.17688), which achieved 37.9mm for P1. The paper also highlights strong performance in the real P1 error with 2D ground truth (P1+), achieving 16.4mm, demonstrating the model's robustness to upstream 2D detection inaccuracies.

For the MPI-INF-3DHP dataset, MixTGFormer continued to demonstrate superior performance. It achieved an AUC of 85.4% and an MPJPE of 16.5mm, surpassing the current best reported results by 1.2% in AUC and reducing MPJPE by 1.7mm compared to MotionAGFormer (2604.17688). The PCK metric was also competitive at 98.5%.

Ablation studies provided crucial insights into the architectural contributions:

• Mixformer Block Composition: The combination of MHSA and GCN within each Mixformer Block consistently outperformed configurations using only MHSA ("DoubleAttention") or only GCN ("DoubleGCN"), reducing P1 error by 1.0mm and 0.4mm respectively, underscoring the benefits of hybrid local-global feature capture.
• Connection Order of Mixformer Blocks: The dual-stream design with cross-connected spatio-temporal and temporal-spatial orders (S->T and T->S) achieved the lowest P1 error of 37.9mm, validating the importance of diverse spatio-temporal modeling pathways.
• Encoding Embeddings: The use of spatial encoding embeddings significantly reduced the P1 error by 1.0mm compared to temporal embeddings, indicating the critical role of spatial context. The combination of both temporal and spatial encoding led to a slight increase in P1 error, suggesting specific tuning might be required for optimal integration.
• SE Layer Insertion: Placing the SE Layer between adaptive fusion and the fully connected layer yielded the best performance, resulting in a 0.3mm reduction in P1 error compared to not using it, confirming its role in refining channel-wise information after feature fusion.

Implications and Future Directions

The MixTGFormer architecture represents a significant advancement in 3D human pose estimation, especially within the 2D-to-3D lifting paradigm. Its ability to effectively integrate and interact local skeletal relationships (via GCNs) with global spatio-temporal dependencies (via Transformers) offers a robust solution to a long-standing challenge in the field. The strong numerical results on widely accepted benchmarks demonstrate a clear advantage over previous methods, validating the efficacy of the dual-stream fusion and the enhanced channel attention mechanism.

The theoretical implications center on the successful hybrid modeling of graph-structured data and sequential data, suggesting that a principled combination of specialized architectures can overcome the inherent limitations of individual approaches. This work reinforces the idea that an explicit consideration of both local graph topology and global contextual relationships is paramount for accurate human pose understanding.

Practically, the advancements in 3D human pose estimation have direct benefits across various applications. In motion analysis, improved accuracy can lead to more precise biomechanical assessments. For virtual and augmented reality, more realistic and responsive avatar movements become possible. In activity recognition, a better understanding of 3D poses can enhance the reliability of recognizing complex human actions.

Future developments in AI concerning 3D human pose estimation could build upon the MixTGFormer's foundation by exploring adaptive graph structures that dynamically change based on joint confidence or activity, further refining the local modeling. Additionally, research might focus on optimizing the architectural complexity to enable real-time deployment on resource-constrained devices, potentially through network pruning, quantization, or more efficient attention mechanisms. Expanding the model's robustness to diverse environmental conditions, occlusions, and multi-person scenarios, perhaps through self-supervised learning on larger, unannotated datasets, also represents a promising avenue for continued research.