From Sparse to Dense: Spatio-Temporal Fusion for Multi-View 3D Human Pose Estimation with DenseWarper

Abstract: In multi-view 3D human pose estimation, models typically rely on images captured simultaneously from different camera views to predict a pose at a specific moment. While providing accurate spatial information, this traditional approach often overlooks the rich temporal dependencies between adjacent frames. We propose a novel 3D human pose estimation input method: the sparse interleaved input to address this. This method leverages images captured from different camera views at various time points (e.g., View 1 at time $t$ and View 2 at time $t+δ$), allowing our model to capture rich spatio-temporal information and effectively boost performance. More importantly, this approach offers two key advantages: First, it can theoretically increase the output pose frame rate by N times with N cameras, thereby breaking through single-view frame rate limitations and enhancing the temporal resolution of the production. Second, using a sparse subset of available frames, our method can reduce data redundancy and simultaneously achieve better performance. We introduce the DenseWarper model, which leverages epipolar geometry for efficient spatio-temporal heatmap exchange. We conducted extensive experiments on the Human3.6M and MPI-INF-3DHP datasets. Results demonstrate that our method, utilizing only sparse interleaved images as input, outperforms traditional dense multi-view input approaches and achieves state-of-the-art performance. The source code for this work is available at: https://github.com/lingli1724/DenseWarper-ICLR2026

• The paper introduces a novel sparse interleaved multi-view input paradigm that leverages asynchronous data to achieve temporal super-resolution in 3D pose estimation.
• It employs DenseWarper’s two-stage architecture, integrating epipolar geometry for spatial fusion with deformable temporal convolutions to correct misalignments.
• The approach shows significant accuracy and efficiency improvements over dense methods across benchmarks like Human3.6M and MPI-INF-3DHP, reducing MPJPE by up to 38.2%.

Spatio-Temporal Fusion for Multi-View 3D Human Pose Estimation with Sparse Interleaved Input

Introduction

Multi-view 3D human pose estimation conventionally depends on synchronously acquired, densely sampled images from multiple cameras at each temporal instance. While effective for spatial keypoint localization, this approach incurs redundant computation, underutilizes temporal correlations, and is fundamentally limited by the individual camera frame rate. The paper "From Sparse to Dense: Spatio-Temporal Fusion for Multi-View 3D Human Pose Estimation with DenseWarper" (2605.14525) introduces an alternative paradigm—sparse interleaved multi-view input—enabling temporally offset, view-wise asynchronous sampling to exploit richer spatio-temporal context. This paradigm, instantiated in the Dense Warper architecture, fuses sparse, interleaved visual observations into spatio-temporally coherent pose predictions, offering both higher temporal resolution and efficiency.

Sparse Interleaved Input Paradigm

The core innovation is the input strategy: each camera captures frames at offset time steps, such that at each prediction moment, input images originate from different views and different times. Within an $M$-camera system, inputs for each inference consist of $M$ frames, each from a different camera and a different time instance. This paradigm enables two crucial advancements:

• Temporal Super-Resolution: By interleaving camera acquisition, pose predictions can be generated at an effective frame rate of $M \times F$, where $F$ is the per-camera frame rate, thus breaching the fundamental single-view frame rate ceiling.
• Data Redundancy Mitigation: Selective, sparse temporal sampling reduces redundant computation and storage, as not all cameras need to acquire and process every frame at all time steps.

This input setting inherently introduces missing information and spatio-temporal misalignments, which are addressed by specialized model components.

Dense Warper Architecture

Dense Warper is a two-stage, end-to-end framework orchestrating spatial and temporal fusion to produce dense 3D pose outputs from sparse, asynchronous inputs.

Epipolar Geometry-Based Spatial Fusion

For each interleaved set of 2D keypoint heatmaps (one per view per time), the spatial fusion module rectifies imputed heatmaps across frames via epipolar constraints. Specifically:

• Heatmap Replication: Missing heatmaps are initialized via replication across frames, introducing temporal and spatial drift.
• Cross-View Correction: Epipolar geometry is employed to sweep along corresponding lines in other views: for each heatmap position, the peak response along its epipolar line in all other views is integrated (using the Sampson approximation to ensure geometric consistency), enforcing multi-view spatial alignment.
• Effect: This yields a spatially-consistent, rectified heatmap tensor, even with incomplete temporal and view coverage.

Warper Module for Temporal Fusion

Spatially fused heatmaps remain temporally incoherent due to staggered sampling. The Warper module addresses this via:

• Residual Deformable Temporal Convolutions: For each time frame, differences between spatially-fused and observed heatmaps are processed through multiple stacked residual blocks and multi-rate dilated/deformable convolutions, inferring temporal offsets at each pixel.
• Deformable Warping and Aggregation: Offset maps guide non-linear warping of heatmaps, compensating for inter-frame and inter-view temporal misalignments. Multiple temporally-aligned heatmaps are then aggregated by summation.
• Triangulation: The final dense, spatio-temporally aligned heatmaps are triangulated to yield 3D skeleton predictions.

This layered approach enables robust pose estimation even when inputs are sparsely sampled and temporally staggered across views.

Quantitative Evaluation and Empirical Claims

Datasets and Benchmarks

Comprehensive experiments were conducted on Human3.6M and MPI-INF-3DHP, using both ideal (GT 2D), CPN, and SimpleBaseline pose detectors as 2D keypoint inputs.

Numerical Results

• Human3.6M:
  • With GT 2D input, the approach achieves an average MPJPE of 21.3mm, improving by 10.1% over multi-view dense methods such as Adafuse (23.7mm), and by 25.2% over leading single-view methods.
  • Using SimpleBaseline detections, the method further achieves 22.3mm MPJPE, outperforming Adafuse’s 28.1mm.
  • For P-MPJPE (relative joint accuracy), average error is 19.4mm, yielding up to 6.3% improvement over existing baselines, with best-in-class accuracy on 13 out of 15 action categories.
• MPI-INF-3DHP:
  • On this challenging in-the-wild dataset, DenseWarper attains 65.89mm MPJPE using SimpleBaseline, surpassing single-view and multi-view baselines including Adafuse (78.57mm) and PPT (106.30mm).
• Model Efficiency (Human3.6M):
  • DenseWarper deploys 76.51M parameters and achieves a computational latency of 44.51ms, substantially outperforming larger competitors such as FinePose (269.23M params, 82.24ms latency).
  • Best performance-efficiency ratio: MPJPE per MB is reduced to 0.291, compared to GLA-GCN’s 0.624.

Bold/Contradictory Claims: The paper claims, with empirical evidence, that sparse, asynchronous, interleaved multi-view input—not dense synchronization—is sufficient to support state-of-the-art multi-view 3D human pose estimation. Moreover, temporal super-resolution beyond the native camera frame rate is attainable without degradation in pose accuracy or temporal consistency.

Ablation and Limitations

Ablation studies disentangle the respective contributions of spatial fusion (epipolar geometry) and temporal warping: each module alone confers a marked improvement, with the combination driving a 38.2% error reduction versus baseline on Human3.6M, and 30.25% on MPI-INF-3DHP. The pipeline’s efficacy diminishes with excessively large or non-uniform inter-view temporal gaps, as spatio-temporal correlation extraction becomes unreliable in highly sparse regimes.

Theoretical and Practical Implications

• Theoretical: The results fundamentally question the necessity of temporally dense, synchronized inputs in multi-view 3D estimation, asserting that judicious spatio-temporal sampling—coupled with geometric and temporal alignment mechanisms—can suffice or even outperform traditional paradigms.
• Practical: This unlocks a pathway to reducing hardware synchronization constraints, bandwidth usage, and computational cost in real-world deployments (e.g., markerless motion capture, VR/AR, robotics), where camera frame rates, synchronization fidelity, and storage may be limited.
• Generalizability: The sparse interleaved input and associated fusion mechanisms are directly transferable to other multi-view 3D perception tasks beyond pose estimation, such as multi-object tracking or scene reconstruction.

Future Developments

Future directions include (a) extending the paradigm to non-human 3D tasks and general 3D scene understanding, (b) theoretical analysis of sparse sampling regimes and their information-theoretic limits, and (c) robustness improvements for operation under highly non-uniform or low-frame-rate temporal inputs.

Conclusion

This work rigorously demonstrates that multi-view 3D human pose estimation can transition from a dense synchronous data regime to a sparse, temporally-super-resolved paradigm, without sacrificing accuracy or temporal smoothness. The DenseWarper framework’s architectural innovations—epipolar-based spatial fusion, deformable temporal warping, and real-time, low-latency operation—render it a compelling solution for high-performance and resource-efficient 3D perception, with broad downstream applicability (2605.14525).