Multi-View Pose-Only Estimation

• Multi-view pose-only estimation is a technique that infers 3D pose by focusing solely on pose parameters, often without full shape reconstruction.
• It employs methodologies such as triangulation, graph-based message passing, transformer fusion, and optimization-centric approaches to enforce cross-view geometric consistency.
• The approach is applied in both human and object pose estimation, utilizing calibrated, weakly supervised, and self-supervised regimes to enhance prediction accuracy.

Searching arXiv for recent and foundational papers on multi-view pose estimation to ground the article in cited literature. Multi-view pose-only estimation denotes a class of methods that infer pose from multiple synchronized or otherwise associated views while keeping pose, or pose plus camera parameters, as the primary estimation target rather than treating dense shape reconstruction as the central objective. In the supplied literature, the term spans several regimes: multi-view inference from calibrated cameras for 3D human pose or 6-DoF object pose; training-time use of synchronized multi-view data to improve a single-view estimator; and uncalibrated or weakly calibrated settings in which camera pose, or a distribution over camera poses, is estimated jointly with pose (Wu et al., 2021, Bouazizi et al., 2021, Usman et al., 2021, Jiang et al., 2023, Haugaard et al., 2022). Across these variants, the common motivation is that multiple views mitigate occlusion, self-occlusion, foreshortening, symmetry-induced ambiguity, and depth ambiguity, while enabling geometric consistency constraints that are unavailable to single-view methods (Iqbal et al., 2020, Ma et al., 2022, Ranftl et al., 5 Aug 2025).

1. Scope and problem formulations

Within human pose estimation, multi-view pose-only estimation appears in at least three distinct formulations. The first is direct multi-view inference, where calibrated cameras are used to localize people in 3D and then estimate or refine a 3D skeleton, as in graph-based multi-person pipelines and self-supervised multi-person systems (Wu et al., 2021, Srivastav et al., 2024). The second is multi-view training with single-view inference, where synchronized views provide a weak or self-supervised learning signal but the deployed model remains monocular (Rhodin et al., 2018, Bouazizi et al., 2021, Iqbal et al., 2020). The third is uncalibrated or partially calibrated estimation, where the method avoids fixed camera pose assumptions by estimating camera rotation, camera parameters, or a camera pose distribution jointly with pose (Rhodin et al., 2018, Usman et al., 2021, Luvizon et al., 2019, Jiang et al., 2023).

Within object pose estimation, the same term usually refers to 6-DoF pose estimation from multiple views without jointly solving a SLAM-style camera-and-scene problem. The literature here ranges from test-time fusion of single-view pose hypotheses using voting in $SE(3)$ (Li et al., 2018), to learned multi-view correspondence aggregation under epipolar constraints (Haugaard et al., 2022), to transformer-based early fusion with line-of-sight encodings that is explicitly designed to resolve ambiguities that cannot be solved from a single image or by post-processing independently predicted single-view poses (Ranftl et al., 5 Aug 2025).

A recurrent distinction in the literature is between strict pose-only methods and methods that use shape merely as a prior. Some human-pose systems remain pose-centric while introducing SMPL as a regularizer or auxiliary output, rather than making full body-shape recovery the main target (Dong et al., 2021). Other systems are explicitly model-based and estimate SMPL pose and shape together with camera pose; these are still multi-view pose-centric, but not pose-only in the narrowest geometric sense (Li et al., 2024).

2. Geometric principles and pose representations

The dominant geometric substrate is multi-view consistency. In calibrated settings, this appears as projection between 3D and 2D image planes, transformations between camera coordinate systems, triangulation, and cross-view reprojection losses (Bouazizi et al., 2021). In uncalibrated settings, the geometry may be recovered indirectly through Procrustes-style alignment of predicted poses, optimization of intrinsics and extrinsics, or a probability distribution over camera pose updated from 2D features (Rhodin et al., 2018, Luvizon et al., 2019, Jiang et al., 2023).

Several pose representations recur because they make multi-view supervision tractable. One line of work predicts 3D pose in camera coordinates rather than root-centered coordinates, using a view-frustum space representation in which $(u,v)$ image coordinates and depth are disentangled. In that formulation, the network predicts a relative pose $\mathbf{p}$, per-joint confidence $\mathbf{c}$, and absolute root depth $z_a$, after which absolute 3D joints are recovered by inverse camera projection (Luvizon et al., 2019). Another line uses scale-normalized 2.5D pose, with image-plane coordinates plus relative depth and a normalization constraint based on a fixed bone such as the neck–pelvis distance, so that differentiable reconstruction from 2.5D to 3D can be combined with multi-view consistency without requiring 3D labels (Iqbal et al., 2020).

A third family lifts 2D pose detections directly to 3D. In self-supervised single-person settings, the input is a $2N$-dimensional 2D joint vector and the output is a $3N$-dimensional 3D pose in the camera coordinate system, often with the pelvis as origin (Bouazizi et al., 2021). For multi-person calibrated systems, pose may instead be represented as a set of 3D joints refined from an initial estimate, with residual offset regression

$$\mathbf{X}_k^{\,\text{ref}} = \mathbf{X}_k^{\,\text{init}} + \Delta \mathbf{X}_k$$

used to correct errors after person localization (Wu et al., 2021).

Object-pose methods use analogous geometric parameterizations. Classical multi-view voting methods operate on rigid poses $(R,T)\in SE(3)$ and compare hypotheses using ADD-S-style distances over object model points (Li et al., 2018). Correspondence-based methods represent the problem probabilistically through 2D–3D and 3D–3D correspondences sampled under epipolar constraints (Haugaard et al., 2022). Transformer-based holistic fusion methods incorporate camera geometry through lines of sight, encoded as origin plus direction per pixel, so that cross-view attention can reason in a geometry-aware feature space before regressing rotation and translation (Ranftl et al., 5 Aug 2025).

3. Principal methodological families

A first family comprises triangulation-and-consensus pipelines. In human pose estimation, this includes systems that triangulate pairwise 2D detections into 3D joint candidates, cluster them by confidence-aware majority voting, and then refine the resulting skeletons with reprojection terms and body-model priors (Dong et al., 2021). It also includes self-supervised lifting frameworks that use Direct Linear Triangulation as a differentiable pseudo-label generator and then constrain the network with input triangulation, reprojection, consistency, and output triangulation losses (Bouazizi et al., 2021). For objects, the corresponding pattern is to sample view-consistent 3D–3D correspondences and recover rigid pose hypotheses with Kabsch (Haugaard et al., 2022).

A second family uses graph-structured message passing. In graph-based multi-person 3D human pose estimation, the pipeline follows a coarse-to-fine top-down design: person centers are matched and triangulated across views, refined in 3D, and then each person’s initial 3D pose is corrected by a Pose Regression Graph Module. The pose graph contains projected 2D joints as nodes, skeleton edges within each view, and cross-view edges between the same joint type across views. Message passing is performed with EdgeConv-E,

$$x_v \doteq \max_{v' \in \mathcal{N}(v)} h_{\mathbf{\theta}}\!\left( \text{Concat}(x_v, x_{v'} - x_v, e_{(v, v')})\right),$$

so that both relative node features and explicit edge types condition the propagation (Wu et al., 2021). This setup is intended to learn multi-view geometry and structural relations jointly.

A third family uses transformer-based multi-view fusion. Token-Pruned Pose Transformer applies pruning per view to retain human-area tokens and then performs cross-view fusion only over selected foreground tokens, reducing the cost of global attention while preserving dense attention among human foreground tokens (Ma et al., 2022). MVTOP instead performs early fusion of view-specific features with an encoder–decoder transformer, injecting line-of-sight geometry through Feature Line-of-Sight Encoding and using projective attention to exchange information across views before outputting pose (Ranftl et al., 5 Aug 2025). This suggests a broader methodological shift from late hypothesis aggregation toward geometry-aware early fusion.

A fourth family is optimization-centric. Classical multi-view object pose estimation with known camera poses fuses single-view hypotheses by voting over self-consistency in $(u,v)$0 rather than averaging (Li et al., 2018). Learned “meta-optimizer” approaches approximate bundle adjustment with a feed-forward network that iteratively refines 3D joints and camera parameters from heatmap evidence and current reprojections, while remaining uncertainty-aware through Gaussian-mixture approximations of 2D heatmaps (Usman et al., 2021). Consensus-based optimization for human pose performs alternating updates of camera intrinsics, extrinsics, and poses in camera coordinates so that independently predicted monocular poses agree across views (Luvizon et al., 2019).

4. Supervision regimes and objective functions

The supplied literature shows that multi-view pose-only estimation is not tied to a single supervision regime. Fully supervised calibrated multi-view systems remain important, particularly for multi-person 3D pose estimation where person localization and pose regression are trained with 3D labels and 2D image evidence (Wu et al., 2021). However, weakly supervised and self-supervised variants are especially prominent.

Weakly supervised monocular learning from multi-view images combines a multi-view consistency term, a supervised loss on a small labeled subset, and a regularizer toward an initial supervised model: $(u,v)$1 To avoid degenerate collapse under scale ambiguity, this literature replaces ordinary squared error with a scale-invariant normalized loss,

$(u,v)$2

and estimates cross-view rotations by rotational alignment over predicted poses (Rhodin et al., 2018).

A related weakly supervised formulation predicts a scale-normalized 2.5D pose and enforces confidence-weighted cross-view consistency after rigid alignment: $(u,v)$3 Here $(u,v)$4 is a 2D heatmap loss, $(u,v)$5 is a multi-view consistency term over reconstructed 3D poses, and $(u,v)$6 is a bone-length regularizer. The literature explicitly notes that naive multi-view consistency can admit degenerate solutions, so plausibility constraints and confidence weighting are used to stabilize learning (Iqbal et al., 2020).

Fully self-supervised lifting from 2D detections to 3D relies on a four-loss objective,

$(u,v)$7

where $(u,v)$8 matches predictions to triangulated pseudo-labels, $(u,v)$9 enforces all-view reprojection agreement, $\mathbf{p}$0 aligns predictions across camera frames, and $\mathbf{p}$1 uses a second triangulation of reprojected predictions as a refined self-supervisory target (Bouazizi et al., 2021).

Multi-person self-supervised systems use a different bottleneck strategy. SelfPose3d treats 3D pose as a bottleneck representation, projects predicted joints back to all views, renders them as differentiable 2D Gaussian heatmaps, and compares them to pseudo 2D poses from an off-the-shelf detector. To mitigate pseudo-label noise, it introduces adaptive supervision attention with a hard attention term that drops the worst view for $\mathbf{p}$2 joint supervision and a soft attention term that weights spatial heatmap errors (Srivastav et al., 2024).

Uncalibrated estimation introduces another layer of objectives. Probabilistic triangulation models camera pose as a distribution and updates that distribution by Monte Carlo posterior estimation so that gradients can be back-propagated from 3D pose estimation to the 2D heatmap (Jiang et al., 2023). MetaPose learns a feed-forward surrogate for bundle adjustment by minimizing reprojection error and, optionally, a scale-invariant bone-length prior (Usman et al., 2021). This suggests that uncertainty modeling has become a central device whenever calibration is missing or moving.

5. Calibration, uncertainty, and correspondence

Calibration is a major axis of variation. Many human-pose methods assume synchronized calibrated cameras because calibration enables projection, triangulation, epipolar reasoning, and explicit transformations between camera frames (Wu et al., 2021, Bouazizi et al., 2021, Srivastav et al., 2024). In these settings, the main challenge is not geometry availability but effective fusion under occlusion, noisy detections, or multiple people.

Other methods are explicitly designed to relax or remove fixed calibration assumptions. Weakly supervised monocular learning can jointly estimate camera rotation and human pose, exploiting the fact that pelvis-centered poses make the method independent of camera translation and require only camera intrinsics (Rhodin et al., 2018). Consensus-based optimization estimates extrinsics and intrinsics by alternating Procrustes alignment, translation updates, and intrinsic updates so that predicted poses agree across views, although the method notes that predictions may remain coherent only up to a scale factor when camera intrinsics differ substantially from the training setup (Luvizon et al., 2019). Probabilistic triangulation generalizes calibrated pose-estimation methods to uncalibrated scenes by replacing a fixed camera pose with a maintained and iteratively updated distribution over camera pose (Jiang et al., 2023).

Uncertainty enters both through the pose observations and through the camera model. In MetaPose, each 2D keypoint is treated not as a hard point but as a heatmap-derived probability distribution, compressed into a Gaussian mixture so that multi-modal uncertainty from occlusion can be represented during neural refinement (Usman et al., 2021). In weakly supervised 2.5D learning, joint confidences from 2D heatmaps weight the multi-view consistency objective and downweight unreliable joints under occlusion (Iqbal et al., 2020). In graph-based refinement, projected multi-view joint evidence and skeletal structure priors allow visible views to support views where joints are occluded (Wu et al., 2021).

Correspondence handling differs sharply across methods. Some approaches avoid explicit discrete correspondence resolution and instead project an initial 3D pose into all views, then let graph or transformer layers learn which projected nodes or tokens agree across views (Wu et al., 2021, Ma et al., 2022). Others make correspondence the central object of inference. Multi-view object pose estimation from correspondence distributions samples pixels and 3D model points from learned dense 2D–3D distributions, restricts the second-view pixel search to the epipolar line, triangulates a scene point, and thereby samples 3D–3D correspondences for rigid pose estimation (Haugaard et al., 2022). This literature emphasizes that ambiguities due to symmetry, reflections, or featureless surfaces are best treated as multi-modal distributions rather than deterministic matches.

6. Empirical patterns, applications, and limitations

The empirical record in the supplied source set shows a consistent pattern: multi-view information improves pose accuracy, but the magnitude and mechanism of improvement depend on whether the method uses views only during training, also during inference, or additionally for calibration recovery.

For human pose, graph-based multi-person estimation on CMU Panoptic reports mAP 98.10, mAR 98.70, and MPJPE 15.84 mm, improving over a reproduced VoxelPose baseline with 96.73 mAP, 97.56 mAR, and 17.56 mm MPJPE; the ablations show that PRG improves MPJPE consistently across different initial pose regressors, including 18.12 $\mathbf{p}$3 16.63 mm, 17.78 $\mathbf{p}$4 16.44 mm, and 17.09 $\mathbf{p}$5 15.84 mm (Wu et al., 2021). Self-supervised multi-person SelfPose3d reaches AP$\mathbf{p}$6 = 96.4, AP$\mathbf{p}$7 = 98.5, Recall@500 = 99.6, and MPJPE = 24.5 mm on Panoptic, while reporting Shelf average PCP = 95.1 and Campus average PCP = 87.9 (Srivastav et al., 2024). Weakly supervised 2.5D learning reports 69.1 mm MPJPE on Human3.6M in a weakly supervised setting with H36M multi-view plus MPII 2D, improving to 67.4 mm with additional 3DHP and MQC multi-view data, and 62.8 mm in a semi-supervised setting, improving to 59.7 mm with extra multi-view data (Iqbal et al., 2020). Self-supervised lifting with four losses reports 62.0 mm P-MPJPE on Human3.6M, 58.9 mm on HumanEva after Procrustes alignment, and cross-dataset generalization on MPI-INF-3DHP with PCK 65.9 and AUC 32.5 (Bouazizi et al., 2021). Consensus-based camera-coordinate estimation reports multi-view gains from 50.5 MPJPE with 1 camera to 36.9 MPJPE with 4 cameras (Luvizon et al., 2019). Transformer-based human area fusion reaches 24.4 mm average MPJPE on Human3.6M and 34.1 mm on Ski-Pose, while reducing 4-view fusion computation from 55.1G MACs in Cross-view Fusion to 9.7G MACs (Ma et al., 2022).

For object pose, MVTOP’s synthetic MV-ball benchmark is designed so that the pose cannot be solved from a single view. On that dataset, MVTOP 2v reports mean ADD 0.01185 and rotation error 7.345°, whereas CosyPose 2v remains at mean ADD 1.04312 and rotation error 105.539°; on YCB-V, MVTOP reports AUC of ADD-S = 96.50 (Ranftl et al., 5 Aug 2025). Multi-view correspondence sampling with epipolar geometry on T-LESS reports AR = 0.976 for the full multi-view method with ground truth multi-view detections and AR = 0.986 with multi-view refinement, compared with AR = 0.843 for SurfEmb on ground-truth detections (Haugaard et al., 2022). Classical multi-view 6-DoF voting improves RGB YCB-Video from 75.1 to 80.2 mPCK and RGB-D from 90.6 to 94.3 mPCK when moving from single-view MCN to MV5-MCN, while also improving JHUScene-50 from 33.9 to 38.4 on RGB and from 78.9 to 84.8 on RGB-D (Li et al., 2018).

These results suggest several stable limitations. Calibration assumptions remain decisive in many high-performing systems (Wu et al., 2021, Srivastav et al., 2024). Methods that estimate calibration or camera pose distributions can trade some accuracy for generalizability (Jiang et al., 2023, Luvizon et al., 2019). Performance often degrades when the number of views is reduced, as explicitly reported for SelfPose3d, which drops from AP$\mathbf{p}$8=96.4, MPJPE=24.5 with 5 views to AP$\mathbf{p}$9=66.1, MPJPE=43.5 with 3 views (Srivastav et al., 2024). Several papers also note dependence on the quality of 2D detectors or monocular initializers, especially in occluded scenes or when pseudo labels are noisy (Iqbal et al., 2020, Usman et al., 2021, Dong et al., 2021).

A broader implication is that “multi-view pose-only estimation” is not a single algorithmic template but a convergence zone of geometric consistency, residual refinement, uncertainty-aware fusion, and supervision-efficient learning. The literature in this source set indicates that the field has progressively moved from calibrated late fusion and triangulation toward learned cross-view aggregation, while simultaneously expanding into weakly supervised, self-supervised, and uncalibrated regimes (Li et al., 2018, Bouazizi et al., 2021, Usman et al., 2021, Ranftl et al., 5 Aug 2025).