PersPose: Camera-Aware 3D Pose Estimation

• The paper introduces PersPose, a framework combining perspective encoding and rotation to explicitly handle camera intrinsics and reduce cropping distortions in 3D human pose estimation.
• It employs a staged pipeline that centers the subject using homography-based rotation, constructs dual-channel perspective maps, and leverages HRNet-W48 for predicting 2D heatmaps, depths, and a scale term.
• Quantitative evaluations on benchmarks like 3DPW, Human3.6M, and MPI-INF-3DHP demonstrate significant accuracy improvements, reducing MPJPE by up to 7.54% compared to prior methods.

Searching arXiv for PersPose and directly related papers to ground the article with current citations. PersPose is a monocular 3D human pose estimation framework that combines Perspective Encoding (PE) and Perspective Rotation (PR) to estimate the 3D positions of joints from individual images while explicitly accounting for camera intrinsics and perspective distortions in cropped inputs. The method starts from a single RGB image $I$ of arbitrary size with known full-image intrinsics $K$, applies a homography-based centering transform, constructs a two-channel perspective map under the cropped intrinsics, and predicts 2D joint locations, relative depths, and a scale term for recovering absolute depth. The resulting formulation is presented as a response to a specific limitation of existing 3D HPE approaches: the relative depths of joints cannot be accurately estimated from cropped images without the corresponding camera intrinsics, and the perspective relationship between the 3D scene and the cropped image differs significantly when the subject is far from the image center (Hao et al., 24 Aug 2025).

1. Problem setting and core motivation

PersPose addresses monocular 3D human pose estimation under the assumption that the model receives a single RGB image and the full-image camera intrinsics. The central claim motivating the framework is that existing 3D HPE approaches often use the cropped image alone as input, even though the cropped image by itself does not preserve the information needed to model the perspective relationship induced by the camera. In the formulation presented for PersPose, this missing information is handled explicitly through Perspective Encoding, while geometric distortions caused by off-center subjects are handled through Perspective Rotation (Hao et al., 24 Aug 2025).

The method is organized as a staged pipeline. First, it estimates the 2D center $(u_c,v_c)$ of the person’s bounding box in the original image. Second, it computes a rotation $R\in\mathbb R^{3\times3}$ that brings the corresponding 3D ray onto the positive $z$-axis and warps the original image by the homography $M = K\,R\,K^{-1}$ to produce a centered image $I'$. Third, it crops a fixed $256\times256$ window from the center of $I'$ and derives the corresponding cropped intrinsics $K^{\text{crop}}$. Fourth, it constructs a two-channel map $K$0 by projecting each cropped pixel to the plane $K$1. Fifth, it feeds the cropped RGB image and the perspective map into HRNet-W48, whose decoder heads regress 2D heatmaps, relative depths, and a scale estimate $K$2. Finally, it reconstructs 3D joint coordinates in the centered camera frame and then applies $K$3 to obtain the 3D joints in the original camera coordinate system (Hao et al., 24 Aug 2025).

A plausible implication is that PersPose treats perspective not as an incidental nuisance to be absorbed by a backbone, but as structured signal that should enter the network and the geometric preprocessing explicitly. This interpretation is consistent with the design of both PE and PR.

2. Perspective Rotation

Perspective Rotation is the geometric preprocessing stage used to center the human subject before cropping. The procedure begins by computing the 3D “box-center” ray in the original camera:

$K$4

The target direction is $K$5. The rotation axis $K$6 and angle $K$7 are then defined as

$K$8

The rotation matrix is constructed as $K$9, and the corresponding image-plane warp is the homography

$(u_c,v_c)$0

Applying this homography to the full image yields the centered image $(u_c,v_c)$1 (Hao et al., 24 Aug 2025).

The stated purpose of PR is twofold. First, since the human subject can appear anywhere within the original image, the perspective relationship between the 3D scene and the cropped image differs significantly, which complicates model fitting. Second, the further the subject deviates from the image center, the greater the perspective distortions in the cropped image. PR addresses these issues by centering the subject and thereby reducing perspective distortions (Hao et al., 24 Aug 2025).

After prediction in the rotated frame, the transformation is inverted by

$(u_c,v_c)$2

This makes PR a reversible geometric normalization rather than a change in the target coordinate system. A common misconception would be to read PR as a data-augmentation heuristic. The formulation in PersPose is more specific: PR is a deterministic transformation derived from the camera intrinsics and the subject’s image location, and it is integrated into the reconstruction pipeline rather than used merely to diversify training samples.

3. Perspective Encoding and backbone design

Perspective Encoding is the component that injects the cropped camera intrinsics into the network in a spatially explicit way. After PR, PersPose crops a fixed $(u_c,v_c)$3 window from the center of $(u_c,v_c)$4 and computes the cropped intrinsics

$(u_c,v_c)$5

where $(u_c,v_c)$6, $(u_c,v_c)$7, and $(u_c,v_c)$8 encode the crop (Hao et al., 24 Aug 2025).

Using $(u_c,v_c)$9, the method constructs a two-channel perspective map by projecting every pixel $R\in\mathbb R^{3\times3}$0 in the crop to the plane $R\in\mathbb R^{3\times3}$1:

$R\in\mathbb R^{3\times3}$2

with

$R\in\mathbb R^{3\times3}$3

Channel 1 stores $R\in\mathbb R^{3\times3}$4, and channel 2 stores $R\in\mathbb R^{3\times3}$5. In the implementation described for PersPose, this map is fused at the input stage by processing $R\in\mathbb R^{3\times3}$6 and $R\in\mathbb R^{3\times3}$7 through two parallel convolutional layers with the same kernel size and then adding their feature maps element-wise (Hao et al., 24 Aug 2025).

The backbone is HRNet-W48 with input size $R\in\mathbb R^{3\times3}$8, output spatial map $R\in\mathbb R^{3\times3}$9, and a global pooled vector of length 2048. The decoder has three principal heads for 3D pose: a 2D heatmap head consisting of a $z$0 convolution that produces 14 heatmaps in $z$1 followed by soft-argmax to obtain $z$2, a depth head implemented as $z$3 to output $z$4, and a scale head implemented as $z$5 to output $z$6. The details additionally note two more FC heads for shape and twist in mesh experiments, with the remark “see HybrIK’s analytical IK” (Hao et al., 24 Aug 2025).

This suggests that PersPose uses PE not as metadata appended to a global token or latent vector, but as a dense per-pixel coordinate field that conditions the early feature extractor. In that sense, PE is closer to a camera-aware spatial parameterization than to a late-stage calibration correction.

4. 3D coordinate reconstruction and training objectives

PersPose reconstructs 3D coordinates from the predicted image coordinates, relative depths, and scale. For each joint, the absolute depth is computed as

$z$7

and the 3D coordinates in the centered camera frame are obtained by back-projection:

$z$8

The final output in the original camera coordinate system is then

$z$9

The scale prediction $M = K\,R\,K^{-1}$0 is described as “an estimate of absolute pelvis-to-camera depth per pixel; used to recover absolute depths” (Hao et al., 24 Aug 2025).

The total training objective is

$M = K\,R\,K^{-1}$1

The term $M = K\,R\,K^{-1}$2 is the L1 loss on predicted $M = K\,R\,K^{-1}$3 against ground truth:

$M = K\,R\,K^{-1}$4

The term $M = K\,R\,K^{-1}$5 is L1 on the predicted scale $M = K\,R\,K^{-1}$6 versus the ground-truth absolute pelvis depth ratio. The terms $M = K\,R\,K^{-1}$7 and $M = K\,R\,K^{-1}$8 are small L1/L2 terms on SMPL shape and twist and are used only in mesh experiments. The reported hyperparameters are $M = K\,R\,K^{-1}$9, $I'$0, $I'$1, and $I'$2 (Hao et al., 24 Aug 2025).

The implementation details specify AdamW with initial learning rate $I'$3, learning-rate decay by a factor of $I'$4 every 6 epochs, total training for 70 epochs, and batch size 96 using a mixture of 3DPW, Human3.6M, MPI-INF-3DHP, and COCO, with optional BEDLAM added for “R+B.” Data augmentation follows standard HRNet augmentation with scale $I'$5, rotation $I'$6, and color jitter. Training time is reported as approximately 60 hours on one NVIDIA 3090 (Hao et al., 24 Aug 2025).

5. Quantitative performance and ablation evidence

PersPose is reported to achieve state-of-the-art performance on 3DPW, MPI-INF-3DHP, and Human3.6M. The summary metrics reported for the final 3D pose results are as follows (Hao et al., 24 Aug 2025):

Dataset	Reported results
3DPW	PA-MPJPE 39.1; MPJPE 60.1; PVE 72.4
Human3.6M	PA-MPJPE 28.3; MPJPE 43.0; PVE 52.7
MPI-INF-3DHP	PCK 94.0; AUC 55.2; MPJPE 72.1

The 3DPW result is singled out in the abstract: PersPose achieves an MPJPE of 60.1 mm, which is stated to be 7.54% lower than the previous SOTA approach. For Human3.6M, the MPJPE of 43.0 is reported against 45.7 by ProPose. For MPI-INF-3DHP, the PCK of 94.0 is reported as $I'$7 points over the best prior 91.8 (Hao et al., 24 Aug 2025).

The ablation study isolates the effects of PE and PR under two training regimes, “R” and “R+B,” on 3DPW. Under “R,” the baseline without either component gives depth 45.1 mm, PA-MPJPE 39.8, and MPJPE 62.4; adding PE only gives 44.5, 39.7, and 62.2; adding both PE and PR gives 43.8, 39.1, and 60.1. Under “R+B,” the corresponding numbers are 41.5, 37.8, and 58.4 for none; 41.2, 37.8, and 58.1 for PE only; and 40.0, 37.3, and 57.2 for PE+PR. Similar trends are said to hold on MPI-INF-3DHP. The stated conclusion is that both PE and PR individually reduce depth error and 3D pose error, while the combination gives the biggest gain (Hao et al., 24 Aug 2025).

A plausible implication is that the main gains are not attributable to one module alone. The ablations instead support a decomposition in which PE provides camera-aware conditioning and PR reduces the difficulty of fitting by normalizing the cropping geometry.

6. Interpretation, scope, and distinction from RePose

PersPose is specifically a 3D HPE framework, and its defining contribution lies in making the cropped-image perspective relationship explicit through PE and PR. Its inputs, intermediate representations, and outputs are therefore organized around camera intrinsics, homography-based re-centering, dense perspective maps, and 3D reconstruction in camera coordinates (Hao et al., 24 Aug 2025).

This should be distinguished from RePose, a 2020 single-image pose estimator with a different problem setting and architecture. RePose is a lightweight, end-to-end trainable CNN for human pose estimation from a single image that explicitly encodes the human body’s kinematic structure into its architecture. It uses a U-Net-style multi-scale encoder-decoder, decouples $I'$8 keypoints into independent feature streams at the coarsest $I'$9 level, applies a two-pass message-passing-style kinematic update on those streams, and refines predictions in a hierarchical coarse-to-fine fashion. RePose reports 91.66% mean accuracy on Leeds Sports Pose at [email protected] and 90.29% on MPII at [email protected], with 4 million parameters and 13.5 GFLOPs for a $256\times256$0 input (Isack et al., 2020).

The similarity of the names can obscure the methodological difference. RePose centers on a learnable kinematic prior for lightweight 2D single-person pose estimation, whereas PersPose centers on perspective-aware monocular 3D pose estimation. A common misunderstanding would be to treat PersPose as simply another kinematic-prior architecture. The available description does not support that reading: its stated novelty is the combination of Perspective Encoding and Perspective Rotation, not a graph-based skeletal message-passing mechanism. Conversely, nothing in the PersPose description suggests that it replaces the need for camera intrinsics; the framework is explicitly built around them.