AugLift: Enhanced 3D Pose Estimation

• The paper introduces AugLift, which augments lifting-based 3D pose estimation by incorporating normalized keypoint confidence and depth estimates, achieving up to 10.1% cross-dataset improvement.
• The method preserves the original lifting architecture while expanding the per-joint input from (x,y) to (x,y, confidence, depth), ensuring robustness against domain shifts.
• Experimental results and ablations demonstrate that augmented inputs coupled with camera-distance normalization significantly reduce 2D-to-3D ambiguity and enhance overall pose accuracy.

AugLift is a reformulation of lifting-based monocular 3D human pose estimation in which the standard per-joint 2D input $(x_j,y_j)$ is replaced by an augmented feature $(x_j,y_j,\tilde c_j,\tilde d_j)$, where $\tilde c_j$ is a normalized keypoint detection confidence and $\tilde d_j$ is a keypoint-aligned depth estimate obtained from an off-the-shelf monocular depth model. The method is designed for the regime in which a lifter predicts 3D poses from detected 2D keypoints, and it targets the well-known generalization gap of such systems under cross-dataset and real-world deployment. Across four datasets, AugLift reports an average cross-dataset improvement of $10.1\%$ and an average in-distribution improvement of $4.0\%$, while preserving the underlying lifting architecture apart from its input dimensionality (Warner et al., 9 Aug 2025).

1. Problem formulation and motivation

In standard lifting-based 3D human pose estimation, an RGB image or video $I$ is processed by a 2D pose detector to produce keypoints

$\{(x_j,y_j)\}_{j=1}^K,$

after which a lifter $f$ maps the resulting $2K$-dimensional representation to 3D joint coordinates,

$(x_j,y_j,\tilde c_j,\tilde d_j)$0

with $(x_j,y_j,\tilde c_j,\tilde d_j)$1. The output is typically a root-relative or pelvis-centered 3D pose. This can be done per frame, as in MLP-based SimpleBaseline, or over sequences, as in VideoPose3D, PoseFormer, and MotionBERT (Warner et al., 9 Aug 2025).

The paper attributes poor generalization to two primary factors. First, the inverse problem is ill-posed when only sparse 2D coordinates are provided: many distinct 3D configurations project to similar 2D skeletons, and ambiguity is amplified by occlusion and unusual viewpoints. Using only $(x_j,y_j,\tilde c_j,\tilde d_j)$2 discards both a reliability signal and any cue about scene depth. Second, there are large domain gaps between controlled laboratory datasets and unconstrained settings. Human3.6M is described as a controlled indoor benchmark, whereas 3DPW differs in cameras, backgrounds, clothing, activities, and occlusions; lifters that achieve approximately $(x_j,y_j,\tilde c_j,\tilde d_j)$3–$(x_j,y_j,\tilde c_j,\tilde d_j)$4 mm MPJPE on Human3.6M often exceed $(x_j,y_j,\tilde c_j,\tilde d_j)$5 mm MPJPE on 3DPW in the realistic detection setting (Warner et al., 9 Aug 2025).

A further issue is the disconnect between training conventions and deployment. Architectures are often benchmarked with ground-truth 2D keypoints, but practical systems consume detected keypoints, which include confidence scores correlated with visibility and localization reliability. Standard pipelines ignore these scores. The paper also positions itself against dense image-conditioned lifting, arguing that dense appearance features often hurt generalization because models latch onto dataset-specific backgrounds and textures. AugLift therefore keeps the 2D-to-3D lifting paradigm but enriches it with sparse, keypoint-aligned cues obtained from off-the-shelf models, without additional 3D annotations or sensors (Warner et al., 9 Aug 2025).

2. Input reformulation and the AugLift module

The central reformulation is applied at the per-joint level. Standard lifting uses

$(x_j,y_j,\tilde c_j,\tilde d_j)$6

AugLift replaces this with

$(x_j,y_j,\tilde c_j,\tilde d_j)$7

Stacked across joints, the input to the lifter changes from $(x_j,y_j,\tilde c_j,\tilde d_j)$8 to $(x_j,y_j,\tilde c_j,\tilde d_j)$9 dimensions. The method characterizes this as a sparse enrichment: information is added only at joint locations rather than via per-pixel or per-patch image features (Warner et al., 9 Aug 2025).

The preprocessing pipeline has four stages. First, a 2D detector produces keypoints and confidence scores,

$\tilde c_j$0

Second, a monocular depth model produces a depth map $\tilde c_j$1, and each joint receives a local robust depth estimate

$\tilde c_j$2

In the reported experiments, $\tilde c_j$3 (Warner et al., 9 Aug 2025).

Third, the method applies a bounding-box rescaling intended as camera-distance normalization. It computes the keypoint centroid

$\tilde c_j$4

the box size

$\tilde c_j$5

and a scale factor

$\tilde c_j$6

where $\tilde c_j$7 is the mean training-set box size. The coordinates are then rescaled by

$\tilde c_j$8

This step is model-agnostic and is particularly motivated by cross-dataset differences in subject distance (Warner et al., 9 Aug 2025).

Fourth, confidence and depth are normalized. Confidence is mapped to a centered range,

$\tilde c_j$9

and depth is made root-relative,

$\tilde d_j$0

Depths are then clipped to

$\tilde d_j$1

The final output of the module is therefore

$\tilde d_j$2

The lifting network itself is otherwise unchanged (Warner et al., 9 Aug 2025).

3. Confidence, depth, and architectural integration

The confidence channel is sourced from the 2D detector, specifically RTMPose-L in the reported experiments. Since $\tilde d_j$3, the normalization $\tilde d_j$4 makes $\tilde d_j$5 map to $\tilde d_j$6, which the paper describes as a neutral default. Functionally, the confidence channel acts as a reliability mask: high values indicate that a joint is visible and localized confidently, whereas low values suggest occlusion or noise, encouraging the lifter to infer pose from the rest of the skeleton (Warner et al., 9 Aug 2025).

The depth channel is sourced from Depth Anything v2, which produces a metric depth map $\tilde d_j$7. The use of a local minimum in the neighborhood of each keypoint has two stated motivations. For visible joints, the neighborhood contains body pixels, so the minimum can approximate surface depth. For occluded joints, a foreground occluder often lies in front of the hidden joint, so the minimum furnishes a lower bound on the true depth. The paper argues that even imperfect metric depth carries substantially more geometric information than purely ordinal near/far cues, and it further notes that large-scale depth priors generalize across domains better than a small pose dataset would permit. An oracle study in Appendix E reports that coarse ground-truth ordinal depth cues can reduce cross-dataset MPJPE by approximately $\tilde d_j$8, motivating the use of realistic estimated depth in the deployed method (Warner et al., 9 Aug 2025).

Architecturally, AugLift is a modular add-on. The general sequence formulation changes from

$\tilde d_j$9

to an input

$10.1\%$0

with the same backbone then producing the 3D pose sequence. No auxiliary branches, fusion modules, or separate encoders are introduced. The only structural change is the expansion of the input layer from 2D joint features to 4D joint features (Warner et al., 9 Aug 2025).

This minimal modification is instantiated in four backbones. In SimpleBaseline, the flattened per-frame input changes from $10.1\%$1 to $10.1\%$2. In VideoPose3D, each time step receives $10.1\%$3-dimensional input but the temporal kernel and number of layers are unchanged. In PoseFormer, the per-joint embedding layer is widened from 2D to 4D features. In MotionBERT, only the input projection layer is expanded; the encoder-decoder stack remains unchanged. The training objective also remains standard supervised 3D pose regression, with no extra losses for confidence or depth and no adversarial or reprojection terms added (Warner et al., 9 Aug 2025).

4. Evaluation protocol and quantitative results

The evaluation spans four datasets: Human3.6M, MPI-INF-3DHP, Fit3D, and 3DPW. Human3.6M, 3DHP, and Fit3D are each used for training and testing, whereas 3DPW is used only as an out-of-distribution test set. The experimental design is “train on one dataset, evaluate on all four,” yielding one in-distribution score and three out-of-distribution scores for each source dataset. The primary metric is MPJPE in millimeters; Appendix E also reports P-MPJPE, 3DPCK-150mm, and AUC for additional cross-dataset analyses. All main experiments use the detection setting, meaning that 2D inputs come from RTMPose-L detections rather than ground-truth keypoints, and depth is always sourced from Depth Anything v2 (Warner et al., 9 Aug 2025).

For MotionBERT trained on Human3.6M, the reported MPJPE values are $10.1\%$4 on Human3.6M, $10.1\%$5 on 3DHP, $10.1\%$6 on Fit3D, and $10.1\%$7 on 3DPW. Across all three MotionBERT training regimes, the paper reports an average out-of-distribution improvement of $10.1\%$8 and an average in-distribution improvement of $10.1\%$9. One of the largest drops is observed for Fit3D$4.0\%$03DHP, where MotionBERT improves from $4.0\%$1 to $4.0\%$2 mm, a $4.0\%$3 reduction (Warner et al., 9 Aug 2025).

Backbone	Example ID result	Overall OOD gain
MotionBERT	H36M $4.0\%$4	$4.0\%$5
SimpleBaseline	H36M $4.0\%$6	$4.0\%$7
VideoPose3D	H36M $4.0\%$8	$4.0\%$9
PoseFormer	H36M $I$0	$I$1

Across these four backbones, the mean gain is reported as approximately $I$2 in-distribution and $I$3 out-of-distribution. SimpleBaseline exhibits the largest cross-dataset gain, including a reduction on 3DPW from $I$4 to $I$5 mm. VideoPose3D shows only a $I$6 in-distribution improvement on Human3.6M but still improves on all three out-of-distribution benchmarks. The results are therefore presented as architecture-agnostic rather than backbone-specific (Warner et al., 9 Aug 2025).

The comparison between confidence-only and full AugLift is also central. “Base Model + confidence” sometimes yields small gains and sometimes slight degradations; for example, the SimpleBaseline confidence-only variant is slightly worse on Human3.6M. By contrast, the full $I$7 formulation improves performance consistently across all models and datasets in Table 2. The paper therefore identifies depth as the main driver of the improvement, with confidence acting as a modulator rather than a sufficient cue on its own (Warner et al., 9 Aug 2025).

5. Ablations, qualitative behavior, and practical use

The paper’s ablations analyze three issues in detail: camera-distance normalization, temporal context, and the relative brittleness of motion priors. For bounding-box rescaling, the motivation is that subject distances vary substantially across datasets; the details list mean depths of $I$8 m for Human3.6M and $I$9 m for 3DPW. On 3DPW, SimpleBaseline improves from $\{(x_j,y_j)\}_{j=1}^K,$0 to $\{(x_j,y_j)\}_{j=1}^K,$1 mm with rescaling alone, and from $\{(x_j,y_j)\}_{j=1}^K,$2 to $\{(x_j,y_j)\}_{j=1}^K,$3 mm when AugLift is also used. VideoPose3D improves from $\{(x_j,y_j)\}_{j=1}^K,$4 to $\{(x_j,y_j)\}_{j=1}^K,$5 mm without AugLift and from $\{(x_j,y_j)\}_{j=1}^K,$6 to $\{(x_j,y_j)\}_{j=1}^K,$7 mm with it. The paper notes, however, that PoseFormer is slightly hurt by rescaling, and that MotionBERT already contains camera-invariant mechanisms, so rescaling is unnecessary there (Warner et al., 9 Aug 2025).

Temporal context does not eliminate the benefit of sparse spatial augmentation. For MotionBERT trained on Human3.6M, full AugLift yields an $\{(x_j,y_j)\}_{j=1}^K,$8 out-of-distribution improvement at sequence length $\{(x_j,y_j)\}_{j=1}^K,$9 and an $f$0 improvement at sequence length $f$1. The depth-only marginal gain over the confidence-only model is reported as $f$2 OOD for single-frame input and $f$3 for long temporal context. Appendix D further reports that when VideoPose3D is trained with increasingly long sequences on ground-truth 2D input from Human3.6M, the in-distribution error decreases but the out-of-distribution error on 3DHP, Fit3D, and 3DPW does not improve and often worsens; reversed or variably sped sequences degrade long-sequence models more severely. The stated implication is that motion priors learned from laboratory datasets are brittle, whereas AugLift’s frame-level spatial cues are more robust (Warner et al., 9 Aug 2025).

Qualitative results reinforce this interpretation. Figure 1 visualizes joint confidence and depth maps, showing that occluded joints tend to have low confidence and unreliable depth values. The reported synergy is that confidence allows the model to distrust depth around such joints, while depth provides strong geometric cues where confidence is high. Figure 2, using MotionBERT trained on Human3.6M and tested on 3DHP, shows challenging sitting, crouching, and unusual-viewpoint poses in which the baseline misestimates limb depth and overall posture, whereas AugLift aligns more closely with the ground truth, particularly in limb orientation and depth structure (Warner et al., 9 Aug 2025).

From a deployment standpoint, the method is explicitly intended to be easy to adopt. The practical recipe given is to add a monocular depth estimator, implement the AugLift preprocessing, and retrain the lifter with augmented inputs. The details also provide an example latency budget: approximately $f$4 ms for an HRNet-W48 2D detector, $f$5 ms for MotionBERT, and $f$6 ms for the small version of Depth Anything v2. This suggests that the additional depth pass is a modest overhead and can be amortized in streaming settings by computing depth less frequently and reusing it across frames (Warner et al., 9 Aug 2025).

6. Limitations, conceptual scope, and relation to adjacent work

The paper identifies several limitations. The most immediate is dependence on the quality of the depth estimator: extreme scenes, rare camera setups, non-human subjects, or heavy motion blur can make depth cues misleading. Confidence can mitigate this failure mode but cannot remove it. A second limitation is latency and memory overhead from adding a depth network, which may be problematic under very tight real-time or edge constraints. A third is that hyperparameters such as the neighborhood radius $f$7, depth clipping threshold $f$8, and rescaling scheme may require retuning in new domains. Finally, although AugLift reduces reliance on motion priors, temporal models themselves remain sensitive to out-of-distribution motion patterns, so the broader problem of robust temporal representation remains open (Warner et al., 9 Aug 2025).

The method is positioned against several strands of prior work. Relative to classic lifters such as SimpleBaseline, VideoPose3D, PoseFormer, and MotionBERT, its novelty is not a new backbone but an input-level change from $f$9 to $2K$0. Relative to dense image-conditioned lifting, it deliberately avoids appearance-heavy features that can overfit to backgrounds and clothing. Relative to earlier uses of confidence or depth in 3D pose estimation, it combines confidence with externally estimated depth as an input rather than using depth mainly as supervision or requiring additional depth annotations or multi-view setups. Relative to domain adaptation methods such as PoseDA, it is framed as a preprocessing module that requires neither target labels nor aggregate test-set statistics while still improving cross-dataset performance by $2K$1 on average for MotionBERT (Warner et al., 9 Aug 2025).

The broader literature contains other uses of “lift” terminology that are conceptually separate. LIFT concerns LLM-based pragma insertion for HLS under GNN supervision (Prakriya et al., 29 Apr 2025); Lift3D-VLA concerns lifting vision-language-action models to explicit 3D geometry and dynamics-aware manipulation (Liu et al., 7 Jul 2026); and ALOcc introduces an occlusion-aware adaptive lifting mechanism for semantic occupancy and flow prediction (Chen et al., 2024). These works also use “lifting” to denote an augmentation of an existing pipeline, but they address different problem classes and do not describe AugLift itself.

Within lifting-based 3D human pose estimation, AugLift’s main significance is methodological rather than architectural. It shows that sparse, keypoint-aligned reliability and geometric cues—confidence and depth—can be injected into the canonical lifting pipeline without auxiliary branches, extra sensors, or additional 3D data collection, yet still produce substantial gains in out-of-distribution generalization. The paper’s analysis suggests that this improvement stems from supplying robust frame-level context to a representation that is otherwise highly ambiguous when restricted to 2D keypoint coordinates alone (Warner et al., 9 Aug 2025).