RPEA for 3D Pose Estimation

• The paper's main contribution is introducing RPEA, which leverages a Bayesian MMSE framework using reprojection errors as a likelihood proxy.
• It formulates a joint-wise importance weighting scheme that selects top-K candidates to enhance pose coherence and reduce MPJPE.
• Empirical results demonstrate that RPEA outperforms uniform mean and MAP aggregation with a 4.8% improvement in skeleton structure accuracy.

Reprojection-based Posterior Expectation Aggregation (RPEA) is an inference module introduced within the FMPose3D framework to address the problem of selecting a single, high-quality 3D pose estimate from a set of diverse hypotheses generated for monocular 3D pose estimation scenarios. The approach leverages the Bayesian minimum mean squared error (MMSE) principle, approximates the posterior expectation, and uses camera reprojection errors as an efficient, data-driven proxy for hypothesis likelihood. RPEA is particularly motivated by the inherent one-to-many ambiguity in 2D-to-3D lifting tasks and is designed to aggregate generative model outputs in a principled and empirically effective manner (Wang et al., 5 Feb 2026).

1. Bayesian Foundation and Aggregation Problem

Monocular 3D pose estimation from 2D keypoints is a fundamentally ill-posed task due to both depth ambiguity and the non-injectivity of the camera projection mapping. Generative approaches, such as those leveraged in FMPose3D, address this by sampling $N$ plausible hypotheses $\{H_i\}_{i=1}^N$ from a learned conditional distribution $p_\theta(X^{3D}|Y)$, where $Y$ represents the observed 2D pose and $X^{3D}$ is the latent 3D structure. The challenge is then to aggregate these candidates into a single prediction $\widehat X$ for downstream usage.

Bayesian decision theory prescribes selecting $\widehat X$ to minimize the expected squared error, i.e., using the MMSE estimator:

$$\widehat X^{MMSE} = \mathbb{E}_{p(X^{3D}\mid Y)}[X^{3D}]$$

Since the posterior $p(X^{3D}|Y)$ is intractable in practice, RPEA proposes approximating this expectation via importance weighting, using a data-driven heuristic based on 2D reprojection error as a likelihood surrogate (Wang et al., 5 Feb 2026).

2. Mathematical Derivation and Estimator Formulation

Given $J$ joints, observed $Y \in \mathbb{R}^{J \times 2}$, and sampled hypotheses $\{H_i \in \mathbb{R}^{J\times 3}\}_{i=1}^N$, RPEA constructs weights $w_i$ over hypotheses by leveraging a reprojection consistency loss:

$$L(H_i, Y) = \sum_{j=1}^J \|\Pi(H_{i,j}) - Y_j\|_2^2$$

Here, $\Pi: \mathbb{R}^3 \rightarrow \mathbb{R}^2$ is the camera projection. The unnormalized likelihood proxy for hypothesis $H_i$ is $\exp(-\alpha L(H_i, Y))$, where $\alpha > 0$ is a temperature parameter.

The pose-wise RPEA estimator for the aggregate 3D pose is:

$$\widehat X = \sum_{i=1}^N w_i H_i, \quad\text{where}\quad w_i = \frac{\exp(-\alpha L(H_i, Y))}{\sum_{k=1}^N \exp(-\alpha L(H_k, Y))}$$

Empirical results demonstrate that a joint-wise variant, in which aggregation and weighting are computed separately for each joint, yields further improvements. For each joint $j$, individual joint reprojection errors $L_{i,j}$ are computed, the top $K$ hypotheses according to $L_{i,j}$ are retained, and normalized weights are constructed as:

$$w_{i, j} = \frac{ \exp(-\alpha L_{i,j}) }{ \sum_{H_{k,j} \in \mathcal{H}_{K,j}} \exp(-\alpha L_{k,j}) }$$

The aggregated joint position is then:

$$\widehat X_j^{RPEA} = \sum_{H_{i, j} \in \mathcal{H}_{K, j}} w_{i, j} H_{i, j}$$

The final estimate is $$\widehat X^{RPEA} = (\widehat X_1^{RPEA}, \ldots, \widehat X_J^{RPEA})$$.

3. Reprojection Error-Based Importance Weights

RPEA employs squared 2D reprojection error as an error metric for weighting. For each hypothesis, all joints are projected to 2D via the known (or assumed) camera matrix, and the Euclidean separation from the detected 2D keypoint ground truth is recorded. These errors form the basis for a softmax-style transformation (a radial basis function kernel with temperature $\alpha$) to generate positive weights emphasizing hypotheses that yield 3D proposals consistent with the 2D evidence.

In the joint-wise regime, the top $K$ minimal-error hypotheses for each joint are selected, enabling localized weighting that is robust to outlier samples and maintains a balance between diversity and fidelity to observations. This produces coherent output skeletons and minimizes distortion that might arise from single-joint greedy selection.

4. Algorithmic Workflow

The RPEA procedure can be outlined as follows:

1. For $N$ noise seeds $\{\epsilon_i\}_{i=1}^N$, solve the generative model’s ODE to obtain hypothesis $H_i \in \mathbb{R}^{J\times 3}$.
2. For each joint $j$ in each $H_i$, compute reprojected location $\hat y_{ij} = \Pi(H_{i,j})$ and the squared error $L_{i,j} = \|\hat y_{ij} - Y_j\|^2$.
3. For each joint $j$, determine the indices $\mathcal{I}_{K, j}$ of the $K$ smallest $L_{i,j}$.
4. For each $i \in \mathcal{I}_{K,j}$, assign an unnormalized weight $w_{i, j} = \exp(-\alpha L_{i, j})$ and normalize such that $\sum_{i \in \mathcal{I}_{K, j}} w_{i,j} = 1$.
5. Compute each joint aggregate as $$\widehat X_j = \sum_{i \in \mathcal{I}_{K, j}} w_{i,j} H_{i,j}$$.
6. Form the pose $\widehat X = (\widehat X_1, ..., \widehat X_J)$ as the final 3D prediction.

This process generalizes, with the “pose-wise” form corresponding to $K=N$, i.e., aggregating entire poses holistically rather than per-joint.

5. Empirical Comparison to Mean and MAP Aggregation

A primary empirical observation motivating RPEA is the failure of uniform or maximum likelihood (MAP) aggregation schemes to capitalize on the generative diversity produced by modern models. Three relevant aggregation baselines are:

Aggregation Method	MPJPE @ $N=40$	Remarks
Uniform Mean-Pooling	$\approx$ 49.0 mm	All samples weighted equally
Joint-wise MAP (JPMA)	$\approx$ 49.5 mm	Per-joint minimization but poor structure
RPEA (Joint-wise, Top-K)	47.3 mm	Best MPJPE, coherent structure

Mean-pooling suffers from hypothesis dilution; MAP discards diversity and loses potential for accuracy improvement as $N$ increases. Joint-MAP also degrades anatomical structure, harming metrics like P-MPJPE. In contrast, RPEA assigns higher weights to hypotheses with lower reprojection errors, leading to consistent performance gains and improved skeleton integrity, e.g., RPEA achieves a 4.8% relative improvement over prior state-of-the-art (Wang et al., 5 Feb 2026).

6. Theoretical Justification and Limitations

RPEA is a theoretically principled surrogate for the Bayes-optimal MMSE estimator in settings where the true posterior is intractable. By treating the exponential of the negative reprojection error as a likelihood proxy, RPEA exploits the geometric consistency between the 3D hypothesis and the observed 2D evidence. The reliance on $\alpha$ as a temperature parameter enables control over the sharpness of the weighting, bridging soft and hard selection.

However, as RPEA relies on the accuracy of the projection model and the informativeness of the 2D evidence, it may be sensitive to calibration errors or occlusions unaccounted for by the generative model. Furthermore, the selection of $K$ introduces a hyperparameter that governs diversity versus concentration, and the choice of joint-wise versus pose-wise aggregation impacts skeleton coherence and metric-specific performance.

7. Practical Significance and Broader Impact

RPEA’s demonstration within FMPose3D establishes a blueprint for leveraging generative sample diversity in ill-posed structured prediction problems where only partial observations are available. By unifying Bayesian decision theory with geometric error-based weighting, RPEA offers an architecture-agnostic mechanism that is demonstrably effective across domains, including both human and animal 3D pose estimation. The empirical superiority of RPEA over conventional aggregation strategies illustrates its value in scenarios with inherent posterior ambiguity and where uncertainty quantification is essential (Wang et al., 5 Feb 2026).