Joint Multi-view Body Optimization (JMBO)

• JMBO is an optimization framework that jointly refines a single SMPL-X body model from multiple calibrated views to enforce geometric consistency.
• It employs an alternating optimization strategy, balancing reprojection, silhouette, landmark, and normal consistency to enhance precision, speed, and convergence.
• JMBO serves as a structural, low-frequency anatomical prior that guides downstream clothed-human surface reconstruction and improves metrics like Chamfer distance and pose accuracy.

Joint Multi-view Body Optimization (JMBO) denotes a class of optimization procedures that estimate a single, geometrically consistent human body model or 3D pose by jointly enforcing constraints from multiple views. In its explicit form, JMBO is introduced as the core body-fitting stage of MExECON, where one SMPL-X body is optimized across all calibrated input images before clothed-surface reconstruction (Uğur et al., 21 Aug 2025). In closely related earlier work on multi-person pose estimation, the same essential pattern appears without the term itself: 3D joints and SMPL parameters are jointly refined from multi-view images through an alternating optimization that links confidence-weighted reprojection with body-model regularization (Dong et al., 2021). Across these formulations, the defining property is not merely the use of multiple cameras, but the joint coupling of multi-view evidence with a parametric body prior so that pose, shape, and geometric consistency are solved together rather than in isolation.

1. Definition and conceptual scope

In the narrowest sense, JMBO is the optimization stage in MExECON that “fits a single SMPL-X body model jointly across all input views, enforcing multi-view consistency” (Uğur et al., 21 Aug 2025). The optimized body model is then used as a low-frequency prior for subsequent clothed-human reconstruction. The term therefore refers to a multi-view body-fitting problem in which all images constrain one shared parametric human model.

A broader technical reading is justified by the multi-person pose-estimation pipeline of “Shape-aware Multi-Person Pose Estimation from Multi-View Images,” whose second stage is described as a joint optimization over 3D joints and SMPL parameters, alternating between updating the 3D pose $X$ and the SMPL parameters $\Theta=\{\theta,\beta\}$ until convergence (Dong et al., 2021). This suggests that JMBO is best understood as a methodological family rather than a single implementation: the central motif is a closed optimization loop in which multi-view observations improve the body estimate, and the body estimate in turn regularizes ambiguous or incomplete observations.

Related work on consensus-based multi-view pose fusion and on joint optimization of cameras, bodies, and dense scene representations further enlarges the conceptual perimeter. Consensus-based optimization for 3D human pose estimation formulates multi-view fusion in camera coordinates and alternates over extrinsics, intrinsics, and pose consistency (Luvizon et al., 2019). JOintGS generalizes the same joint-estimation logic to monocular video by simultaneously refining camera extrinsics, human pose parameters, and 3D Gaussian representations under differentiable rendering (Lou et al., 4 Feb 2026). These are not labeled JMBO, but they show that joint optimization over body-related variables and multi-view or multi-frame consistency is a recurring design principle.

2. Core optimization formulations

The most explicit JMBO objective is given in MExECON as a multi-view extension of ECON’s single-view SMPL-X fitting objective: $$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$ with

$\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$

Here, silhouette consistency matches projected body contours to clothed silhouettes in every view,

$$\mathcal{L}_{\text {Silhouette}}= \sum\nolimits_{i=1}^N \|\mathcal{S}_{i}^{\text{b}}-\mathcal{S}_{i}^{\text{c}}\|_1,$$

normal consistency aligns rendered body normals with predicted clothed normals,

$$\mathcal{L}_{\text {Normals}}= \sum\nolimits_{i=1}^N \|\mathcal{N}_i^{\text{b}} - \widehat{\mathcal{N}_i^{\text{c}}}\|_1,$$

landmark consistency uses confidence-weighted 2D landmarks detected with MediaPipe,

$$\mathcal{L}_{\text {Landmarks}}= \sum\nolimits_{i=1}^N \frac{\sum_k c_{i,k}\cdot \|\mathcal{J}_{i,k}^{\text{b}}-\mathcal{J}_{i,k}^{\text{c}}\|_2}{\sum_k c_{i,k}},$$

and head regularization penalizes unrealistic head tilting,

$$\mathcal{L}_{\text{Head}}= 1-\frac{\mathbf{v}_h\cdot \mathbf{v}_y}{\|\mathbf{v}_h\|\,\|\mathbf{v}_y\|}.$$

The optimized variable is a single shared SMPL-X body $\mathcal{M}^{\text{b}}$, initialized from per-view PIXIE predictions and constrained simultaneously by all views (Uğur et al., 21 Aug 2025).

The closely related formulation in the shape-aware multi-person estimator uses a different decomposition. Its total energy is

$$E(X,\theta,\beta) = w_{2D}E_{2D}(X) + w_{shape}E_{shape}(X,\theta,\beta),$$

with a confidence-weighted reprojection term

$\Theta=\{\theta,\beta\}$0

and a body-model regularization term

$\Theta=\{\theta,\beta\}$1

The indicators are defined by thresholding reprojection residuals and joint confidences: $\Theta=\{\theta,\beta\}$2 The body prior is provided by SMPL through

$\Theta=\{\theta,\beta\}$3

where $\Theta=\{\theta,\beta\}$4 is the mesh, $\Theta=\{\theta,\beta\}$5 are pose parameters, $\Theta=\{\theta,\beta\}$6 are shape parameters, and $\Theta=\{\theta,\beta\}$7 is the joint regressor (Dong et al., 2021).

These two formulations reveal an important distinction. In MExECON, the body model itself is the principal optimization target. In the earlier multi-person setting, the optimization couples a nonparametric 3D joint configuration $\Theta=\{\theta,\beta\}$8 with a parametric body prior, and the body model acts primarily as a regularizer and completion mechanism. A plausible implication is that JMBO spans both “body-first” and “pose-plus-body” variants, provided that multi-view evidence and parametric body constraints are solved jointly.

3. Initialization and multi-view evidence construction

JMBO pipelines depend strongly on how cross-view evidence is assembled before or during optimization. In MExECON, each input image $\Theta=\{\theta,\beta\}$9 first yields an independent SMPL-X estimate $$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$0 using PIXIE. The shape and pose parameters from these per-view predictions are averaged to initialize the shared body model $$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$1 (Uğur et al., 21 Aug 2025). This initialization preserves compatibility with ECON and avoids network re-training, which is a stated design goal.

In the multi-person pose-estimation setting, initialization is more explicitly geometric. Off-the-shelf 2D pose detections from OpenPose are triangulated pairwise into 3D joint candidates. For each part label $$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$2, the method solves a linear triangulation system

$$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$3

with $$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$4, and in the implementation each triangulation uses pairs of poses, so $$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$5. The resulting candidate set $$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$6 is then processed by confidence-aware majority voting: the highest-confidence candidate $$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$7 is selected, nearby candidates $$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$8 within distance threshold $$\mathcal{L}_{\text{SMPL-X}}= \mathcal{L}_{\text {Silhouette}} +\lambda_n \mathcal{L}_{\text {Normal}} +\lambda_l \mathcal{L}_{\text {Landmark}} +\lambda_h \mathcal{L}_{\text {Head}},$$9 are grouped, clusters with fewer than three points are discarded as outliers, and the cluster center becomes the representative 3D joint (Dong et al., 2021).

Person-level association is then obtained through a combination of hip-based anchoring and image-space reasoning. The hip is used to determine the number and coarse location of people; a fixed-size, fixed-orientation 3D bounding box is placed around hip candidates; anchors are kept only if the box contains more than 90% of body parts and the average confidence exceeds a threshold; and Part Affinity Fields are used after projection back into the image to separate overlapping people (Dong et al., 2021). This matters because the downstream optimization assumes a reasonably coherent per-person proposal. The earlier clustering and proposal stage therefore functions as the observation-construction layer of JMBO-like refinement.

A concise comparison of these initializations clarifies the range of practice.

Setting	Initialization	Primary evidence source
MExECON	Averaged per-view PIXIE SMPL-X predictions	Silhouettes, normals, landmarks, head term
Shape-aware multi-person pose estimation	Triangulated 3D joint candidates clustered by confidence-aware majority voting	OpenPose detections, triangulation, PAF-based separation
Consensus-based multi-view pose fusion	Monocular absolute pose predictions in camera coordinates	View-frustum prediction and geometric consensus

The table summarizes distinct entry points into joint optimization, but in all cases the optimization depends on cross-view consistency rather than a single-image estimate.

4. Optimization dynamics and coupling mechanisms

The defining algorithmic characteristic of JMBO is the mutual dependence between body variables and multi-view constraints. In the shape-aware multi-person method, this dependence is made explicit through alternating optimization. Starting from an initial 3D pose $\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$0 and zero-initialized SMPL parameters $\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$1, the procedure iterates over two steps: first, optimize the SMPL parameters with $\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$2 fixed using a learned gradient update network,

$\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$3

and then optimize the 3D joints with $\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$4 fixed by minimizing

$\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$5

followed by the update

$\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$6

The optimization is repeated until convergence, and the ablation figure shows precision saturating after about 10 steps (Dong et al., 2021). The important point is structural: the current 3D pose guides SMPL fitting, and the updated SMPL estimate regularizes the next pose update.

MExECON uses a different but conceptually related coupling. The shared SMPL-X body is optimized directly across all views, so the joint variable is already globally constrained. Silhouettes keep the body inside the visible human outline, normals align the body with predicted clothed orientation cues, landmarks align projected joints to image evidence, and the head term stabilizes head orientation, which the paper describes as especially sensitive to landmark fitting (Uğur et al., 21 Aug 2025). Here the coupling is not between separate $\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$7 and $\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$8 variables, but between one body model and several complementary observation channels.

Consensus-based optimization for 3D human pose estimation supplies a broader geometric analogue. For two cameras $\lambda_n = 0.2,\qquad \lambda_l=\lambda_h=0.1.$9 and $$\mathcal{L}_{\text {Silhouette}}= \sum\nolimits_{i=1}^N \|\mathcal{S}_{i}^{\text{b}}-\mathcal{S}_{i}^{\text{c}}\|_1,$$0, the transformed pose is

$$\mathcal{L}_{\text {Silhouette}}= \sum\nolimits_{i=1}^N \|\mathcal{S}_{i}^{\text{b}}-\mathcal{S}_{i}^{\text{c}}\|_1,$$1

and the optimization seeks camera parameters

$$\mathcal{L}_{\text {Silhouette}}= \sum\nolimits_{i=1}^N \|\mathcal{S}_{i}^{\text{b}}-\mathcal{S}_{i}^{\text{c}}\|_1,$$2

that minimize pose disagreement,

$$\mathcal{L}_{\text {Silhouette}}= \sum\nolimits_{i=1}^N \|\mathcal{S}_{i}^{\text{b}}-\mathcal{S}_{i}^{\text{c}}\|_1,$$3

The algorithm alternates between updating rotation via rigid Procrustes alignment, updating translation through a closed-form average solution, cyclically updating camera intrinsics, and recomputing projections (Luvizon et al., 2019). Although this work does not use a body model like SMPL-X, it demonstrates the same principle of alternating consensus optimization across interdependent variables.

5. Anatomical priors, regularization, and downstream reconstruction

A central feature of JMBO is the use of a parametric body model as a structured prior rather than as the final reconstructed surface. In the shape-aware multi-person formulation, the shape term applies only to low-confidence joints, because

$$\mathcal{L}_{\text {Silhouette}}= \sum\nolimits_{i=1}^N \|\mathcal{S}_{i}^{\text{b}}-\mathcal{S}_{i}^{\text{c}}\|_1,$$4

and $$\mathcal{L}_{\text {Silhouette}}= \sum\nolimits_{i=1}^N \|\mathcal{S}_{i}^{\text{b}}-\mathcal{S}_{i}^{\text{c}}\|_1,$$5 otherwise. The paper explicitly states that the parametric model helps correct implausible 3D poses and fill missing joint detections, while updated 3D poses guide better SMPL estimates (Dong et al., 2021). The prior therefore intervenes selectively, chiefly where observation quality is poor.

The same selectivity explains an important empirical note: the paper emphasizes that SMPL helps especially when the 3D joints are low-confidence; for high-confidence joints, the SMPL prior may not help much because of small systematic skeletal differences between OpenPose and SMPL (Dong et al., 2021). This addresses a common misconception that stronger body priors uniformly improve all joints. The reported behavior is more conditional: body-model regularization is most useful in the low-confidence regime.

In MExECON, the optimized body model $$\mathcal{L}_{\text {Silhouette}}= \sum\nolimits_{i=1}^N \|\mathcal{S}_{i}^{\text{b}}-\mathcal{S}_{i}^{\text{c}}\|_1,$$6 is explicitly described as a low-frequency anatomical prior. It is a skinned model that lacks accessories, hair, and clothing detail, and it is “mainly used to guide the reconstruction of the clothed surface” (Uğur et al., 21 Aug 2025). This prior is injected downstream in two ways. First, body normals from $$\mathcal{L}_{\text {Silhouette}}= \sum\nolimits_{i=1}^N \|\mathcal{S}_{i}^{\text{b}}-\mathcal{S}_{i}^{\text{c}}\|_1,$$7 condition clothed-normal prediction. Second, body depths from $$\mathcal{L}_{\text {Silhouette}}= \sum\nolimits_{i=1}^N \|\mathcal{S}_{i}^{\text{b}}-\mathcal{S}_{i}^{\text{c}}\|_1,$$8 guide d-BiNI and IF-Nets+ during surface reconstruction and completion. The relevant pipeline is:

1. JMBO optimizes one SMPL-X body across all views.
2. Front and back clothed normal maps are predicted, using the body normals as conditioning.
3. d-BiNI lifts these normal maps into depth maps using body depths.
4. IF-Nets+ fills missing geometry.
5. Poisson Surface Reconstruction stitches the partial surfaces and completed regions into the final watertight avatar (Uğur et al., 21 Aug 2025).

This makes clear that JMBO is not synonymous with clothed-human reconstruction. It is the body-estimation backbone that stabilizes subsequent detail recovery. The paper is explicit that the optimized SMPL-X body does not directly produce clothing geometry, but instead supplies the structural scaffold for later surface and detail reconstruction (Uğur et al., 21 Aug 2025).

6. Empirical behavior, related formulations, and limitations

The reported empirical gains support the claim that joint multi-view body optimization is not only a conceptual reformulation but an effective optimization strategy. In the multi-person pose-estimation setting, the Association Dataset ablation reports Ours(init) at Precision 83.7, Recall 82.8, F1 83.4, and Ours(final) at Precision 90.1, Recall 89.0, F1 89.2, indicating a large gain from the shape-aware optimization stage (Dong et al., 2021). Against a multi-view variant of SMPLify, the comparison is 78.3 / 77.4 / 77.8 versus 90.1 / 89.0 / 89.2, with learned gradient descent reported to yield about 20x speed-up, convergence in 14 vs 100 iterations, and precision improvement from 78.3% to 90.1%. The convergence plot shows the alternating optimization saturating after roughly 10 steps and reaching over 90% precision.

In MExECON, the body-fitting ablation reports total loss reduction from 0.229 for the single front-view SMPL-X estimate to 0.059 for JMBO, with intermediate performance from a naive multi-view average SMPL-X at 0.113 (Uğur et al., 21 Aug 2025). On the 20-avatar THuman 2.1 test set, ECON reports Chamfer 40.56, P2S Acc. 39.52, P2S Comp. 37.21, and Normal 45.79; 8-view MExECON reports 23.12, 21.26, 22.84, and 39.54; and the paper highlights 42% lower Chamfer distance, 46% lower P2S accuracy error, 38% lower P2S completeness error, and 13% better normal consistency relative to ECON. The 2-view variant lies between ECON and 8-view MExECON, supporting the claim that additional views progressively resolve ambiguities.

Two nearby research directions clarify what JMBO is and is not. Consensus-based multi-view pose estimation in camera coordinates shows that absolute camera-frame poses can be fused geometrically, even for uncalibrated images, through optimization over camera parameters and pose consistency (Luvizon et al., 2019). This is JMBO-like in the sense of multi-view consensus, but it does not center on a shared parametric body prior such as SMPL-X. JOintGS, by contrast, extends joint optimization beyond multi-view body fitting to a bundle-adjustment-like reconstruction framework that jointly optimizes camera extrinsics, human poses, and 3D Gaussian scene-human representations, with foreground-background disentanglement supplying the key stabilization mechanism (Lou et al., 4 Feb 2026). A plausible implication is that JMBO can be regarded as one specialized instance within a larger family of joint human-centered geometric optimization methods.

The limitations reported in the underlying literature are also instructive. MExECON’s optimized body is intentionally low-frequency and cannot itself represent clothing, hair, or accessories (Uğur et al., 21 Aug 2025). JOintGS notes a related limitation of reliance on the SMPL body model, which cannot fully capture fine-grained regions such as hands, faces, and expressive gestures, and identifies more expressive models such as SMPL-X as a longer-term solution (Lou et al., 4 Feb 2026). These observations suggest that JMBO is strongest as a structural prior and optimization scaffold, while high-frequency geometry generally requires additional reconstruction modules.

In aggregate, JMBO names a precise optimization pattern: one shared body hypothesis is constrained by multiple views, and that hypothesis is refined by jointly balancing image evidence, geometric consistency, and anatomical priors. The term is explicit in MExECON, exact in substance in the earlier shape-aware multi-person refinement stage, and closely related to broader consensus and joint-optimization frameworks in multi-view and monocular reconstruction (Uğur et al., 21 Aug 2025).