JOintGS: Joint Optimization for 3D Avatars

• JOintGS is a framework for joint optimization that concurrently refines camera parameters, human poses, and 3D Gaussian representations for robust monocular reconstruction.
• It employs a synergistic refinement loop integrating COLMAP and HMR2.0 initializations with separate foreground and background Gaussian fields to enhance multi-view consistency and temporal accuracy.
• The system achieves state-of-the-art novel-view synthesis performance while demonstrating robustness to noisy initialization and delivering sharper detail in dynamic scenes.

Searching arXiv for the target JOintGS paper and closely related 3DGS/avatar work to ground the article. JOintGS is a framework for reconstructing a high-fidelity, animatable 3D human avatar together with its static scene background from a single, in-the-wild RGB video by jointly optimizing camera extrinsics, human poses, and 3D Gaussian representations from coarse initialization through a synergistic refinement mechanism. Its central premise is that monocular reconstruction in unconstrained settings is limited less by splatting itself than by the inaccuracy of off-the-shelf camera calibration and human pose estimates; JOintGS therefore couples camera refinement, SMPL-based body refinement, foreground-background disentanglement, and pose-conditioned Gaussian optimization in one differentiable system (Lou et al., 4 Feb 2026).

1. Problem setting and research context

JOintGS addresses monocular human reconstruction under conditions in which both camera parameters and body poses are unreliable. The motivating failure modes are explicit: monocular SfM such as COLMAP often drifts in dynamic scenes because human motion violates the static-scene assumption, while off-the-shelf 3D pose regressors such as HMR2.0 provide only coarse SMPL fits with depth ambiguities and temporal jitters (Lou et al., 4 Feb 2026). The framework is therefore formulated around simultaneous refinement rather than fixed preprocessing.

Within the broader 3D Gaussian Splatting literature, JOintGS belongs to a family of methods that replace pipeline-stage isolation with joint optimization. For example, 3R-GS jointly optimizes 3D Gaussians and camera parameters from MASt3R-SfM initialization (Huang et al., 5 Apr 2025), while JGHand uses a joint-driven 3DGS formulation for animatable hand avatars controlled by 3D key points (Sun et al., 31 Jan 2025). JOintGS differs in scope by targeting full-body, in-the-wild monocular human-plus-scene reconstruction, with explicit separation of static background and dynamic foreground and with camera, pose, and Gaussian variables all refined in a single loop (Lou et al., 4 Feb 2026).

A plausible implication is that JOintGS should be read not merely as a rendering method, but as a reconstruction system in which representation, calibration, and kinematic estimation are mutually constraining. This interpretation is consistent with its emphasis on foreground-background disentanglement, multi-view consistency on static regions, and temporal modeling of human-specific non-rigid effects.

2. Representation and initialization

The optimization variables in JOintGS are the per-frame camera extrinsics, the human body parameters, and two Gaussian fields. Camera extrinsics are denoted by $\{T_t=[R_t\mid t_t]\}_{t=1}^T$, SMPL parameters by $\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$, and the Gaussian fields by background $\mathcal G^B=\{g_j^B\}$ and foreground human $\mathcal G^H=\{g_i^H\}$ (Lou et al., 4 Feb 2026). Each Gaussian has center $\mu\in\mathbb R^3$, covariance $\Sigma\in\mathbb R^{3\times3}$, opacity $\alpha\in[0,1]$, and view-dependent color modeled via spherical harmonics coefficients $\mathbf c\in\mathbb R^3$.

Initialization is deliberately coarse. JOintGS first runs COLMAP to estimate per-frame camera poses $\hat T_t^{(0)}$ and HMR2.0 to obtain per-frame SMPL fits $\hat\xi_t^{(0)}=(\hat\theta_t,\hat\beta)$. A static background Gaussian field is then built by splatting a sparse COLMAP point cloud with 10K–50K Gaussians, while a canonical human Gaussian field is instantiated by sampling approximately 10K points on the SMPL mesh in rest pose, each inheriting skinning weights $\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$0 from the nearest SMPL vertex (Lou et al., 4 Feb 2026).

This initialization scheme is structurally significant. The background field is static by construction, whereas the foreground field is canonical and later articulated. That division gives JOintGS two different geometric anchors: sparse but temporally stable scene geometry for camera refinement, and a body prior with skinning structure for human refinement. The resulting decomposition is more constrained than a single undifferentiated Gaussian cloud.

3. Synergistic refinement and foreground-background disentanglement

The core of JOintGS is its synergistic refinement loop. Three pathways are defined explicitly: background-anchored camera refinement, camera-guided pose refinement, and pose-aware Gaussian optimization (Lou et al., 4 Feb 2026). Static Gaussians in $\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$1 enforce multi-view photometric consistency on non-human pixels, providing stable camera updates. Refined cameras then improve projection accuracy for the human Gaussians in $\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$2, which sharpens the losses applied to SMPL parameters. Updated camera and pose estimates in turn improve foreground-background disentanglement, enabling both Gaussian fields to refine geometry and appearance more cleanly.

The background term is written as

$\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$3

where $\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$4 is rendered from $\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$5 only and $\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$6 is the human mask. The foreground term is

$\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$7

where $\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$8 is rendered from $\{\xi_t=(\theta_t,\beta)\}_{t=1}^T$9 given the current SMPL parameters (Lou et al., 4 Feb 2026).

The static background is also described as a volumetric density field,

$\mathcal G^B=\{g_j^B\}$0

with a corresponding color field given by spherical harmonics coefficients. This formalization clarifies why the background can act as a camera anchor even when the foreground is highly dynamic: it carries temporally shared geometry and appearance that can be enforced across frames (Lou et al., 4 Feb 2026).

This suggests that foreground-background disentanglement is not only a scene decomposition device but also the mechanism that stabilizes joint optimization. In JOintGS, better masks improve camera refinement, better cameras improve SMPL alignment, and better SMPL alignment improves mask separation. The method is therefore cyclical rather than sequential.

4. Objective function and temporal residual modeling

The full objective is

$\mathcal G^B=\{g_j^B\}$1

The rendering loss is

$\mathcal G^B=\{g_j^B\}$2

with $\mathcal G^B=\{g_j^B\}$3, $\mathcal G^B=\{g_j^B\}$4, and $\mathcal G^B=\{g_j^B\}$5. The prior term is

$\mathcal G^B=\{g_j^B\}$6

with $\mathcal G^B=\{g_j^B\}$7. The regularization term contains LBS weight regularization, canonical geometry anchoring, and dynamics regularization: $\mathcal G^B=\{g_j^B\}$8

$\mathcal G^B=\{g_j^B\}$9

$\mathcal G^H=\{g_i^H\}$0

weighted by $\mathcal G^H=\{g_i^H\}$1, $\mathcal G^H=\{g_i^H\}$2, and $\mathcal G^H=\{g_i^H\}$3 (Lou et al., 4 Feb 2026).

Beyond standard LBS skinning, JOintGS adds two lightweight MLPs to capture residual effects that the skeletal model does not explain directly. The Temporal Offset Module predicts pose-conditioned deformation residuals

$\mathcal G^H=\{g_i^H\}$4

followed by

$\mathcal G^H=\{g_i^H\}$5

The Residual Color Module predicts illumination-related color shifts

$\mathcal G^H=\{g_i^H\}$6

Both networks are shallow, hash-encoded MLPs whose outputs are regularized by $\mathcal G^H=\{g_i^H\}$7 (Lou et al., 4 Feb 2026).

The technical role of these modules is narrowly defined. They do not replace the skeletal model; they absorb only fine residual details beyond skeletal motion and static spherical-harmonic appearance. This makes the system more expressive without discarding the strong inductive bias supplied by SMPL and static-background geometry.

5. Optimization schedule and implementation profile

JOintGS uses a three-stage optimization schedule. In the warm-up stage, lasting 5K iterations, only $\mathcal G^H=\{g_i^H\}$8 and $\mathcal G^H=\{g_i^H\}$9 are optimized, with learning rate for Gaussians approximately $\mu\in\mathbb R^3$0. In the independent stage, also 5K iterations, camera updates are enabled from $\mu\in\mathbb R^3$1 and SMPL updates from $\mu\in\mathbb R^3$2 in parallel. In the joint stage, lasting 10K iterations, the full objective $\mu\in\mathbb R^3$3 is optimized with cosine-annealed learning rates down to $\mu\in\mathbb R^3$4 the initial rate (Lou et al., 4 Feb 2026).

Implementation details are correspondingly explicit. Camera poses are initialized by COLMAP without bundle adjustment on dynamic pixels. SMPL pose and shape are initialized by HMR2.0 and aligned in scale via RANSAC to COLMAP depths. The network backbone uses tri-plane feature grids with 512 channels per plane, together with two 2-layer MLP decoders with GELU activations for appearance and geometry. Training uses 4096 rays per iteration, randomly sampled from all frames (Lou et al., 4 Feb 2026).

Evaluation is reported on NeuMan, consisting of 6 sequences of 10–20 seconds each with an 80/10/10 train/val/test split, and EMDB, consisting of 10 sequences of 200 frames each with the same split (Lou et al., 4 Feb 2026). These dataset choices are consistent with the method’s emphasis on uncontrolled monocular capture, moving humans, and jointly reconstructed scene context.

A plausible implication is that the staged schedule is necessary because the variables are strongly coupled but differently conditioned. Gaussian appearance and geometry can be stabilized before camera and SMPL corrections become dominant; only then does full joint refinement become numerically favorable.

6. Empirical results, robustness, and limitations

On NeuMan, JOintGS reports human-only novel-view synthesis performance of 34.84 dB PSNR, 0.984 SSIM, and 0.010 LPIPS. The best prior listed in the provided results is Vid2Avatar-Pro at 32.71 dB, giving a gain of 2.13 dB. On EMDB, JOintGS achieves 30.99 dB, compared with 28.95 dB for ODHSR (Lou et al., 4 Feb 2026). The abstract summarizes the gain on NeuMan as 2.1 dB over state-of-the-art methods while maintaining real-time rendering.

Ablation results indicate that the synergistic design is structurally important rather than cosmetic. On NeuMan, the full model yields 34.84 dB average human PSNR. Removing the Temporal Dynamics module reduces performance to 34.23 dB, a drop of 0.61 dB. Removing synergistic refinement by optimizing all variables jointly from iteration 0 reduces performance to 31.38 dB, a drop of 3.46 dB (Lou et al., 4 Feb 2026). This strongly suggests that staged coordination between camera, pose, and Gaussian updates is central to the method.

Robustness to noisy initialization is also reported directly. When Gaussian noise with standard deviation $\mu\in\mathbb R^3$5 is added to $\mu\in\mathbb R^3$6 and $\mu\in\mathbb R^3$7, JOintGS shows a 0.9 dB PSNR drop at $\mu\in\mathbb R^3$8, whereas HUGS drops by 3.7 dB (Lou et al., 4 Feb 2026). On full-image rendering for NeuMan, the recovered background Gaussians enable 32.52 dB PSNR, compared with 25.94 dB for HUGS (Lou et al., 4 Feb 2026). Qualitatively, the method is described as producing crisper facial wrinkles, sharper clothing folds, and more accurate silhouettes than prior methods.

The limitations are stated in two parts. First, avatar fidelity remains bounded by SMPL’s low-dimensional shape space, so hands and faces remain less detailed. Second, long dynamic occlusions, such as back-turns, can still cause local drifting in the Gaussian field (Lou et al., 4 Feb 2026). The cited future directions are correspondingly targeted: replacing SMPL with a more expressive model such as SMPL-X or parametric face/hands, integrating learned temporal priors for long occlusions, and incorporating learned illumination models to disentangle reflectance and lighting.

Taken together, JOintGS occupies a specific position in 3DGS-based human reconstruction: it is a monocular in-the-wild system whose main contribution lies in making camera refinement, body refinement, and Gaussian optimization interdependent rather than sequential. Its reported gains on NeuMan and EMDB, together with its explicit robustness analysis, indicate that this interdependence is not merely an implementation choice but the operative principle of the framework (Lou et al., 4 Feb 2026).