Diff3R: Optimization-Aware 3D Gaussian Splatting

• Diff3R is a feed-forward 3D Gaussian Splatting framework that integrates test-time optimization to enhance reconstruction from sparse views.
• It employs a bilevel optimization approach with a differentiable inner loop to predict initial parameters and per-parameter uncertainty for refined scene representations.
• The method utilizes matrix-free implicit differentiation and a preconditioned conjugate gradient solver to efficiently compute gradients, leading to robust and high-quality renderings.

Searching arXiv for Diff3R and closely related 3DGS optimization papers. Diff3R is a framework for feed-forward 3D Gaussian Splatting (3DGS) that is explicitly trained to work with test-time optimization rather than in spite of it. It addresses the tension between fast feed-forward inference in sparse-view settings and the higher rendering quality often obtained by per-scene optimization. The central mechanism is a differentiable 3DGS optimization layer inserted into the training loop, so that the network learns to predict an initialization $\Theta_0$ and per-parameter uncertainty or regularization weights $\boldsymbol{\Lambda}$ that are optimized at test time into a refined scene representation $\Theta^*$ (Liu et al., 1 Apr 2026).

1. Problem setting and conceptual objective

In 3DGS, a scene is represented as a large set of 3D Gaussians

$$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$

with center $\mu_i\in\mathbb{R}^3$, scale matrix $S_i$, rotation $R_i$, opacity $\alpha_i\in[0,1]$, and SH color coefficients $c_i$. Rendering from a camera $P$ involves projecting these Gaussians and alpha-blending,

$\boldsymbol{\Lambda}$0

Diff3R is situated between two established 3DGS paradigms: per-scene 3DGS optimization, which is highly accurate but slow and requires dense multiview supervision, and feed-forward 3DGS, which predicts $\boldsymbol{\Lambda}$1 directly from a few views in one forward pass but is limited by priors and often lacks fine details. The paper characterizes naïve test-time optimization in sparse-view settings as prone to severe overfitting, so the objective is not merely to optimize a predicted scene, but to train the predictor so that its output is well suited for subsequent optimization (Liu et al., 1 Apr 2026).

This objective is implemented by treating test-time optimization itself as part of learning. Instead of training a network to output final Gaussians, Diff3R trains the network to output an initialization and a regularization structure such that a short inner optimization on the context views yields Gaussians with improved novel-view quality. This reformulation is expressed as a bilevel optimization problem. A plausible implication is that Diff3R shifts the target of feed-forward prediction from zero-shot reconstruction quality to optimization-aware reconstructability.

2. Architecture, bilevel formulation, and optimization layer

Given a small set of context images with optional camera parameters,

$\boldsymbol{\Lambda}$2

a feed-forward network $\boldsymbol{\Lambda}$3 predicts

$\boldsymbol{\Lambda}$4

together with per-parameter regularization weights

$\boldsymbol{\Lambda}$5

where $\boldsymbol{\Lambda}$6 contains one weight for each scalar component of $\boldsymbol{\Lambda}$7. In practice, the paper uses a DPT head to predict per-pixel Gaussians and a small MLP to predict $\boldsymbol{\Lambda}$8 from intermediate feature maps. The framework is demonstrated with a DepthSplat-based pose-given pipeline and a Depth Anything v3–based pose-free pipeline (Liu et al., 1 Apr 2026).

The differentiable optimization layer bridges feed-forward prediction and test-time optimization through four steps. First, the model predicts $\boldsymbol{\Lambda}$9 and $\Theta^*$0 from the context views. Second, it runs an inner test-time optimization of $\Theta^*$1 to obtain $\Theta^*$2 that fits the context views under a proximal regularizer anchored at $\Theta^*$3. Third, it evaluates a novel-view outer loss on the rendered image $\Theta^*$4 against the target novel view $\Theta^*$5. Fourth, it backpropagates through the entire pipeline, including the optimization layer, via implicit differentiation (Liu et al., 1 Apr 2026).

The baseline inner objective is

$\Theta^*$6

where $\Theta^*$7 stacks rendering residuals over the context images. The proximal term is crucial because it bounds the inner optimization, prevents Gaussians from drifting arbitrarily, and makes the inner optimum dependent on $\Theta^*$8. The outer objective evaluates the optimized Gaussians on a held-out novel view:

$\Theta^*$9

In practice, the photometric term follows standard 3DGS and uses L1+SSIM rather than pure L2, although the theory is presented in the least-squares form for analytical clarity (Liu et al., 1 Apr 2026).

3. Implicit differentiation and matrix-free optimization

The main technical difficulty is computing gradients through the converged inner optimization without unrolling all optimization steps. Diff3R addresses this with the Implicit Function Theorem and a Gauss–Newton approximation. At the optimum $$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$0, the optimality condition is

$$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$1

Differentiating this condition with respect to $$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$2 yields

$$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$3

so that

$$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$4

Under the Gauss–Newton approximation for a residual-squared loss, $$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$5, giving

$$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$6

with $$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$7 (Liu et al., 1 Apr 2026).

The required gradient is computed through the vector–Jacobian product

$$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$8

Defining

$$\Theta = \{ \mu_i, S_i, R_i, \alpha_i, c_i \}_{i=1}^N$$9

this is equivalent to solving the linear system

$\mu_i\in\mathbb{R}^3$0

Directly forming $\mu_i\in\mathbb{R}^3$1 is infeasible at 3DGS scale, so Diff3R uses a matrix-free Preconditioned Conjugate Gradient solver. PCG requires only matrix–vector products of the form $\mu_i\in\mathbb{R}^3$2, implemented through forward Jacobian–vector products, transposed-Jacobian–vector products, and addition of the damping term. The implementation reuses CUDA kernels from 3DGS-LM, normalizes $\mu_i\in\mathbb{R}^3$3 by the number of residuals for stability, and employs a Levenberg–Marquardt-style scaling and a Jacobi-type preconditioner derived from diagonal structure (Liu et al., 1 Apr 2026).

The paper emphasizes two consequences. First, the method avoids storing activations for every inner optimization step, which would be infeasible at 3DGS scale. Second, it computes the exact gradient at the converged point under the Gauss–Newton approximation with constant memory and manageable compute. Empirically, this enables training with large numbers of Gaussians and inner steps in the 50–100 range.

4. Uncertainty-aware regularization and robustness

Diff3R generalizes the global scalar proximal weight to per-parameter regularization weights. The inner loss becomes

$\mu_i\in\mathbb{R}^3$4

Here each scalar Gaussian parameter $\mu_i\in\mathbb{R}^3$5 has its own $\mu_i\in\mathbb{R}^3$6. Large $\mu_i\in\mathbb{R}^3$7 indicates a confident parameter that is strongly anchored to initialization, while small $\mu_i\in\mathbb{R}^3$8 indicates an uncertain parameter that is free to move during optimization. These weights are predicted from image features and learned only from outer novel-view loss; no ground-truth uncertainty is provided (Liu et al., 1 Apr 2026).

With this modification, the Gauss–Newton system becomes

$\mu_i\in\mathbb{R}^3$9

The analytical gradients are

$S_i$0

and

$S_i$1

The second expression provides the interpretive core of the uncertainty model: if a parameter’s optimization movement aligns with the direction favored by outer loss, the gradient decreases its regularization weight, allowing more movement next time; if the movement is detrimental, the regularization is increased (Liu et al., 1 Apr 2026).

The paper attributes two forms of robustness to this mechanism. In under-constrained regions, such as areas visible from only one viewpoint, the model often predicts higher $S_i$2, anchoring Gaussian means to their prior positions and mitigating overfitting. It also improves robustness to exposure mismatches and other input outliers by reducing the tendency of optimization to introduce artifacts, floaters, or view-dependent distortions that fit the context views at the expense of novel-view quality. The visualizations of $S_i$3 in monocular regions support this interpretation.

5. Integration with feed-forward 3DGS and empirical performance

The optimization layer is described as model-agnostic: it treats the 3DGS pipeline as a black-box renderer $S_i$4 and requires Gaussian parameters, residuals, and Jacobians of residuals with respect to those parameters. In the pose-given setting, Diff3R is integrated with DepthSplat. The input is 2 context images with known poses; the network predicts per-pixel depth or 3D points, then Gaussians $S_i$5, while a small MLP predicts $S_i$6. In the pose-free setting, the paper uses a Depth Anything v3–based 3DGS system with 4 unposed images, followed by the same uncertainty-aware refinement mechanism (Liu et al., 1 Apr 2026).

Training follows a two-stage schedule. In Stage 1, the base network is pretrained with only the outer loss and no inner optimization, so $S_i$7. In Stage 2, inner optimization is activated for 50–100 steps, the refined scene is evaluated with the outer loss, and an auxiliary proxy loss

$S_i$8

is added to prevent degenerate initializations that become acceptable only after extreme test-time optimization. The effective regularization is further scaled by a moving average $S_i$9 of the diagonal of $R_i$0 through

$R_i$1

so that parameter types with different sensitivities are regularized on different scales (Liu et al., 1 Apr 2026).

The principal quantitative results reported for RealEstate10K and ScanNet++ are summarized below.

Setting	Baseline after TTO	Diff3R after TTO
RE10K, pose-given, 2 context views, PSNR	DepthSplat: 27.694	27.949
RE10K, pose-given, 2 context views, LPIPS	DepthSplat: 0.116	0.112
RE10K, pose-free, 4 unposed views, PSNR	DA3: 22.582	22.806
RE10K, pose-free, 4 unposed views, LPIPS	DA3: 0.205	0.202
ScanNet++, DSLR, pose-free, PSNR	DA3: 24.29	24.64
ScanNet++, iPhone, pose-free, PSNR	DA3: 24.43	24.57

On RE10K with known poses, zero-shot performance is similar to DepthSplat, but the gain after test-time optimization is much larger: DepthSplat improves by $R_i$2 dB, whereas Diff3R improves by $R_i$3 dB. On RE10K with unposed views, AnySplat degrades under test-time optimization, DA3 improves by $R_i$4 dB, and Diff3R improves by $R_i$5 dB. On ScanNet++, Diff3R yields a final PSNR of $R_i$6 on DSLR and $R_i$7 on iPhone, with better or comparable SSIM and LPIPS relative to the DA3 baseline. The qualitative interpretation offered in the paper is sharper renderings, fewer artifacts, better geometry, and increased robustness to exposure variations (Liu et al., 1 Apr 2026).

Ablations isolate the contribution of the two main design elements. Without the proximal term, the model still improves over the DepthSplat baseline but remains inferior to the full system, indicating that unconstrained optimization still overfits. Without learnable $R_i$8, optimization is described as too conservative. Full Diff3R achieves the largest gain, which the paper uses to argue that both the proximal anchor and the learned per-parameter regularization are necessary.

6. Limitations, computational profile, and place in the literature

Diff3R improves test-time optimization at the cost of substantially more expensive training. The paper states that training is 4–5$R_i$9 slower per step than pure feed-forward training, with about 65% of the additional time coming from the 50–100 inner optimization steps. Test-time cost does not include the implicit-gradient machinery; it is similar to other test-time-optimization-enabled 3DGS pipelines, because only the standard inner optimization is run during inference. The approach scales to typical 3DGS scene sizes through matrix-free PCG and optimized GPU kernels, and the paper describes the memory footprint as manageable precisely because optimization is not unrolled (Liu et al., 1 Apr 2026).

The paper also identifies methodological limits. Diff3R inherits the limitations of 3DGS and sparse-view novel-view synthesis: extremely sparse views or severe occlusions may still produce artifacts. The uncertainty model is diagonal and does not represent correlations between parameters or Gaussians. The inner optimization is approximate because it uses Gauss–Newton and finite stopping, so the assumption of full convergence is not exact. These constraints delimit the scope of the current formulation rather than negating its empirical gains.

Within the literature, Diff3R is positioned relative to original per-scene 3DGS, feed-forward 3DGS methods such as PixelSplat, MVSplat, DepthSplat, and GAUSSIAN LRM, and optimization-aware alternatives such as G3R and QuickSplat. Its distinguishing features are an explicit bilevel formulation for 3DGS test-time optimization, implicit gradients at the inner optimum via the Implicit Function Theorem, and a learned per-parameter uncertainty or regularization model. It is further aligned with differentiable optimization traditions associated with MAML, implicit meta-learning, BA-Net, VGGSfM, and Theseus, but applies those ideas to dense 3DGS with hundreds of thousands to millions of Gaussians. This suggests that Diff3R’s main contribution is not a new renderer or a new feed-forward predictor in isolation, but a way to make optimization itself a trainable and uncertainty-aware component of feed-forward 3DGS (Liu et al., 1 Apr 2026).