Learn2Splat: Meta-Learned 3D Optimizer

• Learn2Splat is a meta-learned optimizer for 3D Gaussian Splatting that leverages geometric and spatial relationships to enable rapid convergence and robust performance.
• The framework employs a latent state architecture with a kNN-Point Transformer and update MLP to decouple update directionality and magnitude, ensuring stable optimization over long horizons.
• It generalizes across datasets, view configurations, and resolutions without re-training, outperforming traditional methods like Adam and SGD with up to 24× speedup and notable PSNR gains.

The Learn2Splat framework is a meta-learned optimizer designed to accelerate and stabilize optimization for 3D Gaussian Splatting (3DGS). Unlike standard optimizers such as Adam or SGD, which apply independent, structure-agnostic parameter updates, Learn2Splat leverages the rich geometric and spatial relationships inherent in 3DGS, enabling faster convergence and robust long-horizon stability. Through a meta-learning scheme featuring a latent state architecture, a checkpoint buffer, and an optimizer rollout strategy, Learn2Splat achieves strong early and final novel view synthesis performance while generalizing across datasets, resolutions, and both sparse and dense view regimes without retraining or fine-tuning (Pearl et al., 15 May 2026).

1. 3D Gaussian Splatting Optimization and Associated Challenges

3D Gaussian Splatting [20] represents a scene as a collection of $G$ anisotropic Gaussians:

$\mathcal{G} = \{ G_m \}_{m=1}^G,$

where each $G_m$ encompasses a 3D center $P_m \in \mathbb{R}^3$, scale $S_m \in \mathbb{R}^3$, rotation $q_m \in \mathbb{R}^4$, opacity $o_m \in [0,1]$, and spherical harmonics color $SH_m \in \mathbb{R}^{d \times 3}$. For each camera view $i$, rendering produces image $I_i(\mathcal{G})$, with photometric losses (e.g., $\mathcal{G} = \{ G_m \}_{m=1}^G,$0 and SSIM) driving optimization:

$\mathcal{G} = \{ G_m \}_{m=1}^G,$1

Traditional optimization with Adam or SGD is consistently stable but slow, requiring thousands of iterations and providing no inductive bias for scene geometry or inter-Gaussian dependencies. Feed-forward learned initializers (“FFNs”) enable rapid one-shot predictions of Gaussian parameters but lack the iterative quality improvements of optimizer-based methods. Previous learned optimizers (e.g., G3R [14], QuickSplat [29], ReSplat [47]) predict structured, correlated updates based on network input features, but their training is typically limited to short horizons (e.g., 4–24 steps) and requires learning rate schedules to avoid post-training divergence or degradation.

Learn2Splat is specifically engineered to (a) accelerate early convergence, (b) maintain stability over many thousands of optimization steps (well beyond the training horizon), and (c) support zero-shot generalization to new datasets, camera configurations, and view counts.

2. Learn2Splat Optimizer Architecture

At each inner-loop iteration $\mathcal{G} = \{ G_m \}_{m=1}^G,$2, Learn2Splat represents the parameter state as $\mathcal{G} = \{ G_m \}_{m=1}^G,$3 (all per-Gaussian parameters) and $\mathcal{G} = \{ G_m \}_{m=1}^G,$4 (per-Gaussian latent vectors). Gradients $\mathcal{G} = \{ G_m \}_{m=1}^G,$5 are computed with respect to the current loss. Adam-like moment estimates are formed:

• $\mathcal{G} = \{ G_m \}_{m=1}^G,$6
• $\mathcal{G} = \{ G_m \}_{m=1}^G,$7
• $\mathcal{G} = \{ G_m \}_{m=1}^G,$8

A composite feature tensor is constructed:

$\mathcal{G} = \{ G_m \}_{m=1}^G,$9

Two parallel branches process $G_m$0:

• kNN-Point Transformer Branch: For each Gaussian, a Transformer computes new latent states $G_m$1 based on spatially local (k=4) neighborhood attention, enabling local geometric context propagation.
• State-Scale MLP Branch: Predicts a non-negative scalar $G_m$2 used to scale $G_m$3, yielding $G_m$4.

An Update MLP processes $G_m$5 to yield a direction $G_m$6 (unit $G_m$7-normalized) and a learned magnitude $G_m$8. The actual parameter update is $G_m$9, leading to the next parameter state:

$P_m \in \mathbb{R}^3$0

This design decouples directionality and magnitude, exploits latent spatial-temporal correlations, and permits adaptive, context- and state-dependent step sizes. Only the latent $P_m \in \mathbb{R}^3$1 remains differentiable across inner-loop steps, with all other components detached from autograd to enable robust meta-learning.

Pseudocode Sketch:

$S_m \in \mathbb{R}^3$4

3. Meta-Learning for Long-Horizon Stability

Conventional learned optimizers are meta-trained to optimize a loss after a short fixed number of steps $P_m \in \mathbb{R}^3$2. Learn2Splat addresses long-horizon divergence and instability through:

3.1. Loss Ensemble

The meta-loss at each outer iteration combines:

• Rendering Loss ($P_m \in \mathbb{R}^3$3): Weighted sum of $P_m \in \mathbb{R}^3$4 and LPIPS between target images and intermediate renders across the trajectory.
• Low-Visibility Supervision ($P_m \in \mathbb{R}^3$5): Penalizes parameter updates where gradients are negligible or where the update sign clashes with that of normalized Adam, preventing drift in ill-constrained dimensions.
• Stability Loss ($P_m \in \mathbb{R}^3$6): Penalizes any step where $P_m \in \mathbb{R}^3$7 error on target views increases.

Combined, the outer-objective is:

$P_m \in \mathbb{R}^3$8

3.2. Checkpoint Buffer and Optimizer Rollout

A buffer $P_m \in \mathbb{R}^3$9 of up to 20 intermediate scene states is maintained:

• With probability $S_m \in \mathbb{R}^3$0, meta-iterations sample initial states from $S_m \in \mathbb{R}^3$1, otherwise, scenes are freshly initialized via SfM (for dense) or FFN (for sparse).
• After the standard unroll, the current optimizer (“frozen” weights) is rolled out for $S_m \in \mathbb{R}^3$2 further steps (with $S_m \in \mathbb{R}^3$3 ramping up to 50), simulating late-stage optimization and exposing the meta-learner to sequences far beyond the training horizon.

3.3. Complete Meta-Training Loop

The meta-training loop alternates unrolling, outer-loop updates, extended frozen rollouts, and buffer management. This ensures the optimizer is exposed to diverse, non-stationary optimization phases and converges gracefully, never requiring explicit learning rate schedules or auxiliary stabilization.

4. Implementation and Zero-Shot Generalization

Learn2Splat is meta-trained under two distinct initialization regimes yet applies the same meta-learning procedure:

• Dense-View Regime (L2SP): Scenes are initialized from a Structure-from-Motion (SfM) point cloud (COLMAP). Training involves context-view subsampling, random SfM point dropout (for robustness), and proceeds on the DL3DV dataset at low resolution for 50k meta-steps.
• Sparse-View Regime (L2SS): Initialization leverages ReSplat’s FFN outputs from 8 fixed, forward-facing views. Training uses the same view set at each step, also on DL3DV at low resolution for 50k meta-steps.

At test time:

• L2SS is evaluated directly (zero-shot) on dense or non-training view settings (including high-resolution and more views).
• L2SP is evaluated both on sparse configurations and across datasets. No fine-tuning, learning rate schedules, or dataset-specific adaptation are required; typical runs span 2,000–30,000 steps before saturation.

5. Experimental Evaluation

Experiments are conducted on DL3DV, RealEstate10K, DTU, LLFF, and Mip-NeRF360. Metrics include PSNR, SSIM, and LPIPS on novel/held-out views.

Results summary:

Setting	Method	Early Convergence	Final PSNR	Relative Speedup
Sparse, Forward-Facing	L2SS	Reaches 3DGS* in 10× fewer iterations	+0.3 dB vs 3DGS*	3–10× faster at 80% gain
Dense, Large-Baseline	L2SP	+2.1 dB vs Adam (early)	+2.1 dB vs Adam	3–24× faster to 80% gain
Dense, Zero-shot General.	L2SP	Matches/surpasses 3DGS	0–0.35 dB difference (after correction)	3–24× faster
RealEstate10K (zero-shot)	L2SS	Outperforms 3DGS* by 0.6 dB	—	3–5× faster

• LO baseline (ReSplat-architecture with time-encode & scheduled LR) matches L2SS at very early steps but degrades sharply past 100 steps.
• ReSplat overfits to the context views, leading to target view degradation after ∼10 steps.
• Qualitatively, Learn2Splat produces crisper geometry and sharper edges in an order of magnitude fewer steps.

A plausible implication is that Learn2Splat’s generalization and stability capabilities are tied to the explicit inclusion of gradient norm information and trajectory-aware regularization losses.

6. Unified 3DGS Training and Evaluation Framework

The “gsplat” framework introduced with Learn2Splat provides:

• Support for both sparse/dense view settings and multiple initialization strategies (SfM, FFN).
• Plug-in support for both standard and learned optimizers, meta-learning buffer/rollout, and long-horizon regimes.
• Consistent benchmarking on DL3DV (140 scenes), RealEstate10K (72 scenes), DTU (15 scenes), LLFF (8 scenes), and Mip-NeRF360 (9 scenes).
• Standardized evaluation protocols: 2,000 inner steps (unless otherwise stated), with PSNR/SSIM/LPIPS and runtime profiling.

Table of supported features:

Optimizer / Feature	Standard Optimizers	Learn2Splat
Long-horizon meta-learning	No	Yes
Buffer/rollout mechanisms	No	Yes
Sparse/dense regime support	Yes	Yes
Unified benchmarking framework	Yes (via gsplat)	Yes (via gsplat)

7. Conclusions, Limitations, and Prospects

Key results:

• Specialized, meta-learned optimizers outpace and outperform general-purpose optimizers for 3DGS, with up to 24× speedup and 0.3–2 dB higher PSNR.
• Long-horizon stability arises from meta-architectural innovations: explicit gradient norm encoding, low-visibility and stability losses, and buffer-based training on all optimization phases.
• Zero-shot generalization is demonstrated over datasets, resolutions, view regimes, and camera distributions without re-training or time-dependent scheduling.

Limitations:

• On large, complex scenes (e.g., millions of Gaussians), update magnitudes may become overly conservative, limiting late-stage gains.
• The L2SS variant can overfit to the fixed input-view pattern, underperforming on random-view or denser sampling.

Future directions outlined include:

• Unified optimizers handling both sparse and dense regimes, potentially via view-position encoded latents.
• Integrated learnable densification/pruning within the optimizer loop.
• Per-parameter-group adaptive scaling as opposed to the current per-Gaussian amplitude.
• Incorporation of second-order or quasi-Newton information for sharper late-stage convergence.

(Pearl et al., 15 May 2026)