RegGS: 3D Gaussian Registration Framework

• RegGS is a 3D Gaussian registration framework that incrementally aligns local Gaussian maps into a unified, globally consistent scene representation.
• It employs an entropy-regularized Sinkhorn algorithm to match Gaussian mixtures in Sim(3) space, ensuring robust camera pose estimation and scale alignment.
• Its joint refinement using photometric consistency and depth geometry enhances reconstruction quality and mitigates topological distortions in sparse-view settings.

RegGS is a 3D Gaussian registration-based framework for reconstructing scenes from sparse, unposed images by aligning local 3D Gaussians generated by a feed-forward network into a globally consistent 3D Gaussian representation (Cheng et al., 10 Jul 2025). It is designed for a setting in which optimization-based 3DGS methods struggle with sparse views because of limited prior knowledge, while feed-forward Gaussian approaches are constrained by input formats and therefore cannot easily incorporate more input views. RegGS addresses this gap with an incremental coarse-to-fine registration strategy in which local Gaussian maps are interpreted as Gaussian mixture models, aligned in $\mathrm{Sim}(3)$ space with an entropy-regularized Sinkhorn algorithm for the Mixture 2-Wasserstein distance, and then refined with photometric consistency and depth geometry, jointly recovering camera poses and scene structure (Cheng et al., 10 Jul 2025).

1. Problem formulation and conceptual position

RegGS targets unposed sparse-view reconstruction, a regime in which standard 3DGS pipelines are underconstrained because camera poses are unknown, overlap may be limited, depth is ambiguous, and the scene scale can drift (Cheng et al., 10 Jul 2025). The paper explicitly notes that optimization-only 3DGS can suffer from topological discontinuities and scale ambiguity. By contrast, feed-forward Gaussian predictors can infer coherent local 3D Gaussian structure directly from images thanks to learned priors from large datasets, but they usually accept only a fixed small number of images as input. RegGS therefore asks whether local 3D Gaussian reconstructions inferred from small image subsets can be registered and fused into a single scene.

The method is incremental. A pretrained feed-forward Gaussian model first produces a main Gaussian map from two initial images. Each subsequent image yields a sub Gaussian map. RegGS then estimates a $\mathrm{Sim}(3)$ transform that aligns the submap to the current global map, merges the transformed Gaussians, and repeats this process until a full scene representation is obtained. This formulation makes registration, rather than direct end-to-end reconstruction or pure optimization from scratch, the organizing principle of the pipeline. The resulting system is simultaneously a reconstruction method, a pose-estimation method, and a 3DGS fusion method.

A central representational choice is to treat local Gaussian maps as Gaussian mixture models. For registration, RegGS does not use color information, including spherical harmonic coefficients, because color is unstable under lighting changes and viewpoint changes. Instead, it relies on Gaussian means, covariances, and opacity-normalized weights, making the registration stage explicitly geometric rather than appearance-driven (Cheng et al., 10 Jul 2025).

2. Gaussian mixture representation and registration in $\mathrm{Sim}(3)$

For the main map $\mathcal{G}^A$ and submap $\mathcal{G}^B$, RegGS writes the corresponding Gaussian mixture models as

$$G^A = \sum_{i=1}^{M} w_i^A \mathcal{N}(\mu_i^A, \Sigma_i^A), \qquad G^B = \sum_{k=1}^{N} w_k^B \mathcal{N}(\mu_k^B, \Sigma_k^B),$$

with

$\sum_i w_i^A = 1, \qquad \sum_k w_k^B = 1.$

The weights are obtained by opacity normalization. This abstraction preserves anisotropic structure through $\Sigma$ and gives RegGS a richer alignment signal than point-based matching.

The registration target is a similarity transform rather than a rigid transform: $$T^* = \arg\min_{T \in \mathrm{Sim}(3)} \mathcal{D}\left(\mathcal{G}^A, T(\mathcal{G}^B)\right).$$ The inclusion of scale is essential because local Gaussian maps produced by feed-forward predictors may differ in global scale. Under a $\mathrm{Sim}(3)$ transform with rotation $\mathrm{Sim}(3)$0, translation $\mathrm{Sim}(3)$1, and scale $\mathrm{Sim}(3)$2, RegGS transforms a submap Gaussian by

$\mathrm{Sim}(3)$3

The optimization variables are parameterized as

$\mathrm{Sim}(3)$4

where $\mathrm{Sim}(3)$5 is a unit quaternion. The paper reports that quaternion parameterization converges faster than Lie algebra rotation in its experiments (Cheng et al., 10 Jul 2025).

This formulation makes local-to-global registration the mechanism through which camera pose estimation emerges. Once a local Gaussian submap is aligned to the global map, the associated camera pose is induced and refined through the same transform. This suggests that, in RegGS, pose recovery is not a separate preprocessing stage but an intrinsic byproduct of Gaussian scene registration.

3. Mixture 2-Wasserstein distance and entropy-regularized Sinkhorn optimization

The core alignment metric is the Mixture 2-Wasserstein distance, denoted $\mathrm{Sim}(3)$6, defined over Gaussian mixtures in $\mathrm{Sim}(3)$7 space (Cheng et al., 10 Jul 2025). For a pair of Gaussian components, RegGS uses the squared $\mathrm{Sim}(3)$8-Wasserstein cost

$\mathrm{Sim}(3)$9

This cost jointly captures displacement of means and mismatch of covariances.

At the mixture level, RegGS solves a discrete optimal transport problem: $\mathrm{Sim}(3)$0 where $\mathrm{Sim}(3)$1 is the Gaussian $\mathrm{Sim}(3)$2 cost and $\mathrm{Sim}(3)$3 is the set of transport plans satisfying the marginal constraints

$\mathrm{Sim}(3)$4

The paper notes that $\mathrm{Sim}(3)$5 is an upper bound on the exact Wasserstein distance between the full mixture distributions.

To make this practical and differentiable, RegGS introduces entropy regularization: $\mathrm{Sim}(3)$6 The Sinkhorn kernel is

$\mathrm{Sim}(3)$7

with iterative updates

$\mathrm{Sim}(3)$8

After $\mathrm{Sim}(3)$9 iterations,

$\mathcal{G}^A$0

and the regularized transport cost is

$\mathcal{G}^A$1

The paper gives complexity $\mathcal{G}^A$2 and states that $\mathcal{G}^A$3 is typically sufficient. It also reports practical engineering measures: GPU tensorization, batch-parallel Cholesky decomposition, log-domain Sinkhorn updates, covariance regularization

$\mathcal{G}^A$4

and scale normalization before optimization. The theoretical discussion further states that, as $\mathcal{G}^A$5, the gradient of the regularized objective converges to a subgradient direction of the exact Wasserstein objective. This underwrites the use of entropy regularization as a coarse alignment mechanism rather than merely a numerical convenience (Cheng et al., 10 Jul 2025).

4. Joint registration, coarse-to-fine reconstruction, and pose recovery

RegGS does not rely on $\mathcal{G}^A$6 alone. It combines the structural alignment term with photometric consistency and depth geometry in a joint registration module: $\mathcal{G}^A$7 The structural term is

$\mathcal{G}^A$8

The photometric term is

$\mathcal{G}^A$9

and the depth term is

$\mathcal{G}^B$0

In the paper’s interpretation, $\mathcal{G}^B$1 supplies the coarse global alignment, photometric consistency sharpens local registration, and depth geometry stabilizes scale and suppresses topological distortions (Cheng et al., 10 Jul 2025).

The operational pipeline is incremental. RegGS initializes the main map from two images, predicts a sub Gaussian map for each additional input image, estimates an initial relative scale by comparing rendered depths of submaps and the main map, performs coarse registration with $\mathcal{G}^B$2, refines with the joint loss, merges the aligned submap into the global set, and finally applies global refinement of the merged 3DGS with pruning. The paper describes this as a coarse-to-fine registration strategy rather than a graph-based or bundle-adjustment-style formulation.

Camera poses are evaluated with ATE RMSE, but structurally they are recovered through local-to-global map alignment. The paper explicitly states that training-frame poses and sub-Gaussian scale estimation are jointly optimized during processing. This makes RegGS a registration-centric alternative to methods that first estimate poses and then reconstruct.

5. Empirical performance and ablation evidence

RegGS is evaluated on RE10K and ACID under sparse-view protocols with 2, 8, 16, and 32 input views, using PSNR, SSIM, LPIPS, ATE RMSE, and $\mathcal{G}^B$3 distance as metrics (Cheng et al., 10 Jul 2025). On RE10K, the reported novel-view synthesis results are 24.272 PSNR / 0.853 SSIM / 0.174 LPIPS for 2 views, 26.691 / 0.877 / 0.185 for 8 views, 28.663 / 0.913 / 0.147 for 16 views, and 28.332 / 0.912 / 0.151 for 32 views. On ACID, the reported results are 24.291 / 0.703 / 0.237 for 2 views, 25.764 / 0.753 / 0.252 for 8 views, 27.745 / 0.834 / 0.201 for 16 views, and 26.772 / 0.774 / 0.243 for 32 views. The paper emphasizes that the advantage grows as views become sparser.

Pose estimation results are especially strong. On RE10K, ATE RMSE is 0.023 for 8 views, 0.041 for 16 views, and 0.078 for 32 views. On ACID, the corresponding values are 0.020, 0.038, and 0.095. The paper interprets this as evidence that local-to-global Gaussian registration is highly effective as a pose estimator, especially when overlap is limited.

The ablation study on RE10K with 16 views shows that all three loss components are necessary. Without photometric loss, performance drops to ATE 1.184, PSNR 16.06, SSIM 0.52, LPIPS 0.44, and $\mathcal{G}^B$4. Without depth, it reports ATE 0.160, PSNR 20.97, SSIM 0.72, LPIPS 0.29, and $\mathcal{G}^B$5. Without $\mathcal{G}^B$6, it reports ATE 1.151, PSNR 19.41, SSIM 0.67, LPIPS 0.31, and $\mathcal{G}^B$7. The full RegGS configuration reports ATE 0.098, PSNR 23.09, SSIM 0.79, LPIPS 0.23, and $\mathcal{G}^B$8. A further module ablation shows that removing joint registration yields ATE 1.164, PSNR 11.41, SSIM 0.34, LPIPS 0.60, and normalized $\mathcal{G}^B$9, indicating that the joint registration module is the core reason the method works.

The supplementary runtime analysis shows the practical cost of this design. At 2 views, RegGS takes 259s and uses 3.9 GB. At 16 views, it takes 57 min and uses 12.1 GB. At 64 views, it reaches 28.703 PSNR, takes 165 min, and still uses 12.1 GB. The paper explicitly identifies $$G^A = \sum_{i=1}^{M} w_i^A \mathcal{N}(\mu_i^A, \Sigma_i^A), \qquad G^B = \sum_{k=1}^{N} w_k^B \mathcal{N}(\mu_k^B, \Sigma_k^B),$$0 computation as the main computational bottleneck.

6. Limitations and place within Gaussian-splatting research

The paper is explicit about several limitations of RegGS (Cheng et al., 10 Jul 2025). The method depends on the quality of the upstream feed-forward Gaussian predictor; if that predictor produces abnormal local Gaussians, especially in challenging regions such as reflective surfaces, registration and fusion can fail. Runtime increases significantly with the number of input views because of $$G^A = \sum_{i=1}^{M} w_i^A \mathcal{N}(\mu_i^A, \Sigma_i^A), \qquad G^B = \sum_{k=1}^{N} w_k^B \mathcal{N}(\mu_k^B, \Sigma_k^B),$$1. Very large inter-frame motion can cause registration failure. Final global refinement can introduce visible noise artifacts even when PSNR is high.

Within the broader Gaussian-splatting literature, RegGS occupies a specific position. It is neither a purely feed-forward reconstruction method nor a purely optimization-based 3DGS pipeline. Instead, it treats registration of local Gaussian maps as the organizing problem for sparse, unposed scene reconstruction. This distinguishes it from Graph-GSReg, which reformulates registration of independently reconstructed 3DGS scenes as graph registration over object-level 3D scene graphs and then refines the merged scene with self-supervised test-time optimization (Lee et al., 29 Jun 2026), and from RadGS-Reg, which reconstructs vertebral Radiative Gaussians from biplanar X-rays and converts CT/X-ray registration into a 3D/3D registration problem in medical imaging (Shen et al., 28 Aug 2025). By contrast, GRGS, RigGS, REArtGS, and REArtGS++ use Gaussian splatting for relightable human novel-view synthesis, rigging of articulated objects, or articulated-object reconstruction with geometric and motion constraints rather than unposed sparse-view registration (Sun et al., 27 May 2025, Yao et al., 21 Mar 2025, Wu et al., 9 Mar 2025, Wu et al., 21 Nov 2025).

This suggests that RegGS is best understood as a registration-centric bridge between generalizable local Gaussian prediction and globally consistent multi-view 3DGS. Its main contribution is not a new rendering primitive, but a reconstruction strategy: treat local Gaussian outputs as Gaussian mixture models, align them in $$G^A = \sum_{i=1}^{M} w_i^A \mathcal{N}(\mu_i^A, \Sigma_i^A), \qquad G^B = \sum_{k=1}^{N} w_k^B \mathcal{N}(\mu_k^B, \Sigma_k^B),$$2 with entropy-regularized optimal transport, refine them with rendered photometric and depth consistency, and thereby recover both scene structure and camera poses from sparse, unposed views.