GS$^{2}$CI: Robust Gaussian Splatting For Snapshot Compressive Imaging via Large Vision Model Priors

Abstract: Snapshot Compressive Imaging (SCI) offers an efficient solution for high-speed video acquisition and, under exposure-time camera--scene relative motion, multi-view scene capture by compressing temporal or spatial information into a single 2D measurement. While recent studies have explored SCI for 3D scene reconstruction, existing methods struggle with significant challenges due to information loss, limited viewpoint diversity, and the computational burden of jointly optimizing 3D representations and camera poses. In this work, we propose a novel framework that reconstructs high-quality 3D scenes from a single SCI measurement by leveraging 3D Gaussian Splatting (3DGS) and the powerful priors of large-scale vision foundation models (VFMs). Our primary reconstruction combines measurement-derived 3D VFM initialization with SCI-aware Gaussian optimization. After coarse-stage convergence, an auxiliary 2D VFM provides pseudo-view supervision at synthesized viewpoints for local appearance refinement. To further address the instability caused by ambiguous SCI supervision during 3DGS optimization, we introduce Opacity-Guided Splitting and Growth Regulation (OSGR), an SCI-specific densification strategy that augments split candidates using local opacity statistics, discourages loss-compensating opacity inflation through mean-opacity regulation, and bounds representation growth with explicit candidate-ratio and Gaussian-count constraints. Extensive experiments across multiple benchmarks demonstrate that our method achieves the strongest overall performance, combining leading reconstruction quality and robustness to viewpoint variation with competitive computational efficiency.

• The paper introduces GS²CI, which combines multiplicity-aware proxy views, VGGT geometry and pose priors, SCI-aware 3D Gaussian optimization, and opacity-guided splitting to reconstruct 3D scenes from a single compressed measurement.
• GS²CI achieves leading reconstruction quality across benchmarks, including up to 38.52 dB PSNR, while reducing per-scene runtime to about 68.4 minutes compared with 746.84 minutes for SCINeRF.
• The method remains robust across mask ratios up to 0.75 and compression ratios up to 32, but its static-scene formulation performs less reliably on dynamic sequences and depends on accurate coding masks and vision-model initialization.

Motivation and problem setting

Snapshot Compressive Imaging (SCI) encodes $N$ temporally or spatially varying observations into a single 2D measurement through coded modulation, following the standard forward model $$\mathbf{Y} = \sum_{i=1}^{N} \mathbf{X}_i \odot \mathbf{M}_i + \mathbf{Z}$$. When the latent frames arise from rigid camera–scene relative motion during exposure, the measurement multiplexes genuinely distinct viewpoints, so SCI can in principle support 3D reconstruction from a single snapshot. Existing decoders—both model-based and learning-based—are trained as video reconstruction systems on low-diversity datasets such as DAVIS and lack explicit modeling of depth, occlusion, and view-dependent appearance; they consequently degrade under wide-baseline viewpoint variation. Prior 3D-SCI methods based on NeRF or 3DGS jointly optimize scene representation and camera poses from scratch, which is computationally expensive (SCINeRF requires roughly 12.45 hours per scene) and prone to overfitting the modulated views.

The paper proposes GS²CI, a framework that reconstructs a static 3D scene and its camera trajectory from one SCI measurement by injecting priors from large-scale vision foundation models (VFMs) into a 3DGS pipeline. The central claim is that VFM-provided geometry, appearance, and motion priors regularize the severely under-constrained SCI inverse problem sufficiently to make joint pose-and-scene optimization tractable and robust.

Method

Measurement-derived proxy views and 3D VFM initialization

Because neither COLMAP nor VGGT accepts a multiplexed measurement directly, the method constructs geometry-oriented proxy views $\widetilde{\mathcal{X}}$ from $\mathbf{Y}$ and the known masks $\mathcal{M}$. Routing is determined by the mean measurement multiplicity $\mu_{\mathcal{M}}$ with a fixed threshold $\tau_\mu = 4$: for low multiplicity, Energy-Normalized Initialization (ENI) divides by per-pixel mask overlap counts, remodulates with each mask, and completes omitted locations via linear/nearest-neighbor interpolation; for high multiplicity, Adaptive Mask-Decoding Initialization (AMDI) solves an adaptive, trace-normalized ridge regression per pixel over the smallest spatial window whose local mask Gram matrix is numerically full-rank and well-conditioned. The proxy sequence feeds a frozen VGGT (VGGT's track-based pipeline), followed by bundle adjustment with validity checks that fall back to pre-BA estimates, expanded tracking queries, or dense direct predictions when BA is underconstrained. All criteria are fixed across scenes and require no ground-truth information.

SCI-aware Gaussian optimization and auxiliary refinement

Gaussians and per-view SE(3) pose corrections (parameterized in $\mathfrak{se}(3)$) are jointly optimized against the full differentiable SCI forward operator—rendering at estimated poses, then encoding with the masks—using an $\ell_1$/SSIM data term plus a mean-opacity penalty. After coarse convergence, a frozen 2D VFM (DiFix3D+) generates pseudo-ground-truth targets at synthesized poses obtained by SO(3)/translation interpolation between adjacent anchors plus boundary extrapolation. Because independently generated targets may be locally inconsistent with the 3D scene, photometric residuals are modulated by a soft alpha-based support weight derived from the pointwise minimum of target-construction and current-render alpha maps, retaining only weakly weighted supervision along rays with low accumulated alpha.

OSGR densification

The key SCI-specific contribution is Opacity-Guided Splitting and Growth Regulation (OSGR). Under SCI supervision, the optimizer can absorb part of the multiplexed residual by inflating individual Gaussian opacities before spatial support is resolved, producing local opacity peaks that vanilla adaptive density control does not address. OSGR augments the standard split-candidate set with Gaussians whose opacity exceeds their local neighborhood average (three nearest neighbors, capped at 5% of the population by deviation magnitude), applies transmittance-preserving child-opacity corrections after splitting, penalizes global mean opacity to discourage renewed inflation, and bounds growth via a candidate-ratio budget and a hard Gaussian-count cap fixed at the population reached at iteration 7,000. Without the count cap, runs exhaust GPU memory before completing training—an explicit demonstration that unconstrained densification is unstable under SCI supervision.

Experimental results

Experiments span simulated SCI measurements (NeRF Synthetic, DeblurNeRF, DTU, LLFF, DAVIS; CR = 8, mask ratio 0.25 unless stated) and real SCI data from SCINeRF, evaluated with PSNR/SSIM/LPIPS on a single RTX 4090.

Reconstruction quality. On six benchmark scenes, GS²CI achieves the best PSNR on all six and the best LPIPS on four, e.g., 38.52 dB / 0.9890 SSIM / 0.0098 LPIPS on Vender versus SCINeRF's 36.40 dB / 0.0290 LPIPS, and 37.78 dB on Tanabata versus SCIGS's 35.12 dB. Against EfficientSCI, the leading video-SCI decoder, gains exceed 5 dB on several scenes.

Efficiency. Total runtime is approximately 68.4 minutes per scene (4 min 45 s initialization, 53.67 min coarse training, ~10 min fine stage), compared with 746.84 minutes for SCINeRF under its official schedule—a reduction of more than an order of magnitude while improving quality. Rendering reaches 407.9 FPS.

Controlled densification comparison. With the shared SCI objective, VFM initialization, pose protocol, and budget held fixed, OSGR attains 29.80 dB / 0.9005 SSIM / 0.1090 LPIPS averaged over 20 cases, versus 29.05 / 0.8821 / 0.1276 for SCI-MCMC and roughly 27.8–27.9 dB for SCI-3DGS and SCI-RevADC. This isolates OSGR—not merely the 3DGS backbone—as the source of the improvement.

Comparison with SCISplat. Since no public implementation exists, the authors reproduce SCISplat$^\dagger$ under identical conditions; it fails to initialize on three of twenty cases. On the matched seventeen cases, GS²CI improves PSNR by 3.29 dB and SSIM by 0.0538, reduces LPIPS by 0.0694 and training time by 32.16 minutes, and lowers every trajectory metric (e.g., ATE RMSE of 0.04235 versus 0.08023).

Robustness studies. The method ranks first at every tested mask ratio (0.125–0.75) and at both CR = 16 and CR = 32, where the margin over all baselines widens (21.85 dB versus 21.28 dB best baseline at CR = 32). Trajectory accuracy is the lowest among all compared methods on every metric (ATE 0.0424, RPE$$\mathbf{Y} = \sum_{i=1}^{N} \mathbf{X}_i \odot \mathbf{M}_i + \mathbf{Z}$$0 0.893°). Ablations confirm each component contributes: removing VFM points, VFM poses, opacity splitting, opacity regulation, or pose refinement degrades all metrics, and removing the count cap causes out-of-memory failures. Three-seed repetition yields a PSNR standard deviation of about 0.044 dB, indicating low run-to-run variance. Adaptive proxy routing improves over forced ENI by 4.40 dB on average across strong-mixing settings, validating the multiplicity-based routing rule.

Auxiliary refinement. The 2D VFM stage yields modest but consistent gains, concentrated on extrapolated views (25.19 → 25.39 dB), with alpha-based support weighting maintaining or improving every metric. The authors note this stage contributes less than the coarse-stage components, positioning it as local appearance refinement rather than a primary driver.

Limitations and open questions

The static-scene formulation is the principal limitation. On two DAVIS dynamic sequences, GS²CI ranks second behind SCI-MCMC on every metric, and the authors attribute this to the misspecified static forward model: compensatory primitives supported by only a subset of views can become local opacity outliers that satisfy OSGR's splitting criterion despite lacking temporal support, potentially diverting bounded representation capacity toward ghost contours. None of the current mechanisms distinguishes static-geometry opacity peaks from motion-compensation peaks. Additional caveats include reliance on known coding masks, dependence on VGGT proxy-view quality (with fallbacks required when bundle adjustment is underconstrained), and the modest absolute gain of the fine stage. Open questions left by the paper are whether motion-aware cross-view support can be incorporated into the splitting criterion and whether extension to 4D Gaussians resolves dynamic SCI without sacrificing the count-constrained stability that OSGR provides.

Conclusion

GS²CI reformulates single-measurement SCI as a 3D reconstruction problem regularized by frozen vision foundation models, combining multiplicity-aware proxy construction, VGGT-based geometry and pose initialization, SCI-aware joint Gaussian-pose optimization with the OSGR densification strategy, and lightweight diffusion-based pseudo-view refinement. The controlled comparisons demonstrate that the density-control design, not the 3DGS backbone alone, accounts for the gains, and the framework reduces per-scene optimization time by roughly an order of magnitude relative to SCINeRF while improving reconstruction, novel-view synthesis, and trajectory accuracy across mask ratios up to 0.75 and compression ratios up to 32. Dynamic scenes remain outside the current formulation's reliable operating range.