GenWildSplat: Sparse 3D Reconstruction

• GenWildSplat Architecture is a feed-forward model for sparse-view 3D reconstruction, leveraging multi-view geometry estimation, per-view appearance modulation, and occlusion reasoning.
• It integrates vision transformers, DPT-based decoders, and a unified pipeline that constructs a canonical 3D Gaussian field through voxelization and Gaussian merging.
• Empirical results show real-time inference with high fidelity, robust handling of unknown camera poses, dynamic occluders, and varying lighting conditions.

GenWildSplat architecture refers to a class of feed-forward models for generalizable, sparse-view 3D reconstruction from unconstrained image collections, designed to address the reconstruction of photorealistic, relightable, and temporally consistent 3D Gaussian fields in the wild. These architectures are characterized by their ability to operate without known camera intrinsics or extrinsics, to handle sparse and unposed image sets, and to provide per-view appearance modulation that adapts to varying lighting conditions while handling transient occlusion (Fujimura et al., 23 Apr 2026, Gupta et al., 30 Apr 2026, Park et al., 30 Apr 2026).

1. High-Level Pipeline and Methodological Innovations

GenWildSplat integrates multi-view geometry estimation, 3D Gaussian splatting, appearance adaptation, and occlusion reasoning in a unified pipeline. Given a set of $V$ real-world, unposed input images $\{I_i\}_{i=1}^V$, the architecture estimates per-view depth maps $D_i$, camera intrinsics and extrinsics $(K_i, E_i)$, constructs a canonical 3D Gaussian field $G_c$, and predicts view-specific photometric attributes through an appearance adapter. A transient-object mask $M_i$ is used to exclude occluders from the loss calculation.

The principal workflow is:

• Feature Encoding:

Input images are passed through a vision transformer (VGGT or ViT variants), extracting per-image and aggregated multi-view features.

• Prediction of Geometry and Camera:

Dedicated DPT-based heads decode features into depth $D_i$, camera parameters $(K_i, E_i)$, and per-pixel Gaussian parameters (center $\mu$, scale $s$, rotation $\{I_i\}_{i=1}^V$0, opacity $\{I_i\}_{i=1}^V$1, and canonical color coefficients $\{I_i\}_{i=1}^V$2).

• Voxelization and Gaussian Merging:

Gaussian parameters across views are projected into a shared canonical space and merged within voxels to yield a compact set $\{I_i\}_{i=1}^V$3.

• Appearance Conditioning:

A per-view lighting code $\{I_i\}_{i=1}^V$4 is extracted by a dedicated CNN, and an appearance adapter MLP modulates the canonical color coefficients to yield $\{I_i\}_{i=1}^V$5 per Gaussian, achieving disentanglement of geometry and appearance.

• Semantic Segmentation:

Pretrained segmentation networks (e.g., YOLOv8x-seg) generate transient masks $\{I_i\}_{i=1}^V$6 which exclude occluding objects from both training and evaluation.

• 3D Gaussian Splatting & Rendering:

The final rendering splats the adapted 3D Gaussians into the image under computed camera parameters, using an accelerated, differentiable 3DGS rasterizer.

This design enables real-time, scene-generalizable inference, robust to both camera pose uncertainty and severe view sparsity (Fujimura et al., 23 Apr 2026, Gupta et al., 30 Apr 2026).

2. Core Network Components

GenWildSplat comprises multiple specialized deep learning modules organized as follows:

Component	Input	Output
Vision Geometry Transformer	$\{I_i\}_{i=1}^V$7 (images)	Feature tensor $\{I_i\}_{i=1}^V$8
Depth Head ($\{I_i\}_{i=1}^V$9)	$D_i$0	Depth maps $D_i$1
Camera Head ($D_i$2)	$D_i$3 (global pooled)	Intrinsics $D_i$4, Extrinsics $D_i$5
Gaussian Head ($D_i$6)	$D_i$7	Per-pixel Gaussian parameters $D_i$8
Voxelization+Merging	Raw Gaussian outputs	Compact set $D_i$9
Light Encoder ($(K_i, E_i)$0)	$(K_i, E_i)$1	Lighting code $(K_i, E_i)$2
Appearance Adapter ($(K_i, E_i)$3)	$(K_i, E_i)$4	Adapted color $(K_i, E_i)$5
Semantic Segmentation	$(K_i, E_i)$6	Transient mask $(K_i, E_i)$7

The Geometry Transformer (VGGT) is composed of 24 alternating transformer layers with frame-attention and global-attention. The depth, camera, and Gaussian heads are U-Net-like DPT decoders with channel-specific heads. The appearance adapter is a five-layer MLP modulating spherical-harmonic color coefficients (Gupta et al., 30 Apr 2026).

3. 3D Gaussian Field Representation and Rendering

Each canonical Gaussian $(K_i, E_i)$8 is parameterized by:

• $(K_i, E_i)$9: spatial center
• $G_c$0: anisotropic covariance
• $G_c$1: spherical-harmonic canonical color coefficients

Given a target view $G_c$2, the appearance adapter adjusts $G_c$3 to $G_c$4 conditioned on $G_c$5. Rendering uses ellipsoidal splats projected into the target view, compositing colors via an "EWA splat" or volumetric rendering kernel. For each image pixel $G_c$6 along ray $G_c$7, the calculation employs:

$G_c$8

Closed-form integration is applied over each Gaussian ellipsoid's support (Gupta et al., 30 Apr 2026, Fujimura et al., 23 Apr 2026).

4. Appearance Adaptation and Transient Occlusion Reasoning

Lighting and appearance variation are addressed through:

• A 2D CNN-based Light Encoder producing compact per-view lighting codes $G_c$9 from each input image.
• A Appearance Adapter MLP transforming $M_i$0 for view-specific spherical-harmonic color coefficients.
• Pretrained semantic segmentation provides binary transient masks $M_i$1 per image, actively excluding occluded or dynamic content in the training loss and during evaluation.

Ablation reveals that omission of the appearance adapter precipitates significant PSNR drops (from 15.84 to 13.76), while exclusion of transient-object masking also degrades reconstruction quality (Gupta et al., 30 Apr 2026). This demonstrates that geometry-appearance decoupling and transient exclusion are critical for high-fidelity, generalizable 3D reconstruction under challenging real-world conditions.

5. Loss Functions, Training Curriculum, and Hyperparameters

Masked photometric and perceptual losses are computed for each view:

$M_i$2

$M_i$3

No regularizers are used beyond inherited AnySplat priors. Curriculum learning proceeds in three stages: synthetic appearance variation, diverse synthetic geometry (without occlusion), and synthetic occlusion with real scene fine-tuning. Typical training uses 40,000 iterations, batch sizes and learning rates are not explicitly stated, and all modules are initialized from AnySplat or DPT pretraining where applicable (Gupta et al., 30 Apr 2026).

6. Extensions: Sparsity-Aware Gaussian Replication and View Refinement

Advances such as sparsity-aware Gaussian replication (SAGR) and view refinement diffusion models have been introduced to further improve reconstructions from extremely sparse image collections:

• SAGR identifies low-density regions by computing an opacity accumulation map $M_i$4 over image pixels. Gaussians projected into pixels with $M_i$5 are replicated along principal axes, with each duplicate inserted back into $M_i$6 with small spatial offsets ($M_i$7), increasing representational power in poorly observed regions (Park et al., 30 Apr 2026).
• Reference-guided view refinement employs a diffusion U-Net with cross-attention, conditioned on transient masks and reference renders, refining corrupted renders and serving as a synthetic view generator for additional supervisory signals.

This approach incorporates additional terms in the total loss:

$M_i$8

Default values: $M_i$9200k Gaussians, $D_i$0, replication factor $D_i$1, U-Net channels [64,128,256,512], loss weights $D_i$2, $D_i$3, $D_i$4, mask threshold $D_i$5, LoRA rank $D_i$6 (Park et al., 30 Apr 2026).

7. Empirical Results, Generalization, and Limitations

Empirical evaluations across PhotoTourism, MegaScenes, and DL3DV datasets show that GenWildSplat achieves state-of-the-art results for feed-forward, pose-free 3D reconstruction, with robust generalization across illumination, occlusion, and viewpoint sparsity. The architecture achieves real-time inference (sub-second to 3-second per scene) with no per-scene optimization and maintains scene geometry and appearance fidelity even in the presence of dynamic distractors and lighting changes.

Ablations confirm the necessity of the appearance adapter, occlusion handling, and curriculum training. The precise implementation of feature channels in VGGT and DPT, learning rates, and voxelization details are not fully specified and may require careful empirical tuning for reproduction (Gupta et al., 30 Apr 2026).

These results position GenWildSplat as a reference framework for generalizable 3DGS methods in unconstrained settings, informing future research in robust 3D scene understanding and relightable reconstruction from highly sparse and unposed imagery.