Feed-Forward Gaussian Splatting Encoder

• Feed-Forward Gaussian Splatting Encoder is a neural module family that amortizes scene reconstruction by mapping images or video to renderable Gaussian primitives in one forward pass.
• Various architectures differ in pixel alignment, latent space design, and multi-view fusion, jointly predicting camera parameters and Gaussian attributes without per-scene optimization.
• These methods enhance efficiency by reducing redundant primitives, improving rendering speed and scalability across 3D and 2D scene representations.

A feed-forward Gaussian splatting encoder is a learned mapping from images or video to a renderable Gaussian scene representation that is produced in a single forward pass rather than by per-scene optimization. In the surveyed literature, the encoder may jointly predict camera parameters and Gaussian attributes, may operate in 3D Gaussian Splatting or 2D Gaussian Splatting form, and may be conditioned on multi-view images, monocular video, unposed image sequences, or appearance codes. The surveyed papers use the term for several distinct modules, which suggests that it denotes a family of modules rather than a single canonical design (Jiang et al., 29 May 2025, Chen et al., 2024, Zhang et al., 3 Apr 2026, Itkin et al., 16 Apr 2026).

1. Definition, scope, and problem setting

The common objective is to replace scene-specific reconstruction loops with amortized inference. In uncalibrated novel-view synthesis, AnySplat predicts “everything in one shot”: a set of 3D Gaussian primitives encoding scene geometry and appearance, and the corresponding camera intrinsics and extrinsics for each input image (Jiang et al., 29 May 2025). In pose-free sequential reconstruction, PreF3R reconstructs a 3D Gaussian field within a canonical coordinate frame directly from a sequence of unposed images, using a spatial memory network instead of optimization-based global alignment (Chen et al., 2024). In monocular SLAM, Flash-Mono uses a recurrent feed-forward frontend that directly predicts camera pose and a dense 2D Gaussian Splatting representation for each frame, while a backend performs map fusion and lightweight refinement (Zhang et al., 3 Apr 2026).

The problem setting varies substantially across domains. Driving-scene systems such as SpectralSplat retain a feed-forward geometric backbone while modifying the color pathway so that the same geometry can be rendered under different lighting or weather conditions (Herau et al., 3 Apr 2026). Human-centric systems such as PointSplat start from calibrated multi-view RGB images and foreground masks, then directly infer Gaussian primitives from an input point set in 3D space (Guo et al., 30 Jun 2026). Unconstrained Internet-photo settings such as WildSplatter jointly learn 3D Gaussians and appearance embeddings from sparse input views with unknown camera parameters and varying lighting conditions (Fujimura et al., 23 Apr 2026).

A recurring motivation is that classical pixel-aligned feed-forward 3DGS pipelines inherit redundancy from the image grid. Multiple papers identify rigid pixel-to-Gaussian or voxel-to-Gaussian parameterizations as the main reason that primitive count scales with image resolution or the number of views, producing redundant Gaussians and limiting compactness, rendering speed, and downstream applicability (Zhang et al., 3 Apr 2026, Itkin et al., 16 Apr 2026, Lao et al., 26 Mar 2026, Kim et al., 22 Mar 2026).

2. Principal architectural families

The encoder families in current work differ mainly in where correspondence is resolved, how locality is represented, and whether Gaussian prediction is tied to pixels, points, or latent scene tokens.

Representative paper	Encoder formulation	Distinguishing idea
AnySplat (Jiang et al., 29 May 2025)	Alternating-Attention Transformer + DPT heads + differentiable voxelization	Joint pose and 3DGS prediction from uncalibrated image collections
PreF3R (Chen et al., 2024)	ViT encoder-decoder + spatial memory network	Pose-free canonical 3D Gaussian field from variable-length ordered image sequences
Flash-Mono (Zhang et al., 3 Apr 2026)	Recurrent frontend with hidden state and cross-attention decoders	Streaming prediction of pose and dense 2D Gaussian surfels for monocular SLAM
GlobalSplat (Itkin et al., 16 Apr 2026)	Fixed-size global latent scene tokens with dual-branch iterative attention	“Align first, decode later” with a scene-centric latent bottleneck
SparseSplat (Zhang et al., 3 Apr 2026)	Entropy-based sparse sampling + 3D-local attribute predictor	Pixel-unaligned, content-adaptive Gaussian placement
PointSplat (Guo et al., 30 Jun 2026)	Point-Image Transformer over ray-cast point anchors	Object-centric 3D prediction directly in point space
WildSplatter (Fujimura et al., 23 Apr 2026)	Depth Anything 3-based ViT with geometry and appearance branches	Pose-free 3DGS with global appearance embeddings

Pixel-aligned encoders remain the most direct formulation. AnySplat predicts per-pixel depth and Gaussian parameters, then lifts them into 3D using predicted cameras (Jiang et al., 29 May 2025). PreF3R similarly predicts a dense Gaussian parameter map, one Gaussian per pixel, but does so in a pose-free canonical frame (Chen et al., 2024). WildSplatter also uses a per-pixel geometry branch, with Gaussian centers computed from predicted rays and depths (Fujimura et al., 23 Apr 2026).

A second family breaks the pixel grid explicitly. SparseSplat samples sparse 2D pixels probabilistically from an entropy map, back-projects them to 3D anchor points, and predicts Gaussian attributes from local 3D neighborhoods rather than from single-pixel features (Zhang et al., 3 Apr 2026). PointSplat is more strongly object-centric: it constructs a coarse visual hull, ray-casts through a voxelized point cloud to remove redundant interior points and establish explicit 2D–3D correspondences, and then predicts Gaussian parameters from fused point and image tokens (Guo et al., 30 Jun 2026). “Off The Grid” replaces pixel centers with continuous sub-pixel primitive detection, using heatmap expectation to regress 2D centers inside image patches (Moreau et al., 17 Dec 2025).

A third family resolves multi-view redundancy before explicit geometry is decoded. GlobalSplat compresses all views into a constant-size global latent scene representation with 2048 latent tokens of dimension 512 and four iterative encoder blocks, then decodes those scene tokens into Gaussians only after cross-view alignment in latent space (Itkin et al., 16 Apr 2026). This is architecturally distinct from view-aligned lifting or post hoc merging.

3. Encoded state and Gaussian parameterization

Despite architectural variation, the predicted state is structurally consistent across most papers. Standard 3DGS outputs include a 3D mean or center, opacity, anisotropic scale, rotation, and spherical-harmonic color coefficients. AnySplat uses the parameterization

$$\bigl\{(\boldsymbol{\mu}_g, \sigma_g, \boldsymbol{r}_g, \boldsymbol{s}_g, \boldsymbol{c}_g)\bigr\}_{g=1}^G,$$

where $\boldsymbol{\mu}_g$ is the center, $\sigma_g$ controls opacity, $\boldsymbol{r}_g$ is an orientation quaternion, $\boldsymbol{s}_g$ is anisotropic scale, and $\boldsymbol{c}_g$ stores spherical-harmonic color coefficients (Jiang et al., 29 May 2025). SparseSplat predicts $\{\alpha,\mathbf{s},\mathbf{q},\mathbf{c}\}$ for each 3D anchor point, where $\alpha$ is opacity, $\mathbf{s}\in\mathbb{R}^3$ is 3D scale, $\mathbf{q}\in\mathbb{R}^4$ is rotation as a quaternion, and $\boldsymbol{\mu}_g$0 is the spherical-harmonics color coefficient tensor (Zhang et al., 3 Apr 2026).

Some encoders predict more than Gaussian attributes. AnySplat jointly predicts camera intrinsics and extrinsics for every input image (Jiang et al., 29 May 2025). Flash-Mono predicts camera pose, a dense 2D Gaussian surfel map, and an updated hidden state: $\boldsymbol{\mu}_g$1 Its hidden state is not an auxiliary by-product; it is the mechanism that accumulates geometry, appearance context, and pose-related cues across frames (Zhang et al., 3 Apr 2026). PreF3R similarly extends the feed-forward encoder beyond Gaussian regression by predicting dense pointmaps, confidence maps, and Gaussian parameters within a canonical coordinate system (Chen et al., 2024).

The encoded state may also include learned global scene or appearance variables. GlobalSplat’s core state is a fixed set of scene tokens, independent of the number of input frames (Itkin et al., 16 Apr 2026). SpectralSplat introduces a global appearance embedding $\boldsymbol{\mu}_g$2, derived from DINOv2 features, and uses it to factor color into an appearance-agnostic base stream and an appearance-conditioned adapted stream (Herau et al., 3 Apr 2026). WildSplatter likewise uses a global appearance token $\boldsymbol{\mu}_g$3, with $\boldsymbol{\mu}_g$4, broadcast spatially and concatenated with local features to predict target-conditioned spherical-harmonic coefficients (Fujimura et al., 23 Apr 2026).

Not all feed-forward Gaussian splatting encoders target volumetric 3D ellipsoids. Flash-Mono explicitly adopts 2D Gaussian surfels rather than 3D Gaussian ellipsoids because 3D ellipsoids tend to produce noisy geometry and “floaters” in SLAM; each surfel is defined by position, color, opacity, rotation, and 2D scale (Zhang et al., 3 Apr 2026). LGTM uses textured 2D Gaussian primitives with learnable per-primitive color and alpha textures, thereby decoupling geometric complexity from rendering resolution (Lao et al., 26 Mar 2026).

4. Supervision, objectives, and optimization-aware training

Differentiable rendering is the dominant supervision mechanism, but the loss design depends strongly on whether the encoder is pose-free, recurrent, appearance-disentangled, or budget-controlled.

AnySplat is trained without explicit 3D supervision such as SfM point clouds or MVS geometry. Instead it combines a self-supervised rendering loss, a geometry consistency loss between predicted and rendered depth, a pose distillation loss from VGGT pseudo poses, and a depth distillation loss from VGGT pseudo depth, with $\boldsymbol{\mu}_g$5, $\boldsymbol{\mu}_g$6, $\boldsymbol{\mu}_g$7, and $\boldsymbol{\mu}_g$8 in the full objective (Jiang et al., 29 May 2025). Flash-Mono uses a weighted sum of pose, geometry, and rendering losses, where pose is supervised in quaternion-plus-translation form, geometry uses confidence-weighted depth supervision, and rendering includes MSE, LPIPS, and depth terms (Zhang et al., 3 Apr 2026). PreF3R combines confidence-weighted pointmap regression with masked MSE on rendered images, which directly couples structural accuracy and photorealism (Chen et al., 2024).

Other encoders rely on RGB supervision only. SparseSplat is trained end-to-end with RGB supervision only, using pixelwise reconstruction error and LPIPS on rendered images; there is no separate explicit supervision on Gaussian parameters (Zhang et al., 3 Apr 2026). PointSplat is likewise trained end-to-end with RGB supervision only, using $\boldsymbol{\mu}_g$9 and $\sigma_g$0 (Guo et al., 30 Jun 2026).

Several papers add task-specific supervisory signals that alter what the encoder is encouraged to represent. SpectralSplat augments standard feed-forward supervision with invariant base-color loss, augmented reconstruction loss, cross-appearance swap loss, and a base-color loss anchored to pseudo-ground-truth intrinsic albedo. The base stream receives gradients only from $\sigma_g$1 and $\sigma_g$2, not from the photometric losses, specifically to prevent appearance leakage into the canonical pathway (Herau et al., 3 Apr 2026). EcoSplat adds a pseudo ground-truth importance mask $\sigma_g$3 and supervises opacity with a BCE-based importance-aware opacity loss so that the encoder can rank primitives under a user-specified target primitive count $\sigma_g$4 (Park et al., 21 Dec 2025). F4Splat supervises predicted densification scores using the view-space positional gradient of the rendering loss, converting that gradient norm into a scalar target $\sigma_g$5 via a logarithmic mapping (Kim et al., 22 Mar 2026).

A separate line of work makes the training objective explicitly optimization-aware. ForeSplat argues that standard feed-forward 3DGS is trained solely for zero-step rendering error, ignoring whether its output constitutes a good initialization for the downstream optimizer. It therefore proposes an optimization-aware framework in which the encoder is trained to produce initializations designed for rapid and effective refinement (Li et al., 21 May 2026). This suggests that, in predict-then-refine pipelines, the encoder may be evaluated not only by zero-step reconstruction quality but also by the quality of the optimization trajectory it induces.

5. Compactness, scalability, and Gaussian allocation

A central axis of encoder design is how primitive count scales with resolution, view count, and scene complexity. The literature presents several incompatible solutions: sparse sampling, point-centric prediction, global latent bottlenecks, differentiable voxelization, textured primitives, and learned budget control.

SparseSplat addresses redundancy by abandoning pixel-aligned prediction and allocating Gaussians according to image entropy. On DL3DV, DepthSplat reports 24.17 PSNR, 0.816 SSIM, 0.152 LPIPS, and 688k Gaussians, whereas SparseSplat-150k reports 24.20 PSNR, 0.817 SSIM, 0.168 LPIPS, and 150k Gaussians; the paper summarizes this as state-of-the-art rendering quality with only 22% of the Gaussians, and still reasonable rendering quality with only 1.5% of the Gaussians (Zhang et al., 3 Apr 2026). PointSplat reduces inter-view redundancy by predicting a single object-centric 3D representation, reporting 71k Gaussians against 204k for GS-LRM, 212k for DepthSplat, and 86k for AnySplat, while reaching 27.18 PSNR / 0.891 SSIM / 0.071 LPIPS on DNA-Rendering and 27.62 PSNR / 0.887 SSIM / 0.084 LPIPS on ActorsHQ (Guo et al., 30 Jun 2026).

GlobalSplat uses a fixed latent bottleneck and a coarse-to-fine training curriculum that gradually reveals Gaussian capacity. Its final 16K model uses 2048 tokens and 8 Gaussians per token, for 16,384 Gaussians total. On RealEstate10K with 24 views, GlobalSplat16K reports 28.53 PSNR, 0.883 SSIM, 0.140 LPIPS, 16K Gaussians, 1.79 GB peak memory, 77.88 ms inference time, and a 3.8 MB footprint (Itkin et al., 16 Apr 2026). This is a scene-centric alternative to dense view-aligned prediction.

LGTM addresses a different scalability barrier: high-resolution synthesis. Prior feed-forward Gaussian splatting methods that predict pixel-aligned primitives experience quadratic primitive-count growth as output resolution increases. LGTM decouples geometry from rendering resolution by predicting a compact geometric Gaussian grid plus per-primitive texture maps. At 4K, the benchmark reports LGTM at $\sigma_g$6 using $\sigma_g$7 primitives and $\sigma_g$8 textures with 5.51 GB peak memory and 175.10 ms total time, compared to about 3.06 GB and 119.13 ms for a low-resolution NoPoSplat 2DGS baseline at only $\sigma_g$9 output (Lao et al., 26 Mar 2026).

AnySplat tackles dense-view inflation by differentiable voxelization, quantizing Gaussian centers into voxels and aggregating attributes with confidence-weighted softmax weights. The paper reports that this reduces redundant primitives by about 30–70% while lowering memory use and smoothing gradients (Jiang et al., 29 May 2025). EcoSplat instead keeps the pixel-aligned formulation but learns a primitive ranking conditioned on a user-specified budget $\boldsymbol{r}_g$0. On RE10K with 24 views at 5% Gaussians, it reports 24.72 dB PSNR / 0.822 SSIM / 0.183 LPIPS; in the same setting it keeps only 78K Gaussians, and the paper notes that compared with GGN it uses about 7× fewer primitives while improving PSNR by around 9 dB (Park et al., 21 Dec 2025).

6. Applications, adjacent compression pipelines, and open limitations

Feed-forward Gaussian splatting encoders now appear in several adjacent research programs. In monocular SLAM, Flash-Mono combines a feed-forward prediction frontend, a 2DGS mapping backend, and a hidden-state-based loop closure module, reporting a 10x speedup over optimization-based GS-SLAM and about 20 refinement iterations in the backend (Zhang et al., 3 Apr 2026). In driving scenes, SpectralSplat preserves the reconstruction quality of the underlying backbone while enabling controllable appearance transfer and temporally consistent relighting across driving sequences (Herau et al., 3 Apr 2026). In human telepresence, PointSplat is explicitly motivated by immersive live streaming systems where compactness affects transmission cost, memory, and rendering speed (Guo et al., 30 Jun 2026).

A common misconception is that feed-forward Gaussian splatting is intrinsically pixel-aligned. The literature no longer supports that view. SparseSplat is explicitly pixel-unaligned (Zhang et al., 3 Apr 2026); PointSplat predicts directly in 3D space from a point set (Guo et al., 30 Jun 2026); “Off The Grid” detects Gaussian primitives at sub-pixel coordinates rather than on a dense rigid grid (Moreau et al., 17 Dec 2025); and GlobalSplat resolves cross-view correspondences in a fixed latent scene representation before any explicit Gaussian decoding (Itkin et al., 16 Apr 2026).

A second misconception is that feed-forward implies the absence of any refinement. Several systems are predict-then-refine. ForeSplat is motivated precisely by the observation that feed-forward 3DGS is often followed by downstream optimization, and it trains the initialization to be refinement-friendly (Li et al., 21 May 2026). Flash-Mono separates a direct-prediction frontend from a lightweight refinement backend (Zhang et al., 3 Apr 2026). AnySplat also exposes optional post-optimization over Gaussian and camera parameters after feed-forward prediction (Jiang et al., 29 May 2025).

A third adjacent direction concerns compression of encoder outputs or encoder latents. TinySplat wraps existing feedforward 3DGS generators with a training-free compression stage and reports over 100× compression, comparable quality to FCGS with only 6% of the storage size, 25% of the encoding time, and 1% of the decoding time (Song et al., 11 Jun 2025). CodecSplat moves compression inside the Gaussian-generation pipeline by coding an intermediate 2D Gaussian-generation feature, reaching 23.56–26.36 dB on DL3DV with 20.00–107.77 KiB per scene and 24.76–27.05 dB on RealEstate10K with 3.37–12.51 KiB per scene (Yu et al., 25 May 2026). FCGS and LocoMoco instead compress existing 3DGS assets in a generalizable feed-forward pass, with FCGS reporting over 20× compression and LocoMoco reporting about 10% better BD-Rate than FCGS (Chen et al., 2024, Liu et al., 30 Nov 2025).

The limitations remain domain-specific but recurrent. GlobalSplat notes that a fixed Gaussian budget may be insufficient for city-scale or unbounded scenes and that static scenes only are considered (Itkin et al., 16 Apr 2026). PreF3R requires substantial frame overlap and is sensitive to scale estimation (Chen et al., 2024). WildSplatter uses a single global appearance embedding, which can cause slight color drift and limited modeling of localized complex effects such as shadows (Fujimura et al., 23 Apr 2026). PointSplat notes memory constraints in unbounded scenes and identifies temporally consistent 4D representations as an open problem (Guo et al., 30 Jun 2026). These limitations suggest that the next phase of encoder research is likely to center on adaptive capacity, stronger long-context geometry priors, and tighter integration of feed-forward prediction with downstream optimization or compression.