ViewSplat: Feed-forward 3D Gaussian Splatting

• ViewSplat is a feed-forward 3D Gaussian splatting framework that leverages anisotropic Gaussians for both coarse-to-fine and dynamic view-adaptive novel-view synthesis.
• The hierarchical variant applies a three-stage, error-aware fusion process using MVS cost volumes and residual depth offsets to improve reconstruction fidelity.
• The dynamic variant utilizes target-view-conditioned residuals via pixel-wise dynamic MLPs for adapting Gaussian attributes from unposed images for efficient rendering.

ViewSplat denotes feed-forward 3D Gaussian splatting architectures for novel-view synthesis, and the name has been used in two distinct but related senses in the recent literature. In one consolidated description, “ViewSplat” refers to the same three-stage, coarse-to-fine architecture as HiSplat, a hierarchical 3D Gaussian splatting framework for generalizable sparse-view reconstruction that performs novel view synthesis using only two-view reference images (Tang et al., 2024). In a later paper, ViewSplat denotes a view-adaptive 3D Gaussian splatting network for novel view synthesis from unposed images, in which canonical Gaussians are refined on-the-fly through target-view-conditioned residual updates produced by pixel-wise dynamic MLPs (Jeong et al., 26 Mar 2026). In both usages, the underlying representation is a set of anisotropic 3D Gaussians rendered by a 3DGS rasterizer, but the mechanisms for improving fidelity differ: one emphasizes hierarchical coarse-to-fine correction, whereas the other emphasizes view-adaptive dynamic splatting.

1. Scope of the designation

A consolidated description tied to HiSplat states that “ViewSplat” refers to the same three-stage, coarse-to-fine architecture, with “ViewSplat” used as the name of a hierarchical 3D-Gaussian-splatting pipeline for generalizable two-view reconstruction. The 2026 paper titled “ViewSplat: View-Adaptive Dynamic Gaussian Splatting for Feed-Forward Synthesis” uses the same term for a different pipeline centered on dynamic, target-view-conditioned residual refinement (Tang et al., 2024).

Designation	Input regime	Distinguishing mechanism
ViewSplat as HiSplat	two-view reference images	hierarchical 3D Gaussians via a coarse-to-fine strategy
ViewSplat (2026)	$N$ unposed images	view-adaptive dynamic splatting with pixel-wise dynamic MLPs

This dual usage is significant because it prevents a common conflation. The hierarchical variant is organized around multi-stage Gaussian generation and inter-scale correction. The dynamic variant is organized around canonical Gaussians plus target-view residual offsets for 3D position, scale, rotation, opacity, and color. A plausible implication is that “ViewSplat” should be interpreted by publication context rather than by name alone.

2. Gaussian formulation and image formation

In the hierarchical formulation, each Gaussian $j$ is parameterized by $\mu_j \in \mathbb{R}^3$, $\Sigma_j \in \mathbb{R}^{3\times 3}$, $c_j \in \mathbb{R}^3$, and $\alpha_j \in [0,1]$. When projecting to an image, the contribution of Gaussian $j$ to pixel $p$ is

$$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$

where $\Pi$ is the $j$0 projection and $j$1 is the pixel coordinate. Feed-forward prediction proceeds from a pixel-aligned feature $j$2, from which a small CNN head regresses the center via un-projecting the predicted depth $j$3 along the camera ray $j$4, opacity through a sigmoid head, covariance through predicted positive axis-scales, and color as a small vector or SH coefficients (Tang et al., 2024).

In the dynamic formulation, a single anisotropic 3D Gaussian primitive is

$j$5

with $j$6, where $j$7 are scale factors and $j$8 is derived from a rotation quaternion $j$9. In practice the representation stores $\mu_j \in \mathbb{R}^3$0, $\mu_j \in \mathbb{R}^3$1, $\mu_j \in \mathbb{R}^3$2, opacity $\mu_j \in \mathbb{R}^3$3, and color as $\mu_j \in \mathbb{R}^3$4 spherical-harmonic coefficients $\mu_j \in \mathbb{R}^3$5. Rendering is expressed as

$\mu_j \in \mathbb{R}^3$6

where a 3DGS rasterizer projects each anisotropic Gaussian into the target view, splats its elliptical footprint into screen space, and accumulates color with alpha compositing in front-to-back order (Jeong et al., 26 Mar 2026).

The shared formalism is the use of anisotropic Gaussians as explicit scene primitives. The divergence lies in how those primitives are predicted and corrected: multi-scale staged prediction in the hierarchical system, and target-view-conditioned residual adaptation in the dynamic system.

3. Hierarchical coarse-to-fine reconstruction

The hierarchical ViewSplat builds three stages $\mu_j \in \mathbb{R}^3$7, ordered low-to-high resolution and coarse-to-fine. Stage 1 operates at input resolution $\mu_j \in \mathbb{R}^3$8 and uses an MVS-style cost-volume stereo block. Given cross-view features $\mu_j \in \mathbb{R}^3$9 from a U-Net+Transformer backbone, the method sweeps $\Sigma_j \in \mathbb{R}^{3\times 3}$0 depth planes, warps other views into view $\Sigma_j \in \mathbb{R}^{3\times 3}$1, builds a cost volume, predicts depth by softmax over the cost-volume slice, and then forms $\Sigma_j \in \mathbb{R}^{3\times 3}$2 from a small CNN on $\Sigma_j \in \mathbb{R}^{3\times 3}$3. These Gaussians capture the scene “skeleton,” including large-scale structure such as blocks, walls, and major surfaces (Tang et al., 2024).

Stages 2 and 3 operate at higher spatial resolutions $\Sigma_j \in \mathbb{R}^{3\times 3}$4 and $\Sigma_j \in \mathbb{R}^{3\times 3}$5. Rather than predicting independently, each stage $\Sigma_j \in \mathbb{R}^{3\times 3}$6 re-uses stage $\Sigma_j \in \mathbb{R}^{3\times 3}$7: it renders the previous-stage Gaussians back into the input views, computes an error map

$\Sigma_j \in \mathbb{R}^{3\times 3}$8

and applies an Error Aware Module (EAM) that ingests $\Sigma_j \in \mathbb{R}^{3\times 3}$9 to predict a small residual depth offset $c_j \in \mathbb{R}^3$0 and $c_j \in \mathbb{R}^3$1. The fractional offset is

$c_j \in \mathbb{R}^3$2

with $c_j \in \mathbb{R}^3$3 and the updated depth given by

$c_j \in \mathbb{R}^3$4

This design localizes regions where coarse Gaussians fail and adds corrective, smaller Gaussians.

Fusion across stages is handled by the Modulating Fusion Module (MFM). For each previous stage $c_j \in \mathbb{R}^3$5, the module concatenates upsampled fine-stage features with previous Gaussian features,

$c_j \in \mathbb{R}^3$6

computes a gating weight

$c_j \in \mathbb{R}^3$7

and updates opacity through $c_j \in \mathbb{R}^3$8. This suppresses old Gaussians in regions where fine Gaussians can better explain the appearance. The resulting architecture explicitly models Gaussians at three resolutions, lets each finer stage attend to and correct the errors of coarser splats, and fuses them through learned opacity gates.

4. View-adaptive dynamic splatting

The dynamic ViewSplat takes as input $c_j \in \mathbb{R}^3$9 unposed images $\alpha_j \in [0,1]$0. A shared geometry transformer backbone, such as MASt3R or VGGT, produces multi-view feature tokens. Two Dense Prediction Transformer heads decode these tokens into a “Gaussian center” head that predicts per-pixel 3D centers $\alpha_j \in [0,1]$1 and a “Gaussian parameter” head that predicts per-pixel attributes $\alpha_j \in [0,1]$2, $\alpha_j \in [0,1]$3, $\alpha_j \in [0,1]$4, and $\alpha_j \in [0,1]$5. These define base canonical Gaussians

$\alpha_j \in [0,1]$6

aligned to the coordinate frame of view 1. A lightweight pose head regresses each view’s relative extrinsic $\alpha_j \in [0,1]$7 in a single forward pass using 6D rotation and 4D translation (Jeong et al., 26 Mar 2026).

The defining component is the view-dependent head, also DPT-based, which outputs the weights of pixel-wise dynamic MLPs. At render time, each view MLP is instantiated and receives a 4-D target-pose feature $\alpha_j \in [0,1]$8, where $\alpha_j \in [0,1]$9 is the unit vector from $j$0 to the target camera center and $j$1 is the log-distance. Each view MLP has one hidden layer of $j$2 units with ReLU and predicts residuals $j$3, $j$4, $j$5, $j$6, and $j$7.

These residuals are applied element-wise:

$j$8

In compact form,

$j$9

where $p$0 denotes per-component addition with quaternion re-normalization. The paper explicitly frames this as a shift from static primitive regression to view-adaptive dynamic splatting. A common misconception is that the mechanism is limited to view-dependent color modeling; in fact, the residual updates cover 3D position, scale, rotation, opacity, and color.

5. Objectives, optimization, and reported performance

The hierarchical ViewSplat supervises the rendered novel-view image from each intermediate fused Gaussian set. With ground-truth novel image $p$1 and stage-$p$2 rendering $p$3, the loss is

$p$4

with default $p$5 and $p$6. In the two-view setting on RealEstate10K / ACID, the reported stage-3 performance is PSNR $p$7 versus single-scale MVSplat $p$8 $p$9, SSIM $$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$0 versus $$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$1, and LPIPS $$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$2 versus $$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$3. For zero-shot cross-dataset evaluation, trained on RealEstate10K and tested on Replica, stage 3 reports PSNR $$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$4 versus PixelSplat $$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$5 $$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$6 (Tang et al., 2024).

The dynamic ViewSplat uses a total loss

$$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$7

where

$$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$8

and

$$G_j(p) = \alpha_j \cdot c_j \cdot N_2(u_p; \Pi(\mu_j), \Pi \Sigma_j \Pi^T + \sigma_{\text{image}}^2 I_2),$$9

The hyper-parameters are $\Pi$0 and $\Pi$1. The implementation uses PyTorch with a CUDA-accelerated 3DGS renderer; MASt3R is used for SPFSplat/V2 and VGGT for the V2-L variant; image resolution is $\Pi$2 and $\Pi$3 for V2-L; initialization comes from pre-trained SPFSplat or SPFSplatV2 with the view-dependent head zero-initialized; the optimizer is Adam with learning rate $\Pi$4, batch size $\Pi$5, a frozen backbone, and a curriculum that progressively increases inter-view frame distance. On RealEstate10K with SPFSplatV2-L backbone, the reported performance is PSNR $\Pi$6 dB, SSIM $\Pi$7, and LPIPS $\Pi$8, compared with static SPFSplatV2-L at $\Pi$9, corresponding to $j$00 dB PSNR. Efficiency on one RTX 4090 is reported as inference time $j$01 s $j$02 FPS) and rendering $j$03 FPS. Trained on RE10K and evaluated zero-shot on ACID and DTU, the method shows consistent $j$04–$j$05 dB gains. Ablations report that the full residual set $j$06 is needed and that decoupling $j$07 and $j$08 causes collapse (Jeong et al., 26 Mar 2026).

These two performance profiles reflect different operating assumptions. The hierarchical system emphasizes sparse two-view generalization and cross-dataset behavior under a coarse-to-fine schedule. The dynamic system emphasizes unposed-image synthesis, single-forward-pass pose regression, and residual adaptability at render time.

6. Position within the broader Gaussian-splatting landscape

ViewSplat belongs to a broader family of feed-forward Gaussian-splatting systems that differ in supervision, coordinate assumptions, and deployment context. “Cross-View Splatter” is a feed-forward method that predicts pixel-aligned Gaussian splats for outdoor scenes captured at ground level and by satellite, fusing orthorectified satellite views with GPS-tagged ground photos in a unified 3D coordinate frame. Its architecture alternates Attn_frame and Attn_global layers with $j$09 bidirectional cross-attention layers $j$10, and its reported results include Tanks & Temples combined PSNR $j$11 versus ground-only $j$12 in sparse settings (Turkulainen et al., 19 May 2026).

At the systems level, “SplatBus” is a Gaussian Splatting Viewer Framework via GPU Interprocess Communication. It separates a Gaussian-splatting renderer from one or more interactive viewers, using two TCP channels and a CUDA IPC channel for bulk image data, and is intended to integrate 3DGS outputs with external clients such as Unity, Blender, Unreal Engine, and OpenGL viewers (Xu et al., 21 Jan 2026).

This wider context clarifies several points of interpretation. ViewSplat is not synonymous with all feed-forward 3D Gaussian splatting. It is also not uniformly tied to a single camera assumption: the hierarchical variant relies on sparse reference views with an MVS-style cost volume, whereas the dynamic variant explicitly targets unposed images and regresses relative extrinsics in a single forward pass. The term therefore identifies a line of methods centered on feed-forward Gaussian prediction and rasterization, but the concrete realization depends on whether the goal is hierarchical sparse-view reconstruction or view-adaptive dynamic synthesis.