StructSplat: 3D Gaussian Splatting Framework

• StructSplat is a dual-framework approach that provides fast 3D Gaussian reconstruction from uncalibrated images while also optimizing sparse GPU computations.
• It decouples geometry, semantic, and texture cues through dedicated encoders and reassembling blocks to ensure precise view synthesis and robust camera alignment.
• The framework achieves significant performance gains, with improvements such as a +5.67 dB PSNR boost over competitors and strong cross-dataset generalization.

StructSplat is the designation for two distinct but related frameworks: (1) a 3D Gaussian splatting architecture for generalizable scene reconstruction from uncalibrated sparse views in computer vision (Zhao et al., 26 Jun 2026), and (2) a code-generation scheme for optimized sparse GPU computation under the SPLAT umbrella, focusing on attention kernels in deep learning (Gupta et al., 2024). This article focuses primarily on the 3D reconstruction framework, while establishing conceptual links to structured GPU computation where appropriate.

1. Structured 3D Gaussian Splatting from Uncalibrated Views

StructSplat introduces a feed-forward, generalizable method for fast 3D Gaussian reconstruction given a set of N uncalibrated source images $$\{I^s_i \in \mathbb{R}^{H \times W \times 3}\}_{i=1..N}$$, without requiring test-time knowledge of camera intrinsics or extrinsics (Zhao et al., 26 Jun 2026). Unlike prior methods that perform per-scene optimization or assume known camera parameters, StructSplat predicts both scene geometry and camera poses while maintaining high fidelity and generalization across domains.

The output of StructSplat is a global set of 3D Gaussian primitives

$$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$

in a consistent world coordinate frame, enabling differentiable rendering of novel target views ($\hat{I}^t = \mathrm{Render}(\mathcal{G}, p^t)$ for arbitrary $p^t$).

2. Multi-Stream Feature Representation and Decoding Architecture

The core architectural advance in StructSplat is the explicit decoupling of geometric, semantic, and texture cues, each captured by a dedicated encoder:

• Geometry features: encoded via a VGGT-based multi-view transformer yielding multi-view consistent scene tokens and depth predictions.
• Semantic features: encoded using a frozen DINOv3 network, which computes per-image, multi-scale semantic maps, supporting object- and context-level regularization.
• Texture features: captured by a compact convolutional encoder producing high-resolution maps for precise texture recovery.

Feature fusion is performed in the Gaussian decoding pathway via "Reassembling" blocks, which concatenate and project geometry and semantic features. Texture features are injected at a late stage, immediately before the prediction heads for each Gaussian's attributes (depth $d$, opacity $\alpha$, color $c$, scale $s$, and rotation $r$).

3. Pixel-Aligned Feature Injection and Semantic Priors

Fine-grained, view-dependent appearance is reconstructed by injecting texture features pixel-aligned with the input images. Formalization:

• Let $G_i(u,v) \in \mathbb{R}^{C_\text{geo}}$ (geometry), $$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$0 (semantics), $$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$1 (texture).
• Reassembling block: $$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$2.
• Texture injection: $$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$3.
• The output $$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$4 supplies the local parameters for each Gaussian primitive.

Semantic priors, introduced via DINOv3, substantially reduce ambiguity under few-view conditions: ablation indicates a PSNR increase from 20.61 dB (geometry only) to 26.24 dB (+ semantics), with an additional rise to 28.05 dB when including texture features.

4. Camera Alignment and Leakage-Free Pose Estimation

Jointly predicting camera parameters and scene geometry from uncalibrated images creates the risk of feature "leakage" due to transformer cross-attention involving both source and target views. To mitigate this, StructSplat maintains two streams:

• (A) "Mixed" with all source + target images,
• (B) "Source-only" with just source images.

Alignment is performed post-hoc by:

• Solving for rotation $$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$5 that maximizes

$$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$6

• Solving for scaling $$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$7 and translation $$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$8 minimizing

$$\mathcal{G} = \{(\mu_j, \alpha_j, c_j, s_j, r_j)\}_{j=1..N \cdot H \cdot W}$$9

The target pose is remapped as $\hat{I}^t = \mathrm{Render}(\mathcal{G}, p^t)$0. Omitting this alignment degrades performance by 0.71 dB PSNR.

5. Rendering, Training Regimen, and Quantitative Results

Gaussian rendering follows the differentiable splatting paradigm established by 3DGS: primitives are projected as ellipses on the image plane, composited in depth order. Each primitive encodes 3D center ($\hat{I}^t = \mathrm{Render}(\mathcal{G}, p^t)$1), opacity ($\hat{I}^t = \mathrm{Render}(\mathcal{G}, p^t)$2), color ($\hat{I}^t = \mathrm{Render}(\mathcal{G}, p^t)$3), anisotropic scale ($\hat{I}^t = \mathrm{Render}(\mathcal{G}, p^t)$4), and rotation ($\hat{I}^t = \mathrm{Render}(\mathcal{G}, p^t)$5 as a quaternion).

Training is performed using source and target views, with the following loss for all views:

$\hat{I}^t = \mathrm{Render}(\mathcal{G}, p^t)$6

No explicit depth or 3D supervision is used. On DL3DV with 2 source views, StructSplat achieves 28.045 dB PSNR, surpassing AnySplat (22.377 dB) by +5.67 dB; similar gains are seen in cross-dataset generalization.

Main Performance Comparison (DL3DV, 2 views)

Method	PSNR (↑)	SSIM (↑)	LPIPS (↓)
AnySplat	22.377	0.716	0.150
Depth Anything 3	20.715	0.615	0.226
StructSplat (ours)	28.045	0.888	0.091

Cross-dataset gains: +1.94 dB PSNR on ACID, +1.72 dB on RealEstate10K.

6. Implementation Notes

Significant components include:

• Geometry encoder (VGGT, multi-view consistent tokens and predicted depth).
• Semantic encoder (frozen DINOv3, multi-scale tokens).
• Texture encoder (4-layer conv, $\hat{I}^t = \mathrm{Render}(\mathcal{G}, p^t)$7 native resolution, upsampled).
• Gaussian decoder (modified DPT heads with Reassembling + texture injection).
• Camera decoder (4 Reassembling blocks + fully connected to quaternion and translation).
• Activation functions per attribute: softplus, sigmoid, and normalization per specification.
• Implementation: NVIDIA H100, 2.5 training days (2-view), 73 GB peak memory, 1.535B parameters, 165 ms inference time, image resolution 256×256.

7. Relation to Structured Sparse Computation (“StructSplat” in Attention)

In the context of GPU-accelerated attention, StructSplat denotes a code-generation framework based on the affine-compressed-sparse-row (ACSR) format, optimized for masks with row-regular sparsity patterns (e.g., strided, windowed, blocked). The framework performs a just-in-time analysis of sparsity structure and dispatches optimized kernels for R-SDDMM, softmax, and R-SpMM stages.

Notable properties (Gupta et al., 2024):

• O(N) metadata storage versus O(nnz) in standard CSR.
• JIT detection of structure, with fallback for non-regular masks.
• Achieves 2–6× speedup over cuSPARSE/cuBLAS (10–50% density), and geometric mean speedups of 2.05× (vs. Triton) and 4.05× (vs. TVM) on A100 GPUs.

This suggests that the structured approach to both 3D scene representation and sparse GPU computation reflects a growing emphasis on architectural specialization for regularity in scientific and visual computing.

---

References:

• "StructSplat: Generalizable 3D Gaussian Splatting from Uncalibrated Sparse Views" (Zhao et al., 26 Jun 2026)
• "SPLAT: A framework for optimised GPU code-generation for SParse reguLar ATtention" (Gupta et al., 2024)