Papers
Topics
Authors
Recent
Search
2000 character limit reached

StructSplat: Generalizable 3D Gaussian Splatting from Uncalibrated Sparse Views

Published 26 Jun 2026 in cs.CV | (2606.28321v1)

Abstract: We present StructSplat, a feed-forward and generalizable 3D Gaussian reconstruction framework that operates directly on uncalibrated images without requiring camera parameters. Existing methods either rely on per-scene optimization or assume known camera poses, and often entangle geometry and appearance within a unified backbone, limiting reconstruction fidelity and generalization. Our key idea is to adopt a structured representation that organizes geometry, semantic, and texture cues with explicit roles in the reconstruction process. Specifically, we introduce a pixel-aligned feature injection mechanism to enable accurate texture modeling from 2D observations, incorporate semantic-aware priors to improve global consistency, and design a camera alignment strategy to prevent information leakage and improve generalization. Experiments show that our method significantly outperforms prior approaches on challenging benchmarks. On DL3DV, our method achieves 28.045 PSNR, surpassing AnySplat (22.377) by +5.67 dB. In cross-dataset evaluation, our method achieves +1.94 dB over AnySplat on ACID and +1.72 dB on RealEstate10K. Project page: https://structsplat.github.io Code: https://github.com/J-C-Zhao/StructSplat

Summary

  • The paper introduces a novel structured multi-pathway architecture that decouples geometry, semantic, and texture cues to improve 3D scene reconstruction.
  • It employs a robust camera alignment module to prevent information leakage, achieving state-of-the-art metrics with significant gains in PSNR and SSIM.
  • Ablation studies confirm that integrating dedicated semantic and texture features consistently enhances both global structure and local detail fidelity.

StructSplat: Generalizable 3D Gaussian Splatting from Uncalibrated Sparse Views

Introduction and Motivation

The StructSplat framework addresses the challenge of generalizable 3D scene reconstruction from sparse, uncalibrated multi-view images without relying on camera intrinsics or extrinsics. Existing feed-forward models for novel view synthesis typically require explicit camera parameters or per-scene optimization, limiting scalability and generalization. Moreover, most approaches entangle geometry and appearance within unified backbones, resulting in suboptimal recovery of high-frequency textures and reduced robustness under unposed conditions.

StructSplat introduces a structured multi-pathway design that organizes geometry, semantic, and texture cues into distinct components with specialized roles. This decoupling is motivated by empirical evidence that geometry-oriented features lack the local fidelity needed for appearance reconstruction (Figure 1), thus necessitating additional pathways for semantic and texture information. Figure 1

Figure 1: Geometry-oriented features, lacking dedicated appearance modeling, result in poor color fidelity and texture preservation even on training views.

Methodology

Overall Architecture

Given a set of uncalibrated images, StructSplat extracts three pathways of visual cues: (1) 3D geometry features from a VGGT-based geometry encoder, (2) semantic priors from a pretrained DINOv3 encoder, and (3) pixel-aligned high-resolution texture features. Two parallel decoders operate on these features: the Gaussian decoder predicts all camera-centric Gaussian attributes per source pixel, while the camera decoder estimates camera parameters. A crucial downstream alignment module resolves spatial inconsistencies, mapping all predictions into a global, unified coordinate system, after which differentiable 3D Gaussian splatting is used for feed-forward rendering. Figure 2

Figure 2: Schematic of StructSplatโ€”uncalibrated images are processed into structured cues and decoded into Gaussians and aligned camera poses for global scene rasterization.

Structured Representation and Feature Decoupling

StructSplat's architecture assigns explicit roles to each feature pathway:

  • Geometry Encoder: VGGT encodes global structure and inter-view correspondences suitable for scene assembly and depth prediction.
  • Semantic Encoder: Pretrained DINOv3 features guide Gaussian prediction via object-driven priors, aiding in semantic disambiguation and structural consistency in challenging scenes.
  • Texture Encoder: Lightweight convolutions preserve spatial fidelity and capture high-frequency textures, which are late-fused into the Gaussian prediction pipeline through a pixel-aligned feature injection mechanism.

Ablations demonstrate the necessity of this separation (see Figure 3): incorporating semantic features immediately boosts global structure, while texture cues further improve local sharpness and color fidelity. Figure 3

Figure 3: Ablation studyโ€”geometry-only yields artifacts, semantics restore global layouts, and texture features enable sharp high-frequency details.

Gaussian Prediction and Late Feature Fusion

The Gaussian decoder head is a modified DPT structure that prioritizes sequential reassembly and late-stage fusion. Semantic and geometric features are concatenated and spatially aligned through projection layers; only in the final stage are texture features injected to maximize appearance fidelity without corrupting geometric priors (Figure 4). Figure 4

Figure 4: Gaussian decoder architecture, illustrating spatial alignment, multi-feature fusion, and late texture injection to preserve details.

Camera Alignment and Leakage Prevention

A critical innovation is the camera alignment module (Figure 5), which prevents accidental information leakage between source and target views during pose estimation and supervision. Two streamsโ€”one with mixed source/target views, one source-onlyโ€”yield independent extrinsic estimates, which are then aligned via quaternion-based rotation and translation minimization. This procedure guarantees that Gaussian attribute prediction depends solely on source views, thus enforcing fair evaluation and genuine generalization. Figure 5

Figure 5: Camera alignment strategyโ€”source and target features are strictly separated, preventing target-view information leakage in training or inference.

Quantitative and Qualitative Results

StructSplat achieves state-of-the-art results among methods that do not use any camera parameters. On DL3DV, StructSplat outperforms the prior art (AnySplat) by +5.67 dB in PSNR (28.045 vs. 22.377) and simultaneously yields superior SSIM (0.888 vs. 0.716) and reduced LPIPS (0.091 vs. 0.150). Consistent gains are observed across ACID (+1.94 dB over AnySplat) and RealEstate10K (+1.72 dB), underlining the method's cross-dataset robustness.

Qualitatively, StructSplat recovers sharper boundaries, more detailed textures, and globally consistent layouts in diverse benchmarks, surpassing strong baselines which often produce blurred or structurally distorted results (Figure 6). Figure 6

Figure 6: StructSplat yields higher-fidelity, less-blurred, and artifact-free reconstructions from uncalibrated sparse input relative to competing methods.

In more challenging cross-dataset and sparse-view settings, the method remains stable, as numerically validated in multi-view and high-resolution experiments. Camera alignment affords lower absolute trajectory errors compared to alternative alignment schemes.

Analysis and Ablations

Ablation studies detail the contributions of each architectural element. The incorporation of semantic features into a geometry-only baseline yields a +5.6 dB PSNR jump, and adding texture cues provides another +1.8 dB. Removing camera alignment substantially degrades all metrics, confirming its necessity for generalization.

StructSplat also demonstrates strong implied geometric understanding, generating accurate depth maps from RGB-only supervision, without explicit depth or pose constraints (Figure 7). Figure 7

Figure 7: StructSplat depth maps, produced without ground-truth depth supervision, accurately reflect global 3D geometry across diverse scenes.

Implications and Future Directions

StructSplat advances feed-forward, generalizable 3D reconstruction in practical uncalibrated scenarios by (1) decoupling scene cues into geometry, semantics, and texture, (2) preventing information leakage through robust alignment, and (3) achieving strong fidelity and generalization in the absence of camera parameters. This work weakens the dependency of 3D scene understanding pipelines on calibrated data, promoting wider applicability in unconstrained, real-world settings.

Theoretical implications include the utility of structured, modular representations for disentangling attributes in neural implicit models, and the importance of architectural isolations to prevent cross-view information leakage. Practically, StructSplat's techniques can be extended to dynamic scenes, higher input sparsity, and semantic-aware generation for editing tasks.

Potential future directions involve augmenting the semantic pathway with scene-level or language-conditioned priors, improved handling of view-dependent reflectance or lighting, and scaling to dense urban or dynamic environments.

Conclusion

StructSplat introduces a robust architecture for pose- and intrinsic-free 3D Gaussian splatting from sparse and uncalibrated views. Through structured feature decoupling, pixel-aligned texture injection, and strict camera alignment, StructSplat delivers clear improvements over the existing state of the art in both quantitative and qualitative metrics, and sets a foundation for more flexible generalizable 3D vision systems.

Paper to Video (Beta)

No one has generated a video about this paper yet.

Sign Up to Generate All Videos Subscribe on YouTube

Whiteboard

No one has generated a whiteboard explanation for this paper yet.

Sign Up to Generate

Explain it Like I'm 14

StructSplat: A simple, teen-friendly guide

What is this paper about?

This paper introduces StructSplat, a computer vision method that turns a few regular photos into a 3D scene you can look around inโ€”without knowing any camera settings (like where the camera was or its lens details). It represents the 3D world using many tiny, soft โ€œblobsโ€ (called 3D Gaussians) that carry color and shape. When these blobs are โ€œsplattedโ€ (projected) onto an image, they create realistic pictures from new viewpoints.

What questions is the paper trying to answer?

StructSplat aims to solve three practical problems:

  • Can we build a good 3D model from a few photos when we donโ€™t know the cameraโ€™s position or lens information (uncalibrated images)?
  • How can we keep sharp textures and accurate colors (appearance), while also getting the overall shape right (geometry)?
  • How do we train the system fairly so it doesnโ€™t accidentally โ€œpeekโ€ at the target image itโ€™s supposed to predict (avoid information leakage), and still work well on new, unseen scenes (generalization)?

How does StructSplat work? (Methods explained simply)

Think of the system as three specialists looking at the same photos, each noticing different things, and then a builder assembling a 3D scene from their notes.

  • Geometry encoder (the โ€œarchitectโ€): Figures out the overall 3D structureโ€”where surfaces and shapes are. It uses a powerful neural network to understand how multiple pictures relate in 3D.
  • Semantic encoder (the โ€œplannerโ€): Understands what kinds of things are in the scene (like walls, furniture, trees), providing high-level context so the model stays consistent, even when the photos are few or tricky.
  • Texture encoder (the โ€œartistโ€): Focuses on fine details like colors and textures. It keeps features aligned to image pixels so each pixel can directly influence the color and look of the 3D blobs (this is called pixel-aligned feature injection).

These three sets of features are combined in a decoder that predicts, for each pixel, the properties of a tiny 3D blob (its position, size, rotation, transparency, and color). At first, blobs are placed in each cameraโ€™s local space; then theyโ€™re transformed into a single, shared 3D world.

To produce images from new viewpoints, the method โ€œrendersโ€ the blobs by splatting them onto an image planeโ€”imagine gently pressing soft, colored dots onto a canvas from a particular angle. Because this rendering is differentiable (the computer can compute how to improve the blobs by comparing the rendered image to a real photo), the system learns to make the 3D scene look right.

What does โ€œuncalibratedโ€ and โ€œfeed-forwardโ€ mean?

  • Uncalibrated: The method doesnโ€™t need to know the cameraโ€™s position or lens details in advance.
  • Feed-forward: It predicts the 3D scene in a single pass instead of slow, per-scene fine-tuning. This makes it faster and more usable in practice.

Preventing โ€œinformation leakageโ€ (no cheating)

During training, itโ€™s easy for a neural network to accidentally use the target image (the one itโ€™s supposed to predict) to help itself, which would make it look better than it really is. The paper introduces a camera alignment strategy:

  • It runs two parallel streams: one with only source images (the input photos), and one mixed with source + target images (for estimating camera poses).
  • It estimates camera poses in both streams and then mathematically aligns the target camera into the source-only coordinate systemโ€”without letting the source features โ€œseeโ€ the target image.
  • This keeps training fair and improves how well the method works on new data.

What did they find? Why is it important?

StructSplat beats previous camera-free methods by a large margin on multiple benchmarks, especially when only a few photos are available.

Here are some simple takeaways:

  • On the DL3DV dataset (a standard 3D test set), StructSplat scored much higher image quality (PSNR 28.045) than a leading camera-free baseline (AnySplat at 22.377). Higher PSNR and SSIM and lower LPIPS mean clearer, sharper, more accurate images.
  • It also performed better on other datasets it wasnโ€™t trained on (ACID and RealEstate10K), showing it generalizes well to new scenes.
  • Adding the semantic encoder (high-level scene understanding) gave a big quality boost. Adding the texture encoder (pixel-aligned detail) made the results even sharper and more colorful.
  • The camera alignment step reduced pose errors a lot and improved image quality, confirming itโ€™s key to stable, fair training.

Why it matters:

  • You can make convincing 3D scenes from a handful of casual photos or video framesโ€”no camera calibration needed.
  • Itโ€™s fast (feed-forward) and works well on new scenes, which is vital for real-world uses.

What could this be used for?

  • Creating 3D models for games, movies, or virtual tours from phone videos.
  • Augmented and virtual reality, where quick, reliable 3D scene building is important.
  • Robotics and mapping, where cameras often move unpredictably and camera details arenโ€™t known.

Limitations and whatโ€™s next

The authors note challenges when:

  • There are extremely few views, heavy occlusions, or tricky lighting and shiny surfaces (view-dependent effects).
  • Scenes are dynamic (moving objects) or extremely large.

They plan to improve geometry and appearance modeling and extend the method to dynamic and large-scale scenes.

Quick glossary

  • 3D Gaussian splatting: Representing a scene with many soft, colored 3D blobs that, when projected, form an image.
  • Uncalibrated images: Photos taken without known camera poses or lens details.
  • Feed-forward: One-pass prediction without per-scene fine-tuning.
  • Geometry vs. appearance: Structure (shape/depth) vs. look (color/texture).
  • Semantics: High-level understanding of โ€œwhat is whereโ€ (e.g., this is a chair, this is a wall).
  • Information leakage: When training accidentally uses the target image, making results look unfairly better.
  • PSNR/SSIM/LPIPS: Image quality scores; higher PSNR/SSIM and lower LPIPS are better.
View Paper Prompt  View All Prompts 

Knowledge Gaps

Knowledge gaps, limitations, and open questions

Below is a concise, actionable list of what remains missing, uncertain, or unexplored in the paper.

  • Training supervision for the camera decoder is unclear: Does StructSplat require ground-truth camera intrinsics/extrinsics during training, or is pose learning fully self-supervised from photometric losses? Provide ablations isolating the need for pose supervision and its effect on generalization.
  • Intrinsics modeling and lens distortion are not specified: The method claims to be intrinsic-free, but how focal length/FOV, principal point, and lens distortion are represented and learned is not detailed. Evaluate robustness to varied FOVs and real-world distortion (barrel/pincushion/rolling shutter).
  • Arbitrary view control at inference is under-defined: Without user-supplied camera parameters, how are novel viewpoints specified for interactive NVS? Provide a mechanism and evaluation for user-controlled view synthesis (e.g., canonical scene coordinates + view sampler).
  • Single-view reconstruction capability is not evaluated: Assess performance with one source image (N=1), quantify failure modes, and identify what priors are critical when geometry is severely underconstrained.
  • Extreme sparsity and severe occlusions are acknowledged but not quantified: Provide stress tests across controlled sparsity and occlusion regimes to characterize breakdown points and how semantics/texture injection mitigate them.
  • View-dependent appearance and lighting remain difficult: Current Gaussian attributes appear primarily diffuse (no explicit SH or specular terms). Investigate per-Gaussian BRDFs, learned SH coefficients, or reflectance-decomposition to handle specularities and illumination changes.
  • Domain shift and semantics dependence are underexplored: The semantic encoder relies on DINOv3 features; quantify how domain mismatch (e.g., nighttime, outdoor, texture-poor scenes) affects performance and whether domain-adaptive semantic priors help.
  • High-resolution scalability is untested: All evaluations are center-cropped to 256ร—256. Analyze reconstruction quality, memory, and runtime at higher resolutions (e.g., 512p/1080p/4K) and propose memory-aware Gaussian culling/merging strategies.
  • Computational efficiency and memory footprint are not reported: Provide training/inference speed, GPU memory usage, and model size; profile the overhead of semantic and texture encoders and the camera alignment module.
  • Gaussian count explosion and pruning are not addressed: Predicting per-pixel Gaussians can produce very large sets. Explore pruning, clustering, or hierarchical representations to keep rendering real-time while preserving quality.
  • Camera alignment assumes a single global similarity transform between โ€œmixedโ€ and โ€œsource-onlyโ€ pose sets: If per-view biases exist (e.g., non-uniform errors), this alignment may be insufficient. Evaluate more flexible alignment (e.g., joint similarity + per-view residuals or lightweight bundle adjustment) and characterize failure cases.
  • Leakage prevention is only partially validated: Beyond the reported PSNR/SSIM/LPIPS and ATE improvements, perform adversarial tests that intentionally allow leakage to quantify its impact on generalization, and report cross-dataset gains attributable solely to the alignment strategy.
  • Quantitative 3D geometry evaluation is missing: Report depth error, pose error (beyond ATE pos/rot), and reprojection error to substantiate geometric fidelity, not just NVS metrics.
  • Robustness to camera motion artifacts is not studied: Evaluate rolling-shutter effects, motion blur, and auto-exposure/white-balance inconsistencies commonly found in handheld video.
  • Handling of dynamic, non-rigid scenes is future work: Specify how the structured representation and camera alignment would extend to time-varying geometry (e.g., per-frame Gaussians, motion fields) and provide initial baselines.
  • Large-scale scene reconstruction is not demonstrated: Assess drift, loop closure, and memory/runtime on long trajectories or large environments; consider hierarchical/global map structures and pose refinement at scale.
  • Ambiguity in intrinsic/scale estimation persists: The translation-scale alignment and rotation alignment reduce global mismatches but do not fully resolve inherent scale/FOV ambiguities. Explore learned scale priors, metric depth constraints, or scene-size regularization.
  • Sensitivity to the number of views beyond 6 is unknown: Provide scaling curves for 8โ€“16+ source views, and analyze diminishing returns and computational trade-offs.
  • Occlusion reasoning and depth ordering across views are not explicitly modeled: Investigate occlusion-aware aggregation or visibility reasoning to reduce ghosting and texture bleeding in multi-view fusion.
  • Uncertainty is not modeled: Add predictive uncertainty for camera parameters and Gaussian attributes to identify unreliable regions and enable confidence-weighted rendering or downstream tasks.
  • Limited ablations on feature-injection design: Evaluate alternative injection stages, multi-scale fusion strategies, and cross-view texture aggregation to understand where and how appearance cues help most.
  • Comparison to posed upper bounds is missing: Benchmark against strong posed feed-forward methods to quantify the residual performance gap attributable to the intrinsic/pose-free constraint.
  • Optional use of known intrinsics is not explored: Show whether supplying intrinsics (when available) improves performance and how the framework adapts between intrinsic-free and intrinsics-known regimes.
  • Real-world โ€œin-the-wildโ€ evaluation lacks quantitative metrics: Provide quantitative results on real videos with approximate ground truth (e.g., via SfM proxies) or human perceptual studies to substantiate qualitative claims.
  • Reproducibility details are sparse for camera decoder and losses: Specify loss functions for camera estimation, training curriculum, and how cameras/poses are supervised or constrained to facilitate faithful reimplementation.
View Paper Prompt  View All Prompts 

Practical Applications

Below are practical, real-world applications derived from StructSplatโ€™s findings, methods, and innovations. Each item states the sector, specific use case, potential tools/products/workflows, and key assumptions or dependencies.

Immediate Applications

These can be deployed now with existing models and infrastructure.

  • Sector: AR/VR and 3D content creation
    • Use case: Mobile โ€œpose-freeโ€ capture of rooms, objects, and scenes from casual photos or short videos for instant 3D viewing.
    • Tools/products/workflows: A mobile capture SDK that uploads N uncalibrated frames โ†’ StructSplat inference (cloud or edge) โ†’ export 3D Gaussians โ†’ view in a real-time splat renderer (Unity/Unreal plugin).
    • Assumptions/dependencies: Static or quasi-static scenes, sufficient view overlap, access to GPUs or accelerated inference, reliance on pre-trained geometry/semantic encoders (e.g., VGGT, DINOv3) with appropriate licensing; quality is best near the training domain and may be limited by resolution (e.g., 256ร—256 metrics in paper).
  • Sector: Real estate and interior design
    • Use case: Rapid, photorealistic virtual walkthroughs of properties from smartphone videos without camera calibration.
    • Tools/products/workflows: โ€œStructSplat Cloudโ€ API integrated into listing platforms; workflow: agent records a short video โ†’ cloud inference โ†’ embeddable viewer with novel-view synthesis and measurements.
    • Assumptions/dependencies: Enough coverage of spaces; static lighting helps; output fidelity depends on scene complexity and view sparsity.
  • Sector: E-commerce and marketplaces
    • Use case: 3D product listings from user-generated, uncalibrated photos for higher conversion and lower return rates.
    • Tools/products/workflows: โ€œPose-free 3D Captureโ€ widget for seller apps โ†’ batch inference โ†’ 3D viewer with turntable-like novel views.
    • Assumptions/dependencies: Good texture detail and multi-view coverage; small, highly specular items may need controlled lighting or additional views.
  • Sector: Film, VFX, and previsualization
    • Use case: Fast set/prop digitization from scout footage, bypassing SfM pipelines in low-texture regions.
    • Tools/products/workflows: On-set laptop/edge device pipeline โ†’ ingest frames โ†’ StructSplat โ†’ splat asset for previs; optional DCC integration (Blender/Unreal).
    • Assumptions/dependencies: Static sets; photorealism sufficient for previs; dynamic elements and complex view-dependent effects remain challenging.
  • Sector: Robotics (mapping and planning)
    • Use case: Quick environment models for navigation or simulation from uncalibrated cameras (e.g., low-cost mobile platforms).
    • Tools/products/workflows: ROS node wrapping StructSplat; pipeline: record short exploration video โ†’ reconstruct gaussians โ†’ export occupancy/mesh approximations for planning.
    • Assumptions/dependencies: Non-dynamic scenes; sufficient overlap; coarse metric scale may require external cues; reliability decreases with extreme occlusions.
  • Sector: Cultural heritage and education
    • Use case: Digitize artifacts or small historical sites from visitor photos for virtual exhibits or classroom materials.
    • Tools/products/workflows: โ€œStructSplat Museumโ€ kiosk or ingestion pipeline from public images โ†’ 3D recon โ†’ web viewer for interactive learning.
    • Assumptions/dependencies: Curated image sets improve output; permission and privacy considerations for public-sourced content.
  • Sector: Software engineering and datasets
    • Use case: Pose bootstrapping and leakage-free training for generalizable 3D models; camera-alignment module as a reusable component.
    • Tools/products/workflows: Incorporate StructSplatโ€™s camera alignment strategy into training pipelines to prevent target-view leakage and improve fair evaluation; export estimated camera parameters for downstream tasks.
    • Assumptions/dependencies: Access to the alignment routines; consistency of source-only and mixed-stream predictions; minor scale ambiguities may persist.
  • Sector: Media and social platforms
    • Use case: โ€œVideo-to-3Dโ€ user features for immersive posts and stories without calibration.
    • Tools/products/workflows: Server-side batch inference; web/mobile splat viewer; shareable 3D embeds.
    • Assumptions/dependencies: Compute budget at scale; content moderation; privacy of indoor scenes.
  • Sector: Public safety and emergency response (policy-relevant)
    • Use case: Rapid situational modeling from citizen smartphone videos during incidents for response coordination.
    • Tools/products/workflows: Intake portal โ†’ StructSplat inference โ†’ 3D scene viewer for command centers; optional coarse measurements for triage.
    • Assumptions/dependencies: Static scenes or short time windows; legal and privacy frameworks for user-submitted media; robust handling of diverse devices.
  • Sector: Academia (vision/graphics research and teaching)
    • Use case: Teaching labs in novel view synthesis and 3DGS; reproducible benchmarks on uncalibrated sparse views; ablation of texture/semantic encoders.
    • Tools/products/workflows: Course kits and Colab notebooks; plug-in datasets (DL3DV, ACID, RealEstate10K) with StructSplat baseline; use ATE metrics to study camera alignment.
    • Assumptions/dependencies: Students need GPUs or access to cloud; licensing for pretrained backbones; careful domain matching.

Long-Term Applications

These require further research, scaling, or productization (e.g., dynamic scenes, higher fidelity, robust generalization, multi-sensor fusion).

  • Sector: AR headsets and spatial computing
    • Use case: On-device, real-time, pose-free 3D reconstruction for persistent AR anchors and spatial experiences.
    • Tools/products/workflows: Hardware-accelerated StructSplat variants; streaming inference; live updates to splat scenes.
    • Assumptions/dependencies: Efficient on-device models; better handling of dynamic scenes and lighting; energy constraints on wearables.
  • Sector: Autonomous vehicles and advanced SLAM
    • Use case: Replacing or augmenting SLAM with generalizable reconstruction that tolerates unknown intrinsics and sparse views.
    • Tools/products/workflows: Multi-sensor fusion (IMU/LiDAR/RGB) + StructSplat-derived priors; robust scale estimation; semantic guidance for navigation.
    • Assumptions/dependencies: Safety-critical validation; strong robustness to motion blur, dynamics, and occlusions; regulatory acceptance.
  • Sector: City-scale digital twins and facilities management
    • Use case: Continuous, calibration-free capture of buildings and infrastructure for maintenance, planning, and simulation.
    • Tools/products/workflows: Fleet capture from mobile devices; cloud-based incremental updates; integration with BIM/GIS.
    • Assumptions/dependencies: Scalability of training/inference; domain shift handling; standardized data schemas and privacy policies.
  • Sector: Healthcare (endoscopy and minimally invasive imaging)
    • Use case: 3D scene reconstruction from uncalibrated endoscopic videos for preoperative assessment and training simulators.
    • Tools/products/workflows: Clinical pipeline: ingest OR video โ†’ reconstruct anatomy โ†’ VR/AR visualization; combine with segmentation/detection.
    • Assumptions/dependencies: Medical-grade validation; robust handling of specularities, fluids, and extreme dynamics; regulatory compliance (HIPAA, CE/FDA).
  • Sector: Industrial inspection and maintenance
    • Use case: Calibration-free 3D reconstruction of machinery/plant areas for remote inspection, anomaly detection, and worker training.
    • Tools/products/workflows: Worker mobile capture โ†’ StructSplat โ†’ semantic overlays with defect heatmaps; digital twin integration.
    • Assumptions/dependencies: Handling reflective/low-texture surfaces; integration with safety protocols; domain adaptation for factories.
  • Sector: Simulation and robotics training
    • Use case: Rapid generation of photorealistic training environments from casual videos to reduce sim-to-real gaps.
    • Tools/products/workflows: โ€œSplat-to-Simโ€ pipeline: reconstruct โ†’ convert to simulation-ready assets โ†’ behavior learning.
    • Assumptions/dependencies: Improved view-dependent effects and lighting models; standardized asset conversion; scalability.
  • Sector: Education and cultural preservation at scale
    • Use case: Global programs to convert public media into immersive 3D curricula and archives.
    • Tools/products/workflows: Public submission portals; curation tools; ethics review; multilingual access.
    • Assumptions/dependencies: Strong privacy and copyright frameworks; content curation; robust generalization across diverse imagery.
  • Sector: Search and indexing of spatial content
    • Use case: Spatial search engines that index 3D reconstructions of environments for retrieval and navigation.
    • Tools/products/workflows: โ€œSplat Indexerโ€ service that extracts semantics, geometry, and viewable paths; API for spatial queries.
    • Assumptions/dependencies: Reliable semantic priors; legal considerations around mapping private spaces; scalable storage and retrieval.

Notes on broad assumptions and dependencies (applicable across many items):

  • Static or slowly changing scenes and adequate viewpoint coverage are currently important; extreme sparsity and heavy occlusions degrade fidelity.
  • Pre-trained encoders (VGGT, DINOv3) and differentiable splatting libraries are integral; licensing and model availability must be respected.
  • Performance depends on compute (GPU/TPU) and memory; bfloat16/DeepSpeed-style optimizations can help for training and inference services.
  • Camera-alignment strategy is key to generalization and fair training; downstream users should preserve leakage-free pipelines.
  • Domain shift and resolution constraints may require fine-tuning; dynamic scenes, complex lighting, and strong view-dependent effects remain open research areas.
View Paper Prompt  View All Prompts 

Glossary

  • 3D Gaussian Splatting (3DGS): A point-based 3D scene representation and rendering technique that models scenes with anisotropic Gaussians for efficient, high-quality view synthesis. "3D Gaussian Splatting (3DGS)~\cite{kerbl20233d,fang2024mini,yu2024mip} and Neural Radiance Fields (NeRF)~\cite{mildenhall2021nerf,barron2021mip,barron2023zip,wang2023sparsenerf} have significantly advanced 3D representations and enabled high-quality novel view synthesis."
  • Absolute Trajectory Error (ATE): A metric that quantifies the difference between predicted and ground-truth camera trajectories in terms of rotation and translation. "We compare the Absolute Trajectory Error (ATE)~\cite{zhang2018tutorial} of target camera poses obtained using these two alignment strategies in Table~\ref{tab:cp_of_ca}."
  • bfloat16 (BF16): A 16-bit floating-point format that preserves exponent range of FP32 to enable stable mixed-precision training with reduced memory. "we incorporate bfloat16 (BF16) mixed-precision training \cite{bf16} alongside the DeepSpeed optimization library \cite{zero1, zero2, zero3}."
  • Camera alignment: A strategy to align camera parameters estimated from different input sets into a shared coordinate frame to avoid leakage and ensure consistency. "we design a camera alignment strategy to prevent information leakage and improve generalization."
  • Camera-centric: Defined relative to an individual cameraโ€™s local coordinate system rather than a global/world frame. "The Gaussian decoder, equipped with multiple specialized heads, densely predicts the attributes of camera-centric Gaussians."
  • Camera extrinsics: Parameters (rotation and translation) that define a cameraโ€™s pose in 3D space relative to a world frame. "these approaches still require accurate camera intrinsics and extrinsics obtained via SfM pipelines~\cite{colmap_sfm, glomap, dfnet, vggsfm}"
  • Camera intrinsics: Parameters that characterize a cameraโ€™s internal geometry (e.g., focal length, principal point) used for projection. "these approaches still require accurate camera intrinsics and extrinsics obtained via SfM pipelines~\cite{colmap_sfm, glomap, dfnet, vggsfm}"
  • Camera-to-world transformation: The mapping that converts quantities expressed in a cameraโ€™s local frame into a shared world coordinate frame. "Finally, guided by the predicted cameras, a camera-to-world transformation converts camera-centric Gaussians into a global representation, which is rendered via differentiable splatting to synthesize novel views."
  • Cost volume: A 3D tensor storing matching costs across discretized depths used in multi-view stereo aggregation. "construct 3D cost volumes via plane-sweep or homography-based warping"
  • Cross-attention: An attention mechanism that allows features from one set (e.g., target views) to attend to another (e.g., source views), potentially causing leakage. "due to the cross-attention mechanism in the geometry extractor, target image information inevitably leaks into the source features."
  • Cross-dataset evaluation: Testing a model trained on one dataset directly on other datasets to assess generalization. "In cross-dataset evaluation, our method achieves +1.94 dB over AnySplat on ACID and +1.72 dB on RealEstate10K."
  • DeepSpeed: A system for optimizing large-scale model training with features like memory partitioning and parallelism. "we incorporate bfloat16 (BF16) mixed-precision training \cite{bf16} alongside the DeepSpeed optimization library \cite{zero1, zero2, zero3}."
  • DINOv3: A self-supervised vision foundation model producing semantic features useful for downstream tasks. "using a pre-trained vision foundation model (e.g., DINOv3~\cite{dinov3})."
  • DPT (Dense Prediction Transformer): A transformer-based architecture for dense prediction tasks such as depth estimation. "we modify the standard DPT~\cite{dpt} heads to effectively integrate multi-scale semantic features and high-resolution texture cues."
  • Differentiable splatting: A rendering operation that projects Gaussians to the image plane in a differentiable way, enabling end-to-end learning. "rendered via differentiable splatting and rasterization~\cite{kerbl20233d}"
  • Epipolar constraints: Geometric relations between corresponding points in two views that lie along epipolar lines, constraining matching. "leverages epipolar constraints and attention to implicitly encode 3D geometry without explicit cost volumes."
  • Epipolar lines: Lines in one image on which the projection of a 3D point must lie, given its correspondence in another image. "restricting interactions along 1D epipolar lines instead of constructing explicit cost volumes."
  • Feed-forward: Inference performed in a single pass without per-scene optimization loops. "We present StructSplat, a feed-forward and generalizable 3D Gaussian reconstruction framework that operates directly on uncalibrated images without requiring camera parameters."
  • Fusion block: A decoder component that merges multi-scale or multi-source features during prediction. "After passing through cascaded Reassembling and Fusion blocks, high-resolution texture features are injected just before the final prediction layer"
  • Gaussian decoder: The network head that predicts the parameters of Gaussians (e.g., position, color, scale, rotation) from features. "The Gaussian decoder, equipped with multiple specialized heads, densely predicts the attributes of camera-centric Gaussians."
  • Gaussian primitive: An individual Gaussian element used as a 3D scene primitive in the representation. "The decoder outputs the spatial position (parameterized by depth along the camera's z-axis) and the rotation quaternion defined in the local camera coordinate system for each Gaussian primitive."
  • Geometry encoder: A backbone that learns view-consistent geometric features and priors for 3D reconstruction. "Specifically, we introduce a geometry encoder to provide reliable structural guidance"
  • Homography-based warping: Warping features between views using a plane-induced homography for cost volume construction. "construct 3D cost volumes via plane-sweep or homography-based warping"
  • Information leakage: Unintended flow of target-view information into source features during training, harming generalization. "which may introduce target-view information through cross-view attention during rendering, leading to information leakage and degraded generalization."
  • Intrinsic-free: Methods that do not require known camera intrinsic parameters. "some studies~\cite{frozenrecon, noponerf, cf3dgs, inerf, nerfmm, cfnerf} explore pose-free or intrinsic-free reconstruction settings."
  • Lagrange multiplier: An optimization technique for handling constraints, used here for quaternion alignment. "We solve this problem using a Lagrange multiplier formulation."
  • Large-model-based methods: Approaches that rely on models pretrained on massive data to provide strong priors for reconstruction. "More recently, large-model-based methods~\cite{lrm, gs-lrm, lvsm, longlrm, lgm, instant3d} learn strong priors from large-scale data to reconstruct 3D scenes with minimal geometric constraints."
  • Least-squares problem: An optimization problem that minimizes the sum of squared residuals, used for translation alignment. "This can be formulated as a least-squares problem:"
  • LPIPS: A learned perceptual metric that correlates with human judgments of image similarity. "We report PSNR, SSIM~\cite{ssim}, and LPIPS~\cite{lpips} as evaluation metrics."
  • Mixed-precision training: Training with reduced-precision arithmetic to lower memory/computation while maintaining accuracy. "we incorporate bfloat16 (BF16) mixed-precision training \cite{bf16}"
  • Monocular priors: Single-image cues (e.g., depth estimates) used to regularize multi-view reconstruction. "DepthSplat~\cite{depthsplat} further incorporates monocular priors to improve robustness in textureless regions."
  • Neural Radiance Fields (NeRF): A neural representation that maps 3D coordinates and view directions to volume density and color for rendering. "3D Gaussian Splatting (3DGS)~\cite{kerbl20233d,fang2024mini,yu2024mip} and Neural Radiance Fields (NeRF)~\cite{mildenhall2021nerf,barron2021mip,barron2023zip,wang2023sparsenerf} have significantly advanced 3D representations"
  • Novel view synthesis: Rendering images from unseen viewpoints given a set of input views. "have significantly advanced 3D representations and enabled high-quality novel view synthesis."
  • Opacity: The per-Gaussian transparency parameter controlling contribution to the rendered image. "Each Gaussian is parameterized by its center ฮผj\mu_j, opacity ฮฑj\alpha_j, color cjc_j, scale sjs_j, and rotation rjr_j"
  • Per-scene optimization: Scene-specific iterative fitting, often slow and not generalizable across scenes. "they rely on per-scene optimization, requiring iterative fitting for each individual scene."
  • Pixel-aligned feature injection: A mechanism to inject high-resolution 2D features directly into 3D prediction to preserve texture fidelity. "we introduce a pixel-aligned feature injection mechanism to enable accurate texture modeling from 2D observations"
  • Plane-sweep: A technique that projects features across candidate depth planes to build a cost volume. "construct 3D cost volumes via plane-sweep or homography-based warping"
  • Pose-free: Methods that do not require known camera poses (extrinsics) as input. "pose-free or intrinsic-free novel view synthesis~\cite{noposplat,splatt3r,flare,da3}."
  • PSNR: Peak Signal-to-Noise Ratio, a fidelity metric measuring reconstruction error relative to ground truth. "We report PSNR, SSIM~\cite{ssim}, and LPIPS~\cite{lpips} as evaluation metrics."
  • Quaternion: A four-parameter representation of 3D rotation used here for camera and Gaussian orientations. "We seek a unit quaternion ฮ”q\Delta q that aligns qs,iAq^{A}_{\mathrm{s},i} with qs,iBq^{B}_{\mathrm{s},i} for all source views ii."
  • Rasterization: The process of converting 3D primitives into pixel-based images; here used with differentiable splatting. "rendered via differentiable splatting and rasterization~\cite{kerbl20233d}"
  • Reassembling block: A decoder submodule that spatially aligns and merges features before prediction. "The detailed structure of the Reassembling block is shown on the right side of Fig.~\ref{fig:decoder}."
  • Semantic-aware priors: High-level cues derived from semantics that regularize and improve global reconstruction consistency. "incorporate semantic-aware priors to improve global consistency"
  • Semantic encoder: A network extracting high-level, view-invariant semantic features to guide 3D reconstruction. "we introduce a semantic encoder to provide high-level contextual priors for reconstruction."
  • SSIM: Structural Similarity Index, a perceptual image quality metric for structural fidelity. "We report PSNR, SSIM~\cite{ssim}, and LPIPS~\cite{lpips} as evaluation metrics."
  • Structure-from-Motion (SfM): A pipeline that estimates camera poses and sparse geometry from multiple images. "SfM pipelines~\cite{colmap_sfm, glomap, dfnet, vggsfm}"
  • Texture encoder: A lightweight network extracting high-resolution image-space features for accurate color and texture modeling. "we introduce a texture encoder to capture high-frequency appearance details directly from input images."
  • Tokenized 3D features: Discrete feature tokens representing 3D structure for downstream decoding. "The encoder produces tokenized 3D features that encode scene geometry in a view-consistent manner"
  • VGGT: Visual Geometry Grounded Transformer, a geometry-focused backbone used to extract multi-view features. "We employ a VGGT-based encoder to capture global geometric structure"
  • View-dependent appearance: Appearance that changes with viewpoint (e.g., specularities), challenging to model from compact 3D features. "making them insufficient for modeling fine-grained, view-dependent appearance details."
  • Warmup-Stable-Decay (WSD): A learning rate schedule with warmup, stable, and decay phases for robust training. "we employ the Warmup-Stable-Decay (WSD) learning rate scheduler \cite{wsd},"
  • World coordinate system: A shared global frame into which all camera-centric predictions are transformed for consistency. "map these camera-space attributes into a unified world coordinate framework."
View Paper Prompt  View All Prompts 

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Generate Now

Collections

Sign up for free to add this paper to one or more collections.

Sign Up

Tweets

Sign up for free to view the 1 tweet with 32 likes about this paper.

Sign Up for Free