Semantic and Depth-Guided View Transformation

• Semantic and depth-guided view transformation is a technique that integrates semantic cues with depth priors to ensure category consistency and geometric alignment in novel view synthesis.
• The method leverages target-specific Gaussian projections and identity encoding to selectively optimize regions of interest while suppressing background noise.
• Empirical results show that approaches like TSGaussian improve reconstruction quality in sparse-view settings, achieving higher PSNR and SSIM metrics compared to baselines.

Searching arXiv for papers on semantic- and depth-guided view transformation, with emphasis on TSGaussian and closely related methods. Semantic and depth-guided view transformation denotes a family of methods that couple semantic cues with geometric priors so that information lifted, warped, or rendered across viewpoints remains both category-consistent and geometrically aligned. In sparse-view novel view synthesis, this coupling is instantiated by TSGaussian as target-specific 3D Gaussian splatting with semantic constraints and monocular depth priors; in related work, analogous designs appear as hierarchical NeRF guidance, depth-conditioned multiple-plane image prediction, depth-guided 2D-to-3D voxel lifting, and semantics-aware Bird’s-Eye-View transformation (Zhao et al., 2024, Gao et al., 2024, Huang et al., 2020, Zhu et al., 1 Mar 2026, Duan et al., 22 Jul 2025).

1. Conceptual scope and problem setting

The central problem is the transformation of image-derived evidence into a representation that supports novel viewpoints or downstream 3D reasoning under incomplete observation. In the cited literature, this transformation takes several forms: projection of Gaussian primitives into training views, homography-based warping of MPI planes, lifting of image features into voxel or BEV space, and back-projection of monocular depth into point clouds or meshes. Across these settings, semantic guidance constrains which structures should be preserved or emphasized, while depth guidance constrains where those structures should be placed.

A concise cross-section of representative formulations is given below.

System	Transformation form	Semantic and depth guidance
TSGaussian	3D Gaussian projection for target-specific novel view synthesis	SAM+YOLO masks, DEVA IDs, monocular depth priors (Zhao et al., 2024)
HG3-NeRF	Volumetric rendering with hierarchical sampling	CLIP-based semantic guidance and SfM-based geometric guidance (Gao et al., 2024)
Semantic View Synthesis	MPI plane prediction and homography warping	Semantic label map input and synthesized disparity (Huang et al., 2020)
Dr.Occ / SDTR / SDG-OCC	2D-to-3D or BEV view transformation for occupancy or detection	Depth-guided lifting plus semantic or region priors (Zhu et al., 1 Mar 2026, Song et al., 2024, Duan et al., 22 Jul 2025)

Within this landscape, TSGaussian is specialized for reconstructing specified targets with complex structures from sparse views. It explicitly prioritizes computational resources on designated targets while minimizing background allocation, and it does so by tightly coupling 2D semantic prompting, 3D identity clustering, pruning, and monocular depth regularization (Zhao et al., 2024).

2. TSGaussian representation and optimization objective

TSGaussian represents the scene as a set of $M$ Gaussian ellipsoids

$$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$

where $\mu_i\in\mathbb{R}^3$ is the center, $s_i\in\mathbb{R}^3$ the scale, $q_i\in\mathbb{R}^4$ the orientation quaternion, $\alpha_i$ the opacity, $c_i$ the color, and $e_i\in\mathbb{R}^{16}$ a learnable identity encoding. The optimized parameter set is $\theta=\{\mu,s,q,\alpha,c,e\}$ (Zhao et al., 2024).

The full objective decomposes into rendering, semantic, and depth terms:

$$L(\theta)=L_{\text{render}}+\lambda_{\text{sem}}L_{\text{sem}}+\lambda_{\text{depth}}L_{\text{depth}}.$$

The rendering term is the photometric loss

$$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$0

which compares rendered views $$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$1 against ground-truth images $$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$2. The semantic term is written as

$$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$3

and the depth term combines soft-hard depth matching with patch-normalized refinement:

$$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$4

This formulation is notable because the semantic and depth terms are not peripheral regularizers. In TSGaussian they are part of the primary optimization objective that determines grouping, geometry, and capacity allocation. A plausible implication is that the framework treats sparse-view reconstruction as a coupled estimation problem rather than a purely photometric fitting problem.

Critical implementation details further define the optimization regime. Camera poses are obtained by COLMAP pre-processing; initialization uses 10 000 random Gaussians; training runs for 10 000 iterations on a single NVIDIA A6000; optimization uses Adam with learning-rate warm-up followed by cosine decay; and typical weights are $$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$5, $$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$6, $$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$7, $$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$8, $$G=\{g_i(\mu_i,s_i,q_i,\alpha_i,c_i,e_i)\}_{i=1\ldots M},$$9, and $\mu_i\in\mathbb{R}^3$0 (Zhao et al., 2024).

3. Semantic prompting, identity encoding, and target-specific Gaussian control

The semantic pipeline begins with 2D prompting. YOLOv9 is first run on each sparse input image to obtain low-cost bounding boxes around the target class. These boxes are then used as prompts for the Segment Anything Model, producing per-image 2D masks $\mu_i\in\mathbb{R}^3$1 that tightly cover only the object(s) of interest. A zero-shot video tracker, DEVA, assigns consistent instance IDs across views. During 3D training, each Gaussian $\mu_i\in\mathbb{R}^3$2 is projected into each view, and if its projected center falls inside the 2D mask, it is tagged as inside the ROI for that view. These tags determine which Gaussians contribute gradients in $\mu_i\in\mathbb{R}^3$3 and which are eligible for semantic-driven cloning or pruning (Zhao et al., 2024).

The 2D identity loss renders a per-pixel identity feature by $\mu_i\in\mathbb{R}^3$4-weighted blending of nearby Gaussians’ encodings:

$\mu_i\in\mathbb{R}^3$5

followed by a per-class prediction $\mu_i\in\mathbb{R}^3$6 and cross-entropy against the mask label $\mu_i\in\mathbb{R}^3$7:

$\mu_i\in\mathbb{R}^3$8

Because each Gaussian carries a learnable 16-dimensional identity code, the rendered identity field can be optimized directly against tracked instances (Zhao et al., 2024).

TSGaussian then adds 3D spatial consistency regularization. It samples $\mu_i\in\mathbb{R}^3$9 points in the union of all Gaussian ellipsoids, finds the $s_i\in\mathbb{R}^3$0 nearest-neighbor Gaussians for each sample, and minimizes a KL-style divergence between their softmaxed identity predictions:

$s_i\in\mathbb{R}^3$1

This encourages the identity field to vary smoothly in 3D and prevents semantically noisy primitives from scattering across the shape (Zhao et al., 2024).

The semantic signal also governs Gaussian control. During each densification step, view-space position gradients are computed only for Gaussians whose projections lie inside the semantic masks. Gaussians in under-populated ROI cells are cloned to improve local detail. Over-crowded cells have their Gaussians split or pruned, guided by whether their learned $s_i\in\mathbb{R}^3$2 remains consistent with the tracked instance. Any Gaussian whose probability of falling into any mask across all views remains below a small threshold is permanently removed. This suggests that TSGaussian uses semantic supervision not only for classification consistency but also for explicit structural editing of the Gaussian field (Zhao et al., 2024).

4. Depth guidance as a geometric stabilizer

TSGaussian uses a pretrained monocular depth estimator to produce per-view depth maps $s_i\in\mathbb{R}^3$3. Its depth term combines two forms of direct alignment and one form of patch normalization. The hard depth term enforces Gaussian centers to match the nearest-ray depths,

$s_i\in\mathbb{R}^3$4

the soft depth term enforces opacities to match softly blended depth,

$s_i\in\mathbb{R}^3$5

and the global-local term refines small-scale structure through patch normalization,

$s_i\in\mathbb{R}^3$6

The stated purpose is to avoid geometry degradation in challenging novel view synthesis tasks under sparse views (Zhao et al., 2024).

Related sparse-view rendering systems implement the same principle with different supervisory granularity. HG3-NeRF uses Hierarchical Geometric Guidance, where SfM depth does not appear as a direct depth loss; instead, the SfM depth $s_i\in\mathbb{R}^3$7 schedules the near and far planes from local to global regions, thereby mitigating misalignment caused by inherent bias in the depth prior. The same method pairs this with Hierarchical Semantic Guidance, which aligns rendered images and ground-truth views through CLIP embeddings under a coarse-to-fine sampling schedule (Gao et al., 2024). A common misconception is that depth guidance must always be explicit per-pixel supervision; HG3-NeRF provides a counterexample, since it states that HGG does not add a new depth-supervision term.

SIDGaussian introduces a different depth-semantic combination in 3D Gaussian Splatting. It uses DINO-ViT patch features as a fixed semantic regularizer between a training-view render and a side-view render, and it applies local depth regularization by computing locally normalized depth patches and optimizing a patchwise Pearson-correlation loss between rendered depth and DPT pseudo-depth. On LLFF, 3 training views, resolution $s_i\in\mathbb{R}^3$8, it reports PSNR $s_i\in\mathbb{R}^3$9 dB, SSIM $q_i\in\mathbb{R}^4$0, and LPIPS $q_i\in\mathbb{R}^4$1, compared with FSGS at PSNR $q_i\in\mathbb{R}^4$2 dB, SSIM $q_i\in\mathbb{R}^4$3, and LPIPS $q_i\in\mathbb{R}^4$4 (He et al., 20 Jan 2025).

Earlier semantic view synthesis work took a two-step route. “Semantic View Synthesis” predicts the color and disparity of the visible surface from a semantic label map, then uses the synthesized color and depth to impose explicit constraints on MPI prediction. The novel view is rendered by plane homographies and front-to-back alpha compositing, with the paper attributing ghosting and blur in direct methods to incorrect alpha allocation (Huang et al., 2020). This establishes an important historical pattern: semantics define appearance priors, but depth or disparity anchors the visible surface so that view transformation remains geometrically consistent.

5. 2D-to-3D and BEV view transformation in perception systems

Outside image synthesis, semantic and depth-guided view transformation is central to occupancy prediction and multi-camera 3D perception. Dr.Occ formulates this explicitly as a depth-guided 2D-to-3D View Transformer, D$q_i\in\mathbb{R}^4$5-VFormer, backed by dense depth cues from MoGe-2. It back-projects each surround-view pixel into the LiDAR or world frame, voxelizes the resulting point cloud into a geometry-aware occupancy mask, and performs forward projection, backward projection densification, and selective refinement only over voxels with $q_i\in\mathbb{R}^4$6. It then complements geometric alignment with region-guided expert routing through R-EFormer or R$q_i\in\mathbb{R}^4$7-EFormer, where a router selects spatial regions or top-$q_i\in\mathbb{R}^4$8 voxels for further deformable cross-attention (Zhu et al., 1 Mar 2026). On the Occ3D-nuScenes benchmark, the baseline BEVDet4D achieves IoU $q_i\in\mathbb{R}^4$9 and mIoU $\alpha_i$0; adding D$\alpha_i$1-VFormer yields IoU $\alpha_i$2 and mIoU $\alpha_i$3; adding R-EFormer with D$\alpha_i$4-VFormer gives IoU $\alpha_i$5 and mIoU $\alpha_i$6 (Zhu et al., 1 Mar 2026).

SDTR also separates semantics and depth, but in a Transformer-based BEV detection and segmentation pipeline. Its S-D Encoder predicts per-view semantic score maps and depth maps from Stage-5 image features, fuses them by concatenation in the implementation, and uses a Prior-guided Query Builder to inject a global semantic prior into a bank of learnable queries. The reported effect is cumulative: on nuScenes val with ResNet-50, $\alpha_i$7, a baseline Transformer obtains NDS $\alpha_i$8 and mAP $\alpha_i$9; adding the 2D segmentation branch yields $c_i$0; adding the depth branch yields $c_i$1; and adding PQB produces the full SDTR result of $c_i$2 (Song et al., 2024).

SDG-OCC modifies the Lift-Splat-Shoot paradigm more directly. Its view transformation diffuses sparse LiDAR depths within local semantic neighborhoods in the image plane:

$c_i$3

then generates a small number of virtual-point depths around $c_i$4 by bilinear discretization. The stated advantage is that the transform uses actual LiDAR depths plus a class-aware spread rather than a diffuse monocular depth distribution. On Occ3D-nuScenes, the camera-only baseline reports IoU $c_i$5, mIoU $c_i$6; adding the SDG view transform alone gives IoU $c_i$7, mIoU $c_i$8; SDG-Fusion reaches IoU $c_i$9, mIoU $e_i\in\mathbb{R}^{16}$0; and SDG-KL reports IoU $e_i\in\mathbb{R}^{16}$1, mIoU $e_i\in\mathbb{R}^{16}$2 (Duan et al., 22 Jul 2025).

A broader implication of these systems is that semantic-depth guidance is not tied to a single representation. It can govern Gaussians, radiance fields, MPI planes, voxel features, BEV queries, or even interactive 2.5D reconstruction. DepthScape, for example, combines monocular depth reconstruction using MoGE with semantic analysis via a vision-LLM, then extracts geometric anchors such as planes, cylinders, or spheres for direct manipulation in a 2D viewport. The transformation equations are the standard back-projection and reprojection formulas, but the key operational constraint is semantic: extracted anchors reduce placement to semantically meaningful degrees of freedom (Su et al., 1 Dec 2025).

6. Empirical behavior, limitations, and recurring issues

TSGaussian reports evaluation on three public scenes—“bear,” “bouquet,” and “garden,” each rendered from $e_i\in\mathbb{R}^{16}$3 sparse (10–30) views—and on a new Citrus dataset with a smartphone-captured 12-video setup. The reported metrics are as follows (Zhao et al., 2024).

Dataset	PSNR	SSIM / LPIPS
bear	$e_i\in\mathbb{R}^{16}$4	$e_i\in\mathbb{R}^{16}$5 / $e_i\in\mathbb{R}^{16}$6
bouquet	$e_i\in\mathbb{R}^{16}$7	$e_i\in\mathbb{R}^{16}$8 / $e_i\in\mathbb{R}^{16}$9
garden	$\theta=\{\mu,s,q,\alpha,c,e\}$0	$\theta=\{\mu,s,q,\alpha,c,e\}$1 / $\theta=\{\mu,s,q,\alpha,c,e\}$2
Citrus (ours)	$\theta=\{\mu,s,q,\alpha,c,e\}$3	$\theta=\{\mu,s,q,\alpha,c,e\}$4 / $\theta=\{\mu,s,q,\alpha,c,e\}$5

The same report states that on “bear,” PSNR is $\theta=\{\mu,s,q,\alpha,c,e\}$6 dB above 3DGS; on “bouquet,” SSIM is $\theta=\{\mu,s,q,\alpha,c,e\}$7; and on Citrus, PSNR is $\theta=\{\mu,s,q,\alpha,c,e\}$8 dB over the best baseline. Qualitatively, TSGaussian recovers fine branches, petals, and leaf veins with far fewer false-positive Gaussians in the background, while semantic masks overlay cleanly on novel views and baselines show ghosting and fragmentation (Zhao et al., 2024).

Across the broader literature, the main recurring technical issues are misalignment, ambiguity, and capacity waste. Dr.Occ explicitly identifies geometric misalignment in view transformation due to the lack of pixel-level accurate depth estimation and severe spatial class imbalance; its limitations include sensitivity to depth-scale errors for extremely distant voxels and the need to hand-tune region partitions in R-EFormer, though this is alleviated but not fully eliminated by R$\theta=\{\mu,s,q,\alpha,c,e\}$9-EFormer (Zhu et al., 1 Mar 2026). HG3-NeRF notes that noisy poses degrade output and that CLIP-based supervision is global rather than pixel-precise (Gao et al., 2024). “Semantic View Synthesis” identifies exotic materials and strong parallax as failure modes (Huang et al., 2020). DepthScape reports human mask failures, program-format hallucinations, and occasional mis-fits on transparent surfaces or small symmetrical objects (Su et al., 1 Dec 2025).

Several misunderstandings are clarified by these results. First, semantic guidance is not synonymous with semantic segmentation alone: in TSGaussian it also determines densification, splitting, and pruning eligibility; in SDTR it shapes the initial query bank; and in SDG-OCC it restricts depth diffusion to a single semantic class (Zhao et al., 2024, Song et al., 2024, Duan et al., 22 Jul 2025). Second, depth guidance is not a single design choice: it may appear as direct monocular depth penalties, local depth correlation, SfM-conditioned sampling ranges, LiDAR-anchored diffusion, or synthesized disparity that constrains MPI alpha allocation (Zhao et al., 2024, He et al., 20 Jan 2025, Gao et al., 2024, Huang et al., 2020). Third, improvements are often strongest under sparsity or incomplete coverage. This suggests that semantic and depth-guided view transformation is best understood as a strategy for reducing the solution space of ill-posed view synthesis or 2D-to-3D lifting rather than as a generic auxiliary-loss recipe.

In that sense, TSGaussian exemplifies a mature version of the paradigm: 2D instance masks provide target selectivity, identity encodings provide object-level grouping, 3D regularization enforces spatial coherence, pruning suppresses background redundancy, and multi-scale depth losses compensate for missing geometry cues under sparse observation (Zhao et al., 2024).