Semantic TSDF (FAWN) Reconstruction

• Semantic TSDF is a 3D volumetric representation that encodes both signed distance information and semantic attributes for enriched scene understanding.
• FAWN introduces explicit floor and wall normal regularization, aligning surface normals to global up and vertical directions to reduce artifacts.
• The method employs neural TSDF reconstruction pipelines with RGB-D inputs and semantic supervision during training to achieve superior volumetric and semantic consistency.

A semantic TSDF (Truncated Signed Distance Function with semantic integration) refers to a 3D volumetric representation in which each voxel encodes not only the geometric information (distance to the nearest surface) but also semantic attributes—typically class probabilities, instance labels, or other semantic descriptors. The FAWN method ("Floor-And-Walls Normal Regularization for Direct Neural TSDF Reconstruction" (Sokolova et al., 17 Jun 2024)) introduces a semantic TSDF architecture with explicit modeling of scene structure, specifically normal regularization for floor and wall surfaces. The following sections detail the key principles, technical approaches, integration strategies, and canonical applications of semantic TSDF as realized in FAWN and closely related methods.

1. Semantic TSDF: Fundamentals and Definitions

A TSDF encodes $s: \mathbb{R}^3 \rightarrow [-\tau, \tau]$, where $s(p)$ is the signed distance from point $p$ to the nearest object surface, truncated at $\pm \tau$. In a semantic context, each voxel additionally stores semantic quantities such as categorical probability vectors or labels $C(p)$, which may be obtained through projection from 2D semantic segmentation or via direct volumetric prediction.

In FAWN and similar methods, the TSDF is regressed by a neural network from RGB-D or multi-view RGB images, optionally utilizing semantic cues (scene classification, instance recognition, etc.) and prior knowledge (spatial layout, object relationships) to improve volumetric and semantic completion. The resulting semantic TSDF volume $V$ contains both $s(p)$ and $C(p)$ jointly indexed over the voxel grid.

2. Semantic Structure Priors: Floor and Wall Normal Regularization (FAWN)

FAWN leverages architectural scene priors—namely, the planar and horizontal nature of floors and the verticality of walls—for direct TSDF reconstruction. The core technical innovation is the introduction of surface normal regularization loss, which encourages reconstructed surfaces for walls and floors to exhibit physically consistent normals:

• Floor voxels: Surface normals $\vec{n}_p$ are constrained to align with the global up direction $(0,1,0)$ (horizontal plane).
• Wall voxels: Surface normals $\vec{n}_p$ are regularized towards verticality, i.e., the $(x,z)$ plane, such that $|n_y|$ is maximized.

Let $S_F$ and $S_W$ denote the sets of voxels classified as floor and wall. FAWN introduces penalization terms:

$$L_{\text{floor}} = \sum_{p \in S_F} \lambda_F (1 - (\vec{n}_p \cdot \vec{y}_\text{up})^2)$$

$$L_{\text{wall}} = \sum_{p \in S_W} \lambda_W (1 - (\vec{n}_p \cdot \vec{v}_\text{vertical})^2)$$

Where $\lambda_F, \lambda_W$ are scaling hyperparameters, and normals $\vec{n}_p$ are computed via local TSDF gradients:

$\vec{n}_p = \frac{\nabla s(p)}{\|\nabla s(p)\|}$

This regularization eliminates geometric artifacts (holes, pits, hills) and corrects room layout distortions due to incomplete or noisy sensor data. Notably, the normal regularization is applied only during training, and 3D semantics are required solely to select $S_F, S_W$.

3. Neural TSDF Reconstruction Pipelines

FAWN is implemented as a 3D sparse convolutional module compatible with any architecture where TSDF is regressed as output:

• Input: RGB-D or multi-view RGB images.
• Backbone: 2D CNN feature extraction (ResNet/Transformer), feature back-projection into voxel space, 3D encoder-decoder network (U-Net or sparse 3D convolutional architectures).
• Output: TSDF field over the voxel grid; optional semantic label logit prediction for each voxel.
• Loss: Standard TSDF regression (L1 or log-L1), occupancy cross-entropy, plus FAWN surface normal regularization for semantics-aware voxels.

FAWN is designed to be modular; the normal regularization loss is a plug-in module, applied in parallel with conventional TSDF and semantic losses.

4. Training and Semantic Supervision Requirements

FAWN requires 3D semantic supervision only during training. Scene structure detectors identify floor and wall regions in each training sample (either from dense ground-truth labels or from reliable 2D segmentation projected to 3D). During inference (deployment), normal loss and semantic supervision are not needed; the network predicts TSDF and (optionally) semantic labels solely based on image inputs.

This strategy preserves generality—no additional computational cost or input requirements are imposed at runtime. The semantic TSDF is therefore not restricted in downstream applications, e.g., mesh extraction, occupancy mapping, or navigation.

5. Performance and Evaluation

FAWN-modified architectures have demonstrated systematic quality gains over prior semantic or geometry-only TSDF reconstruction methods across standard benchmarks:

• Benchmarks: SCANNET, ICL-NUIM, TUM RGB-D, and 7SCENES.
• Metrics: Surface accuracy, volumetric completion IoU, semantic consistency.

Empirical results in these datasets confirm that enforcing structural priors via normal regularization:

• Reduces surface artifacts and corrects global room geometry.
• Outperforms existing semantic TSDF approaches that use only per-voxel label fusion or simple geometric priors.
• Yields more semantically and metrically coherent reconstructions for floor/wall regions, which are critical for downstream tasks (navigation, object placement, architectural analysis).

6. Broader Context in Semantic TSDF Research

Methods such as Panoptic Multi-TSDFs (Schmid et al., 2021), MDBNet (Alawadh et al., 2 Dec 2024), and classwise entropy-loss frameworks (Ding et al., 25 Mar 2024) contribute complementary strategies for semantic TSDF construction:

Method	Semantic Integration	Structure Priors/Regularization	Requirements / Limitations
FAWN	Floor/wall classification (structural semantics)	Normal regularization	3D semantics only for training
Panoptic Multi-TSDFs	Instance/class submapping	Multi-resolution, object-centric	Per-frame panoptic segmentation
MDBNet	RGB-F-TSDF fusion + residual normalization	No explicit geometry prior	Balanced loss, modality-specific networks
Classwise Entropy models	Semantic feature completion, intra-class entropy	No explicit geometry prior	Requires dense semantic supervision

FAWN distinguishes itself by leveraging explicit geometric priors based on semantics for regularization, rather than relying exclusively on volumetric fusion or post-hoc semantic aggregation.

7. Practical Applications, Limitations, and Future Directions

Semantic TSDFs with regularization such as in FAWN are especially relevant in:

• Room-scale scene reconstruction for architectural modeling or robot navigation.
• Environments with substantial sensor noise, occlusion, or missing data, where semantic structure can regularize ill-posed geometry.
• Downstream tasks requiring physically plausible layouts and interpretable scene semantics (e.g., planning, simulation).

Limitations include the reliance on semantic detection quality during training, potential underfitting in diverse/unconventional scenes, and the applicability of planar/vertical priors beyond standard indoor environments.

Continued development involves extending semantic priors to more complex structures (stairs, doorways), improving automated semantic-3D correspondence, and integrating uncertainty quantification into both TSDF and semantic regularization for robust deployment in diverse and cluttered environments.

---

In sum, semantic TSDF reconstruction frameworks with explicit scene structure prior regularization, as exemplified by FAWN, represent an evolution in leveraging semantic knowledge—not merely for label fusion, but as a mechanism to constrain and improve 3D geometric inference, yielding quantitatively and qualitatively superior scene reconstructions for autonomous systems and spatial AI applications.