SemSight: BEV Semantic Maps for Navigation

• SemSight is a probabilistic model that predicts complete indoor semantic maps and target area heatmaps from sparse, egocentric observations.
• It utilizes a PSPNet-based encoder–decoder with a ResNet50 backbone and pyramid pooling to integrate local details and global context.
• Mask-constrained supervision encourages inferring unseen structures, significantly enhancing target localization and navigation efficiency.

SemSight is a probabilistic bird’s-eye-view prediction model for multi-level scene semantics in target-driven navigation and autonomous exploration. It is designed for the prediction of unexplored indoor regions from partial egocentric observations and a target category query, with the explicit goal of jointly inferring structural layouts, global scene context, and target area distributions. In this formulation, the model completes semantic maps of unknown space while producing a probability heatmap over likely locations of a queried functional area such as a bedroom, thereby coupling scene completion with navigation-oriented semantic priors (He et al., 25 Sep 2025).

1. Problem formulation and semantic scope

SemSight addresses the problem of predicting the semantic layout of unexplored regions in indoor environments for efficient navigation, given partial observations and a navigation target. The stated objective is not limited to estimating geometric occupancy or detecting single objects. Instead, the model operates at the room or area level and aims to recover how spaces such as rooms and corridors are organized, how their boundaries are arranged, and where target functional spaces are likely to appear (He et al., 25 Sep 2025).

At time $t$, the agent receives

$o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$

where $p_t$ is position, $H_t$ is exploration history, $E_t$ is the explored-region mask, $B_t$ is the obstacle mask, and $M_t^{\text{local}}$ is the local semantic map. It also receives a query $q$, the target category. The learning problem is formalized as

$$\mathcal{F}: (o_t, q) \mapsto (\hat{M}_t^{\text{global}}, P_t^q),$$

where $\hat{M}_t^{\text{global}}$ is the predicted completed global semantic map in BEV form, and $o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$0 is a probability heatmap in which each element $o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$1 denotes the likelihood that cell $o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$2 belongs to the queried class $o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$3 (He et al., 25 Sep 2025).

This formulation makes the semantic target explicit. Structural layout, global scene context, and target area distribution are learned jointly rather than as isolated subtasks. A direct consequence is that the output is useful both for environment understanding and for downstream search behavior, since the model predicts not only what unseen space may contain, but also where a robot should preferentially search for a target area.

2. Network architecture and representational design

The core architecture is a PSPNet-based encoder–decoder network. Its inputs are the concatenation of structural channels and semantic channels. The structural channels are the robot’s position, exploration history, obstacle mask, and explored-region mask, for a total of 4 channels. These are concatenated with $o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$4 semantic channels representing partial semantic observations. In the reported experiments, $o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$5, yielding $o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$6 input channels (He et al., 25 Sep 2025).

The encoder uses a ResNet50 backbone to capture local and long-range spatial context from partial input maps. The decoder follows the PSPNet style and uses pyramid pooling to aggregate scene context at multiple scales. This architectural choice is directly tied to the intended inference regime: local detail is needed for wall, doorway, and boundary reasoning, while global context is needed for plausible room arrangement and functional organization (He et al., 25 Sep 2025).

The network produces two outputs. The first is $o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$7, a multi-channel prediction of the complete semantic BEV map over all classes. The second is $o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$8, a single-channel probability map for the queried target category. The latter is central to the navigation use case because it expresses target likelihood directly in map space rather than requiring a separate target localization module.

SemSight’s design explicitly supports three kinds of modeling. It models structural layouts by using obstacle, wall, door, and region boundary information; it leverages global scene context through pyramid pooling; and it predicts target area distributions through a probabilistic heatmap conditioned on the query. The paper states that probabilistic modeling allows the network to represent multiple plausible completions and uncertainties, which is important when inferring unseen regions from sparse evidence (He et al., 25 Sep 2025).

3. Mask-constrained supervision and learning objective

A central methodological feature of SemSight is mask-constrained supervision. Supervision is applied only to unexplored regions, while explored regions are ignored with zero weight. The ground-truth supervision signal is defined as

$o_t = (p_t, H_t, E_t, B_t, M_t^{\text{local}})$9

with $p_t$0 denoting element-wise multiplication (He et al., 25 Sep 2025).

The stated purpose of this strategy is to force the model to infer semantic structure from observed context rather than to copy what has already been seen. The contrast with non-mask supervision is especially important in SemSight’s evaluation: without this constraint, the model learns a trivial “copy observed” behavior and fails to infer unseen structure. In the context of semantic completion, this is not a minor training detail but a definition of what counts as successful prediction.

Training uses a multi-task weighted loss that jointly optimizes global semantic completion and target area prediction:

$p_t$1

where $p_t$2 is the ground-truth binary mask for class $p_t$3 in unexplored regions (He et al., 25 Sep 2025).

Binary cross-entropy is used with class-balanced weights:

$p_t$4

Here $p_t$5 is a class-balanced weight and $p_t$6 is the sigmoid function. The use of class-balanced BCE is explicitly tied to class imbalance in the semantic map prediction problem.

A common misconception would be to interpret SemSight as a generic semantic segmentation model operating over partially observed maps. The training protocol indicates otherwise: it is a completion model whose loss is deliberately restricted to the unknown subset of the environment. That distinction is central to the reported gains.

4. Data simulation and training corpus

SemSight is trained on simulated frontier-driven exploration trajectories built from the RPLAN floorplan dataset. The paper states that training is based on approximately 2,000 diverse annotated layouts, and the abstract describes the resulting corpus as 40,000 sequential egocentric observations paired with complete semantic maps (He et al., 25 Sep 2025).

Each layout is decomposed into 10 semantic channels: 7 room types, along with walls, entrance doors, and outside. The room types are described with examples such as bedroom, living room, and kitchen. Exploration is simulated with an algorithm intended to mimic realistic robotic exploration. At each timestep, the robot obtains position, history, occupancy and exploration masks, and the local semantic region map (He et al., 25 Sep 2025).

For each episode, the first 10–20 exploration steps are sampled. Each partial observation is paired with the global ground-truth semantic map to form a training example $p_t$7. The stated goal is to cover a wide range of partial observability conditions across many layouts and target categories.

This training design is significant because it aligns the data-generation process with the intended deployment regime. The model does not observe arbitrary masked maps; it observes sequential partial maps produced by simulated exploration. A plausible implication is that SemSight’s priors are shaped not only by indoor layout statistics but also by the dynamics of frontier-based observation accumulation.

5. Evaluation metrics and quantitative performance

SemSight is evaluated specifically on unexplored regions using adapted and novel metrics. The metrics are FWIoU (Frequency-Weighted Intersection over Union) for global layout accuracy weighted by class occurrence, PA (Pixel Accuracy or Region Recognition Accuracy) for pixel-wise semantic correctness in unexplored areas, Structural Consistency (SC) for plausibility and connectivity of predicted functional regions, and target-category recall, precision, and F1-score for main functional areas (He et al., 25 Sep 2025).

The most direct quantitative comparison is between non-mask supervision and mask-constrained supervision.

Supervision setting	FWIoU / PA / SC	Interpretation
Baseline, supervision only on observed regions	25.5% / 26.3% / 12.6%	Trivial “copy observed” behavior
SemSight with mask-constrained supervision	69.3% / 86.1% / 64.8%	Stronger unseen-region inference

For functional area prediction, the reported F1-scores are 81.5% for bedroom, 87.7% for living room, 65.2% for kitchen, and 69.7% for bathroom. The paper also states that the model reliably infers plausible room arrangements, wall positions, and door placements from partial context (He et al., 25 Sep 2025).

Qualitatively, the target-area heatmaps evolve during exploration from uncertain or uniform distributions to confident, focused regions as more spatial cues are observed. The predicted global semantic maps are described as displaying functional-area coherence, realistic boundaries, and room interconnectivity typical of homes even under sparse input.

These results support the paper’s central methodological claim: the performance gain is not merely due to a stronger backbone, but to the interaction between architecture and mask-constrained supervision. The ablation directly targets the failure mode of copying observed regions and shows that preventing that behavior materially changes semantic completion quality.

6. Navigation effects, interpretation, and relation to adjacent work

SemSight is also evaluated in closed-loop navigation. When used to guide robots toward target areas, it reduces the average number of steps to find the target by 41.7%, from 57.8 to 33.7. It reduces the exploration ratio by 16.8%, from 17.9% to 14.9% area explored before success, and improves SPL by 16.2%, from 53.7% to 62.4% (He et al., 25 Sep 2025).

These gains are consistent with the model’s output structure. Because SemSight predicts a completed semantic map together with a target probability heatmap, it provides a navigation policy with semantic priors over unexplored space rather than only a geometric estimate of free or occupied cells. The reported interpretation in the paper is that these predictions guide robots to relevant areas faster and more efficiently than standard exploration.

In relation to nearby robotics work, SemSight occupies a distinct position. “Semantic Mechanical Search with Large Vision and LLMs” introduces Semantic Mechanical Search (SMS), which generates a semantic occupancy distribution explicitly using LLMs and combines it with spatial occupancy for object retrieval in shelves (Sharma et al., 2023). SemSight instead predicts a probabilistic BEV completion of room-level scene semantics together with target area distributions for indoor navigation (He et al., 25 Sep 2025). Both works use semantic priors to improve search efficiency, but they operate at different semantic scales: SMS focuses on object relationships in closed-world shelf environments, whereas SemSight focuses on room and area structure in indoor layouts.

SemSight also addresses a narrower question than general semantic map completion. It is not merely reconstructing a map for visualization. Its outputs are explicitly coupled to navigation objectives through target-conditioned probability maps. Conversely, it is not a pure target localization model, because it also completes the full global semantic map. That joint formulation is the model’s defining feature.

The strongest empirical evidence is reported on simulated frontier-driven exploration over RPLAN layouts and on closed-loop simulations. This suggests that the demonstrated benefits are clearest for indoor environments with the architectural and functional regularities represented in that dataset. The paper’s results nevertheless establish that probabilistic inference over multi-level scene semantics can improve both map prediction metrics and navigation efficiency in a unified framework (He et al., 25 Sep 2025).