Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation

Abstract: We introduce a learning-based approach for room navigation using semantic maps. Our proposed architecture learns to predict top-down belief maps of regions that lie beyond the agent's field of view while modeling architectural and stylistic regularities in houses. First, we train a model to generate amodal semantic top-down maps indicating beliefs of location, size, and shape of rooms by learning the underlying architectural patterns in houses. Next, we use these maps to predict a point that lies in the target room and train a policy to navigate to the point. We empirically demonstrate that by predicting semantic maps, the model learns common correlations found in houses and generalizes to novel environments. We also demonstrate that reducing the task of room navigation to point navigation improves the performance further.

• The paper introduces a novel framework that leverages amodal semantic maps to predict unseen room layouts for enhanced navigation.
• It employs a sequence-to-sequence network for map generation and an MLP for target point prediction, integrating multiple data modalities.
• Results demonstrate a RoomNav SPL of 0.35 in validation trials, underscoring the potential for improved navigation in novel environments.

Learning Amodal Semantic Maps for Room Navigation

The paper "Seeing the Un-Scene: Learning Amodal Semantic Maps for Room Navigation" introduces a learning-based framework for room navigation that leverages amodal semantic maps. These maps predict room layouts beyond the agent's current field of view by modeling architectural and stylistic regularities inherent in house designs. This approach contributes to the field of Embodied AI by demonstrating the use of scene priors to improve navigation in previously unseen environments, which is a departure from traditional methods that rely either on pre-constructed maps or sensory input.

The architecture proposed in this paper consists of three main components: Map Generation, Point Prediction, and Point Navigation. The Map Generation component uses a sequence-to-sequence network to predict top-down semantic maps which include information about room type, location, size, and shape. These predictions are amodal, projecting plausible room structures not directly observed by the agent. The approach exploits patterns in house layouts, such as the typical proximity of kitchens to dining rooms, to improve navigational efficiency.

The Point Prediction component reduces the navigation task to point navigation by determining a target point within the desired room. This is achieved through a multi-layer perceptron that combines map predictions, current images, and target room identification to infer a plausible goal location. Consequently, the Point Navigation component uses a pre-trained policy to guide the agent to the predicted target point efficiently, utilizing a depth input for collision avoidance.

The presented results, particularly the RoomNav Success weighted by Path Length (RoomNav SPL), highlight the effectiveness of this strategy. The model achieves a RoomNav SPL of 0.35 in validation trials and 0.33 in test trials. A notable improvement was observed when fine-tuning the point navigation policy, reinforcing the utility of the predicted map guidance.

This research contrasts with classical navigation paradigms, such as SLAM, by emphasizing a prediction-based strategy over environment reconstruction. Unlike SLAM, which often depends on noisy sensors and predefined structures, this learning-based approach dynamically updates its understanding of the environment through iterative predictions, reflecting a more human-like navigation strategy.

The implications for AI are significant. The ability to navigate efficiently in new settings by predicting unseen elements could enhance robotic applications in dynamic environments where pre-mapping is impractical. While this research improves navigation by exploiting domestic architectural consistencies, it also emphasizes the need for models to generalize across diverse cultural and geographical housing designs.

For future work, the paper suggests further enhancing map prediction accuracy and exploring additional environmental features that could be incorporated into semantic representations. Moreover, expanding the diversity of the dataset to include non-domestic and international architectural styles could test the adaptability and robustness of the proposed approach.

In summation, this paper makes a valuable contribution by integrating predictive spatial reasoning into embodied AI, thereby advancing the field towards more autonomous and versatile navigational capabilities within variable and novel environments.