Find a Way Forward: a Language-Guided Semantic Map Navigator

Abstract: In this paper, we introduce the map-language navigation task where an agent executes natural language instructions and moves to the target position based only on a given 3D semantic map. To tackle the task, we design the instruction-aware Path Proposal and Discrimination model (iPPD). Our approach leverages map information to provide instruction-aware path proposals, i.e., it selects all potential instruction-aligned candidate paths to reduce the solution space. Next, to represent the map observations along a path for a better modality alignment, a novel Path Feature Encoding scheme tailored for semantic maps is proposed. An attention-based Language Driven Discriminator is designed to evaluate path candidates and determine the best path as the final result. Our method can naturally avoid error accumulation compared with single-step greedy decision methods. Comparing to a single-step imitation learning approach, iPPD has performance gains above 17% on navigation success and 0.18 on path matching measurement nDTW in challenging unseen environments.