HCSG: Human-Centric Semantic-Geometric Reasoning for Vision-Language Navigation

Abstract: VLN has achieved remarkable progress by scaling data and model capacity. However, the assumption of a static environment breaks down in real-world indoor scenarios, where robots inevitably encounter dynamic pedestrians. Existing human-aware approaches typically treat humans merely as moving obstacles based on implicit visual cues, lacking the explicit reasoning required to interpret human intentions or maintain social norms. To address this, we propose HCSG, the first human-centric framework for VLN. This framework provides a robust foundation for safe, socially intelligent navigation in dynamic human-robot environments that shifts the paradigm from passive collision avoidance to active human behavior understanding. Specifically, HCSG introduces a unified Human Understanding Module that synergizes two key capabilities: (i) geometric forecasting, which predicts human pose and trajectory to anticipate future motion dynamics; and (ii) semantic interpretation, which leverages a Vision-LLM (VLM) to generate natural language descriptions of human actions and intentions. These semantic-geometric representations are fused into the agent's topological map for instruction-conditioned planning. Furthermore, a social distance loss is introduced to enforce socially compliant interaction distances. Extensive experiments on the HA-VLNCE benchmark demonstrate that HCSG significantly outperforms state-of-the-art methods, achieving a 14% improvement in Success Rate and a 34% reduction in Collision Rate. Our project can be seen at https://haoxuanxu1024.github.io/HCSG/.

• The paper introduces HCSG, a unified framework integrating geometric forecasting and semantic interpretation for human-aware vision-language navigation.
• The framework achieves a 14.3% boost in success rate and a 34.5% reduction in collision rate by predicting human trajectories and intents.
• The paper demonstrates sim-to-real reliability with an 82% success rate in real-world scenarios, ensuring safe and instruction-grounded navigation.

Human-Centric Semantic-Geometric Reasoning for Vision-Language Navigation

Motivation and Problem Statement

Vision-Language Navigation (VLN) agents are tasked with interpreting multi-modal instructions and navigating complex environments. Practical deployment, however, reveals a major limitation: the standard VLN paradigm assumes a static world, failing to accommodate the dynamics of human-populated indoor spaces. Existing human-aware approaches primarily treat humans as generic moving obstacles, relying on implicit visual cues and lacking explicit reasoning about human activities, intentions, and social norms. This leads to navigation failures, particularly when instructions reference dynamic social contexts or require sophisticated task grounding, as illustrated in navigation scenarios involving target groups or individual behaviors (Figure 1).

Figure 1: Navigation scenario in a dynamic environment—without semantic and geometric understanding, agents misclassify humans or collide; HCSG integrates both for socially compliant, instruction-grounded navigation.

HCSG Framework Overview

The paper introduces HCSG: Human-Centric Semantic-Geometric Reasoning, a unified framework for VLN that explicitly models human-centric information with dual streams—geometric forecasting and semantic interpretation. The agent builds an online topological graph, enriching static scene features with human-centric attributes upon the detection of humans, enabling instruction-conditioned, socially compliant navigation. The reasoning process is composed of:

• Geometric Reasoning: Forecasts human pose and trajectory using temporal sequences via LSTM encoder-decoders, supplying future-oriented geometric features for navigation.
• Semantic Reasoning: Utilizes a Vision-LLM (VLM) to generate descriptions of human actions and intentions from cropped observations, transforming visual activity into high-level semantic features.
• Feature Fusion: Fuses human-centric semantic and geometric representations with static scene features into the topological graph, processed by a Cross-modal Graph Transformer to select socially compliant waypoints.

  Figure 2: Overview of HCSG—panoramic observations and instructions yield human-centric semantic and geometric features, fused into the scene graph for cross-modal language-guided planning.

Technical Details

Geometric Reasoning

The geometric module decomposes human motion into pose (fine-grained activity) and trajectory (coarse-grained movement), both predicted via LSTM-based encoder-decoders (Figure 3). Future predictions are enforced by supervised losses integrating coordinate, confidence, position, and velocity, encouraging representations to encapsulate imminent human occupancy and movement tendencies.

Figure 3: Pipeline for geometric reasoning—LSTMs predict pose and trajectory over sequences, fusing motion representations for downstream waypoint selection.

Semantic Reasoning

Semantic reasoning leverages zero-shot capabilities of foundation VLMs, converting visual observations into navigation-relevant descriptions of actions, intentions, and social context. Prompts tuned for navigation tasks ensure output relevance. Text embeddings extracted via CLIP encode these descriptions, representing high-level intent complementary to geometric motion parameters.

Figure 4: Human-centric reasoning—semantic stream infers activity, intent, context; geometric stream predicts pose and trajectory evolution.

Social Distance Loss

A dual social safety objective penalizes actual collisions and encroachment within proxemics (front-facing, radial proximity). Proximity loss applies repulsion, while collision loss assigns high penalties for contact. Combined with imitation learning, this objective guarantees instruction-following behavior that respects social norms and maintains safe interaction distances.

Empirical Results

Benchmark Performance

Extensive experiments on HA-VLNCE benchmark demonstrate that HCSG achieves substantial improvements over prior state-of-the-art methods such as BEVBert [an2022bevbert] and ETPNav [an2024etpnav]. On the validation-unseen split, HCSG attains a 14.3% increase in Success Rate and a 34.5% reduction in Collision Rate compared to the best baseline. The results confirm that explicit semantic and geometric modeling enables the agent to execute goal-directed navigation while proactively avoiding collisions and respecting social space.

Figure 5: HCSG avoids a dynamic pedestrian while following instruction; baseline BEVBert collides, illustrating the efficacy of explicit human-centric reasoning.

Figure 6: HCSG recognizes and accomplishes human-referenced VLN tasks; BEVBert misses task-relevant humans.

Ablation Studies

Ablation analyses reveal that:

• Trajectory prediction reduces collision.
• Pose-based features raise task success, especially when the instruction references specific human activities.
• VLM integration improves safety and task grounding beyond geometric cues alone.
• Future-oriented motion features outperform past-oriented encodings, substantiating the predictive modeling paradigm.
• Fine-tuning geometric modules on navigation data yields adaptation and further performance gains.

Sim-to-Real Transfer

Physical deployment on NXROBO Leo validates sim-to-real reliability. HCSG maintains an overall 82% success rate on five real-world scenarios: stationary interactions, dynamic avoidance, and blind-corner anticipation, consistently surpassing baseline approaches.

Figure 7: Real-world qualitative examples—semantic and geometric reasoning drive robust human-aware navigation across interaction, detour, avoidance, and yielding scenarios.

Implications and Future Directions

HCSG fundamentally advances human-aware navigation for real-world embodied agents. By synergizing geometric forecasting and semantic interpretation, agents can reason about social context, future human motion, and interaction-relevant instructions, enabling robust and socially compliant navigation across dynamic environments. Practically, HCSG supports service robots in densely populated spaces and lays groundwork for broader integration of vision-language reasoning in robotics.

Theoretically, the paradigm shift from passive avoidance to active human behavior understanding raises new frontiers: continuous streaming reasoning, adaptive social priors conditioned on richer context, and scalable integration of foundation models (VLMs, LLMs) for grounding complex instructions. Future research may investigate continuous, online refinement of interaction norms, cooperative multi-agent social navigation, and end-to-end learning regimes considering multimodal cues beyond vision-language.

Conclusion

HCSG represents a comprehensive, formally structured approach to human-centric VLN, combining explicit geometric motion forecasting and semantic activity interpretation within a unified planning architecture. The framework markedly improves navigation performance and social safety by addressing both physical and contextual requirements for interaction-grounded behavior. These contributions set a new standard for socially intelligent navigation and offer promising avenues for advancing embodied AI in human-populated domains.