Instruction-as-State: Environment-Guided and State-Conditioned Semantic Understanding for Embodied Navigation

Abstract: Vision-and-Language Navigation requires agents to follow natural-language instructions in visually changing environments. A central challenge is the dynamic entanglement between language and observations: the meaning of instruction shifts as the agent's field of view and spatial context evolve. However, many existing models encode the instruction as a static global representation, limiting their ability to adapt instruction meaning to the current visual context. We therefore model instruction understanding as an Instruction-as-State variable: a decision-relevant, token-level instruction state that evolves step by step conditioned on the agent's perceptual state, where the perceptual state denotes the observation-grounded navigation context at each step. To realize this principle, we introduce State-Entangled Environment-Guided Instruction Understanding (S-EGIU), a coarse-to-fine framework for state-conditioned segment activation and token-level semantic refinement. At the coarse level, S-EGIU activates the instruction segment whose semantics align with the current observation. At the fine level, it refines the activated segment through observation-guided token grounding and contextual modeling, sharpening its internal semantics under the current observation. Together, these stages maintain an instruction state that is continuously updated according to the agent's perceptual state during navigation. S-EGIU delivers strong performance on several key metrics, including a +2.68% SPL gain on REVERIE Test Unseen, and demonstrates consistent efficiency gains across multiple VLN benchmarks, underscoring the value of dynamic instruction--perception entanglement.

• The paper introduces the Instruction-as-State paradigm by dynamically updating semantic representations based on evolving visual cues to improve navigation policies.
• The paper demonstrates the S-EGIU framework’s hierarchical coarse-to-fine processing that refines instructions at both sub-instruction and token levels for better perceptual grounding.
• The paper shows significant performance gains on benchmarks like REVERIE, R2R, and SOON, indicating enhanced path efficiency and generalization in diverse environments.

Instruction-as-State: State-Conditioned Semantic Understanding for Vision-and-Language Navigation

Motivation and Context

Vision-and-Language Navigation (VLN) tasks require embodied agents to interpret natural-language instructions within dynamically evolving visual environments. Current approaches rely predominantly on static, globally encoded instruction representations, yielding suboptimal adaptability as the agent's perception changes throughout navigation. Static semantics hinder an agent’s ability to modulate instruction relevance and semantic interpretation in response to evolving visual cues, spatial context, and newly observable landmarks, resulting in both inefficient trajectories and limited generalization in unseen environments.

The "Instruction-as-State" paradigm, instantiated via State-Entangled Environment-Guided Instruction Understanding (S-EGIU), addresses this by treating instruction semantics not as static embeddings but as a decision-relevant latent state that evolves step-by-step, conditioned on the agent’s perceptual state. This enables dynamic entanglement between language and perception, sharply aligning instruction understanding with what the agent currently observes, and updating which instruction components are active based on real-time visual evidence.

S-EGIU Framework

S-EGIU operationalizes the Instruction-as-State principle using a hierarchical, coarse-to-fine approach composed of two main modules:

• Coarse-Grained Instruction Processing (CGIP): Segments the instruction into sub-instructions and activates the segment whose semantics are most perceptually grounded in the current observation. CGIP leverages both rule-based segmentation and a learnable boundary refinement module to produce clause-level structural priors, followed by a clause-relevance distribution computed via cross-attention between instruction and panoramic visual features.
• Fine-Grained Instruction Processing (FGIP): Refines semantics at the token level within the activated sub-instruction. FGIP employs cross-modal grounding and a local Transformer encoder to adaptively update token importance, preserving internal compositional semantics and grounding them strictly in the agent’s current perceptual evidence. Fusion into the full instruction state is accomplished with a gated residual mechanism that adaptively controls per-token updates based on perceptual support.

This two-stage procedure ensures the instruction state evolves robustly and stably at each navigation step, providing a rich, modulated representation for downstream navigation policy.

Figure 1: Overview of the S-EGIU framework, visualizing the CGIP and FGIP hierarchy and the flow of perception-conditioned instruction state updates.

Experimental Evaluation

S-EGIU is evaluated across three representative VLN benchmarks: R2R, SOON, and REVERIE. Each benchmark stresses different dimensions—concise instructions, long compositional intent sequences, and object-centric grounding, respectively. Extensive comparisons are performed against both standard baselines and mechanism-level controlled plug-ins within the DUET backbone architecture.

Numerical Performance

• REVERIE: S-EGIU achieves a notable +2.68% SPL gain on Test Unseen splits compared to DUET and outperforms several controlled plug-ins representing memory, planning, retrieval, and grounding mechanism families. SPL improves from 36.06 (DUET) to 38.74 (S-EGIU) on Test Unseen, with trajectory length also reduced.
• R2R: On Val Unseen, S-EGIU increases SR from 72% to 74% and SPL from 60% to 61%, with improved trajectory length.
• SOON: S-EGIU achieves higher SPL and reduced trajectory length, especially on long-horizon instructions, demonstrating improved path efficiency.

A backbone generalization study confirms that S-EGIU yields consistent improvements across diverse architectures (DUET, NavGPT2, HAMT, VLN-GOAT), with average gains of +1.5 SR / +1.5 SPL on Val Seen, and +1.5 SR / +1.0 SPL on Val Unseen.

Ablation and Visualization

Ablation studies confirm the complementarity of CGIP and FGIP. CGIP primarily stabilizes navigation decisions through perceptual clause selection, while FGIP independently enhances local grounding quality. Combining both modules yields optimal performance, indicating structured segment selection and fine-grained semantic refinement are synergistic.

Figure 2: Dynamic instruction semantics in VLN; the agent adaptively selects sub-instructions and refines token meanings in response to evolving perception.

Figure 3: Visualization showing panoramic perceptual updates and trajectory, with color-coded instruction tokens highlighting S-EGIU’s state-conditioned semantic refinement.

Failure Modes and Error Analysis

S-EGIU’s generalization gap in unseen environments is attributed to two factors:

1. Clause Selection Robustness: Novel object layouts and ambiguous visual cues can disrupt CGIP, causing premature, late, or incorrect clause transitions, which propagate to subsequent semantic refinement.
2. Perceptual Token Alignment: Appearance or geometry shifts degrade cross-modal grounding quality, resulting in suboptimal token-level semantic emphasis and potential semantic drift.

Recurring error modes include ambiguous landmark discrimination, misaligned sub-instruction transitions, and cumulative semantic drift over long trajectories. These highlight the need for uncertainty-aware grounding and robustness to domain shifts.

Figure 4: Failure case illustrating incorrect final sub-instruction grounding, with color/intensity-coded tokens reflecting misaligned semantic states.

Theoretical and Practical Implications

Modeling instruction understanding as a state variable entangled with perception fundamentally transforms embodied navigation tasks. The stepwise, perception-guided update mechanism reduces semantic ambiguity, improves decision efficiency, and enhances adaptability to unseen scenes. Practically, this enables more reliable navigation in complex, structured, and object-centric environments, with robust gains in path efficiency and generalization. Theoretically, it advances the principle of dynamic language-perception entanglement, opening pathways for richer temporal modeling and uncertainty-aware instruction-state evolution.

Future developments will likely focus on extending S-EGIU with temporal smoothing, domain-invariant priors, and explicit uncertainty modeling, further enhancing robustness and transferability in real-world scenarios.

Conclusion

The Instruction-as-State paradigm, instantiated through S-EGIU, provides a principled and effective mechanism for aligning instruction semantics with the agent’s dynamic perceptual state. Empirical evidence demonstrates consistent efficiency gains across datasets and architectures, substantiating the value of trajectory-conditioned instruction-state updates. Despite residual limitations in boundary refinement and sensitivity to perceptual ambiguity, S-EGIU establishes a strong foundation for adaptive, robust instruction understanding in VLN, with substantial implications for future embodied AI research.