STE-VLN: Navigating with Structured States

• STE-VLN is a non-standard umbrella that unifies diverse VLN approaches using enriched state representations and large-scale synthetic supervision.
• It integrates formulations across Street View, indoor, and aerial environments by employing language-centric embodiment, self-training, and structured spatial reasoning.
• Empirical studies show significant improvements in trajectory reasoning, landmark extraction, and control accuracy by enhancing traditional navigation methods with explicit state structures.

STE-VLN is best understood as a non-standard umbrella label within vision-and-language navigation research rather than as a single canonical architecture. In the cited literature, the label is used explicitly for Street-View-based Vision-and-Language Navigation, and it is also invoked more loosely for methods centered on self-training or self-exploration, semantic-topo-metric spatial representations, and structured state evolution. This suggests that STE-VLN denotes a family of VLN systems that strengthen standard instruction-conditioned navigation with richer state representations, large-scale synthetic supervision, or language-centric embodiment in environments ranging from Manhattan Street View to Matterport3D and aerial outdoor scenes (Schumann et al., 2023, Rawal et al., 2024, Gao et al., 2024, Qiao et al., 2023, Qi et al., 20 Jun 2025, Chen et al., 2022).

1. Scope and nomenclature

Within the supplied corpus, STE-VLN is not attached to one universally accepted expansion. One line of work uses it directly for Street-View-based VLN in Google Street View graphs; other lines treat it as a useful descriptor for self-training or self-exploration style VLN, for semantic-topo-metric reasoning, or for structured state-evolution methods. A plausible implication is that the term functions more as a research nexus than as a fixed benchmark name.

Paper	Relation to STE-VLN	Core setting
VELMA (Schumann et al., 2023)	Explicitly framed as Street-View-based VLN	Manhattan Street View graph
AIGeN (Rawal et al., 2024)	Not mentioned, but tightly related through synthetic instruction generation	Indoor VLN on R2R, REVERIE, HM3D
STMR + LLM (Gao et al., 2024)	Framed as a semantic-topo-metric instance of STE-VLN	Aerial VLN for UAVs
VLN-PETL (Qiao et al., 2023)	Not explicit, but close to history-aware STE-style adaptation	Indoor VLN with HAMT
SEvol (Chen et al., 2022)	Closely aligned through structured state evolution	Indoor VLN on R2R and R4R
VLN-R1 (Qi et al., 20 Jun 2025)	Conceptually parallel through end-to-end LVLM policies and reward-driven post-training	Continuous VLN-CE in Habitat

The acronym also admits a common source of confusion. In "Training Quantised Neural Networks with STE Variants: the Additive Noise Annealing Algorithm," STE denotes the Straight-Through Estimator for quantised neural network training rather than any navigation formulation. That usage is technically unrelated to VLN, even though the acronym is identical (Spallanzani et al., 2022).

2. Task formulations associated with STE-VLN

The most explicit STE-VLN formulation in the supplied material is the Street View setting used by VELMA. The environment is a directed graph $G=\langle V,E\rangle$ over lower Manhattan with 29,641 panoramas, where each edge $\langle v,v'\rangle\in E$ has a heading label $\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$. The navigation state is $s_t=(v_t,\alpha_t)$, the action space is $$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$, and success is defined when the stop node is within one graph hop of the goal. VELMA also modifies the Touchdown transition function so that $\texttt{forward}$ preserves heading semantics and $\texttt{turn\_around}$ yields $s_{t+1}=(v_t,\alpha_t-180^\circ)$, addressing semantically misaligned actions at irregular intersections (Schumann et al., 2023).

A second formulation appears in aerial VLN. Here the initial UAV pose is $P=[x,y,z,\phi,\theta,\psi]$, observations are egocentric RGB and depth images, and the agent chooses from $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$ with action parameters given by a yaw change in $\langle v,v'\rangle\in E$0 degrees and/or a distance of $\langle v,v'\rangle\in E$1–$\langle v,v'\rangle\in E$2 meters, plus a Stop action. An episode is successful if the UAV stops within $\langle v,v'\rangle\in E$3 m of the target. This formulation departs from graph-based panorama navigation by requiring online semantic mapping and explicit metric reasoning in continuous outdoor space (Gao et al., 2024).

Indoor VLN variants associated with STE-VLN retain the Matterport-style graph abstraction or Habitat continuous embodiment. AIGeN works with the standard R2R and REVERIE setting, where trajectories are sequences of panoramic images and the deployed agent must stop within a 3 m radius of the goal. SEvol operates on Matterport3D panoramas split into 36 direction bins $\langle v,v'\rangle\in E$4, with candidate navigable directions plus STOP. VLN-R1 moves beyond the nav-graph in Habitat/Matterport3D, using egocentric RGB video only and predicting low-level actions from $\langle v,v'\rangle\in E$5 while still evaluating under VLN-CE metrics such as SR, OS, SPL, NE, and TL (Rawal et al., 2024, Chen et al., 2022, Qi et al., 20 Jun 2025).

3. Representational paradigms

A defining feature of STE-VLN-style work is the replacement of a flat instruction-conditioned state with a more explicit linguistic, topological, or object-centric representation. VELMA adopts a fully language-mediated embodiment strategy: the LLM never receives pixels directly. Instead, a prompt $\langle v,v'\rangle\in E$6 contains the task description, instruction, past textual observations, step indices, and prior actions. Landmark phrases are first extracted from the instruction with GPT-3, then CLIP scores their visibility in five panoramic directions, standardized by a z-score over training panoramas. Landmarks whose standardized score exceeds $\langle v,v'\rangle\in E$7 are verbalized as strings such as “There is [$\langle v,v'\rangle\in E$8] on your [$\langle v,v'\rangle\in E$9]”, while graph degree yields “There is a [$\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$0]-way intersection” when $\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$1. The result is a language-only policy over a street graph, with visual grounding delegated to a landmark-verbalization pipeline (Schumann et al., 2023).

The aerial semantic-topo-metric approach encodes space differently. Instruction-related landmarks are extracted from language, localized by Grounding DINO and Tokenize Anything, filtered by TF-IDF cosine similarity with threshold $\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$2, projected into a 3D semantic point cloud through depth, and fused into a top-down map. That map is converted into a $\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$3 integer matrix $\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$4, with one cell per $\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$5 m square over a $\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$6 local region. The matrix stores semantic category indices, the UAV’s current orientation token at $\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$7, and special markers for past trajectory. The prompt explicitly tells the LLM that “The distance between adjacent numbers is 5 meters in the real world.” This converts semantics, topology, and metric scale into a discrete textual artifact designed for LLM reasoning (Gao et al., 2024).

SEvol formalizes structured state evolution at the object level. At each step it detects objects, builds a fully connected layout graph $\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$8, and assigns node features

$\alpha_{\langle v,v'\rangle}\in[0^\circ,360^\circ)$9

with GloVe class embeddings and orientation encodings. Edge weights encode relative heading: $s_t=(v_t,\alpha_t)$0 A Reinforced Layout clues Miner selects a subgraph $s_t=(v_t,\alpha_t)$1, and a Structured Evolving Module maintains a layout memory matrix $s_t=(v_t,\alpha_t)$2 through a matrix-GRU. The structured state is then

$s_t=(v_t,\alpha_t)$3

after which attention pooling and fusion with the decoder LSTM produce a structure-aware navigation state $s_t=(v_t,\alpha_t)$4. This directly addresses the loss of object-level environment layout in one-dimensional hidden vectors (Chen et al., 2022).

4. Supervision, adaptation, and optimization

One major STE-VLN theme is the expansion of supervision beyond human instructions. AIGeN addresses the language bottleneck by generating synthetic instructions from trajectories. Its generator is a GPT-2 decoder conditioned on ResNet-152 trajectory features and object detections, while its discriminator is a BERT encoder scoring whether an instruction is real for the same trajectory. The adversarial losses are

$s_t=(v_t,\alpha_t)$5

AIGeN then generates synthetic instructions for 217K randomly sampled HM3D trajectories and uses those pairs to pre-train DUET before fine-tuning on REVERIE and R2R. Although STE-VLN is not named in that paper, the mechanism is directly aligned with self-training or self-exploration style VLN because it enlarges the instruction side of the supervision signal over unlabeled trajectories (Rawal et al., 2024).

A second strand concerns efficient task adaptation of history-aware VLN transformers. VLN-PETL freezes HAMT and introduces three task-specific modules—Language Encoder Adapter, Historical Interaction Booster, and Cross-modal Interaction Booster—plus LoRA on attention query and value projections. The trainable fraction is only $s_t=(v_t,\alpha_t)$6 on R2R, $s_t=(v_t,\alpha_t)$7 on REVERIE, $s_t=(v_t,\alpha_t)$8 on NDH, and $s_t=(v_t,\alpha_t)$9 on RxR. The HIB and CIB modules are explicitly designed for the two factors repeatedly emphasized in STE-style navigation systems: trajectory history and cross-modal grounding (Qiao et al., 2023).

A third strand is reward-driven post-training for end-to-end embodied policies. VLN-R1 formulates LVLM navigation as prediction of a short action sequence from instruction, current observation, and sampled history memory. Supervised fine-tuning aligns Qwen2-VL outputs with expert six-step action strings, and reinforcement fine-tuning uses Group Relative Policy Optimization with a Time-Decayed Reward: $$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$0 where the text states that an exponentially decaying $$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$1 term prioritizes earlier correct actions. This turns multi-step action matching into a verifiable reward signal for continuous-environment VLN-CE (Qi et al., 20 Jun 2025).

5. Empirical record

The empirical literature grouped here under STE-VLN reports gains in several distinct regimes. In Street View, VELMA shows that a text-only LLM policy with verbalized observations can outperform prior vision-heavy baselines on geographically disjoint Manhattan splits. The strongest reported model, VELMA-RBL, achieves on test Touchdown SPD $$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$2, KPA $$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$3, and TC $$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$4, and on test Map2seq SPD $$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$5, KPA $$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$6, and TC $$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$7. The paper describes this as roughly $$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$8–$$\{\texttt{forward},\texttt{left},\texttt{right},\texttt{stop},\texttt{turn\_around}\}$$9+ relative TC improvement depending on the baseline, and it also shows that CLIP-based landmark verbalization materially improves performance over no-image ablations (Schumann et al., 2023).

In aerial VLN, the semantic-topo-metric framework reports on AerialVLN-S validation unseen NE $\texttt{forward}$0 m, SR $\texttt{forward}$1, and OSR $\texttt{forward}$2, compared with the best listed baseline OSR values of $\texttt{forward}$3 for CMA and $\texttt{forward}$4 for LAG on that split. On 100 unseen scenes, the ablation between Topo, Metric, and STMR yields NE/SR/OSR of $\texttt{forward}$5, $\texttt{forward}$6, and $\texttt{forward}$7, respectively. Real-world UAV tests on 10 outdoor scenes report SR $\texttt{forward}$8 and OSR $\texttt{forward}$9, versus SR/OSR $\texttt{turn\_around}$0 for MapGPT and $\texttt{turn\_around}$1 for NavGPT (Gao et al., 2024).

In indoor VLN, AIGeN shows that instruction generation can improve a downstream navigator even without altering the navigation architecture. On REVERIE Val Unseen, DUET + AIGeN reports TL $\texttt{turn\_around}$2, SPL $\texttt{turn\_around}$3, SR $\texttt{turn\_around}$4, OSR $\texttt{turn\_around}$5, RGS $\texttt{turn\_around}$6, and RGSPL $\texttt{turn\_around}$7, compared with DUET’s SPL $\texttt{turn\_around}$8 and RGSPL $\texttt{turn\_around}$9. On the instruction-generation side, AIGeN with detections reaches BLEU-1 $s_{t+1}=(v_t,\alpha_t-180^\circ)$0, METEOR $s_{t+1}=(v_t,\alpha_t-180^\circ)$1, ROUGE $s_{t+1}=(v_t,\alpha_t-180^\circ)$2, CIDEr $s_{t+1}=(v_t,\alpha_t-180^\circ)$3, and SPICE $s_{t+1}=(v_t,\alpha_t-180^\circ)$4, while adversarial fine-tuning increases novelty from $s_{t+1}=(v_t,\alpha_t-180^\circ)$5 to $s_{t+1}=(v_t,\alpha_t-180^\circ)$6 and raises diversity metrics from Div-1 $s_{t+1}=(v_t,\alpha_t-180^\circ)$7, Div-2 $s_{t+1}=(v_t,\alpha_t-180^\circ)$8 to Div-1 $s_{t+1}=(v_t,\alpha_t-180^\circ)$9, Div-2 $P=[x,y,z,\phi,\theta,\psi]$0 (Rawal et al., 2024).

Parameter-efficient and end-to-end continuous variants also report strong results. VLN-PETL reaches on R2R Test Unseen SR $P=[x,y,z,\phi,\theta,\psi]$1 and SPL $P=[x,y,z,\phi,\theta,\psi]$2, effectively matching full fine-tuning at SR $P=[x,y,z,\phi,\theta,\psi]$3 and SPL $P=[x,y,z,\phi,\theta,\psi]$4, and on RxR Validation Unseen it reports nDTW $P=[x,y,z,\phi,\theta,\psi]$5 and sDTW $P=[x,y,z,\phi,\theta,\psi]$6, slightly surpassing full fine-tuning at nDTW $P=[x,y,z,\phi,\theta,\psi]$7 and sDTW $P=[x,y,z,\phi,\theta,\psi]$8 (Qiao et al., 2023). VLN-R1 on R2R-CE Val-Unseen reports SR $P=[x,y,z,\phi,\theta,\psi]$9, OS $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$0, SPL $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$1, NE $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$2, and TL $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$3 for the Qwen2-VL-7B model after reinforcement fine-tuning, improving over its own SFT-only configuration at SR $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$4, OS $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$5, SPL $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$6, NE $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$7, and TL $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$8 (Qi et al., 20 Jun 2025).

SEvol contributes a separate empirical argument: richer state can matter as much as additional data. On R2R test-unseen with augmentation, EnvDrop improves from SR $$\{\text{right},\text{left},\text{up},\text{down},\text{forward},\text{backward}\}$$9, SPL $\langle v,v'\rangle\in E$00 to SR $\langle v,v'\rangle\in E$01, SPL $\langle v,v'\rangle\in E$02 after adding SEvol, while NvEM improves from SR $\langle v,v'\rangle\in E$03, SPL $\langle v,v'\rangle\in E$04, NE $\langle v,v'\rangle\in E$05 to SR $\langle v,v'\rangle\in E$06, SPL $\langle v,v'\rangle\in E$07, NE $\langle v,v'\rangle\in E$08. Ablations show that SEM alone already helps, and adding the RL-based RLM further improves val-unseen SR/SPL from $\langle v,v'\rangle\in E$09 to $\langle v,v'\rangle\in E$10 without augmentation (Chen et al., 2022).

6. Limitations, misconceptions, and open directions

The most immediate limitation is terminological. Because STE-VLN is not a universally standardized acronym in the supplied corpus, readers must infer its intended scope from context. In some papers it refers to Street View embodiment; in others it is only an interpretive bridge to self-training, semantic-topo-metric, or structured-state methods. A common misconception is to equate this STE with the Straight-Through Estimator literature; however, the quantised-network ANA paper uses STE in an entirely different sense and does not concern navigation (Spallanzani et al., 2022).

A second limitation is dependence on an upstream perception stack. VELMA depends on GPT-3 landmark extraction and CLIP visibility scoring, AIGeN depends on Mask2Former detections on the last panoramic view, STMR depends on Grounding DINO, Tokenize Anything, depth, and pose, and SEvol depends on object detection with Faster R-CNN. The supplied papers repeatedly report that detector quality affects navigation quality, and several methods note that missed or misclassified landmarks can derail reasoning (Schumann et al., 2023, Rawal et al., 2024, Gao et al., 2024, Chen et al., 2022).

A third issue is the tension between discrete control, long-horizon context, and real-world deployment. VELMA notes that Street-View trajectories average about $\langle v,v'\rangle\in E$11 steps and stress LLM context limits. VLN-R1 explicitly acknowledges that its action space is discrete and therefore limits fine-grained control, even though it operates in a continuous Habitat environment. The aerial LLM framework similarly relies on coarse grid cells and templated prompt structure, which may not capture all continuous geometry. This suggests that future STE-VLN systems will likely need tighter integration of structured spatial memory with continuous control heads and more scalable memory management (Schumann et al., 2023, Gao et al., 2024, Qi et al., 20 Jun 2025).

Finally, the current directions are complementary rather than settled competitors. AIGeN implies a language-side route to scaling; VLN-PETL shows that history-aware backbones can be adapted with $\langle v,v'\rangle\in E$12–$\langle v,v'\rangle\in E$13 trainable parameters; SEvol argues for explicit object-level structured memory; VELMA and the aerial STMR framework reframe navigation as language-centric reasoning over verbalized or serialized spatial structure; and VLN-R1 shows that LVLM policies can be refined by verifiable rewards in continuous environments (Rawal et al., 2024, Qiao et al., 2023, Chen et al., 2022, Schumann et al., 2023, Gao et al., 2024, Qi et al., 20 Jun 2025). The clearest synthesis is that STE-VLN names a converging research program: VLN systems that strengthen instruction following with explicit state structure, expanded supervision, or language-mediated embodiment in order to improve grounding, long-horizon reasoning, and transfer.