MLANet: Hierarchical Multi-Level Attention for VLN

• The paper introduces MLANet, which leverages hierarchical attention by integrating low-level word semantics and high-level sub-instruction cues for effective 3D navigation.
• MLANet employs a fast sub-instruction algorithm that segments instructions 70x faster than traditional methods, ensuring real-time applicability.
• The multi-level attention module fuses visual inputs with dual GRU memories and uses a peak attention loss to refine action prediction in continuous VLN.

Hierarchical Multi-Level Attention Network (MLANet) most commonly denotes the continuous vision-and-language navigation model introduced for VLN-CE, where an agent must navigate in free 3D space from egocentric RGB-D observations under natural-language instructions. In that formulation, MLANet combines fast sub-instruction generation, dual-level language grounding, and an auxiliary attention-shaping objective so that navigation policy can exploit both global route semantics and local step-wise grounding. Its central premise is that continuous VLN requires more than flat word attention: the agent must understand both low-level word semantics and higher-level sub-instruction structure, while selecting the currently relevant sub-instruction during long action sequences (He et al., 2023).

1. Continuous VLN setting and architectural motivation

MLANet is designed for the continuous setting of vision-and-language navigation rather than the discrete graph-based setting. In VLN-CE, the agent receives a natural-language instruction and egocentric RGB-D observations at each step, and predicts primitive actions such as move forward, turn left, turn right, or stop. The paper characterizes this setting as harder than discrete VLN because the agent acts in free 3D space, must execute in real time, must interpret long multi-clause instructions, and must predict long action sequences without relying on graph nodes or teleportation-style movement (He et al., 2023).

The model’s motivation is explicitly hierarchical. Word-level attention alone is presented as insufficient because isolated words carry low-level semantics, whereas navigation instructions are naturally organized into higher-level chunks such as sub-instructions. The paper argues that a model attending only to words can over-distribute weight onto irrelevant or weakly informative tokens, described as “noise words,” and can fail to group words into coherent sub-goals. MLANet therefore separates instruction understanding into low-level word semantics and high-level sub-instruction semantics, using both to guide action prediction (He et al., 2023).

At the pipeline level, MLANet is an encoder-decoder model with six stages: raw instruction $I=\{i_1,\dots,i_L\}$, sub-instructions $S=\{s_1,\dots,s_N\}$ produced by FSA, RGB and depth inputs, modality-specific encoders, dual GRU memory states, a Multi-Level Attention (MLA) module, and a GRU-based action decoder with Peak Attention Loss (PAL) as auxiliary supervision. The encoders are specified as CLIP ViT-B/32 for RGB, ResNet for depth, BiLSTM for the raw instruction, and the CLIP text Transformer for sub-instructions (He et al., 2023).

2. Fast Sub-instruction Algorithm and the FSASub dataset

The first stage of MLANet is the Fast Sub-instruction Algorithm (FSA), which converts a raw instruction into sub-instructions without manual annotation. FSA is explicitly described as annotation-free and designed to satisfy the real-time requirement of continuous VLN. Its procedure first removes invalid characters and markup, then performs coarse segmentation with a sentence tokenizer, and finally refines the segments using part-of-speech tagging and heuristic boundary rules denoted NeedRefine. The algorithm outputs a set $O=\{(S_i,tokens_i)\}_{i=1}^N$, where each $S_i$ is a sub-instruction and $tokens_i$ is its token sequence (He et al., 2023).

The resulting dataset is called FSASub. Its role is practical as well as representational: it supplies the high-level semantic units used by MLANet during navigation, and it is also intended for online use rather than merely offline preprocessing. The paper compares FSASub with an older chunking method, FGSub, used in FGR2R-based processing. According to the reported numbers, generating FGSub takes about 8.8 hours, whereas FSASub takes only 7.3 minutes, which the paper describes as roughly 70x faster and in one place as over 72x faster. FSASub is also reported to segment more instructions successfully and to yield more sub-instructions per instruction on average (He et al., 2023).

A common misunderstanding is to treat FSA as a learned parser. The paper does not make that claim. FSA is instead a heuristic refinement procedure built for efficiency and annotation-free operation. The stated limitation is correspondingly specific: sub-instruction segmentation relies on heuristic refinement rules and POS tagging rather than learned parsing (He et al., 2023).

3. Multi-Level Attention module

The Multi-Level Attention module is the core fusion mechanism of MLANet. It integrates visual state with two semantic levels of language: low-level instruction features $\tilde I$ from the BiLSTM and high-level sub-instruction features $\tilde S$ from the CLIP text Transformer. The module is built around two GRU-based navigation memories,

$$h^{(h)}_t = \text{GRU}_h(\text{Concat}(v_t,\tilde{a}_{t-1});h^{(h)}_{t-1}),$$

$$h^{(l)}_t = \text{GRU}_l(\text{Concat}(v_t,\tilde{a}_{t-1});h^{(l)}_{t-1}),$$

where $v_t$ is the fused visual feature at time $S=\{s_1,\dots,s_N\}$0 and $S=\{s_1,\dots,s_N\}$1 is the embedding of the previous action. The paper interprets $S=\{s_1,\dots,s_N\}$2 as tending to store short-range, word-aligned object and direction cues, while $S=\{s_1,\dots,s_N\}$3 captures longer-horizon trajectory context aligned with sub-instructions (He et al., 2023).

Attention is then applied separately at the two semantic levels: $S=\{s_1,\dots,s_N\}$4

$S=\{s_1,\dots,s_N\}$5

$S=\{s_1,\dots,s_N\}$6

This yields a fused instruction feature $S=\{s_1,\dots,s_N\}$7. The high-level branch is meant to identify which sub-instruction is currently relevant, while the low-level branch refines immediate grounding within that sub-instruction. The paper states that this arrangement reduces the word-grouping problem and mitigates the adverse effects of noise words, because the model reasons over meaningful instruction chunks instead of only over isolated tokens (He et al., 2023).

The action decoder consumes both language levels, visual information, and action history: $S=\{s_1,\dots,s_N\}$8

$S=\{s_1,\dots,s_N\}$9

The corresponding action loss is

$O=\{(S_i,tokens_i)\}_{i=1}^N$0

The paper also notes a design preference for multi-head attention over simple soft attention, on the grounds that it better aligns multimodal and multi-level information (He et al., 2023).

4. Peak Attention Loss and optimization

PAL is introduced to regularize the high-level attention over sub-instructions. The stated problem is that, even when MLA provides global instruction understanding, the high-level attention distribution can remain too diffuse, assigning moderate mass to several sub-instructions at once. For long trajectories, this is undesirable because the agent must estimate the current sub-goal rather than maintain an indiscriminate mixture across the route (He et al., 2023).

Let $O=\{(S_i,tokens_i)\}_{i=1}^N$1 denote the high-level attention over $O=\{(S_i,tokens_i)\}_{i=1}^N$2 sub-instructions. MLANet first finds the index of maximal attention,

$O=\{(S_i,tokens_i)\}_{i=1}^N$3

then builds a Gaussian-like target centered at $O=\{(S_i,tokens_i)\}_{i=1}^N$4,

$O=\{(S_i,tokens_i)\}_{i=1}^N$5

The PAL objective encourages $O=\{(S_i,tokens_i)\}_{i=1}^N$6 to approximate a single dominant peak with limited leakage to neighboring sub-instructions: $O=\{(S_i,tokens_i)\}_{i=1}^N$7 The total training loss is

$O=\{(S_i,tokens_i)\}_{i=1}^N$8

with $O=\{(S_i,tokens_i)\}_{i=1}^N$9 and $S_i$0 chosen empirically, and $S_i$1 increased linearly during training (He et al., 2023).

The paper emphasizes that PAL is not a hard one-hot constraint. Its Gaussian-shaped target preserves a principal focus while allowing neighboring sub-instructions to retain some mass. That soft concentration is intended to improve local perception without removing contextual continuity. Alternative target shapes—constant, linear, quadratic, cubic, and Gaussian—are studied, with Gaussian reported as best overall, and $S_i$2 selected as a good balance. The same study reports that a 512-dimensional MLA with 8 heads works well (He et al., 2023).

Training is conducted on VLN-CE using train, val-seen, val-unseen, and test. The reported implementation details are: CLIP ViT-B/32 for the RGB encoder and sub-instruction encoder, 256-dimensional projections for RGB, depth, word, and sub-instruction features, 512-dimensional GRU hidden and fused representations, dropout 0.25, Adam, batch size 5, learning rate $S_i$3, inflection weighting coefficient 3.2, and 45 epochs with evaluation every 5 epochs. Auxiliary training strategies include imitation learning, DAgger, EnvDrop-style augmentation, and PPO fine-tuning (He et al., 2023).

5. Empirical behavior, benchmarks, and ablations

MLANet is evaluated on VLN-CE, the continuous reconstruction of R2R in Habitat. The paper reports the standard metrics TL (trajectory length), NE (navigation error), nDTW, OSR (oracle success rate), SR (success rate), and SPL (success weighted by inverse path length). In imitation learning, compared with Seq2Seq, MLANet improves SPL on val-seen from 0.21 to 0.28 and on val-unseen from 0.15 to 0.21. On the official test leaderboard, MLANet reports TL 7.42, NE 6.78, OSR 0.39, SR 0.34, and SPL 0.32. The paper states that these results are better than the listed baselines including CMA, LAW, WPN+DN, and HPN+DN on most metrics, while noting that HPN+DN has slightly better NE (He et al., 2023).

The ablation studies isolate the contributions of FSA, MLA, and PAL. Removing FSA eliminates sub-instruction information and degrades performance. Replacing MLA with simple averaging or concatenation of instruction features reduces SR and SPL, which the paper interprets as evidence that hierarchical multi-level attention is necessary for robust grounding. Setting the PAL coefficient to zero also hurts performance, especially SPL, indicating that the shape of high-level attention matters for navigation quality (He et al., 2023).

The comparison between FSASub and FGSub serves as both an efficiency and quality ablation. FSASub is reported to have a higher segment ratio, a higher average number of sub-instructions, slightly better navigation results, and far lower preprocessing time. This supports the model’s stated emphasis on real-time applicability in continuous VLN (He et al., 2023).

The paper also records several limitations and assumptions. The backbone remains RNN-based, which the authors suggest may restrict richer use of sub-instructions and multimodal context relative to Transformer-based alternatives. The method can fail on rare observations or unusual language. The environment assumes a forward-facing RGB-D camera and a fixed action set, and success is judged by ending within 3 meters of the goal. Unlike graph-based approaches, MLANet does not exploit discrete navigation-graph priors; the paper notes that this may help explain why some graph-based methods obtain slightly better NE in certain cases (He et al., 2023).

6. Broader meaning of hierarchical multi-level attention

The broader literature suggests that “hierarchical multi-level attention network” denotes an architectural family rather than a single canonical block. In CTR prediction, DHAN replaces DIN’s flattened item attention with a hierarchy from item $S_i$4 attribute $S_i$5 overall interest and reports substantial AUC gains over DIN, while also arguing that hierarchical interest modeling and temporal interest evolution are complementary (Xu et al., 2020). In mixed cooperative and competitive MARL, HAMA uses inter-agent attention within groups and inter-group attention across groups inside an actor-critic framework, with transferability tied to relational aggregation rather than fixed-size concatenation (Ryu et al., 2019). In multi-relational graphs, BR-GCN uses node-level attention within each relation-specific neighborhood and relation-level attention to integrate relation-specific embeddings, explicitly characterizing bi-level attention as more powerful than uni-level attention on highly multi-relational data (Iyer et al., 2024).

Comparable motifs appear in other domains. PMANet extracts a CharBERT-based feature pyramid across Transformer layers, applies dynamic layer-wise attention, and then uses spatial pyramid pooling for malicious URL detection (Liu et al., 2023). MHAL ties token-level and sentence-level classification through a shared multi-head attention mechanism, enabling sentence supervision to flow to word-level predictions and supporting zero-shot sequence labeling (Pislar et al., 2020). Document-level NMT has used hierarchical attention over words within previous sentences and then sentences within the document, integrated into the encoder and decoder with a context gate (Miculicich et al., 2018). Earlier scene-labeling work combined hierarchical CNN features, contextual recurrent modeling, and attention-based fusion across levels (Fan et al., 2016). A later face-reconstruction model explicitly titled “Hierarchical MLANet” uses attention inside ResNet bottlenecks and progressive attention fusion across stages for 3D face reconstruction from single images (Cao, 12 Sep 2025).

Within that wider landscape, the continuous-VLN MLANet is a specific instantiation whose hierarchy is semantic rather than purely structural or topological. Its levels are word semantics and sub-instruction semantics; its auxiliary objective is not generic attention regularization but a peak-shaped constraint tied to sequential sub-goal execution; and its operational target is action selection in free 3D navigation rather than classification, recommendation, or representation learning. A plausible implication is that its distinctive contribution lies less in introducing hierarchy per se than in aligning multi-level attention with the temporal requirements of continuous navigation (He et al., 2023).