JanusVLN: Dual-Memory RGB Navigation

• The paper presents JanusVLN’s dual implicit neural memory system that decouples semantic and spatial cues for efficient RGB-only navigation.
• It employs separate semantic and spatial encoders with incremental key-value caching to reduce redundancy and maintain fixed-size memory.
• Empirical evaluations on VLN benchmarks demonstrate improved success rates and path efficiency, with ablations highlighting the critical role of pretrained 3D priors.

Searching arXiv for the specified paper to ground the article and cite the correct arXiv ID. JanusVLN is a vision-and-language navigation framework that decouples visual-semantic and spatial-geometric representations through a dual implicit neural memory for streaming, RGB-only navigation. In the formulation studied in "JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation" (Zeng et al., 26 Sep 2025), an embodied agent receives an egocentric RGB stream $O_T = \{x_0, \ldots, x_t\}$ with $x_t \in \mathbb{R}^{3 \times H \times W}$ and a natural language instruction $I$ of $l$ tokens, and at each timestep selects a low-level action $$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$. The framework extends a Multimodal LLM with an RGB-to-3D spatial encoder, builds fixed-size implicit memories via key–value caching, and incrementally updates those memories using an initial anchor and a sliding window. Its central claim is that compact, implicit neural states can preserve both “what” and “where” more effectively than explicit textual maps or stored frame histories while reducing inference overhead (Zeng et al., 26 Sep 2025).

1. Problem formulation and motivation

Vision-and-Language Navigation requires an embodied agent to navigate through unseen environments, guided by natural language instructions and a continuous video stream. In continuous VLN, each action corresponds to a small physical change, such as a $30^\circ$ rotation or a $25$ cm forward step. The problem is therefore not only instruction following, but also long-horizon accumulation of scene understanding under partial observability.

JanusVLN is motivated by limitations in recent MLLM-based VLN systems. Prior methods typically either build textual cognitive maps, consisting of object nodes and relation edges, or store raw historical frames. The paper argues that text-only maps sacrifice precise spatial relationships, including depth, layout, and orientation, and become cluttered as trajectories get long. Frame histories, by contrast, require reprocessing the entire sequence at each step, which incurs substantial computational redundancy and memory growth. Both approaches are described as leading to spatial information loss, memory bloat, and inefficient inference (Zeng et al., 26 Sep 2025).

A second motivation concerns representation bias in standard VLM and MLLM vision encoders. Most inherit 2D image-text pretraining, emphasizing semantics—“what”—while under-modeling 3D geometry—“where, layout, relative pose.” JanusVLN is designed around the observation that RGB frames still contain perspective, occlusion, and geometric cues. The framework therefore injects 3D priors into an MLLM without requiring active depth sensors or precomputed point clouds. This suggests a reformulation of RGB-only VLN in which geometry is not an auxiliary modality, but an implicit property to be extracted and retained.

2. Dual-memory principle and conceptual contribution

The core contribution is a dual implicit neural memory inspired by hemispheric specialization of the human brain: a “left” semantic system and a “right” spatial system. JanusVLN instantiates this analogy as two compact, fixed-size memories. The visual-semantic memory $M_{\text{sem}}$ stores high-level semantic structure via KV caches from the MLLM’s semantic encoder, while the spatial-geometric memory $M_{\text{spa}}$ stores geometry-aware structure via KV caches from a 3D spatial encoder, specifically VGGT (Zeng et al., 26 Sep 2025).

This decoupling is operational rather than merely descriptive. The system uses a dual-encoder pipeline, one purely semantic and one purely spatial-geometric, and fuses their outputs into a spatially-aware feature $F_t$ that is then consumed by the action policy of the backbone LLM. Unlike explicit memory approaches, JanusVLN does not cache raw images or generated text. It caches deeply processed neural states, namely transformer keys and values, so that historical information can be reused directly by attention.

The paper positions this as a distinct memory paradigm. Earlier MLLM-based VLN methods store texts or frames and perform full-sequence re-encoding at each step. JanusVLN instead builds streaming KV caches per encoder, retains only initial “attention sinks” and a sliding window of recent frames, and directly attends to cached KVs for each new frame. The stated effect is elimination of redundant extraction under a bounded memory budget. A plausible implication is that JanusVLN reframes navigation memory from an external symbolic record into an internalized latent state that is both task-conditioned and architecture-native.

3. Architecture: semantic encoder, spatial encoder, and fusion

The visual-semantic encoder is the vision branch of Qwen2.5-VL 7B. For an input frame $x_t \in \mathbb{R}^{3 \times H \times W}$0, it produces patch tokens

$x_t \in \mathbb{R}^{3 \times H \times W}$1

JanusVLN then applies the Qwen2.5-VL spatial merging strategy, grouping adjacent $x_t \in \mathbb{R}^{3 \times H \times W}$2 patches into one token, yielding

$x_t \in \mathbb{R}^{3 \times H \times W}$3

The spatial-geometric encoder is VGGT, a feed-forward RGB-to-geometry model pre-trained on pixel–3D point cloud pairs. VGGT consists of an encoder for single-image features, a fusion decoder for cross-frame interaction, and a head that predicts a point map $x_t \in \mathbb{R}^{3 \times H \times W}$4 and a confidence map $x_t \in \mathbb{R}^{3 \times H \times W}$5. JanusVLN uses the encoder and fusion decoder outputs as geometric tokens rather than the predicted 3D outputs themselves:

$x_t \in \mathbb{R}^{3 \times H \times W}$6

After alignment with Qwen2.5-VL merging, adjacent $x_t \in \mathbb{R}^{3 \times H \times W}$7 blocks in $x_t \in \mathbb{R}^{3 \times H \times W}$8 are concatenated to form

$x_t \in \mathbb{R}^{3 \times H \times W}$9

Fusion is additive and spatially aware:

$I$0

with $I$1 controlling spatial contribution. The fused visual features $I$2 are combined with instruction tokens and passed into the Qwen2.5-VL LLM for action prediction in the discrete action space. Both vision encoders are frozen during training, while the Qwen2.5-VL 7B LLM is fine-tuned with a small learning rate, together with the fusion MLP (Zeng et al., 26 Sep 2025).

The encoder-choice ablation clarifies the role of this design. Adding DINOv2 or SigLIP2 yields only marginal changes, and adding VGGT with random initialization gives no gain, whereas adding VGGT pretrained on pixel↔3D pairs raises performance substantially. This indicates that the improvement is tied to 3D priors rather than parameter count alone.

4. Implicit neural memory and incremental KV caching

JanusVLN defines implicit neural memory as the cached key–value tensors produced inside transformer attention blocks. Let $I$3 denote the encoder type, and let $I$4 be the token matrix at timestep $I$5, where $I$6. For attention projections $I$7, the paper defines

$I$8

with each tensor in $I$9.

The two memories are

$l$0

$l$1

The initial memory retains the first $l$2 frames permanently:

$l$3

The sliding memory maintains the most recent $l$4 frames in FIFO order:

$l$5

for $l$6, otherwise all $l$7 frames. The final caches used at time $l$8 are

$l$9

Because $$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$0 and $$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$1 are fixed, with implementation values $$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$2 and $$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$3, the memory is fixed-size (Zeng et al., 26 Sep 2025).

Attention uses the cached KVs directly:

$$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$4

For VGGT, the cached $$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$5 is used in the fusion decoder via cross-attention. The semantic encoder attends to $$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$6 analogously, and the LLM also retains KV caches from instruction tokens and visual tokens with the same initial-plus-sliding mechanism. The first $$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$7 frames function as “attention sinks,” serving as global anchors with sustained high attention weights, while the window tracks local dynamics.

The computational consequence is explicit. Let $$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$8 be the fixed memory size in frames. Without caching, each step reprocesses all $$a_{t+1} \in \{\text{Move\_Forward}, \text{Turn\_Left}, \text{Turn\_Right}, \text{Stop}\}$$9 frames and memory grows as $30^\circ$0. With dual implicit memory, per-step cost is $30^\circ$1 and memory is $30^\circ$2 regardless of $30^\circ$3. The paper reports measured inference time on the current frame as follows: VGGT recomputation requires $30^\circ$4 ms at $30^\circ$5 frames and $30^\circ$6 ms at $30^\circ$7 frames, with out-of-memory beyond approximately $30^\circ$8 frames on a $30^\circ$9 GB GPU; cached memory requires $25$0 ms at $25$1 frames. The reported reduction in inference overhead is $25$2–$25$3, with performance saturating around $25$4 frames (Zeng et al., 26 Sep 2025).

5. Training regime, benchmarks, and empirical performance

The training objective is the navigation action loss, with no auxiliary spatial losses or contrastive objectives:

$25$5

and

$25$6

JanusVLN fine-tunes only the Qwen2.5-VL 7B LLM and the fusion MLP, using learning rates $25$7 for the LLM and $25$8 for the projection MLP. The semantic encoder and the spatial encoder are frozen. Evaluation is conducted on the VLN-CE benchmarks R2R-CE (Val-Unseen) and RxR-CE (Val-Unseen), built from Matterport3D scenes in Habitat, using Navigation Error (NE), Oracle Success (OS), Success Rate (SR), Success-weighted Path Length (SPL), and normalized DTW (nDTW), with SR and SPL treated as primary metrics (Zeng et al., 26 Sep 2025).

Following StreamVLN, the framework uses a subset of ScaleVLN containing $25$9K trajectories, approximately $M_{\text{sem}}$0K image–action pairs, and DAgger data comprising $M_{\text{sem}}$1K trajectories, approximately $M_{\text{sem}}$2K image–action pairs, collected on standard R2R-CE and RxR-CE. The paper also reports a zero-extra-data variant, JanusVLN*.

On R2R-CE Val-Unseen, selected reported lines are: GridMM with SR $M_{\text{sem}}$3 and SPL $M_{\text{sem}}$4 using Pano+Odo+Depth; DreamWalker with SR $M_{\text{sem}}$5 and SPL $M_{\text{sem}}$6 using Pano+Odo+Depth; g3D-LF with SR $M_{\text{sem}}$7 and SPL $M_{\text{sem}}$8 using Odo+Depth+RGB; NaVid-4D with SR $M_{\text{sem}}$9 and SPL $M_{\text{spa}}$0 using Depth+RGB; NaVid with SR $M_{\text{spa}}$1 and SPL $M_{\text{spa}}$2 using single RGB and $M_{\text{spa}}$3K external samples; Uni-NaVid with SR $M_{\text{spa}}$4 and SPL $M_{\text{spa}}$5 using single RGB and $M_{\text{spa}}$6K; NaVILA with SR $M_{\text{spa}}$7 and SPL $M_{\text{spa}}$8 using single RGB and $M_{\text{spa}}$9K; StreamVLN with SR $F_t$0 and SPL $F_t$1 using single RGB and approximately $F_t$2K; JanusVLN* with SR $F_t$3 and SPL $F_t$4 using single RGB and $F_t$5K extra; and JanusVLN with SR $F_t$6 and SPL $F_t$7 using single RGB and $F_t$8K extra. The paper summarizes these comparisons by stating that, against multi-input methods, JanusVLN improves SR by $F_t$9–$x_t \in \mathbb{R}^{3 \times H \times W}$00 using single RGB only, and against methods using more RGB training data, it improves SR by $x_t \in \mathbb{R}^{3 \times H \times W}$01–$x_t \in \mathbb{R}^{3 \times H \times W}$02 (Zeng et al., 26 Sep 2025).

On RxR-CE Val-Unseen, the reported selected lines are: Reborn with SR $x_t \in \mathbb{R}^{3 \times H \times W}$03, SPL $x_t \in \mathbb{R}^{3 \times H \times W}$04, and nDTW $x_t \in \mathbb{R}^{3 \times H \times W}$05 using Pano+Odo+Depth; Uni-NaVid with SR $x_t \in \mathbb{R}^{3 \times H \times W}$06 and SPL $x_t \in \mathbb{R}^{3 \times H \times W}$07 using single RGB; NaVILA with SR $x_t \in \mathbb{R}^{3 \times H \times W}$08, SPL $x_t \in \mathbb{R}^{3 \times H \times W}$09, and nDTW $x_t \in \mathbb{R}^{3 \times H \times W}$10 using single RGB; StreamVLN with SR $x_t \in \mathbb{R}^{3 \times H \times W}$11, SPL $x_t \in \mathbb{R}^{3 \times H \times W}$12, and nDTW $x_t \in \mathbb{R}^{3 \times H \times W}$13 using single RGB and approximately $x_t \in \mathbb{R}^{3 \times H \times W}$14K; JanusVLN* with SR $x_t \in \mathbb{R}^{3 \times H \times W}$15, SPL $x_t \in \mathbb{R}^{3 \times H \times W}$16, and nDTW $x_t \in \mathbb{R}^{3 \times H \times W}$17 using $x_t \in \mathbb{R}^{3 \times H \times W}$18K extra; and JanusVLN with SR $x_t \in \mathbb{R}^{3 \times H \times W}$19, SPL $x_t \in \mathbb{R}^{3 \times H \times W}$20, and nDTW $x_t \in \mathbb{R}^{3 \times H \times W}$21 using $x_t \in \mathbb{R}^{3 \times H \times W}$22K. The paper states that on RxR-CE, JanusVLN improves SR by $x_t \in \mathbb{R}^{3 \times H \times W}$23–$x_t \in \mathbb{R}^{3 \times H \times W}$24 over prior methods, with strong generalization.

6. Ablations, qualitative behavior, and limitations

The component ablation on R2R-CE with $x_t \in \mathbb{R}^{3 \times H \times W}$25K extra data attributes a distinct role to each memory. Full JanusVLN reports NE $x_t \in \mathbb{R}^{3 \times H \times W}$26, OS $x_t \in \mathbb{R}^{3 \times H \times W}$27, SR $x_t \in \mathbb{R}^{3 \times H \times W}$28, and SPL $x_t \in \mathbb{R}^{3 \times H \times W}$29. Removing spatial implicit memory reduces SR to $x_t \in \mathbb{R}^{3 \times H \times W}$30 and SPL to $x_t \in \mathbb{R}^{3 \times H \times W}$31. Removing semantic implicit memory reduces SR to $x_t \in \mathbb{R}^{3 \times H \times W}$32 and SPL to $x_t \in \mathbb{R}^{3 \times H \times W}$33. Removing both memories yields SR $x_t \in \mathbb{R}^{3 \times H \times W}$34 and SPL $x_t \in \mathbb{R}^{3 \times H \times W}$35. The paper concludes that both memories are indispensable and complementary: spatial memory yields large SPL gains, associated with path efficiency, while semantic memory boosts SR (Zeng et al., 26 Sep 2025).

The extra-encoder ablation addresses a likely misconception that gains come merely from adding another encoder. With no extra encoder, SR is $x_t \in \mathbb{R}^{3 \times H \times W}$36 and SPL is $x_t \in \mathbb{R}^{3 \times H \times W}$37. Adding DINOv2 or SigLIP2 gives only marginal changes, with SR at most $x_t \in \mathbb{R}^{3 \times H \times W}$38. Adding VGGT with random initialization gives SR $x_t \in \mathbb{R}^{3 \times H \times W}$39, effectively no gain. Adding VGGT pretrained on pixel↔3D pairs raises SR to $x_t \in \mathbb{R}^{3 \times H \times W}$40 and SPL to $x_t \in \mathbb{R}^{3 \times H \times W}$41. The stated conclusion is that gains derive from 3D priors, not parameter count.

Fusion and data ablations further constrain interpretation. The appendix reports that $x_t \in \mathbb{R}^{3 \times H \times W}$42 in $x_t \in \mathbb{R}^{3 \times H \times W}$43 yields the best performance, while concat and cross-attention fusion are competitive but slightly inferior. The data ablation reports that even without extra data, JanusVLN* outperforms several prior methods that use additional data, and that ScaleVLN or DAgger each improve performance, with the combination further boosting SR and SPL. This suggests that the framework is data-efficient within the reported training regime.

Qualitative results are presented through real-world case studies on a Unitree Go2 robot equipped with an Insta360 X5 front RGB camera, with inference performed on a remote server using an NVIDIA A10 GPU. The reported examples include “go to the farthest yellow stool,” which requires monocular distance discrimination; “stop beside the green potted plant (not in front of it),” which requires relational spatial reasoning; and “the stool beside the orange cabinet,” which requires spatial association. The paper notes that the spatial memory can be visualized as inferred depth and point clouds, and that attention sinks stabilize global scene anchors as trajectories evolve.

The limitations identified are bounded-memory limitations on very long instructions and extremely extended trajectories, possible stress on VGGT’s spatial priors under highly dynamic environments or fast camera motion, and the fact that the framework is RGB-only by design. Proposed future directions include adaptive memory policies such as learned retention and content-aware eviction, motion-aware geometry, temporal consistency constraints, lightweight egomotion estimation, multi-sensor integration, and transfer to other continuous embodied tasks such as manipulation and search. These are presented as extensions of the implicit-memory paradigm rather than as resolved capabilities.