LE-Nav: Scene-Aware Navigation Framework

• LE-Nav is an interpretable, scene-aware navigation framework that combines multi-modal large language models with CVAE to adapt planner hyperparameters.
• It features a runtime pipeline of sensors, scene reasoning, hyperparameter modulation, and motion planning to maintain safety and efficiency in dynamic environments.
• The framework demonstrates robustness through adaptive tuning and packet-loss augmentation, achieving superior navigation metrics compared to conventional methods.

Searching arXiv for LE-Nav and related papers to ground the article. LE-Nav is an interpretable and scene-aware navigation framework that leverages multi-modal LLM reasoning and conditional variational autoencoders to adaptively tune planner hyperparameters for service robots operating in diverse and dynamic environments (Wang et al., 15 Jul 2025). In the cited formulation, its runtime pipeline is Sensors → Scene Reasoning ([MLLM](https://www.emergentmind.com/topics/qwen2-5-vl-3b-multimodal-large-language-model-mllm) + Visual Model) → CVAE-Based Hyperparameter Modulator → Motion Planner ([TEB](https://www.emergentmind.com/topics/timed-elastic-bands-teb-local-planner) or DWA) → Robot Control, with scene understanding updated at 0.5 Hz and planner behavior modulated through [ROS](https://www.emergentmind.com/topics/reasoning-over-space-ros-framework) dynamic_reconfigure rather than by replacing the underlying planner (Wang et al., 15 Jul 2025). The name also appears in a distinct, narrower sense in event-to-LiDAR localization, where LEAR can serve as the pose-refinement core of an online LE-Nav pipeline for 6-DoF localization in GPS-denied scenarios (Chen et al., 2 Mar 2026). This dual usage makes it important to distinguish the 2025 framework for planner adaptation from the 2026 localization module that is explicitly described as a possible LE-Nav component.

1. System definition and architectural scope

In the 2025 formulation, LE-Nav is defined as an interpretable and scene-aware navigation framework that leverages multi-modal LLM reasoning and conditional variational autoencoders to adaptively tune planner hyperparameters (Wang et al., 15 Jul 2025). Its stated objective is to address settings in which conventional navigation systems that rely on fixed parameters often fail to generalize across scenarios, resulting in degraded performance and reduced social acceptance. The framework is designed for service robots in unstructured environments where both physical layouts and social contexts change over time and across locations.

The runtime architecture is specified as a sequence of five stages: Sensors → Scene Reasoning (MLLM + Visual Model) → CVAE-Based Hyperparameter Modulator → Motion Planner (TEB or DWA) → Robot Control (Wang et al., 15 Jul 2025). The sensing stack comprises an onboard camera (RealSense D435i) for semantic scene analysis and 2D LiDAR (RS-LiDAR-16) for localization and basic obstacle detection. Scene reasoning operates at 0.5 Hz. A small visual detector (YOLO-like) assists the MLLM in counting and localizing pedestrians, and the MLLM, exemplified by Qwen-VL-Max at 0.5 Hz, produces a fixed-size scene rating tensor $s^t \in \mathbb{R}^{5}$.

The motion-planning backend is TEB or DWA with 9 dynamically reconfigured weights, and the architecture explicitly preserves planner safety guarantees: Guarantees collision-free output; hyperparameter changes only modulate planner behavior (Wang et al., 15 Jul 2025). This is central to the framework’s design. Rather than learning control outputs end to end, LE-Nav learns a mapping from semantic and social scene descriptors to planner hyperparameters. A plausible implication is that the framework is intended to retain the predictability and formal structure of classical local planners while incorporating adaptive behavior.

2. Scene reasoning, prompt design, and scene representation

The scene-reasoning subsystem combines a visual detector with an MLLM to produce a structured scene rating tensor $s^t \in \mathbb{R}^{5}$ (Wang et al., 15 Jul 2025). The tensor comprises exactly five quantities:

1. ped_present ∈ {0,1}
2. ped_density ∈ {0,…,N}
3. ped_direction ∈ {discrete bins}
4. ped_proximity ∈ ℝ⁺
5. env_complexity ∈ [0,1]

This representation is the sole semantic interface between perception and hyperparameter adaptation. The framework therefore constrains the MLLM output to a fixed and parseable schema rather than allowing unrestricted textual reasoning.

The prompt design is divided into a one-shot induction phase and a chain-of-thought deduction phase (Wang et al., 15 Jul 2025). In the induction phase, the prompt begins by defining the task—“You are a navigation expert. Given an RGB image and domain rules…”—and provides exactly one example pairing of input and output. The example output is a JSON object with the five keys: $s^t \in \mathbb{R}^{5}$5

In the deduction phase, the prompt uses a stepwise template:

“Please think step-by-step: 1. Detect foreground moving agents via the visual model → count them → set ped_density. 2. Examine their optical flow vectors → set ped_direction. 3. Using bounding-box size heuristic → estimate distance → ped_proximity. 4. From the LiDAR scan density and open-space ratio → set env_complexity. 5. Finally, output a JSON mapping exactly five keys to numerical values.”

The paper states that This two-stage prompt (one-shot exemplars + CoT) enforces a uniform, parseable output (Wang et al., 15 Jul 2025). The role of chain-of-thought here is therefore operational rather than expository: it structures the intermediate reasoning needed to convert heterogeneous visual and range cues into the five-dimensional scene rating tensor.

3. Conditional variational autoencoder formulation

The hyperparameter modulator is a Conditional VAE trained on scene ratings and expert-selected planner settings (Wang et al., 15 Jul 2025). The input sequence is denoted Let s^{t:t−t_w} be the sequence of scene ratings; H^t ∈ ℝ⁹ the (normalized) hyperparameters at time t. The architectural decomposition consists of:

• Condition Encoder: a small Transformer that embeds a short history s^{t−t_w: t}
• Encoder q_ϕ(z|s,H): [MLP](https://www.emergentmind.com/topics/multi-layer-perceptron-mlp) that maps (s, expert‐chosen H) to latent z ∈ ℝ^d
• Prior p_θ(z|s): MLP that maps s → parameters of N(μ_p,σ_p²)
• Decoder p_θ(H|s,z): MLP that maps (s,z) → normalized hyperparameters H_norm ∈ [0,1]^9

The probabilistic specification is given explicitly as

$$q_\phi(z|s,H)=\mathcal{N}(z;\mu_q(s,H), \operatorname{diag}(\sigma_q^2(s,H)))$$

$$p_\theta(z|s)=\mathcal{N}(z;\mu_p(s), \operatorname{diag}(\sigma_p^2(s)))$$

$$p_\theta(H|s,z)=\mathcal{N}(H; f_{dec}(s,z), \sigma_r^2 I)$$

with σ_r fixed (Wang et al., 15 Jul 2025). The loss per time step is reported as

$$L(s,H;\theta,\phi) = D_{KL}[ q_\phi(z|s,H) \,\|\, p_\theta(z|s) ] + \gamma \cdot || f_{dec}(s,z_1…z_N) - H ||^2_2 / N$$

where z_n = μ_q + σ_q ∘ ε_n, ε_n∼𝒩(0,I), N is # of Monte Carlo samples, γ balances reconstruction vs KL.

This formulation makes the latent variable $z$ scene-conditional at inference through the learned prior $p_\theta(z|s)$, while training couples scene ratings to expert hyperparameter choices through the posterior $q_\phi(z|s,H)$. The decoder output is a normalized nine-dimensional vector subsequently rescaled to planner-specific physical units. This suggests that stochasticity in $z$ is used to represent multiple plausible expert-like tunings for the same scene description rather than deterministic regression to a single configuration.

4. Training regime and runtime adaptation

The training dataset is defined in terms of expert-tuned sequences gathered separately for the two planners (Wang et al., 15 Jul 2025). For TEB, the paper reports 53 expert‐tuned sequences → 44 train / 9 val; for DWA, 56 sequences → 47 train / 9 val. Each sequence contains 0.5 Hz image + dynamic_reconfigure hyperparams. Preprocessing consists of Min–max scale each of the 9 hyperparams into [0,1].

A distinctive robustness measure is Random packet‐loss augmentation: with probability p randomly drop 1–2 recent s frames and mask in the Transformer’s attention (Wang et al., 15 Jul 2025). The optimization setup is specified as batch size = 16, epochs = 8000, Optimizer = AdamW, lr = 1e−3, GPU memory < 0.5 GB, and No additional regularization beyond packet‐loss masking. The explicit absence of further regularization is notable because it places the main burden of robustness on the sequence model and the packet-loss augmentation strategy.

At runtime, adaptation occurs every 2 s (0.5 Hz) through a six-step procedure (Wang et al., 15 Jul 2025):

1. Acquire scene rating sᵗ from MLLM
2. Compute prior p_θ(z|sᵗ), draw k samples {z_i}
3. Decode each to H_norm,i = f_dec(sᵗ,z_i)
4. Rescale H_i = H_min + H_norm,i * (H_max−H_min)
5. Optionally select best H_i via a simple criterion (e.g. closest to last expert tuning or lowest variance) or just apply the first sample
6. Push H_i into ROS dynamic_reconfigure for TEB/DWA

The paper emphasizes that The entire mapping is learned end‐to‐end within the [CVAE](https://www.emergentmind.com/topics/conditional-variational-autoencoders-cvae-23f53af5-31ba-4720-b618-bc30e7f489b2); no hand‐tuned interpolation is needed (Wang et al., 15 Jul 2025). In practical terms, LE-Nav therefore treats planner adaptation as conditional generative modeling over expert hyperparameter choices rather than as rule-based scheduling or direct policy learning.

5. Evaluation, quantitative results, and deployment

Evaluation is reported at three levels: hyperparameter-generation accuracy, real-world navigation performance, and user study outcomes (Wang et al., 15 Jul 2025). For hyperparameter generation, the metrics are Top-Err (best of k=10 samples) and Mean-Err (average of k) in % error vs expert parameters, conditioned on three [MLLMs](https://www.emergentmind.com/topics/autoregressive-multimodal-large-language-models-mllms) under packet loss. The reported results are as follows.

Planner	Input MLLM	Top-Err / Mean-Err
TEB	GPT-4o	3.60% / 3.77%
TEB	Kimi	8.05% / 8.14%
TEB	QWEN	6.70% / 6.86%
DWA	GPT-4o	3.84% / 4.03%
DWA	Kimi	9.12% / 9.42%
DWA	QWEN	4.16% / 4.83%

The packet-loss ablation states that Without packet‐loss augmentation, Top-Err nearly doubles under realistic loss patterns (dropping 1–2 frames) (Wang et al., 15 Jul 2025). This is one of the clearest causal findings reported for the system.

Real-world navigation is evaluated in five scenarios (narrow gate, tight turns, dynamic crossing, static efficiency, dense crowd) against TEB-Prog / TEB-Cons, DWA-Prog / DWA-Cons, DADWA [ref. 24], and LE-Nav-TEB / LE-Nav-DWA (Wang et al., 15 Jul 2025). The metrics are:

1. Failed-to-plan (timeouts or collisions)
2. Ego progress (s) – lower is better
3. Risk rate = % of timesteps with TTC=∞, plus relaxed thresholds at TTC\>5s, TTC\>2s
4. Acceleration (m/s²)
5. Jerk (m/s³)
6. Composite Score (Eq. (7))

The composite score is defined as

$s^t \in \mathbb{R}^{5}$0

with α={0.8,0.1,0.1}, β={0.5,1.0,1.5} (Wang et al., 15 Jul 2025).

A highlighted result for Scenario (a): narrow gate) is that TEB-Prog and DADWA both fail 3/3 trials → Score=0, whereas LE-Nav-TEB: zero failures, Ego 34.35 s, Score=0.7957 (best) and LE-Nav-DWA: Score=0.7739 (Wang et al., 15 Jul 2025). More generally, Across all five tasks, LE-Nav variants (both TEB & DWA) achieve the highest composite Score within their planner class, with zero or minimal [planning](https://www.emergentmind.com/topics/dynamic-heuristic-biasing-planning) failures and strong trade‐offs of safety vs efficiency.

The user study consists of 10 blind trials in a corridor with real users and pedestrians (Wang et al., 15 Jul 2025). Questions were scored on a 1–5 scale for both pedestrians and wheelchair users. The reported result is that LE-Nav-DWA scored highest in all five questions, followed by LE-Nav-TEB, then DADWA. The paper also reports qualitative feedback: Pedestrians noted smoother, earlier obstacle avoidance under LE-Nav, and Users reported higher perceived safety and predictability.

Deployment details are explicit. The platform is a Smart wheelchair with RealSense D435i + RS-LiDAR-16, running on an Onboard laptop: Intel i5-12400F + RTX 3060, using the ROS MoveBase stack for localization, global planning, TEB/DWA local planning; the Offboard MLLM server operates at 0.5 Hz; CVAE runs locally at real-time (Wang et al., 15 Jul 2025). The documented baseline failure cases are also specific: DADWA: large [sim-to-real gap](https://www.emergentmind.com/topics/sim-to-real-gap) → misses narrow-gate turn in real trials; Human-tuned single “Progressive/Conservative” fails dynamic crossing and dense crowds; and Direct LLM→hyperparameters (no CVAE) sticks to defaults, no scene adaptation.

6. Relationship to LEAR and the event-to-LiDAR usage of “LE-Nav”

A separate paper describes how LEAR can be cast into a “LE-Nav” navigation module that fuses event-camera data with a prebuilt LiDAR map to produce 6-DoF poses in GPS-denied scenarios (Chen et al., 2 Mar 2026). In that formulation, the term “LE-Nav” does not denote the MLLM→CVAE→planner architecture. Instead, it denotes an online localization pipeline in which LEAR serves as the pose-refinement core.

The underlying sensing and optimization problem are defined differently from the 2025 navigation framework. The event camera outputs an asynchronous stream

$s^t \in \mathbb{R}^{5}$1

where (x_i,y_i) are pixel coordinates, t_i the timestamp, and p_i∈{+1,−1} the polarity (Chen et al., 2 Mar 2026). Events are aggregated over a short window $s^t \in \mathbb{R}^{5}$2 into an event image

$s^t \in \mathbb{R}^{5}$3

The world point cloud is P_w={P_j∈ℝ^3}_{j=1}^M, and given an initial pose guess T_init∈SE(3), points are projected into the camera frame and onto the image plane to form a depth image I_D (Chen et al., 2 Mar 2026). The stated objective is to Estimate the refined pose T∈SE(3) so that events in I_EV align to points in I_D.

LEAR itself consists of two branches—a Depth-Edge Detector and a Dense Event-Depth Flow Estimator—coupled by cross-modal feature fusion and iterative refinement (Chen et al., 2 Mar 2026). The edge branch uses a Holistically-Nested Edge Detector (HED) backbone to produce a per-pixel edge probability map, while the flow branch uses a modified [RAFT](https://www.emergentmind.com/topics/retrieval-augmented-fine-tuning-raft) encoder-decoder with an all-pairs correlation volume Corr(F_D,F_EV) and a recurrent GRU-based updater for K iterations to predict dense 2D flow. Cross-Modal Fusion (CFF) injects edge geometry into flow estimation by fusing F_D^i and F_ED^i at shared scales i∈{½,¼,1/8}, and Iterative Feature Refinement (IFR) jointly refines flow and edge features over K steps.

Pose recovery is then carried out by [PnP](https://www.emergentmind.com/topics/predict-and-perturb-pnp) + RANSAC, solving

$s^t \in \mathbb{R}^{5}$4

using a robust solver (e.g. Huber loss + RANSAC) (Chen et al., 2 Mar 2026). The paper’s explicit deployment interpretation is:

• At each time step t, take the last pose estimate T_{t−1} as T_init.
• Project the prebuilt LiDAR map into a live depth image I_D and form I_EV from recent events.
• Run the dual-branch LEAR network to obtain refined flow and edges.
• Recover T_t via the PnP module.
• Feed T_t back into the map-projection for the next frame.

This event-to-LiDAR version of LE-Nav is evaluated on M3ED and DSEC, with metrics Flow [EPE](https://www.emergentmind.com/topics/expected-prediction-entropy-epe) (pixel) and Localization translation error (cm) & rotation error (deg), mean & median (Chen et al., 2 Mar 2026). Reported results include that On M3ED, LEAR reduces median translation by ~15–20% and rotation by ~20–25% over EVLoc, and On DSEC, LEAR achieves up to 36% lower translation and 29% lower rotation error vs. EVLoc, and up to 61%/44% vs. I2D-Loc. An ablation further reports: Oracle (masked depth by gt edges) EPE=1.66px, trans=3.31 cm, rot=0.22°; Baseline (no edge) EPE=6.10px, trans=8.11 cm, rot=0.97°; +CFF: EPE→5.75px, trans=7.20 cm, rot=0.83°; +CFF+IFR: EPE→5.52px, trans=6.85 cm, rot=0.74°.

The coexistence of these two usages means that “LE-Nav” can refer either to a service-robot navigation framework based on MLLM reasoning and CVAE-based planner adaptation (Wang et al., 15 Jul 2025), or to an online event-camera–LiDAR localization pipeline whose core is LEAR (Chen et al., 2 Mar 2026). The papers themselves support both interpretations, but they concern different technical problems: planner hyperparameter modulation in one case, and 6-DoF pose refinement in the other.