---
title: 'LOVON: Legged Open-Vocabulary Navigator'
url: https://www.emergentmind.com/topics/lovon
type: topic
---

# LOVON: Legged Open-Vocabulary Navigator

Searching arXiv for the LOVON paper and closely related navigation work.
arXiv search query: "LOVON Legged Open-Vocabulary Object Navigator"
LOVON, short for **Legged Open-Vocabulary Object Navigator**, is an end-to-end operating system for legged robots that couples high-level language planning, open-vocabulary object perception, and a language-to-motion controller to execute long-horizon object navigation tasks in unstructured, dynamic environments [2507.06747]. The framework is designed for missions of the form $T_l$, such as “go to the chair, then the backpack, then follow the person at medium speed,” where the robot must decompose a long-horizon instruction into ordered subgoals, identify objects specified in free-form language, navigate toward static or moving targets, and remain robust to motion blur, occlusions, temporary target loss, and “blind zones” in the robot’s field of view. The system was validated in simulation and on multiple legged platforms—Unitree Go2, B2, and H1-2—with an emphasis on autonomous navigation, task adaptation, and robust task completion under real-world disturbances [2507.06747].

## 1. Problem formulation and scope

LOVON addresses object navigation in open-world environments where long-horizon task execution requires both open-world object detection and high-level task planning [2507.06747]. In the formulation used by the system, the robot receives a mission $T_l$ expressed in natural language and must transform it into an ordered sequence of executable subgoals. This entails three coupled requirements: semantic interpretation of free-form instructions, perception of detector-recognizable object instances despite synonymy and paraphrase, and motion control that remains stable under the perturbations characteristic of legged locomotion.

The motivation for this formulation is that legged robots already exhibit strong mobility on rugged terrain, but typical systems optimize isolated behaviors such as walking or short-range tracking rather than unified long-horizon autonomy. LOVON therefore combines three components that are often treated separately: **LLM-based hierarchical planning**, **open-vocabulary visual detection**, and a **Language-to-Motion Model (L2MM)** that maps language plus perception to continuous velocity commands [2507.06747].

The paper defines LOVON’s operating setting in terms of ordered mission execution rather than map-based navigation. The robot must search, detect, and navigate toward moving or static targets, but also recover from disturbances such as temporary detector failure, target displacement, and partial observability. A central design choice is that navigation is **vision-centric with direct control, not map-based**; LiDAR/SLAM is not used or reported in the system [2507.06747].

## 2. Architectural organization and dataflow

LOVON is organized as a modular pipeline comprising an LLM planner, an Instruction-Object Extractor, a detector, and a transformer-based controller [2507.06747]. The core interfaces are explicit:

- The **LLM task planner** $f_{LLM}$ takes a system description $I_{sys}$, a long-sequence task $T_l$, and feedback $O_f$ from the controller, and returns structured mission instructions:
  $$
  I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).
  $$
  The implementation uses **DeepSeek-R1** as planner and data-generation assistant.

- The **Instruction-Object Extractor (IOE)** $f_{IOE}$ maps each instruction to a detector-recognizable class $I_{object} \in C$:
  $$
  I_{object} = f_{IOE}(I_m).
  $$
  The IOE is a two-layer transformer with dimension $64$, $2$ layers, and $4$ heads.

- The **visual detector** $f_{det}$ processes LVF-filtered RGB frames and returns object name, confidence, normalized center, and normalized box size:
  $$
  O_m, C_p, O_{xy}, O_{wh} = f_{det}(I_{RGB}, I_{object}).
  $$
  The experiments use **YOLO-11 (Ultralytics)** for speed and efficiency, although DETR/DINO/GroundingDINO are described as compatible.

- The **Language-to-Motion Model (L2MM)** is a transformer encoder-decoder whose Base configuration uses dimension $256$, $4$ layers, $8$ heads, and FFN $=1024$. Its encoder input is
  $$
  I_{encoder} = \{I_{m0}, I_{m1}, O_p, C_p, O_{xy}, O_{wh}, S_m, S_s\},
  $$
  concatenated with `[SEP]`. It produces three heads: a motion vector head $D_{motion}$ with
  $$
  V_m = [v_x, v_y, \theta],
  $$
  a mission state head $D_{mission}$ with $S_m \in \{\text{running}, \text{success}\}$, and a search state head $D_{search}$ with $S_s \in \{\text{searching\_0}, \text{searching\_1}\}$.

The dataflow is sequential. First, the LLM decomposes $T_l$ into ordered atomic instructions $I_i$. Second, the IOE grounds each instruction to a class that the detector can recognize. Third, a perception loop captures RGB images at approximately $15$ Hz, applies Laplacian Variance Filtering (LVF), runs detection, and smooths confidence and bounding boxes with a Moving Average Filter (MAF). Fourth, the L2MM combines current and previous instructions with detection features and state variables to output velocities and status. Fifth, the robot executes $V_m$, and the resulting mission/search status is fed back to the planner as $O_f$ for progression or adaptation [2507.06747].

| Module | Input | Output |
|---|---|---|
| LLM planner $f_{LLM}$ | $I_{sys}, T_l, O_f$ | $I_{ins}=\{I_i\}$ |
| IOE $f_{IOE}$ | $I_m$ | $I_{object}\in C$ |
| Detector $f_{det}$ | $I_{RGB}, I_{object}$ | $O_m, C_p, O_{xy}, O_{wh}$ |
| L2MM | Language, detection features, states | $V_m, S_m, S_s$ |

A distinctive element is the way open-vocabulary behavior is achieved despite using a closed-set detector. The IOE expands synonyms and paraphrases to the specific class in $C$ that YOLO-11 recognizes. This enables open-vocabulary grounding at the instruction level while preserving the speed of a conventional detector [2507.06747].

## 3. Perception stabilization and execution logic

LOVON’s robustness claims depend substantially on two mechanisms: **Laplacian Variance Filtering** and a **functional execution logic** implemented as a state machine [2507.06747].

LVF is used before detection to suppress motion-blurred frames that would otherwise destabilize confidence estimates. For an image $I_{RGB}$, the procedure is: convert to grayscale $I_{gray}$, compute the Laplacian $L = \nabla^2 I_{gray}$, compute the variance $\mathrm{Var}(L)$, and classify the frame as blurred if
$$
\mathrm{Var}(\nabla^2 I_{gray}) < T_{blur}.
$$
The paper fixes the blur threshold at
$$
T_{blur} = 150
$$
after empirical calibration. If a frame falls below this threshold, it is replaced with the last qualified clear frame. Bounding boxes and confidence scores are then smoothed with a moving average filter. The combination of thresholding and MAF improves the qualified frame ratio by approximately $25\%$, whereas thresholding alone yields approximately $15\%$, across object categories and robot speeds of $0.3/0.5/0.7$ m/s [2507.06747].

The execution logic uses two discrete state variables. The **mission state** is $S_m \in \{\text{running}, \text{success}\}$, and the **search state** is $S_s \in \{\text{searching\_0}, \text{searching\_1}\}$. Representative motion mappings are explicitly specified:

- $\text{success} \rightarrow [0,0,0]$
- $\text{running} \rightarrow [v_x, 0, \theta_{corr}]$
- $\text{searching\_0} \rightarrow [0,0,-0.3]$ rad/s
- $\text{searching\_1} \rightarrow [0,0,+0.3]$ rad/s

When the detector returns a valid target after LVF and MAF, the system enters **running** and applies heading correction $\theta_{corr}$ based on target-center offset. If the target is temporarily lost, confidence drops, or no detection is returned, the controller alternates between **searching\_0** and **searching\_1**, which rotate left and right to sweep blind zones in the camera field of view. Once the target is reacquired, the controller returns to running. Mission completion occurs when the normalized bounding-box size $O_{wh}$ exceeds an object-dependent success threshold; at that point the system sets $S_m=\text{success}$ and $V_m=[0,0,0]$, after which the planner advances to the next instruction [2507.06747].

This state logic is also the system’s disturbance-recovery mechanism. If the robot is kicked or the target is displaced, object loss triggers search behavior; once reacquisition occurs, the L2MM resumes running. The paper attributes part of this robustness to LVF, which prevents unstable frames from causing false losses [2507.06747].

## 4. Hierarchical planning, dataset generation, and learning

LOVON’s hierarchical planning is driven by prompt-based LLM decomposition. The planner ingests $I_{sys}$, $T_l$, and $O_f$, and emits an ordered list of instructions with explicit targets and action parameters. One example given is: “Approach the chair at 0.4 m/s; then find the backpack at 0.3 m/s; next follow the person at 0.5 m/s.” The planner may reconfigure tasks and monitor feedback to adapt order, speeds, or switching criteria [2507.06747].

The training and inference boundary is sharply defined. **DeepSeek-R1** is used for planning and dataset generation but is **not fine-tuned**. **YOLO-11** is pre-trained and used as-is. The trainable components are the **L2MM** and the **IOE** [2507.06747].

The training dataset contains **1M samples** with a **4:1 train/test** split and is generated in **<15 minutes on CPU (Intel i9-12900KF)** by combining synonym expansion for detection classes, instruction paraphrasing for language diversity, and category-specific success-threshold adaptation via the LLM [2507.06747]. Training is performed on an **RTX 3080 Ti** with dropout $0.1$, learning rate $10^{-4}$, batch size $512$, sequence length $64$, and $25$ epochs. The reported training times are approximately **1 hour** for the L2MM and **30 minutes** for the IOE [2507.06747].

The L2MM is optimized with a weighted motion loss and state-classification losses. The motion-vector term is
$$
L_{MSE} = (\beta/N)\sum_i \|V_{m,\mathrm{pred}}^i - V_{m,\mathrm{true}}^i\|^2
$$
with $\beta=10$, and the mission/search state terms use cross-entropy:
$$
L_{CE} = -\sum_i y_i \log p_i.
$$
An ablation study reports that larger models reduce $\sigma_v$ and $\epsilon_v$ but with saturating gains, that too small a $\beta$ undervalues motion loss, that too large a $\beta$ harms state inference, and that `[SEP]` tokens are critical for separating modalities [2507.06747].

A noteworthy consequence of the data-generation pipeline is that the system achieves scalability without retraining the LLM or detector. This suggests a design in which language diversity is absorbed largely through synthetic supervision and instruction grounding, rather than by end-to-end optimization of all modules.

## 5. Evaluation, benchmarks, and empirical behavior

LOVON is evaluated in simulation on the **Gym-UnrealCV** benchmark, where the maximum episode length is **500** steps and the tracker visible region is a **90-degree sector with radius 750 cm** [2507.06747]. The paper reports **Episode Length (EL)** and **Success Rate (SR)** across **100 trials**, and it also states the standard definition of SPL, although SPL is not used in the study.

Simulation baselines include **DiMP, SARL, AD-VAT, AD-VAT+, TS, RSPT, EVT, TrackVLA**, and LOVON. Averaged over **ParkingLot, UrbanCity, UrbanRoad, SnowVillage**, LOVON achieves **near-perfect SR (≈1.00) and EL ≈500 in most scenes with only 1.5 hours of training**, outperforming or matching state-of-the-art methods. The paper gives **EVT: mean ≈487.75/0.94** and **TrackVLA: 500/1.00 but 360 hours training** as comparison points [2507.06747].

Real-world experiments focus on four regimes: **open-world adaptation**, **long-horizon multi-goal tracking**, **dynamic tracking**, and **robustness to disturbances**. The reported tasks include navigation to large objects such as cars, medium objects such as people, and small objects such as bags; sequential multi-goal tasks handled by the LLM planner; following moving targets across flat roads, spiral stairs, and wild grass; and quick relocalization after target movement or robot kicking [2507.06747].

The ablation results quantify the contribution of filtering and search-state design. For the “filter method and number of states” comparison, the paper contrasts **Case 1 (3 states, no filtering)**, **Case 2 (4 states, no filtering)**, and **Case 3 (4 states + LVF)** using the metrics **Number of searching cycles $N_s$** and **search time $T_s$** at **4 m and 6 m**. For a **Backpack at 6 m**, the reported values are **Case 1 $N_s=5.95$, $T_s=178$ s** versus **Case 3 $N_s=1.00$, $T_s=32.57$ s**, which the paper summarizes as approximately **5× faster** [2507.06747].

These empirical results support the paper’s claim that LVF plus dual-direction searching reduces search instability for hard-to-detect objects and that the combined architecture supports long-sequence tasks involving real-time detection, search, and navigation toward open-vocabulary dynamic targets [2507.06747].

## 6. Platforms, deployment characteristics, limitations, and related systems

LOVON is deployed on **Unitree Go2, B2, and H1-2** and runs core on-board inference on **NVIDIA Jetson Orin** hardware [2507.06747]. The perception stack uses the robots’ built-in cameras together with an **Intel RealSense D435i**; detection uses RGB streams, while the IMU is present for locomotion but is **not explicitly used by LOVON’s perception**. The detector, IOE, LVF, and L2MM run on-board at approximately the camera rate of **15 Hz**. The paper does not provide explicit FPS or energy figures, although LVF and MAF are described as lightweight [2507.06747].

The platform abstraction is deliberately simple: LOVON emits generic velocity commands $V_m=[v_x,v_y,\theta]$ into the robot’s native velocity-control interface. The system is therefore described as **plug-and-play**, with portability arising from detector-independent IOE and LVF modules, generic velocity outputs, and reliance on pre-trained planner/detector plus a lightweight L2MM trained on synthetic and augmented data [2507.06747]. This portability is demonstrated across the three reported legged platforms.

The paper also delineates important limitations. The planner may misinterpret ambiguous instructions, and the feedback loop is **not foolproof**. YOLO-11 is efficient but closed-set; synonym mapping mitigates vocabulary mismatch but cannot detect truly unseen classes, and the authors identify **GroundingDINO or other open-vocabulary detectors** as a future path. On-board LVF+YOLO+L2MM runs at approximately camera rate, but heavier LLMs may require off-board compute or smaller local models. Very fast, highly occluded targets stress reacquisition, and the vision-and-control-centric design does not address general obstacle avoidance, SLAM, or 3D obstacle reasoning, all of which are identified as future directions [2507.06747].

In the paper’s own related-work framing, LOVON is positioned against open-vocabulary object-goal navigation systems such as **OVON/HM3D-OVON, VLM-Nav, and VLFMs**, and against trackers such as **TrackVLA**. The stated differences are that LOVON focuses on legged robots in unstructured outdoor and indoor scenes, directly outputs velocities through a language-to-motion transformer without explicit map/SLAM, adds LVF for vision stabilization under legged motion, and demonstrates plug-and-play deployment across distinct legged platforms [2507.06747].

A naming clarification is also useful. LOVON should not be conflated with **LOVO**, an unrelated system for complex object queries in large-scale video datasets [2507.14301]. The two works address different problem domains: LOVON concerns long-horizon embodied navigation for legged robots, whereas LOVO concerns open-vocabulary retrieval over video corpora.

Source: https://www.emergentmind.com/topics/lovon