LOVON: Legged Open-Vocabulary Navigator

• LOVON is a legged robot navigation system that integrates LLM-based hierarchical planning, open-vocabulary object detection, and language-to-motion control to execute long-horizon tasks.
• It employs Laplacian Variance Filtering and a moving average filter to stabilize vision and maintain robust performance under motion blur and disturbances.
• Evaluations in simulation and on multiple legged platforms show that LOVON achieves near-perfect success rates with reduced search cycles and rapid target reacquisition.

Searching arXiv for the LOVON paper and closely related navigation work. arXiv search query: "LOVON Legged Open-Vocabulary Object Navigator" LOVON, short for Legged Open-Vocabulary Object Navigator, is an end-to-end operating system for legged robots that couples high-level language planning, open-vocabulary object perception, and a language-to-motion controller to execute long-horizon object navigation tasks in unstructured, dynamic environments (Peng et al., 9 Jul 2025). The framework is designed for missions of the form $T_l$, such as “go to the chair, then the backpack, then follow the person at medium speed,” where the robot must decompose a long-horizon instruction into ordered subgoals, identify objects specified in free-form language, navigate toward static or moving targets, and remain robust to motion blur, occlusions, temporary target loss, and “blind zones” in the robot’s field of view. The system was validated in simulation and on multiple legged platforms—Unitree Go2, B2, and H1-2—with an emphasis on autonomous navigation, task adaptation, and robust task completion under real-world disturbances (Peng et al., 9 Jul 2025).

1. Problem formulation and scope

LOVON addresses object navigation in open-world environments where long-horizon task execution requires both open-world object detection and high-level task planning (Peng et al., 9 Jul 2025). In the formulation used by the system, the robot receives a mission $T_l$ expressed in natural language and must transform it into an ordered sequence of executable subgoals. This entails three coupled requirements: semantic interpretation of free-form instructions, perception of detector-recognizable object instances despite synonymy and paraphrase, and motion control that remains stable under the perturbations characteristic of legged locomotion.

The motivation for this formulation is that legged robots already exhibit strong mobility on rugged terrain, but typical systems optimize isolated behaviors such as walking or short-range tracking rather than unified long-horizon autonomy. LOVON therefore combines three components that are often treated separately: LLM-based hierarchical planning, open-vocabulary visual detection, and a Language-to-Motion Model (L2MM) that maps language plus perception to continuous velocity commands (Peng et al., 9 Jul 2025).

The paper defines LOVON’s operating setting in terms of ordered mission execution rather than map-based navigation. The robot must search, detect, and navigate toward moving or static targets, but also recover from disturbances such as temporary detector failure, target displacement, and partial observability. A central design choice is that navigation is vision-centric with direct control, not map-based; LiDAR/SLAM is not used or reported in the system (Peng et al., 9 Jul 2025).

2. Architectural organization and dataflow

LOVON is organized as a modular pipeline comprising an LLM planner, an Instruction-Object Extractor, a detector, and a transformer-based controller (Peng et al., 9 Jul 2025). The core interfaces are explicit:

• The LLM task planner $f_{LLM}$ takes a system description $I_{sys}$, a long-sequence task $T_l$, and feedback $O_f$ from the controller, and returns structured mission instructions:

$I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$

The implementation uses DeepSeek-R1 as planner and data-generation assistant.

• The Instruction-Object Extractor (IOE) $f_{IOE}$ maps each instruction to a detector-recognizable class $I_{object} \in C$:

$I_{object} = f_{IOE}(I_m).$

The IOE is a two-layer transformer with dimension $T_l$0, $T_l$1 layers, and $T_l$2 heads.

• The visual detector $T_l$3 processes LVF-filtered RGB frames and returns object name, confidence, normalized center, and normalized box size:

$T_l$4

The experiments use YOLO-11 (Ultralytics) for speed and efficiency, although DETR/DINO/GroundingDINO are described as compatible.

• The Language-to-Motion Model (L2MM) is a transformer encoder-decoder whose Base configuration uses dimension $T_l$5, $T_l$6 layers, $T_l$7 heads, and FFN $T_l$8. Its encoder input is

$T_l$9

concatenated with [SEP]. It produces three heads: a motion vector head $f_{LLM}$0 with

$f_{LLM}$1

a mission state head $f_{LLM}$2 with $f_{LLM}$3, and a search state head $f_{LLM}$4 with $f_{LLM}$5.

The dataflow is sequential. First, the LLM decomposes $f_{LLM}$6 into ordered atomic instructions $f_{LLM}$7. Second, the IOE grounds each instruction to a class that the detector can recognize. Third, a perception loop captures RGB images at approximately $f_{LLM}$8 Hz, applies Laplacian Variance Filtering (LVF), runs detection, and smooths confidence and bounding boxes with a Moving Average Filter (MAF). Fourth, the L2MM combines current and previous instructions with detection features and state variables to output velocities and status. Fifth, the robot executes $f_{LLM}$9, and the resulting mission/search status is fed back to the planner as $I_{sys}$0 for progression or adaptation (Peng et al., 9 Jul 2025).

Module	Input	Output
LLM planner $I_{sys}$1	$I_{sys}$2	$I_{sys}$3
IOE $I_{sys}$4	$I_{sys}$5	$I_{sys}$6
Detector $I_{sys}$7	$I_{sys}$8	$I_{sys}$9
L2MM	Language, detection features, states	$T_l$0

A distinctive element is the way open-vocabulary behavior is achieved despite using a closed-set detector. The IOE expands synonyms and paraphrases to the specific class in $T_l$1 that YOLO-11 recognizes. This enables open-vocabulary grounding at the instruction level while preserving the speed of a conventional detector (Peng et al., 9 Jul 2025).

3. Perception stabilization and execution logic

LOVON’s robustness claims depend substantially on two mechanisms: Laplacian Variance Filtering and a functional execution logic implemented as a state machine (Peng et al., 9 Jul 2025).

LVF is used before detection to suppress motion-blurred frames that would otherwise destabilize confidence estimates. For an image $T_l$2, the procedure is: convert to grayscale $T_l$3, compute the Laplacian $T_l$4, compute the variance $T_l$5, and classify the frame as blurred if

$T_l$6

The paper fixes the blur threshold at

$T_l$7

after empirical calibration. If a frame falls below this threshold, it is replaced with the last qualified clear frame. Bounding boxes and confidence scores are then smoothed with a moving average filter. The combination of thresholding and MAF improves the qualified frame ratio by approximately $T_l$8, whereas thresholding alone yields approximately $T_l$9, across object categories and robot speeds of $O_f$0 m/s (Peng et al., 9 Jul 2025).

The execution logic uses two discrete state variables. The mission state is $O_f$1, and the search state is $O_f$2. Representative motion mappings are explicitly specified:

• $O_f$3
• $O_f$4
• $O_f$5 rad/s
• $O_f$6 rad/s

When the detector returns a valid target after LVF and MAF, the system enters running and applies heading correction $O_f$7 based on target-center offset. If the target is temporarily lost, confidence drops, or no detection is returned, the controller alternates between searching_0 and searching_1, which rotate left and right to sweep blind zones in the camera field of view. Once the target is reacquired, the controller returns to running. Mission completion occurs when the normalized bounding-box size $O_f$8 exceeds an object-dependent success threshold; at that point the system sets $O_f$9 and $I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$0, after which the planner advances to the next instruction (Peng et al., 9 Jul 2025).

This state logic is also the system’s disturbance-recovery mechanism. If the robot is kicked or the target is displaced, object loss triggers search behavior; once reacquisition occurs, the L2MM resumes running. The paper attributes part of this robustness to LVF, which prevents unstable frames from causing false losses (Peng et al., 9 Jul 2025).

4. Hierarchical planning, dataset generation, and learning

LOVON’s hierarchical planning is driven by prompt-based LLM decomposition. The planner ingests $I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$1, $I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$2, and $I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$3, and emits an ordered list of instructions with explicit targets and action parameters. One example given is: “Approach the chair at 0.4 m/s; then find the backpack at 0.3 m/s; next follow the person at 0.5 m/s.” The planner may reconfigure tasks and monitor feedback to adapt order, speeds, or switching criteria (Peng et al., 9 Jul 2025).

The training and inference boundary is sharply defined. DeepSeek-R1 is used for planning and dataset generation but is not fine-tuned. YOLO-11 is pre-trained and used as-is. The trainable components are the L2MM and the IOE (Peng et al., 9 Jul 2025).

The training dataset contains 1M samples with a 4:1 train/test split and is generated in <15 minutes on CPU (Intel i9-12900KF) by combining synonym expansion for detection classes, instruction paraphrasing for language diversity, and category-specific success-threshold adaptation via the LLM (Peng et al., 9 Jul 2025). Training is performed on an RTX 3080 Ti with dropout $I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$4, learning rate $I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$5, batch size $I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$6, sequence length $I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$7, and $I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$8 epochs. The reported training times are approximately 1 hour for the L2MM and 30 minutes for the IOE (Peng et al., 9 Jul 2025).

The L2MM is optimized with a weighted motion loss and state-classification losses. The motion-vector term is

$I_{ins} = f_{LLM}(I_{sys}, T_l, O_f).$9

with $f_{IOE}$0, and the mission/search state terms use cross-entropy:

$f_{IOE}$1

An ablation study reports that larger models reduce $f_{IOE}$2 and $f_{IOE}$3 but with saturating gains, that too small a $f_{IOE}$4 undervalues motion loss, that too large a $f_{IOE}$5 harms state inference, and that [SEP] tokens are critical for separating modalities (Peng et al., 9 Jul 2025).

A noteworthy consequence of the data-generation pipeline is that the system achieves scalability without retraining the LLM or detector. This suggests a design in which language diversity is absorbed largely through synthetic supervision and instruction grounding, rather than by end-to-end optimization of all modules.

5. Evaluation, benchmarks, and empirical behavior

LOVON is evaluated in simulation on the Gym-UnrealCV benchmark, where the maximum episode length is 500 steps and the tracker visible region is a 90-degree sector with radius 750 cm (Peng et al., 9 Jul 2025). The paper reports Episode Length (EL) and Success Rate (SR) across 100 trials, and it also states the standard definition of SPL, although SPL is not used in the study.

Simulation baselines include DiMP, SARL, AD-VAT, AD-VAT+, TS, RSPT, EVT, TrackVLA, and LOVON. Averaged over ParkingLot, UrbanCity, UrbanRoad, SnowVillage, LOVON achieves near-perfect SR (≈1.00) and EL ≈500 in most scenes with only 1.5 hours of training, outperforming or matching state-of-the-art methods. The paper gives EVT: mean ≈487.75/0.94 and TrackVLA: 500/1.00 but 360 hours training as comparison points (Peng et al., 9 Jul 2025).

Real-world experiments focus on four regimes: open-world adaptation, long-horizon multi-goal tracking, dynamic tracking, and robustness to disturbances. The reported tasks include navigation to large objects such as cars, medium objects such as people, and small objects such as bags; sequential multi-goal tasks handled by the LLM planner; following moving targets across flat roads, spiral stairs, and wild grass; and quick relocalization after target movement or robot kicking (Peng et al., 9 Jul 2025).

The ablation results quantify the contribution of filtering and search-state design. For the “filter method and number of states” comparison, the paper contrasts Case 1 (3 states, no filtering), Case 2 (4 states, no filtering), and Case 3 (4 states + LVF) using the metrics Number of searching cycles $f_{IOE}$6 and search time $f_{IOE}$7 at 4 m and 6 m. For a Backpack at 6 m, the reported values are Case 1 $f_{IOE}$8, $f_{IOE}$9 s versus Case 3 $I_{object} \in C$0, $I_{object} \in C$1 s, which the paper summarizes as approximately 5× faster (Peng et al., 9 Jul 2025).

These empirical results support the paper’s claim that LVF plus dual-direction searching reduces search instability for hard-to-detect objects and that the combined architecture supports long-sequence tasks involving real-time detection, search, and navigation toward open-vocabulary dynamic targets (Peng et al., 9 Jul 2025).

6. Platforms, deployment characteristics, limitations, and related systems

LOVON is deployed on Unitree Go2, B2, and H1-2 and runs core on-board inference on NVIDIA Jetson Orin hardware (Peng et al., 9 Jul 2025). The perception stack uses the robots’ built-in cameras together with an Intel RealSense D435i; detection uses RGB streams, while the IMU is present for locomotion but is not explicitly used by LOVON’s perception. The detector, IOE, LVF, and L2MM run on-board at approximately the camera rate of 15 Hz. The paper does not provide explicit FPS or energy figures, although LVF and MAF are described as lightweight (Peng et al., 9 Jul 2025).

The platform abstraction is deliberately simple: LOVON emits generic velocity commands $I_{object} \in C$2 into the robot’s native velocity-control interface. The system is therefore described as plug-and-play, with portability arising from detector-independent IOE and LVF modules, generic velocity outputs, and reliance on pre-trained planner/detector plus a lightweight L2MM trained on synthetic and augmented data (Peng et al., 9 Jul 2025). This portability is demonstrated across the three reported legged platforms.

The paper also delineates important limitations. The planner may misinterpret ambiguous instructions, and the feedback loop is not foolproof. YOLO-11 is efficient but closed-set; synonym mapping mitigates vocabulary mismatch but cannot detect truly unseen classes, and the authors identify GroundingDINO or other open-vocabulary detectors as a future path. On-board LVF+YOLO+L2MM runs at approximately camera rate, but heavier LLMs may require off-board compute or smaller local models. Very fast, highly occluded targets stress reacquisition, and the vision-and-control-centric design does not address general obstacle avoidance, SLAM, or 3D obstacle reasoning, all of which are identified as future directions (Peng et al., 9 Jul 2025).

In the paper’s own related-work framing, LOVON is positioned against open-vocabulary object-goal navigation systems such as OVON/HM3D-OVON, VLM-Nav, and VLFMs, and against trackers such as TrackVLA. The stated differences are that LOVON focuses on legged robots in unstructured outdoor and indoor scenes, directly outputs velocities through a language-to-motion transformer without explicit map/SLAM, adds LVF for vision stabilization under legged motion, and demonstrates plug-and-play deployment across distinct legged platforms (Peng et al., 9 Jul 2025).

A naming clarification is also useful. LOVON should not be conflated with LOVO, an unrelated system for complex object queries in large-scale video datasets (Liu et al., 18 Jul 2025). The two works address different problem domains: LOVON concerns long-horizon embodied navigation for legged robots, whereas LOVO concerns open-vocabulary retrieval over video corpora.