Vision-Language Navigation for Agricultural Robots

• The paper introduced AgriVLN and the A2A benchmark to bridge the gap between conventional VLN datasets and real agricultural environments.
• It decomposes lengthy, realistic instructions into atomic subtasks using an interpretable Subtask List (STL) to guide low-level robot control.
• Subsequent extensions, like T-araVLN and MDE-AgriVLN, enhance navigation performance by addressing spatial perception challenges and instruction ambiguity.

Vision-and-Language Navigation for Agricultural Robots (AgriVLN) is an agricultural adaptation of vision-and-language navigation in which a robot receives natural-language navigation instructions and front-facing RGB observations, then outputs low-level control actions to reach a target position in real agricultural environments. The line was established by "AgriVLN: Vision-and-Language Navigation for Agricultural Robots," which introduced both the AgriVLN baseline and the A2A benchmark as a domain-specific testbed for farm, greenhouse, forest, mountain, garden, and village scenes captured from a low-mounted quadruped platform (Zhao et al., 10 Aug 2025). Subsequent work extended the setting toward instruction refinement, depth-aware reasoning, spatial memory reuse, and robustness to mistaken instructions, yielding a compact but rapidly developing research area within agricultural robotics (Zhao et al., 8 Sep 2025, Zhao et al., 3 Dec 2025, Zhao et al., 16 Oct 2025, Zhao et al., 1 Jun 2026).

1. Domain-specific problem setting

AgriVLN was proposed in response to a mismatch between mainstream VLN benchmarks and actual agricultural deployment conditions. The motivating observation is that agricultural robots are already used in tasks such as phenotyping, spraying, and harvesting, yet still heavily rely on manual driving or fixed rail systems, limiting mobility and adaptability. Existing VLN benchmarks were developed mainly for indoor homes, streets, or aerial scenes: R2R, REVERIE, and VLN-CE use residence-like environments with panoramic or RGB-D sensing and camera heights around $1.5\,\mathrm{m}$; TouchDown uses street-view data from a car-mounted camera at about $2.5\,\mathrm{m}$; AerialVLN uses synthetic aerial data from drones. By contrast, agricultural robots typically operate with a single low front-facing RGB camera mounted near $0.38\,\mathrm{m}$, in outdoor scenes with clutter, lighting variation, and irregular terrain (Zhao et al., 10 Aug 2025).

The original AgriVLN paper also argues that imitation-learning and reinforcement-learning VLN methods often generalize poorly to unseen environments and are hard to interpret, which makes them less suitable for the variety and unpredictability of real farms. This suggests that agricultural VLN is not merely a domain-transfer problem from indoor navigation, but a reconfiguration of sensing geometry, scene structure, and instruction style. In that framing, AgriVLN is both a benchmark problem and a systems question about how to ground long, practical, spoken-like directions in low-viewpoint agricultural perception (Zhao et al., 10 Aug 2025).

2. A2A benchmark and formal task definition

The A2A benchmark, short for Agriculture to Agriculture, defines AgriVLN as a continuous navigation task. It contains $1{,}560$ evaluation episodes across six scene classes: $372$ farm episodes, $258$ greenhouse, $384$ forest, $198$ mountain, $258$ garden, and $90$ village. Data were collected using a Unitree Go2 Air quadruped robot. In each episode, experts manually drove the robot along the best route from start to destination while recording the built-in front-facing RGB stream at $2.5\,\mathrm{m}$0 resolution, around $2.5\,\mathrm{m}$1 FPS, from a camera height of $2.5\,\mathrm{m}$2. Ground-truth actions are annotated at each time step as one of four low-level controls: FORWARD, LEFT ROTATE, RIGHT ROTATE, and [STOP](https://www.emergentmind.com/topics/self-taught-optimizer-stop); consecutive identical actions are clustered into temporal intervals and stored in JSON (Zhao et al., 10 Aug 2025).

Formally, given an instruction

$2.5\,\mathrm{m}$3

and the current RGB frame $2.5\,\mathrm{m}$4, the agent must output

$2.5\,\mathrm{m}$5

The instructions are explicitly long and realistic. Their lengths range from $2.5\,\mathrm{m}$6 to $2.5\,\mathrm{m}$7 words; the benchmark description reports an average length of $2.5\,\mathrm{m}$8, while another comparison setting in the paper reports $2.5\,\mathrm{m}$9. The average trajectory length is about $0.38\,\mathrm{m}$0, and each episode contains on average $0.38\,\mathrm{m}$1 subtasks, with subtask counts ranging from $0.38\,\mathrm{m}$2 to $0.38\,\mathrm{m}$3. A2A is evaluation-only rather than a training corpus, which is why the original paper treats its size as sufficient for zero-shot-style assessment even though it is smaller than older VLN datasets (Zhao et al., 10 Aug 2025).

Evaluation in the original benchmark uses Success Rate (SR) and Navigation Error (NE), and also introduces Independent Success Rate (ISR) to measure how often subtasks are solved individually. Random performance is reported as near useless, while human control is near perfect, supporting the claim that A2A is both difficult and valid as an evaluation environment (Zhao et al., 10 Aug 2025).

3. Baseline AgriVLN architecture and subtask decomposition

The baseline AgriVLN system is intentionally lightweight. Rather than training a large end-to-end policy from scratch, it uses a vision-LLM as a decision-making module,

$0.38\,\mathrm{m}$4

where $0.38\,\mathrm{m}$5 is the prompt for decision making, $0.38\,\mathrm{m}$6 is a state transition, and $0.38\,\mathrm{m}$7 is the model’s reasoning. In the reported experiments, GPT-4.1 mini is used both as the instruction decomposition model and as the decision-making model, accessed through API. The system is prompted with carefully designed templates so that it can understand both the current scene and the navigation context, then emit a low-level control action rather than a high-level abstract plan (Zhao et al., 10 Aug 2025).

A central contribution is the Subtask List (STL) module, introduced to address the failure of single-prompt reasoning on long instructions. STL decomposes the full instruction into atomic subtasks,

$0.38\,\mathrm{m}$8

with each subtask represented as

$0.38\,\mathrm{m}$9

Here $1{,}560$0 is the step number, $1{,}560$1 the textual description, $1{,}560$2 the start condition, $1{,}560$3 the end condition, and $1{,}560$4 the state. The state set is pending, doing, and done. The decomposition prompt follows three principles: the particle principle, requiring each subtask to be indivisible; the synonymity principle, requiring semantic equivalence between the original instruction and the resulting subtask list, written as $1{,}560$5; and the connection principle, requiring continuity, written as $1{,}560$6. During execution, if no subtask is in the doing state, the agent focuses on the first pending one; otherwise it focuses on the current doing subtask. The stated benefit is that the model attends to one actionable unit at a time instead of reprocessing the entire long instruction at every time step (Zhao et al., 10 Aug 2025).

This design makes interpretability an explicit part of the baseline. The paper presents STL as analogous to a human to-do list, and qualitatively reports that AgriVLN converts an instruction into ordered subtasks, then updates states from pending to doing to done as navigation progresses. That interpretability claim is integral to the method’s positioning against less transparent policy-learning alternatives (Zhao et al., 10 Aug 2025).

4. Reported performance, scene effects, and failure characteristics

On the full A2A benchmark, the GPT-4.1 mini baseline without STL achieves $1{,}560$7 and $1{,}560$8, whereas AgriVLN with STL improves to $1{,}560$9 and $372$0. On the challenging subset with at least three subtasks, the baseline drops to $372$1, while AgriVLN with STL reaches $372$2. In comparison with prior instruction-decomposition methods from VLN, SIA-VLN reports $372$3 and $372$4, DILLM-VLN reports $372$5 and $372$6, and AgriVLN reports the best overall SR at $372$7 with $372$8. The ablations show that removing STL sharply hurts performance, especially for instructions with three or more subtasks, where performance can collapse to $372$9 in one setting. The same study also reports that GPT-4.1 mini outperforms Gemini-1.5 Flash and Llama-4 Maverick in this setup (Zhao et al., 10 Aug 2025).

The original paper emphasizes that the higher SR with slightly worse NE reflects a conservative navigation style: the model tends to stop safely rather than risk overshooting. This is an important interpretive point because it clarifies that SR and NE do not move monotonically together in AgriVLN. Scene type also matters. Performance varies significantly across farm, greenhouse, forest, mountain, garden, and village scenes, likely because of differences in clutter, obstacle density, and lighting; one reported scene-wise breakdown gives SR values ranging from $258$0 in farms down to $258$1 in forests. The main failure cases identified are ambiguous instructions and inaccurate estimation of spatial distance (Zhao et al., 10 Aug 2025).

A common misconception is that AgriVLN’s main bottleneck is linguistic parsing alone. The empirical record is more specific: long instructions degrade step-tracking; scene clutter affects perception; and distance estimation remains unstable under monocular, low-mounted sensing. The benchmark therefore exposes a coupled failure surface rather than a single-language bottleneck (Zhao et al., 10 Aug 2025).

5. Major extensions of the AgriVLN line

Later papers preserve the core agricultural VLN formulation while targeting different failure modes: instruction ambiguity, weak monocular spatial perception, lack of cross-episode memory, and mistaken human instructions.

Method	Added module	Headline reported result
T-araVLN	Instruction Translator	A2A: SR $258$2, NE $258$3
MDE-AgriVLN	Monocular Depth Estimation	A2A: SR $258$4, NE $258$5
SUM-AgriVLN	Spatial Understanding Memory	A2A: SR $258$6, NE $258$7
IMAC-AgriVLN	Instruction Mistake Awareness and Correction	A2A-MI: SR $258$8, NE $258$9, AR $384$0

T-araVLN introduces an Instruction Translator that rewrites a raw instruction $384$1 into a refined instruction $384$2 before navigation. The translator is guided by five principles: Inessential Removal, Errors Revision, High-Low Separation, Representational Rotation, and Representational Movement. On full A2A, the GPT-4.1-based translator improves performance from AgriVLN’s $384$3, $384$4 to $384$5, $384$6; on the two-subtask subset, it reaches $384$7 and $384$8 (Zhao et al., 8 Sep 2025).

MDE-AgriVLN targets spatial reasoning under monocular perception by inserting a monocular depth estimation module between the RGB input and the VLM decision process. It represents depth both as a coarse depth matrix in meters and as a pseudo-color depth map, using Depth Pro as the selected estimator in the final system. In that paper’s local-model setup, the AgriVLN baseline is reported at $384$9, $198$0, while MDE-AgriVLN with Depth Matrix reaches $198$1, $198$2. The paper also introduces average prompt-token and completion-token usage as reasoning-cost measures, and identifies cross-modal misalignment and $198$3 to $198$4 seconds of additional latency per time step as main limitations (Zhao et al., 3 Dec 2025).

SUM-AgriVLN addresses the fact that practical agricultural navigation may involve repeated instructions in the same scene, whereas baseline AgriVLN treats episodes independently. Its SUM module reconstructs 3D scene geometry from sampled frames using VGGT, processes the reconstruction with trimesh, and stores frontal and oblique memory views in a Spatial Memory Bank. On full A2A, SUM-AgriVLN improves SR from $198$5 to $198$6 with a slight NE increase from $198$7 to $198$8. The oblique-memory variant is the representative model in the main comparison, and the paper explicitly notes failure modes including ghosting in dynamic scenes, the 2D memory bottleneck, the need for pre-exploration, and diminishing gains on highly complex tasks (Zhao et al., 16 Oct 2025).

IMAC-AgriVLN moves the line into a more realistic robustness setting by dropping the assumption that instructions are always correct. It introduces A2A-MI, which keeps the same scenes and images as A2A but inserts three mistake classifications into each instruction—descriptive adjective, concrete noun, and behavioral verb—tripling the evaluation set to $198$9 instructions. For the AgriVLN baseline in that protocol, performance drops from $258$0, $258$1 on clean A2A to $258$2, $258$3 on mistaken instructions, corresponding to about $258$4 on SR and $258$5 on NE on average. IMAC analyzes the instruction and current image to detect and minimally correct mistakes, improving the mistaken-instruction result to $258$6, $258$7, with average Awareness Rate $258$8 (Zhao et al., 1 Jun 2026).

6. Conceptual position within agricultural robot navigation

AgriVLN occupies a distinct place within agricultural navigation because it combines language grounding with low-level control in realistic field imagery. It differs from adjacent vision-based navigation systems that do not use language. A representative example is the crop-row exit work "Leaving the Lines Behind," which decomposes row switching into three subtasks—exiting the current crop row, turning around towards the field, and re-entering the next crop row—and focuses on the first stage using RGB local feature matching and a depth-based headland distance estimate. That system uses SIFT descriptors, KNN matching with $258$9, a ratio-test threshold of $90$0, and halts when $90$1; it reports a $90$2 median absolute error in Stage 1, $90$3 median absolute error in soil headland, $90$4 in verdant headland, and a $90$5 error margin for complete row exit into the headland (Silva et al., 2023).

The relationship between that line and AgriVLN is complementary rather than competitive. The crop-row exit paper does not study language, but it shows that agricultural navigation can benefit from explicit state decomposition, localized visual correspondence, and geometry-aware stopping conditions. A plausible implication is that future AgriVLN systems may combine language-conditioned subtask planning with specialized visual or geometric controllers for distance-sensitive maneuvers, especially in structured row and headland transitions (Silva et al., 2023).

Several limitations recur across the AgriVLN literature. A2A is evaluation-only and relatively small compared with some general VLN datasets, so it functions as a focused domain benchmark rather than a broad-scale training corpus. Robustness remains incomplete: the original paper identifies ambiguous instructions and poor spatial distance estimation; IMAC-AgriVLN shows that agents often assume instructions are correct even when the scene conflicts with the text; MDE-AgriVLN shows that monocular depth can help but introduces cross-modal alignment and latency issues; and SUM-AgriVLN depends on mostly static scenes and an initial exploration phase (Zhao et al., 10 Aug 2025, Zhao et al., 1 Jun 2026). Taken together, these results indicate that agricultural VLN is best understood as a systems problem spanning instruction structure, geometric perception, memory, and fault-tolerant language grounding rather than a narrow benchmark variant of indoor VLN.