LeCropFollow: Latent Space Planning for Navigation in Unstructured Crop Fields

Abstract: Unstructured navigational features, such as irregular planting or discontinuities, remain the primary failure mode for under-canopy agricultural robots. Existing geometric approaches often fail in these scenarios because they compress high-dimensional visual data into deterministic spatial references, effectively discarding the uncertainty and semantic context required to navigate ambiguous terrain. To address this, we present LeCropFollow, a visual navigation framework that bypasses explicit geometric modeling in favor of a learned latent representation. By integrating a self-supervised semantic heatmap extractor with TD-MPC2, a Model-Based Reinforcement Learning (MBRL) planner, our system optimizes trajectories directly within a latent manifold. The framework operates over the uncompressed heatmap signal, preserving the semantic context that geometric reductions discard. We demonstrate that this representational shift enables zero-shot transfer from simplified simulation to the physical world without fine-tuning. Extensive field experiments in late-stage corn fields show that LeCropFollow matches state-of-the-art baselines in unstructured rows but significantly outperforms them in plantation gaps, achieving a 2.4x reduction in semantic failures compared to keypoint-based methods. These results suggest that latent planning offers a robust alternative to geometric estimation for operations in heterogeneous agricultural environments. Code, models, and data available: https://felipe-tommaselli.github.io/lecropfollow .

• The paper introduces a latent space planning framework that bypasses explicit geometric state estimation for improved navigation in unstructured crop fields.
• It employs model-based reinforcement learning with full semantic heatmap inputs, enabling robust trajectory optimization in challenging environments.
• Experimental results show state-of-the-art gap traversal and a significant reduction in semantic failures compared to traditional geometry-based methods.

LeCropFollow: Latent Space Planning for Navigation in Unstructured Crop Fields

Introduction and Motivation

Autonomous ground robots for under-canopy agricultural navigation face significant challenges due to the unstructured and discontinuous nature of typical crop environments. While current onboard perception frameworks leverage LiDAR or camera data, classical path-following pipelines rely on geometric feature extraction (e.g., row lines, vanishing points) to define spatial references for downstream control. This geometric compression, although computationally tractable and effective in uniform fields, leads to high failure rates in highly occluded or structurally irregular sections, such as gaps or severe erosion, due to a deterministic over-reduction of the underlying semantic context.

The paper "LeCropFollow: Latent Space Planning for Navigation in Unstructured Crop Fields" (2606.31941) systematically addresses this limitation by proposing a navigation framework that discards explicit geometric state estimation. Instead, it operates entirely in the latent space of uncompressed semantic heatmaps, leveraging a model-based reinforcement learning (MBRL) architecture for real-time trajectory optimization.

Figure 1: Schematic view of the LeCropFollow framework, which uses learned latent world models and uncompressed heatmap signals for navigation in unstructured agricultural fields—bypassing GNSS reliance.

Methodological Framework

LeCropFollow couples a fixed, self-supervised semantic perception backbone—RowFollowNet—with a TD-MPC2-based latent world model and planning stack. The input pipeline accepts monocular RGB camera observations and processes them through the frozen backbone to yield pixelwise semantic heatmaps for left row, right row, and vanishing point classes. Critically, these three-channel heatmaps are fed without further reduction into a learned encoder, maintaining the full spatial uncertainty and contextual information arising from ambiguous or occluded visual signals.

Figure 2: System overview: RGB inputs pass through a frozen semantic backbone to generate heatmaps which, concatenated with previous actions, form the encoder input for latent space planning and control.

Successive pipeline components—latent encoder, policy prior, world model, reward, and value function—are trained in simulation using RL. The action space consists of linear and angular velocities for a unicycle model robot, and the reward structure prioritizes safety and stability over blind velocity tracking, incorporating gating terms for collision and actuation constraint violations.

The framework structures the navigation task as a finite-horizon planning problem in a learned latent manifold, optimized online via Model Predictive Path Integral (MPPI) control over imagined rollouts. Only the first action of the planned sequence is executed, enabling receding horizon re-planning and robust correction under dynamic changes or perception dropouts.

Perceptual Architecture and Training Environment

Figure 3 illustrates the model’s capacity to encode uncertainty: in structured scenes, semantic peaks are sharp and localized; in occluded or degenerate cases, high spatial variance in the heatmap directly communicates signal ambiguity.

Figure 3: Contrasting high-confidence and high-uncertainty scenarios; structured rows yield sharp Gaussian peaks, while occlusions produce diffuse heatmap predictions ingested directly by the encoder.

Simulation training is conducted in Gazebo using minimally realistic crops (randomly colored geometric cylinders). This presents the perception stack with intentionally noisy and out-of-domain data, enforcing invariance and robustness before deployment. Episodes terminate on collision or completion of a fixed forward traversal, and all downstream models are exposed to the entirety of the heatmap activations, driving the policy to exploit these raw cues.

Figure 4: The simulated training environment, highlighting both the geometric arrangement (left) and the perception overlay (right) with egocentric RGB and heatmap signals.

Experimental Validation and Empirical Results

Field trials were conducted on a skid-steer TerraSentia robot in late-stage corn environments encompassing both Flowering and Harvested phases. Baseline comparisons included CropFollow++ (vision-only, keypoint-based) and CROW (LiDAR+vision). Critically, both baselines depend on explicit geometry extraction (either from heatmap mode estimation or LiDAR-inertial-odometry fusion), whereas LeCropFollow operates in a purely reactive manner without odometry or LiDAR, consuming only monocular visual observations and previous actions.

Validation environments involved navigation across multiple row types and particularly emphasized unstructured regimes, notably an 8.7 m plantation gap with complete left-row degradation.

Figure 5: Experimental field environments used for evaluation, marking the traversed rows and the unstructured gap for ablation studies.

Unstructured Resiliency and Gap Traversal

In structured settings, all methods demonstrated similar maximum gap-free traversal distances, supporting the parity of latent-space planning and high-performance geometric state estimation under controlled conditions. However, in the critical unstructured gap regime, the distinction is pronounced. LeCropFollow traversed the gap in 14 out of 15 runs (93.3%), compared to substantial failures for CropFollow++ (6.7%) and partial successes for CROW (53.3%). Ablation studies confirm that the benefit arises directly from planning over the learned world model with full, unreduced heatmap input: horizon ablation or replacement with deterministic prior rapidly degraded results.

Figure 6: Gap traversal ablation—collisions per run for geometric and learned method variants, highlighting the performance drop from horizon or planner ablations.

Failure Mode and Robustness Analysis

A systematic breakdown of collision causes reveals a dominant reduction (>2.4×) in semantic failures for LeCropFollow over baselines. Perception and occlusion-induced errors, which dominate in keypoint-centric methods due to their forced geometric fits under ambiguity, are efficiently mitigated by direct consumption of the full heatmap dispersion. The residual failure rate is almost entirely attributable to physical/mechanical phenomena (e.g., terrain-induced actuation limit violations or bad start states), rather than representational deficiencies.

Figure 7: Failure analysis: Distribution and categorization of collision causes, showing strong reduction of semantic failures using LeCropFollow relative to geometric baselines.

Theoretical and Practical Implications

This work offers a strong empirical argument against over-compressed geometric state pipelines in unstructured agricultural field domains. By retaining high-dimensional, uncertainty-preserving visual representations downstream of perception, LeCropFollow achieves robust gap traversal and generalizes in zero-shot transfer to physical environments from simulation, with no fine-tuning. The results suggest that planning dynamics over information-rich latent manifolds, as opposed to explicit geometric cues, affords improved robustness, particularly in rare-event scenarios and ambiguous visual contexts.

From an AI and robotics perspective, these findings emphasize the critical role of uncertainty propagation and distributional representation in closed-loop control under partial observability. This provides support for broader adoption of latent world model planning—especially in highly stochastic and environmentally diverse settings—while highlighting the limitations of hand-engineered geometric reductions for real-world navigation components.

Future work should incorporate learned or model-based closed-loop actuation constraints into the end-to-end framework, expand the perception input (e.g., depth, event data), and validate in multi-agent and multi-class crop morphologies to characterize scalability and sample efficiency.

Conclusion

LeCropFollow establishes a robust, MBRL-driven framework for autonomous navigation in challenging agricultural environments. By discarding deterministic geometric state estimation in favor of full spatial heatmap propagation into a latent model-based planner, the system achieves state-of-the-art gap traversal, significant semantic error reduction, and zero-shot sim-to-real transfer in late-season field deployments. This work motivates future research on distributional reinforcement learning, uncertainty-driven planning, and advanced latent visual representation integration for unstructured field robotics.