DeepMEL: End-to-End Visual Teach & Repeat

• DeepMEL is a deep-learning system that integrates stereo visual odometry and path-relative localization via end-to-end relative pose regression, eliminating explicit feature matching.
• It employs a modified AlexNet backbone with spatial pyramid pooling to extract fixed-size features from 12-channel stereo images, enhancing robustness across lighting and seasonal changes.
• Experimental results on outdoor driving datasets show that DeepMEL achieves VO and localization accuracy comparable to VT&R even under dramatic day-night and winter-spring transitions.

Searching arXiv for the DeepMEL paper and closely related VT&R work. DeepMEL is a deep-learning incarnation of the Stereo Visual Teach and Repeat (VT{R}) pipeline for vision-based path following, designed to replace hand-engineered feature matching and multi-experience selection with a single end-to-end neural network. It learns to regress 3-DOF relative motions for both visual odometry (VO) and path-relative localization by harvesting ground-truth relative poses from a spatio-temporal VT{R} pose graph built using stereo VO and multi-experience localization across lighting, weather, and seasonal change. Experiments on two outdoor driving datasets indicate that the system achieves VO and localization accuracy comparable to VT{R} itself, while localizing directly across radically different conditions such as day↔night and winter↔spring without intermediate “bridging” experiences (Gridseth et al., 2020).

1. Position within vision-based path following

Vision-based path following allows robots to autonomously repeat manually taught paths. In the VT{R} setting, Stereo Visual Teach and Repeat accomplishes accurate and robust long-range path following in unstructured outdoor environments across changing lighting, weather, and seasons by relying on colour-constant imaging and multi-experience localization. DeepMEL inherits that operational context but shifts the computational burden from a hand-engineered pipeline to learned relative-pose regression (Gridseth et al., 2020).

The central design objective is unification. Rather than maintaining separate mechanisms for stereo VO, feature correspondence, and experience selection, DeepMEL uses the same overall architecture for both VO and localization, training two separate instances for the two tasks. This suggests a compilation of the behavior of a spatio-temporal localization system into a supervised regressor whose labels already encode the environmental invariances discovered by VT{R}. A plausible implication is that the model internalizes cross-condition alignment from the pose graph supervision rather than from an explicit matching module.

The target state variable is a robot-frame 3-vector,

$\xi = [x,y,\theta]^T,$

representing forward offset $x$, lateral offset $y$, and yaw change $\theta$. In this formulation, VO predicts the relative motion between temporally adjacent keyframes, while localization predicts the relative pose of the live frame with respect to the teach map.

2. Network architecture and representation

DeepMEL takes as input a pair of stereo keyframes, each consisting of four RGB images—left/right and reference/target—yielding a 12-channel image tensor of size $512 \times 384 \times 12$. Before entering the network, images are converted to colour-constant representations per Paton et al. (2015) to normalize illumination statistics prior to network input (Gridseth et al., 2020).

The backbone adopts the convolutional feature extractor of AlexNet, specifically conv1–conv5, with modified channel dimensions and strides:

• conv1: 12→96 channels, kernel $11 \times 11$, stride 4
• conv2: 96→256 channels, kernel $5 \times 5$, stride 1
• conv3: 256→384 channels, kernel $3 \times 3$, stride 1
• conv4: 384→384 channels, kernel $3 \times 3$, stride 1
• conv5: 384→256 channels, kernel $3 \times 3$, stride 1

After conv5, a Spatial Pyramid Pooling (SPP) layer collates feature maps into a fixed 4-level representation with $x$0, $x$1, $x$2, and $x$3 bins, preserving spatial information without warping to $x$4. Fully connected layers mirror AlexNet’s FC6 (4096), FC7 (4096), and FC8 (1000), followed by a final regression head of dimension 3 (Gridseth et al., 2020).

Two task-specific instantiations are trained. The VO head is trained on temporally adjacent keyframes to predict the relative motion. The localization head is trained on keyframe pairs sampled across different experiences, for example winter versus spring, to predict the relative pose of the live frame with respect to the teach map. The architecture is otherwise the same for both tasks. This division preserves a common representational template while separating supervision regimes.

3. Relative-pose regression and optimization

For VO, given two stereo keyframes at times $x$5 and $x$6, DeepMEL predicts the 3-DOF motion $x$7. The corresponding relative transformation in $x$8 is written as

$x$9

This explicit parameterization constrains the prediction target to planar motion with yaw, consistent with path-following on outdoor vehicle trajectories (Gridseth et al., 2020).

Training uses a weighted squared-error loss,

$y$0

with

$y$1

so that yaw error receives 10× the weight assigned to $y$2 and $y$3. No explicit cross-season or cross-condition losses are added; multi-experience alignment is encoded implicitly in the training labels. This suggests that generalization across appearance shifts is expected to emerge from supervised relative-pose consistency rather than from a dedicated domain-alignment objective.

The optimization procedure uses Adam with $y$4 and $y$5, learning rate $y$6, and batch size 64. Early stopping on validation loss determines the number of epochs, approximately 20–30. The implementation is in PyTorch on an NVIDIA GTX-1080 Ti, with inference at at least 50 Hz (Gridseth et al., 2020).

4. Datasets, supervision, and training regime

The training and evaluation study is conducted exclusively on datasets. Two outdoor driving datasets are used: UTIAS In The Dark and UTIAS Multi-Season (Gridseth et al., 2020).

Dataset	Path and repeats	Conditions
UTIAS In The Dark	250 m path; repeated hourly over 24 h; five held-out repeats for testing	bright day, sun-flare, evening, night with headlights
UTIAS Multi-Season	160 m off-road path; eight held-out repeats	winter↔spring transition; heavy snow to bare ground

Labels are generated from the VT{R} spatio-temporal pose graph. VO labels are sampled from temporal edges between consecutive keyframes. Localization labels are sampled between each repeat keyframe and its corresponding teach keyframe via VT{R} localization, including cases where appearances differ drastically. In this sense, the pose graph functions as a supervisory compiler: the geometric and cross-experience correspondences discovered by VT{R} are transformed into direct regression targets.

No additional geometric or photometric augmentations are applied beyond the colour-constant image conversion inherited from VT{R}. This absence of extra augmentation is notable because it places the burden of invariance on two factors already present in the system: the colour-constant preprocessing and the diversity of the training experiences. A plausible implication is that the reported cross-condition performance is closely tied to the structure of the pose-graph labels and the environmental variability captured during data collection.

5. Empirical performance and closed-loop behavior

Evaluation reports RMSE in meters for $y$7 and $y$8 and in radians for $y$9 over test repeats. Representative VO and localization performance is summarized directly in the reported tables (Gridseth et al., 2020).

On UTIAS In The Dark, VO during daytime yields $\theta$0 m, $\theta$1 m, $\theta$2 rad, while VO at night yields $\theta$3 m, $\theta$4 m, $\theta$5 rad. Localization from day→night yields $\theta$6 m, $\theta$7 m, $\theta$8 rad. On UTIAS Multi-Season, VO under sunny, snow conditions yields $\theta$9 m, $512 \times 384 \times 12$0 m, $512 \times 384 \times 12$1 rad; VO under overcast, no snow yields $512 \times 384 \times 12$2 m, $512 \times 384 \times 12$3 m, $512 \times 384 \times 12$4 rad. Localization from winter→spring reaches up to $512 \times 384 \times 12$5 m, $512 \times 384 \times 12$6 m, $512 \times 384 \times 12$7 rad in the worst case, with typical values below $512 \times 384 \times 12$8 m, below $512 \times 384 \times 12$9 m, and below $11 \times 11$0 rad.

The comparison with Stereo VT{R} is framed in two ways. First, VT{R} itself reports centimeter-level VO/localization over kilometer scales, while DeepMEL achieves comparable VO RMSE of 5–15 cm on 160–250 m paths. Second, unlike VT{R}, which requires intermediate mapping “experiences” to bridge large appearance gaps, DeepMEL localizes directly across day↔night and winter↔spring with no path re-collection. The second point is especially significant because it identifies a capability that the hand-engineered system does not have in the reported setting.

For in-loop path following, DeepMEL alternates network-based VO propagation with weight 0.3 and localization corrections with weight 0.7. Under this scheme, it successfully tracks taught paths in closed-loop simulation. Lateral and heading errors remain within VT{R} tolerances, approximately 10–20 cm and 0.1–0.3 rad, even in worst-case cross-condition scenarios (Gridseth et al., 2020).

6. Scope, limitations, and interpretive significance

The principal conclusion is that a single DNN can subsume both stereo VO and path-relative localization over dramatic environmental change. The learned model achieves “in-the-loop” path-following precision comparable to the heavily engineered VT{R} pipeline. By regressing relative poses directly, DeepMEL obviates the need for explicit feature matching, experience selection, or intermediate bridging maps under radically varying illumination, weather, and season (Gridseth et al., 2020).

The limitations stated for the system are also specific. Current models are trained per path and may not generalize to unseen routes, and performance depends on the diversity of training experiences. Future work is identified as zero-shot transfer to new paths and full on-robot closed-loop evaluation. These limitations bound the interpretation of the results: the reported generalization is across conditions on known routes, not across arbitrary new environments.

A common misconception would be to treat the removal of explicit feature matching as evidence that geometric structure is no longer central. The reported workflow suggests the opposite. Geometry remains foundational, but it is displaced from inference into supervision: the VT{R} spatio-temporal pose graph supplies the relative-pose labels, and the network learns to approximate that geometric localization behavior directly.

7. Name collision and disambiguation

“DeepMEL” also denotes an unrelated framework introduced later for Multimodal Entity Linking. That system is based on multi-agent collaborative reasoning, integrates four specialized agents—Modal-Fuser, Candidate-Adapter, Entity-Clozer, and Role-Orchestrator—and targets end-to-end cross-modal linking in multimodal knowledge graphs rather than robotic localization (Wang et al., 21 Aug 2025).

The two uses of the name refer to distinct research programs. In the robotics context, DeepMEL denotes a relative-pose regression model derived from Stereo Visual Teach and Repeat. In the later language-and-vision context, DeepMEL denotes a multi-agent collaboration framework for multimodal entity linking. The shared acronym therefore should not be read as implying methodological continuity between the two systems.