FOMO25 Challenge: Cross-Season SLAM Benchmark

• FOMO25 Challenge is a benchmarking initiative to evaluate odometry, SLAM, and localization algorithms under extreme seasonal and environmental variations.
• It leverages a comprehensive, multi-sensor dataset with rigorous calibration and diverse terrain scenarios to stress-test UGV navigation.
• Baseline analyses highlight modality-specific strengths and limitations, emphasizing the need for adaptive sensor fusion for robust, long-term navigation.

The FOMO25 Challenge is a research benchmarking initiative designed to rigorously evaluate odometry, SLAM (Simultaneous Localization and Mapping), and cross-season localization algorithms using the FoMo dataset—a comprehensive, multi-sensor, multi-season dataset recorded over a full year in Forêt Montmorency (FoMo), a boreal forest in Quebec, Canada. Targeting the development and assessment of long-term, all-weather robot navigation methodologies, FOMO25 addresses the core challenge of maintaining robust performance under severe seasonal and environmental changes, including deep snow, heavy foliage, and variable terrain (Boxan et al., 9 Mar 2026).

1. Dataset Characteristics and Calibration

The FoMo dataset comprises 64 km of UGV (Uncrewed Ground Vehicle) driving across six distinct routes, each ranging from 300 m to 2.2 km, and sampled in 12 deployments between November 2024 and November 2025. These deployments systematically span extreme conditions: winter (snow depths exceeding 1 m, temperatures down to –20 °C), spring thaw, summer growth (dense vegetation), fall leaf drop, and even night-rain events. Each route was traversed repeatedly to facilitate intra- and inter-sequence cross-seasonal analysis.

The UGV platform is equipped with a dense sensor suite:

• Rotating 3D lidar (RoboSense Ruby Plus, 10 Hz)
• Hybrid solid-state lidar (Leishen LS128S1, 10 Hz)
• FMCW radar (Navtech CIR-304H, 4 Hz)
• Stereo camera (ZED X, 1920×1200 px, 10 Hz)
• Wide-angle monocular camera (Basler ace2, 1920×1200 px, 10 Hz)
• Dual IMUs (Xsens MTi-30 and VectorNav VN100, 200 Hz)
• Wheel-encoder odometry, barometric pressure sensor (100 Hz), microphones (16 kHz)
• Current and voltage monitors

Comprehensive calibration is documented:

• Intrinsics: IMUs via 24 h Allan-variance; Basler with OpenCV checkerboard; ZED X via manufacturer data.
• Extrinsics: Lidar↔Lidar using RANSAC-circle plus ICP; Camera↔Lidar using feature matching (SuperGlue) and chained auxiliary sensors; Radar↔Lidar via SE(2) ICP; IMU↔Camera through batch optimization (Kalibr) with a crane-mounted rig for full 6 DoF excitation.
• Ground Truth: Provided by three Emlid Reach M2 GNSS antennas (UGV) and a static Emlid Reach RS3 base station, PPK-corrected against a Quebec City CORS. Multi-antenna outputs are fused using point-to-Gaussian minimization (see Algorithm 1 in (Boxan et al., 9 Mar 2026)), yielding full 3D TUM-format pose sequences (with timestamp, $\mathbf{x},\mathbf{q}$ and per-timestamp covariance).

Associated metadata includes weather station logs (temperature, snow depth), motor currents (up to 300 A), wheel velocity commands, camera gain logs, and raw sensor streams.

2. Environmental, Seasonal, and Terrain Variation

FOMO25 emphasizes environmental diversity and abrupt appearance changes:

• Snow cover ranges from 0 to over 1 m (with roadside banks above 3 m and single-storm accumulations up to 40 cm overnight). Five out of twelve deployments feature significant snow cover.
• Vegetation: Summer deployments have dense growth impeding camera and GNSS line-of-sight; fall brings substantial appearance changes due to leaf drop.
• Terrain: Includes paved/unpaved roads, steep vegetated hills, rocky quarries, and creek crossings. The UGV alternates between wheels (summer/fall) and tracks (winter) to cope with snow, but still experiences immobilization and wheel slip.
• Perception impact: Lidar scans are markedly altered by snowbanks (impacting ICP-based odometry), camera-based methods struggle with foliage and lighting changes, and radar attenuates the effect of snow/vegetation but suffers from 2D ground geometry limitations. A plausible implication is that multi-modal sensor fusion and adaptive algorithms are required for robust cross-season performance.

3. Benchmarking Tasks and Protocols

The challenge defines a flexible set of allowed tasks, all grounded in cross-seasonal robustness:

• Geometric Odometry: Trajectory estimation (poses over time) from raw sensor data, map-free.
• SLAM/Mapping: Construction of a deployment-specific map (e.g., point cloud) from sensor streams.
• Cross-Seasonal Localization: Localize the UGV in one deployment using a map built in another (e.g., winter-built map, summer operation).
• Long-Term Re-Localization: Evaluate generalization when only cross-seasonal sensor data is available, with no access to GNSS.

The open scope allows teams to focus on single-run odometry or full SLAM, but all methods are expected to demonstrate robustness to appearance and terrain domain shifts.

4. Evaluation Metrics and Submission Procedure

All challenge submissions are assessed with respect to the provided ground truth using standard metrics:

• Absolute Trajectory Error (ATE):

$$\mathrm{ATE} = \sqrt{ \frac{1}{N} \sum_{i=1}^N \| \mathbf{p}_i^{\mathrm{est}} - \mathbf{p}_i^{\mathrm{gt}} \|^2 }$$

where $N$ is the number of poses.

• Relative Pose Error (RPE) (for segment drift, window $\Delta$ steps):

$$\mathrm{RPE}(\Delta) = \sqrt{ \frac{1}{M} \sum_{j=1}^M \| ( \mathbf{p}_{j+\Delta}^{\mathrm{est}} \ominus \mathbf{p}_{j}^{\mathrm{est}} )^{-1} ( \mathbf{p}_{j+\Delta}^{\mathrm{gt}} \ominus \mathbf{p}_{j}^{\mathrm{gt}} ) \|^2 }$$

where $\ominus$ denotes the SE(3) relative pose operator.

• Translational Drift (%): Following KITTI conventions, translation drift is computed for sliding windows of 100–800 m.

Participants are required to:

• Register and download deployment-specific data subsets from https://fomo.norlab.ulaval.ca.
• For each deployment trajectory, submit a TUM-format file:

  <sequence_name>_est.txt

  with lines containing “t x y z qx qy qz qw,” and optionally, per-pose covariance (CSV).
• Package all outputs per team as a ZIP archive.
• Submit prior to the stated deadline (June 1, 2025, 23:59 UTC), with possible consideration for late entries on a secondary leaderboard.

All submissions are automatically aligned to ground truth via Umeyama’s method, and evaluated using FoMo SDK-provided Docker scripts to ensure consistency.

5. Benchmark Results and Baseline Analyses

Preliminary benchmarking, conducted on the Yellow trajectory (representative mixed environment), demonstrates the challenge posed by seasonal shifts:

Method	Mean Translation Drift (%) vs. Jan 29 Map
Proprioceptive	3.9 ± 1.4 (summer), up to 15.3 ± 8.1 (deep snow)
Lidar-Inertial	3.8 ± 1.5 (Jan), spikes to 46.5 ± 28.3 (Oct)
Radar-Gyro TaR	4.2 ± 1.5 (Aug), reaches 32.3 ± 11.2 (Mar)
ORB-SLAM3	<1 % in Oct, fails (rain/night, Nov 03)

Key observations:

• Lidar-inertial pipelines are robust to winter conditions but degrade under summer/fall appearance changes (foliage, snowbank absence).
• Radar-gyro methods show moderate robustness but struggle with rapid heading changes and uneven terrain.
• Visual-inertial SLAM achieves minimal drift under favorable lighting and texture but fails when conditions degrade (rain, night).
• Proprioceptive (wheels+IMU) odometry is stable except during deep snow, where wheel slip and immobilization are prevalent.

This suggests that no single modality or algorithm reliably handles all environmental conditions, and further points to the need for cross-domain adaptation and robust sensor fusion.

6. Challenge Structure, Leaderboard, and Reporting

The FOMO25 Challenge is coordinated as a public benchmarking event:

• The leaderboard displays per-method drift percentages and ATE/RPE for each trajectory pair.
• Participants are expected to include detailed breakdowns of cross-seasonal performance, sensor ablation studies, and qualitative analyses of failure cases (e.g., SLAM loop-closure misses, ICP divergence).
• Upon submission, Docker-based evaluation ensures reproducibility and fair comparison across teams.

A plausible implication is that the standardization of submission and evaluation protocols, together with transparent result reporting, provides a replicable basis for progress in all-season robot navigation.

7. Significance and Outlook

By leveraging the FoMo dataset’s unique combination of sensor diversity, terrain complexity, and rigorous calibration, the FOMO25 Challenge provides an unprecedented testbed for the development and evaluation of odometry, SLAM, and localization algorithms in realistic, seasonally variable, boreal forest environments (Boxan et al., 9 Mar 2026). The explicit focus on cross-season generalization under severe appearance, traction, and visibility shifts sets FOMO25 apart from prior benchmarks. Participation and continued leaderboard reporting will likely advance the state of the art in UGV navigation, with a focus on methods able to withstand real-world, long-duration environmental perturbations.