IRisPath: Thermal–RGB Fusion for Off-road Navigation

• IRisPath is a multimodal off-road navigation framework that fuses thermal and RGB images to produce robust traversability costmaps under varying day and night conditions.
• It employs a targetless calibration method using feature matching and Fourier features for speed conditioning, achieving calibration accuracies of ±1.7 cm in translation and ±0.827° in rotation.
• The framework leverages self-supervised terrain cost regression with dynamic speed adjustments, enhancing performance in agriculture, construction, search and rescue, and defense scenarios.

IRisPath denotes a multimodal off-road navigation framework centered on thermal–RGB traversability estimation, day–night dataset release, and targetless Thermal–LiDAR–RGB extrinsic calibration for autonomous operation under dynamic light and weather conditions. In the cited literature, the term is used explicitly for a 2024 system that couples long-wave infrared sensing, visible-spectrum imagery, kinesthetic self-supervision, and costmap generation for off-road robots, with the stated aim of improving robustness against out-of-domain uncertainty in agriculture, construction, search and rescue, and defence scenarios (Sharma et al., 2024).

1. Problem setting and scope

IRisPath is motivated by the gap between structured on-road autonomy and unstructured off-road mobility. Off-road terrain is heterogeneous, with dirt, gravel, rocks, grass, shrubs, ditches, and roots, and traversability depends not only on scene appearance but also on vehicle properties and speed. Classical semantic-segmentation and occupancy-grid pipelines can overestimate obstacles, while LiDAR-only geometric maps still fail to capture how the vehicle actually experiences terrain. Recent self-supervised traversability methods improve this by regressing ride cost from visual input and kinesthetic feedback, but they remain vulnerable to out-of-domain uncertainty under night, fog, dust, rain, and similar appearance shifts (Sharma et al., 2024).

The core IRisPath response is multimodal redundancy. RGB provides detailed semantic cues in daytime, whereas long-wave infrared remains informative in darkness and is less sensitive to lighting changes. The framework therefore combines RGB and thermal imagery to produce a traversability costmap that remains useful across day and night, and supplements this with a targetless calibration pipeline so that the sensing modalities can be co-registered for fusion (Sharma et al., 2024).

2. Dataset, sensor suite, and calibration stack

IRisPath refers both to a dataset and to a fusion model. The dataset was collected on the Copernicus all-terrain robot and includes LWIR images, RGB images, multiple LiDARs, IMU, GNSS, stereo, wheel odometry, day and night sequences, and pseudo-labels for traversability. The data were also converted to KITTI and nuScenes formats, and post-processed with FastLIO2 plus Scan-Context++ using Livox Mid360 and with LIO-SAM using Velodyne plus GNSS (Sharma et al., 2024).

The sensing stack includes a FLIR Hadron thermal camera at $640 \times 512$ on /ir_cam/image_raw at 24 Hz, a FLIR BlackFly S RGB camera at $2048 \times 1536$ on /camera/image_color at 10 Hz, a ZED-2 stereo camera at $1920 \times 1080$, Livox Mid360 and Livox HAP LiDARs, a Velodyne VLP-32C, an Xsense MTI-680G IMU at 350 Hz, GNSS, and wheel RPM sensing (Sharma et al., 2024).

A distinctive component is the targetless extrinsic calibration method. Rather than relying on a thermal-visible checkerboard, IRisPath estimates $^{rgb}T_{li}$ and $^{ir}T_{li}$ independently and composes them as

$${}^{rgb}T_{ir} = {}^{rgb}T_{li}\,\big({}^{ir}T_{li}\big)^{-1}.$$

The underlying optimization is stated as

$$\arg \min_{{}^{rgb}T_{li}} \sum_{i=1}^{N} \left\| w_i - {}^{rgb}T_{li} \big(\pi^{-1}(x_{rgb,i})\big) \right\|^2,$$

with feature matching between LiDAR intensity images and RGB or IR imagery via SuperGlue or ORB and robust fitting with RANSAC. The reported calibration accuracy is approximately $\pm 1.7\,\text{cm}$ in translation and $\pm 0.827^\circ$ in rotation (Sharma et al., 2024).

3. Fusion model and costmap construction

The model, described in the detailed reconstruction as FuseIsPath, operates on co-registered RGB and IR image patches plus vehicle speed. First, the IR image is warped into the RGB frame through the RGB–IR calibration. The image is then subdivided into patches of size $i \times i$ with stride $2048 \times 1536$0, yielding

$2048 \times 1536$1

Vehicle speed is encoded from the planar velocity norm

$2048 \times 1536$2

using Fourier features

$2048 \times 1536$3

where $2048 \times 1536$4 is sampled from $2048 \times 1536$5 (Sharma et al., 2024).

Architecturally, IRisPath uses two separate ResNet-18 encoders, one for RGB and one for IR, with no weight sharing. The Fourier-encoded velocity is passed through an MLP, and the three feature streams are concatenated:

$2048 \times 1536$6

A prediction head then outputs a scalar traversability cost per patch:

$2048 \times 1536$7

These patch-level predictions are placed back into a 2D grid and resized to form an image-aligned costmap that can be consumed by a downstream planner (Sharma et al., 2024).

4. Self-supervised traversability target and robustness mechanisms

IRisPath is self-supervised rather than manually annotated. Its patch labels are derived from the vertical acceleration signal measured while the vehicle traverses terrain, using the power spectral density of the IMU $2048 \times 1536$8-axis acceleration normalized by speed:

$2048 \times 1536$9

This cost is intended to encode ride roughness and speed dependence, so that the same terrain can become more expensive at higher speed. Training is then cast as scalar regression, conceptually using mean squared error between predicted and target costs (Sharma et al., 2024).

Robustness is pursued through three mechanisms. The first is modality fusion itself: RGB is informative in daytime, while thermal remains informative at night and under degraded visibility. The second is explicit speed conditioning through Fourier features, which helps model non-linear speed dependence. The third is optional test-time adaptation. The paper evaluates TENT and MoASE under synthetic fog, rain, and snow in day and night conditions. Reported costmap errors decrease relative to the source model in all six conditions, with MoASE generally best; for example, SnowNight decreases from 0.156 to 0.109 with TENT and to 0.105 with MoASE, while FogNight decreases from 0.112 to 0.110 and 0.104, respectively (Sharma et al., 2024).

5. Empirical behavior, costmap semantics, and deployment

Qualitative evaluation emphasizes that IR-only inference remains informative at night, whereas RGB-only inference degrades sharply; fusion recovers road structure at night and produces richer costmaps in daytime by combining semantic detail from RGB with robust structure from IR (Sharma et al., 2024).

The learned costmap is speed-sensitive. At low speed, grass and small roughness appear as moderate cost. At high speed, grass becomes substantially more expensive, while minor hard-terrain roughness may be penalized less severely. The reported ablations also indicate that reducing or removing Fourier features makes the costmap less structured and less consistent with speed variation (Sharma et al., 2024).

The output is designed for integration into standard navigation stacks. The paper positions the costmap as a layer compatible with ROS costmap_2d or Nav2, after projection from image coordinates into robot coordinates using LiDAR or stereo depth. It also states that dataset and code are to be available at https://github.com/codeck313/IRisPath, including raw rosbags, KITTI and nuScenes formatted data, calibration files, baseline models, and evaluation scripts (Sharma et al., 2024).

The principal limitations are also explicit: no formal uncertainty estimation, simple late fusion rather than attention-based or transformer fusion, a single geographic collection site, synthetic rather than real adverse-weather evaluation, and limited discussion of synchronization and latency (Sharma et al., 2024).

6. Terminological scope and related uses

Among the cited sources, the title “IRisPath” appears explicitly in the off-road navigation paper just described (Sharma et al., 2024). The same capitalization is not stable across other domains. In digital pathology, “Iris” names a rendering engine, a file format, and a RESTful serving ecosystem rather than an IRisPath-named method (Landvater et al., 21 Apr 2025, Landvater et al., 21 Apr 2025, Landvater et al., 8 Aug 2025). In ocular biometrics and monocular depth estimation, several detailed summaries explicitly state that “IRisPath” does not appear in the original paper or is only an interpretive extension applied to a deformation path, pathology pathway, or priors-to-geometry schedule, rather than the formal name of the method itself (Khan et al., 2022, Wei et al., 5 Jul 2026, Boyd et al., 2022, Cai et al., 17 Mar 2026).

This makes IRisPath, in strict bibliographic usage, primarily an autonomous off-road traversability framework built around thermal–RGB fusion, self-supervised costmap learning, and targetless calibration. Broader uses of the string in adjacent literatures are best treated as non-canonical reinterpretations rather than as a single cross-domain technical lineage (Sharma et al., 2024).