OmniUnet: Multimodal Segmentation for Planetary Rovers

• OmniUnet is a transformer-based multimodal segmentation architecture designed for planetary rover navigation in Martian analog environments.
• It fuses RGB, depth, and thermal imagery using an early fusion strategy with an Omnivore/Swin-style encoder and U-Net-like decoder to assess terrain traversability.
• Developed using data from MaRTA and the Bardenas Reales, it categorizes eight terrain types and offers actionable insights into sensor alignment, classification performance, and embedded deployment trade-offs.

OmniUnet is a transformer-based neural network architecture for semantic segmentation of unstructured terrain from RGB, depth, and thermal imagery, developed for planetary rover navigation in Martian-like environments (Castilla-Arquillo et al., 1 Aug 2025). It operates on synchronized RGB-D-T observations acquired on the Martian Rover Testbed for Autonomy (MaRTA) and targets terrain classes relevant to traversability and obstacle reasoning, including void, compact, bedrock, sandy, gravel, rock, bush, and grass. The model combines early fusion of heterogeneous modalities with an Omnivore/Swin-style encoder and a U-Net-like decoder, and is evaluated on a labeled subset of a multimodal dataset collected in the Bardenas semi-desert in northern Spain, a Mars analog environment featuring sand, bedrock, and compact soil.

1. Problem setting and navigational role

Autonomous planetary rovers must perceive and reason about unstructured terrain where slopes, loose soils, and rocks can degrade mobility, cause slippage, or lead to immobilization. In this setting, a single sensing modality is described as insufficient across the wide range of lighting, geometry, and surface thermophysical conditions encountered in the field. OmniUnet is framed as a response to this constraint through multimodal perception, combining RGB, depth, and thermal cues into a single semantic segmentation pipeline (Castilla-Arquillo et al., 1 Aug 2025).

The modality rationale is explicit. RGB provides rich texture and color, depth encodes 3D structure, and thermal captures surface temperature fields tied to soil thermophysical behavior. Thermal imagery is identified as particularly valuable in Martian-like environments: under solar forcing, granular soils such as sand typically heat up more than compacted or bedrock surfaces because of lower thermal inertia; under low atmospheric pressure, the diurnal amplitude and spatial contrast of surface temperature are enhanced, making thermal differences more discriminative. The paper further states that thermal data have been shown to improve slip estimation and traversability assessment by correlating thermal inertia with trafficability, and to remain informative under low light, glare, or shadows that degrade RGB.

The terrain taxonomy used by OmniUnet comprises eight classes: void, compact, bedrock, sandy, gravel, rock, bush, and grass. Their navigational interpretation is also specified. Compact is generally safest; sand and gravel elevate slip risk; bedrock can be highly traversable but uneven; rocks and bushes constitute obstacles; grass and vegetation affect traction and obscuration. On this basis, semantic segmentation is positioned as a support layer for safe-path selection, obstacle avoidance, and science target detection.

A common misunderstanding would be to treat the model as a generic scene parser without rover-specific semantics. The reported class design is instead operationally tied to mobility risk and environmental affordances. This suggests that OmniUnet is best understood as a perception component within a traversability-oriented autonomy stack rather than as a purely categorical vision model.

2. Architectural design and fusion mechanism

OmniUnet is an encoder-decoder architecture for multimodal semantic segmentation using RGB, depth, and thermal imagery. Its defining design choice is early fusion at the input: a single five-channel tensor $I \in \mathbb{R}^{H \times W \times 5}$ is formed by concatenating RGB, depth, and thermal channels. The stated rationale is twofold: early fusion eliminates the need for separate modality-specific encoders and simplifies deployment, while allowing the transformer to learn cross-modal correlations jointly from the start (Castilla-Arquillo et al., 1 Aug 2025).

After fusion, the model applies a patch tokenizer. The 5-channel image is split into non-overlapping patches and linearly projected to embeddings through a convolutional “patchify” stage. These embeddings form the token sequence consumed by the backbone. The token dimension and patch size are said to follow the Omnivore/Swin configuration used, but are not enumerated.

The transformer encoder is described as an Omnivore/Swin-style backbone employing shifted-window multi-head self-attention. Self-attention is computed within local windows, while alternating shifted layers enable inter-window communication. The encoder extracts hierarchical features, and positional information is handled by Swin’s relative position bias within windows. The paper does not enumerate the number of stages, layers per stage, heads, hidden dimensions, or patch sizes, stating instead that these follow the adopted Omnivore/Swin configuration.

Within each window, scaled dot-product attention is standard:

$$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$

Because OmniUnet uses early fusion, $Q$, $K$, and $V$ are computed from fused features rather than from separate modality-specific streams. The paper further summarizes the encoder-side integration as shared projections $W_Q Z^L$, $W_K Z^L$, and $W_V Z^L$, where $Z^L$ denotes encoder tokens after $L$ layers.

The decoder follows a U-Net paradigm. Encoder features at each resolution are fused with the upsampled decoder output from the coarser scale through skip connections. The fusion operation is concatenation of skip features and upsampled decoder features, followed by two $$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$0 convolution layers with GELU activations for refinement. The final decoder stage emits per-pixel logits, and a softmax produces the segmentation mask. The paper states only that the decoder output is “upsampled”; the specific upsampling method is not specified.

The model is therefore not a late-fusion or explicit cross-modal-attention architecture. There is no modality gating, no separate modality branches, and no reported cross-modal attention across distinct streams. Cross-modal integration occurs implicitly through attention over fused patch embeddings. A plausible implication is that architectural simplicity and deployability were prioritized over modality-specific interpretability.

3. Sensor platform, data acquisition, and annotation

The data acquisition platform is MaRTA, the Martian Rover Testbed for Autonomy, described as a half-scale, six-wheeled ExoMars-like rover. For multimodal capture, the system uses a custom 3D-printed sensor housing aligning an Intel RealSense D435i and an Optris PI-640i thermal camera to facilitate geometric registration (Castilla-Arquillo et al., 1 Aug 2025).

The sensing configuration is reported as follows:

Component	Specification
RealSense D435i RGB	up to 1920×1080, ~69°×42° FOV
RealSense D435i stereo depth	1280×720, ~87°×58° FOV, depth accuracy < 2% at 2 m
Optris PI-640i thermal	LWIR, 8–14 μm; 640×480; 60°×45° FOV; −20°C to 900°C; thermal sensitivity 0.04°C

The housing aligns the RealSense depth plane and the thermal optical axis. Because the sensors have different fields of view, fused frames can contain black borders on the left and right. The paper does not enumerate calibration procedures beyond this alignment. In preprocessing, modalities are processed into a registered, aligned 5-channel image; black borders may appear after RGB/depth–thermal registration due to field-of-view mismatch. No per-modality normalization specifics are listed.

The principal dataset is collected in the Bardenas Reales in northern Spain, described as a Mars analog with sand, bedrock, and compact soil. The collection comprises 24 rover traverses ranging from 6.85 m to 202 m, approximately 1.7 km total coverage, and 36,000 synchronized multimodal RGB-D-T frames. A subset was manually annotated for supervised learning, with 950 images for training and 190 for validation. The public release includes labeled segmentation masks on Zenodo under doi: 10.5281/zenodo.15496884.

The paper also uses the RUGD dataset to provide a baseline with RGB-only inputs because comparable labeled multimodal datasets are scarce. RUGD contains approximately 7,500 labeled RGB images and 25 classes; the experimental split reported is 5,948 training images and 1,487 validation images, for a total of 7,435.

The dataset contribution is presented as a response to a gap in public RGB-D-T terrain annotation for planetary robotics. The paper’s own summary identifies this as a new publicly available labeled multimodal segmentation dataset in Bardenas and, more specifically, as the first public labeled RGB-D-T segmentation dataset in this Mars analog. This suggests that the work is as much a data-infrastructure contribution as a model contribution.

4. Optimization objective and evaluation protocol

Training is performed for 50 epochs with batch size 16 and learning rate $$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$1 on an NVIDIA DGX Station with a Tesla V100 GPU providing 32 GB VRAM. The split is reported as 80/20 train/validation for each dataset, and the retained model is the best epoch according to lowest validation loss. The optimizer, weight decay, scheduler, dropout, regularization choices, and data augmentation are not specified (Castilla-Arquillo et al., 1 Aug 2025).

The training objective is a composite of cross-entropy and Dice losses. Let $$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$2 denote the one-hot ground truth and $$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$3 the softmax probability for class $$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$4. The per-pixel cross-entropy is

$$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$5

For Dice, the paper gives both count-based and probabilistic forms:

$$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$6

and

$$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$7

Dice loss per class is $$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$8, and the total objective is

$$\mathrm{Att}(Q, K, V) = \mathrm{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right)V$$9

where $Q$0 excludes the background or void class. The authors report that $Q$1 ranges from 2 to 0, with 0 indicating perfect agreement.

Evaluation uses pixel accuracy, per-class IoU, and mean IoU. The formulas reported are

$Q$2

$Q$3

and

$Q$4

The paper reports Total PA, Mean PA, and Mean IoU. Confusion matrices are not provided. For the Bardenas multimodal evaluation, the paper notes that validation averages exclude the void class.

The absence of specified optimizer and augmentation details is a reproducibility limitation, although the work also reports a public GitHub repository for training and deployment and a public Zenodo release of labeled masks. A plausible implication is that exact replication of the optimization trajectory may require consulting the codebase rather than the paper text alone.

5. Reported performance and deployment characteristics

The paper reports both RGB-only and multimodal results. On RUGD, used as an RGB-only baseline with OmniUnet, the validation metrics are Total PA 92.58%, Mean PA 39.72%, and Mean IoU 34.90% over 25 classes (Castilla-Arquillo et al., 1 Aug 2025). The authors interpret these results qualitatively as evidence that the architecture can operate on RGB-only inputs, reflecting the generality of the Omnivore/Swin backbone and U-Net decoder.

On the Bardenas multimodal labeled subset, using classes void, compact, grass, bedrock, sandy, gravel, rock, and bush, the validation metrics are Total PA 80.37%, Mean PA 46.59%, and Mean IoU 38.77%, with averages excluding void. The reported per-class pixel accuracies are:

Class	Pixel accuracy
Void	90.48%
Compact	77.44%
Grass	39.04%
Bedrock	28.30%
Sandy	26.57%
Gravel	51.83%
Rock	18.40%
Bush	40.68%

The qualitative interpretation given in the paper is closely tied to thermal sensing. The model is reported to effectively distinguish terrain types with distinct thermal behavior, notably sandy versus compact soil, leveraging thermal contrast associated with lower thermal inertia in sand. In some transition zones, predictions appear more coherent than the ground-truth annotations. Bush detection is described as moderate, whereas rock detection is challenging, with 18.40% pixel accuracy, likely due to variable rock thermal signatures and geometric or appearance diversity. Terrain boundaries and mixed-material patches remain difficult, which the paper attributes to the inherent ambiguity of unstructured natural scenes.

Deployment measurements are reported on both desktop and embedded hardware. Inference on an NVIDIA GeForce RTX 2070 with 8 GB VRAM takes 137.50 ms per frame. On an NVIDIA Jetson Orin Nano with 8 GB RAM, 40 TOPS, and approximately 15 W TDP, embedded inference takes 673 ms per multimodal frame, approximately 1.48 FPS. The paper notes that these timings are reported without TensorRT conversion, mixed precision, pruning, or quantization.

The deployment interpretation is deliberately constrained. For low-speed rover navigation and periodic traversability updates, approximately 1.5 FPS can be acceptable. At the same time, the paper states that throughput on embedded hardware is currently sub-real-time for fast driving, indicating a trade-off between multimodal transformer capacity and latency.

6. Position within related work, limitations, and future directions

OmniUnet is positioned at the intersection of transformer-based segmentation and multimodal fusion for robotics. Relative to transformer segmentation, the paper cites Swin-Unet and UNetFormer as examples of strong single-modality performance using hierarchical attention and U-Net decoders. Relative to multimodal fusion, it notes that prior robotics work often uses separate branches and late or cross-attention fusion, whereas OmniUnet adopts a simpler early-fusion strategy aligned with Omnivore’s design philosophy of a single model across visual modalities, but tailored to dense segmentation (Castilla-Arquillo et al., 1 Aug 2025).

The comparative evidence remains limited. The paper positions OmniUnet relative to transformer-based segmentation, multimodal backbones, and field datasets including RUGD, RELLIS-3D, and CAVS, and demonstrates RGB-only performance on RUGD plus multimodal performance on Bardenas. However, explicit comparisons to other multimodal fusion models on the same dataset are not reported because publicly labeled RGB-D-T terrain datasets are scarce. Likewise, no ablations are reported isolating the contribution of each modality on Bardenas, and there are no experiments comparing early fusion against mid- or late-fusion alternatives.

Several limitations are stated directly. Generalization to off-nominal Martian conditions, including atmospheric dust dynamics, different solar incidence, and thermal emissivity variations, is untested; domain shift remains a concern. Class imbalance and ambiguous boundaries in natural terrain affect per-class accuracy, especially for rarer classes such as rock. Thermal variability tied to time of day, shadowing, and material emissivity, as well as depth noise, can degrade performance. Embedded throughput is currently insufficient for fast driving without further optimization.

Future work is described along four main axes. First, segmentation robustness is to be enhanced as the number of terrain classes grows and visual similarity increases. Second, a specialized obstacle detection module is proposed to complement terrain segmentation. Third, multimodal sensor housings are to be improved, especially for stereo–thermal imaging in low-light conditions. Fourth, broader avenues are noted: self-supervised multimodal pretraining, uncertainty estimation for risk-aware planning, temporal fusion across frames, and domain adaptation to Martian environments.

A recurring misconception would be to interpret the reported results as establishing definitive multimodal superiority across all fusion strategies and planetary settings. The paper does not support that conclusion. What it does support is a practical multimodal transformer U-Net, a custom stereo–thermal rover sensor housing, a public labeled RGB-D-T terrain dataset in a Mars analog setting, and measured embedded deployment performance, all situated within the current scarcity of comparable multimodal terrain benchmarks for planetary robotics.