DooDLeNet: Adaptive RGB–Thermal Segmentation

• DooDLeNet is an advanced RGB–LWIR segmentation model featuring dual DeepLab V3+ branches for dedicated color and thermal feature extraction.
• It introduces a two-stage fusion mechanism that combines confidence weighting for modality reliability with correlation weighting for spatial alignment.
• Empirical evaluations on the MF dataset demonstrate state-of-the-art mIoU improvements and ablation studies confirm the benefit of its adaptive fusion strategy.

DooDLeNet, short for “Double DeepLab Enhanced Feature Fusion,” is an RGB–LWIR thermal semantic segmentation architecture for driving perception that extends DeepLab V3+ to a two-stream setting with modality-specialized processing and a shared segmentation decoder (Frigo et al., 2022). It is designed for feature fusion between color and thermal imagery, using two specialized encoder–decoder branches, a correlation head, and a final shared decoder. Its defining technical contribution is the combination of confidence weighting and correlation weighting for mid-level multimodal fusion, and it reports state-of-the-art mean IoU results on the MF dataset, with an mIoU of $57.3\%$ (Frigo et al., 2022).

1. Problem setting and design rationale

DooDLeNet targets semantic segmentation from paired RGB and LWIR thermal images in driving scenes, where the two modalities provide complementary signals but also exhibit unequal reliability across conditions and known minor parallax (Frigo et al., 2022). The model is explicitly structured to avoid collapsing the problem into a single early-fusion representation. Instead, it separates color and thermal processing into distinct DeepLab V3+ branches and defers final integration to a shared decoder.

The underlying design assumption is that RGB and thermal modalities benefit from specialized feature extractors. The paper states that the model uses “two fully-specialized streams (one for RGB, one for LWIR thermal) and a final shared decoder that fuses modality-specific features in a confidence- and correlation-aware way” (Frigo et al., 2022). This architecture is therefore not merely a 4-channel adaptation of DeepLab. The ablation data reinforce this distinction: “stacked input (4-ch)” reaches an mIoU of $51.1$, whereas the full model reaches $57.3$ (Frigo et al., 2022).

A common simplification is to treat multimodal segmentation as a question of adding channels. The reported ablations argue against that interpretation in this case. A plausible implication is that the primary gain comes from modality-specific representation learning and adaptive fusion, rather than from raw modality concatenation alone.

2. Network topology

The model comprises three major parts: two parallel DeepLab V3+ encoder–decoder branches, a lightweight correlation head, and a shared final DeepLab decoder head (Frigo et al., 2022). One branch takes a 3-channel RGB image, and the other takes a single-channel thermal image. Each specialized branch yields both branch-specific logits and intermediate feature maps used in fusion.

Each specialized branch $m \in \{c,t\}$, with $c$ denoting color and $t$ thermal, follows DeepLab V3+ with a ResNet-101 encoder. The encoder uses a $7 \times 7$ conv1 with stride 2 followed by max-pooling; then conv2_x with stride 1, conv3_x with stride 2, conv4_x with dilated convolution at rate 2, and conv5_x with dilated convolution at rate 4 (Frigo et al., 2022). Atrous Spatial Pyramid Pooling employs parallel $1 \times 1$ and three $3 \times 3$ atrous convolutions with rates $6,12,18$, plus image pooling, producing 256 channels. The decoder concatenates upsampled ASPP output with low-level features from conv2_x reduced to 48 channels by a $51.1$0 convolution, then applies two $51.1$1 convolution layers with 256 channels before bilinear upsampling to one-quarter of the input resolution.

A $51.1$2 convolutional segmentation head on each branch produces preliminary logits

$51.1$3

with $51.1$4 classes (Frigo et al., 2022). In parallel, the model taps feature maps $51.1$5 at two depths: after conv2_x, with spatial size $51.1$6 and 256 channels, and after conv4_x, with spatial size $51.1$7 and 1024 channels. All spatial maps are resized by bilinear interpolation as needed before fusion.

Component	Configuration
Specialized branches	Two DeepLab V3+ encoder–decoder streams
Backbone	ResNet-101
ASPP	$51.1$8, three $51.1$9 atrous convs with rates $57.3$0, image pooling, 256 output channels
Low-level skip	conv2_x reduced to 48 channels
Decoder convs	Two $57.3$1 conv layers, 256 channels
Tapped features	conv2_x: $57.3$2, 256 channels; conv4_x: $57.3$3, 1024 channels
Branch logits	$57.3$4, $57.3$5
Final stage	Shared DeepLab decoder head with bilinear upsampling

This topology implements a division of labor: branch-specific decoders model modality-dependent structure, while the shared decoder produces the ultimate segmentation from fused features. The paper characterizes this as a “double DeepLab architecture with specialized encoder-decoders for thermal and color modalities and a shared decoder for final segmentation” (Frigo et al., 2022).

3. Confidence and correlation fusion

The distinctive component of DooDLeNet is its two-stage feature fusion mechanism. The first stage is confidence weighting, which derives per-pixel modality reliability from branch logits. The second is correlation weighting, which modulates fused features using an estimated matching correlation map intended to mitigate local RGB–thermal misalignment (Frigo et al., 2022).

For confidence weighting, each specialized decoder produces logits $57.3$6 at pixel $57.3$7. These are converted into a per-pixel confidence score

$57.3$8

Collecting these over all pixels yields a confidence map

$57.3$9

The feature maps are then reweighted pointwise: $m \in \{c,t\}$0 After resizing $m \in \{c,t\}$1 to match the corresponding feature-map resolution, the reweighted color and thermal features are concatenated: $m \in \{c,t\}$2

For correlation weighting, the model computes a “matching correlation” map

$m \in \{c,t\}$3

The preliminary logits are first flattened to

$m \in \{c,t\}$4

A raw similarity matrix is then computed: $m \in \{c,t\}$5 The matrix is passed through ReLU and row-wise $m \in \{c,t\}$6-normalization: $m \in \{c,t\}$7 A small $m \in \{c,t\}$8 convolution block $m \in \{c,t\}$9 compresses or reshapes this into a single-channel spatial map

$c$0

with $c$1. The final fused feature map is

$c$2

These fused maps, taken at both tapped depths, are then fed into the shared DeepLab decoder head (Frigo et al., 2022).

The paper interprets the two weighting schemes as complementary. Confidence weighting handles modality reliability, for example “thermal at night, RGB by day,” while correlation weighting handles spatial misalignment (Frigo et al., 2022). This decomposition is central to the architecture’s intended robustness in day/night and imperfectly aligned multimodal settings.

4. Optimization and training protocol

DooDLeNet is trained with the standard pixel-wise cross-entropy applied only to the final shared-decoder output (Frigo et al., 2022). The loss is

$c$3

where

$c$4

and $c$5. The paper explicitly states that it does not introduce additional auxiliary losses on the intermediate decoders.

The optimization recipe uses SGD with momentum $c$6 and weight decay $c$7. The learning rate is initialized at $c$8 and decayed by a factor of $c$9 each epoch for 50 epochs. The batch size is 8 image pairs. Data augmentation consists of random horizontal flip and random crop with size $t$0 (Frigo et al., 2022).

This training setup is notable for its restraint. The architecture includes intermediate modality-specific logits, but these are used for fusion rather than for auxiliary supervision. A plausible implication is that the design emphasizes the final fused prediction as the sole optimization target, keeping the supervision pathway simpler than in many deeply supervised segmentation systems.

5. Evaluation on the MF dataset

The empirical evaluation is conducted on the MF dataset, which contains 1569 RGB–thermal pairs, with 820 daytime and 749 nighttime samples (Frigo et al., 2022). The dataset split is 784 training, 392 validation, and 392 test images. It defines 9 semantic categories: $t$1 Image resolution is $t$2, and the modality pairs have approximate physical alignment with known minor parallax.

The paper compares DooDLeNet to ten previous RGBT segmentation methods and reports class-wise IoU as well as overall mIoU and mAcc. In the summary columns shown, RTFNet-152 attains an mIoU of $t$3, FuseSeg-161 $t$4, FEANet $t$5, and DooDLeNet $t$6 (Frigo et al., 2022). For selected classes, DooDLeNet reports Car $t$7, Person $t$8, Bike $t$9, and Curve $7 \times 7$0. The paper states that DooDLeNet achieves “state-of-the-art mIoU = 57.3% (±2 points over FEANet), notably improving ‘person’ (+1.1%) and ‘bike’ (+1.4%).”

Method	Selected results
RTFNet-152	Car 87.4, Person 70.3, Bike 62.7, Curve 45.3, mIoU 53.2
FuseSeg-161	Car 87.9, Person 71.7, Bike 64.6, Curve 44.8, mIoU 54.5
FEANet	Car 87.8, Person 71.1, Bike 61.1, Curve 46.5, mIoU 55.3
DooDLeNet	Car 86.7, Person 72.2, Bike 62.5, Curve 46.7, mIoU 57.3

These results situate DooDLeNet within the RGBT segmentation literature as a method that emphasizes adaptive feature fusion rather than single-stream multimodal encoding. The strongest gains highlighted in the paper occur on “person” and “bike,” which suggests that fusion quality may be especially consequential for smaller or safety-critical foreground categories. That interpretation remains inferential; the explicit reported fact is the class-wise improvement on those categories.

6. Ablations, interpretation, and limitations

The ablation study isolates the contribution of each fusion component using mIoU on the full test split (Frigo et al., 2022). The reported variants are: only RGB branch, $7 \times 7$1; only thermal branch, $7 \times 7$2; stacked input (4-ch), $7 \times 7$3; unweighted mid-level fusion, $7 \times 7$4; confidence-weight only, $7 \times 7$5; and confidence plus correlation, $7 \times 7$6. The paper concludes that confidence weighting alone yields $7 \times 7$7 mIoU over naïve fusion, and adding correlation yields a further $7 \times 7$8 mIoU.

Variant	mIoU
Only RGB branch	50.7
Only thermal branch	50.1
Stacked input (4-ch)	51.1
Unweighted mid-level fusion	54.7
Confidence-weight only	55.2
Confidence + Correlation (Ours)	57.3

These ablations clarify the model’s internal logic. First, separate modality streams are already advantageous relative to single-stream early fusion. Second, unweighted mid-level fusion is useful but insufficient. Third, the two adaptive weighting mechanisms are complementary rather than redundant. The paper states this directly: “confidence handles modality reliability (e.g. day/night), correlation handles spatial misalignment” (Frigo et al., 2022).

The limitations are equally explicit. The two-stream design doubles encoder cost and therefore increases memory and GPU footprint. The correlation computation scales with $7 \times 7$9, which requires compression. The paper identifies several future directions: lighter backbones such as MobileNet, multi-scale fusion at all decoder stages, joint training of intermediate segmentation heads, and cross-modal transformers for richer feature interaction (Frigo et al., 2022). It also notes that more efficient alignment modules such as deformable convolution or attention could replace the correlation computation.

A common misconception is that DooDLeNet’s performance derives solely from using both RGB and thermal inputs. The ablation evidence indicates a more specific conclusion: multimodality alone does not explain the result, because the 4-channel stacked-input variant remains far below the full model. What distinguishes DooDLeNet is the combination of modality specialization, confidence-aware reweighting, and correlation-aware suppression of poorly aligned features.

7. Position within multimodal semantic segmentation

Within RGB–thermal semantic segmentation, DooDLeNet occupies the design space of mid-level fusion with explicit modality-aware gating (Frigo et al., 2022). It inherits the encoder–decoder and ASPP machinery of DeepLab V3+ while modifying the fusion pathway to account for two multimodal failure modes: uncertain modality-specific predictions and imperfect intermodal correspondence. In this sense, the model is best understood as a structured extension of DeepLab rather than as an entirely new segmentation backbone.

The paper’s “key insights” summarize this position in three terms. First, “Double-DeepLab specialization” allows each modality to learn its own feature extractors, distinguishing “color textures” from “thermal radiometric cues.” Second, “Adaptive weighting” lets the system trust the modality showing more certain segmentation and suppress features from regions where correspondence is weak. Third, the paper frames efficiency and alignment modeling as the principal next challenges (Frigo et al., 2022).

This suggests a broader methodological significance. DooDLeNet exemplifies a class of multimodal segmentation systems in which the fusion operator is conditioned on intermediate task predictions rather than solely on raw features. Such a design treats segmentation confidence and cross-modal compatibility as first-class signals in the architecture. The paper’s reported results on the MF dataset indicate that this strategy is effective in the presence of day/night variability and minor RGB–thermal parallax, which are recurrent conditions in driving perception.