---
title: 'DooDLeNet: Adaptive RGB–Thermal Segmentation'
url: https://www.emergentmind.com/topics/doodlenet
type: topic
---

# DooDLeNet: Adaptive RGB–Thermal Segmentation

DooDLeNet, short for “Double DeepLab Enhanced Feature Fusion,” is an RGB–LWIR thermal semantic segmentation architecture for driving perception that extends DeepLab V3+ to a two-stream setting with modality-specialized processing and a shared segmentation decoder [2204.10266]. It is designed for feature fusion between color and thermal imagery, using two specialized encoder–decoder branches, a correlation head, and a final shared decoder. Its defining technical contribution is the combination of confidence weighting and correlation weighting for mid-level multimodal fusion, and it reports state-of-the-art mean IoU results on the MF dataset, with an mIoU of \(57.3\%\) [2204.10266].

## 1. Problem setting and design rationale

DooDLeNet targets semantic segmentation from paired RGB and LWIR thermal images in driving scenes, where the two modalities provide complementary signals but also exhibit unequal reliability across conditions and known minor parallax [2204.10266]. The model is explicitly structured to avoid collapsing the problem into a single early-fusion representation. Instead, it separates color and thermal processing into distinct DeepLab V3+ branches and defers final integration to a shared decoder.

The underlying design assumption is that RGB and thermal modalities benefit from specialized feature extractors. The paper states that the model uses “two fully-specialized streams (one for RGB, one for LWIR thermal) and a final shared decoder that fuses modality-specific features in a confidence- and correlation-aware way” [2204.10266]. This architecture is therefore not merely a 4-channel adaptation of DeepLab. The ablation data reinforce this distinction: “stacked input (4-ch)” reaches an mIoU of \(51.1\), whereas the full model reaches \(57.3\) [2204.10266].

A common simplification is to treat multimodal segmentation as a question of adding channels. The reported ablations argue against that interpretation in this case. A plausible implication is that the primary gain comes from modality-specific representation learning and adaptive fusion, rather than from raw modality concatenation alone.

## 2. Network topology

The model comprises three major parts: two parallel DeepLab V3+ encoder–decoder branches, a lightweight correlation head, and a shared final DeepLab decoder head [2204.10266]. One branch takes a 3-channel RGB image, and the other takes a single-channel thermal image. Each specialized branch yields both branch-specific logits and intermediate feature maps used in fusion.

Each specialized branch \(m \in \{c,t\}\), with \(c\) denoting color and \(t\) thermal, follows DeepLab V3+ with a ResNet-101 encoder. The encoder uses a \(7 \times 7\) conv1 with stride 2 followed by max-pooling; then conv2\_x with stride 1, conv3\_x with stride 2, conv4\_x with dilated convolution at rate 2, and conv5\_x with dilated convolution at rate 4 [2204.10266]. Atrous Spatial Pyramid Pooling employs parallel \(1 \times 1\) and three \(3 \times 3\) atrous convolutions with rates \(6,12,18\), plus image pooling, producing 256 channels. The decoder concatenates upsampled ASPP output with low-level features from conv2\_x reduced to 48 channels by a \(1 \times 1\) convolution, then applies two \(3 \times 3\) convolution layers with 256 channels before bilinear upsampling to one-quarter of the input resolution.

A \(1 \times 1\) convolutional segmentation head on each branch produces preliminary logits
\[
y_m \in \mathbb{R}^{K \times H/4 \times W/4},
\]
with \(K=9\) classes [2204.10266]. In parallel, the model taps feature maps \(f_m\) at two depths: after conv2\_x, with spatial size \(H/4 \times W/4\) and 256 channels, and after conv4\_x, with spatial size \(H/16 \times W/16\) and 1024 channels. All spatial maps are resized by bilinear interpolation as needed before fusion.

| Component | Configuration |
|---|---|
| Specialized branches | Two DeepLab V3+ encoder–decoder streams |
| Backbone | ResNet-101 |
| ASPP | \(1 \times 1\), three \(3 \times 3\) atrous convs with rates \(6,12,18\), image pooling, 256 output channels |
| Low-level skip | conv2\_x reduced to 48 channels |
| Decoder convs | Two \(3 \times 3\) conv layers, 256 channels |
| Tapped features | conv2\_x: \(H/4 \times W/4\), 256 channels; conv4\_x: \(H/16 \times W/16\), 1024 channels |
| Branch logits | \(y_m \in \mathbb{R}^{K \times H/4 \times W/4}\), \(K=9\) |
| Final stage | Shared DeepLab decoder head with bilinear upsampling |

This topology implements a division of labor: branch-specific decoders model modality-dependent structure, while the shared decoder produces the ultimate segmentation from fused features. The paper characterizes this as a “double DeepLab architecture with specialized encoder-decoders for thermal and color modalities and a shared decoder for final segmentation” [2204.10266].

## 3. Confidence and correlation fusion

The distinctive component of DooDLeNet is its two-stage feature fusion mechanism. The first stage is confidence weighting, which derives per-pixel modality reliability from branch logits. The second is correlation weighting, which modulates fused features using an estimated matching correlation map intended to mitigate local RGB–thermal misalignment [2204.10266].

For confidence weighting, each specialized decoder produces logits \(y_m(x) \in \mathbb{R}^K\) at pixel \(x\). These are converted into a per-pixel confidence score
\[
w_m^{(c)}(x) = \max_{k=1\ldots K} \operatorname{softmax}(y_m^k(x))
= \max_k \frac{\exp(y_m^k(x))}{\sum_{j=1}^{K}\exp(y_m^j(x))}.
\]
Collecting these over all pixels yields a confidence map
\[
C_m \in [0,1]^{H/4 \times W/4}.
\]
The feature maps are then reweighted pointwise:
\[
f'_m(x) = C_m(x) \cdot f_m(x), \quad m \in \{c,t\}.
\]
After resizing \(C_m\) to match the corresponding feature-map resolution, the reweighted color and thermal features are concatenated:
\[
f'_{ct}(x) = [\,f'_c(x)\,\|\,f'_t(x)\,].
\]

For correlation weighting, the model computes a “matching correlation” map
\[
M_{ct} \in \mathbb{R}^{H/4 \times W/4}.
\]
The preliminary logits are first flattened to
\[
\bar{y}_m \in \mathbb{R}^{K \times N}, \quad N=(H/4 \cdot W/4).
\]
A raw similarity matrix is then computed:
\[
S = \bar{y}_t^{\top} \cdot \bar{y}_c \in \mathbb{R}^{N \times N}.
\]
The matrix is passed through ReLU and row-wise \(L_2\)-normalization:
\[
S' = \| \operatorname{ReLU}(S)\|_2.
\]
A small \(1 \times 1\) convolution block \(c(\cdot)\) compresses or reshapes this into a single-channel spatial map
\[
M_{ct} = c(S') \in \mathbb{R}^{H/4 \times W/4},
\]
with \(0 \leq M_{ct}(x) \leq 1\). The final fused feature map is
\[
f''_{ct}(x) = M_{ct}(x)\cdot f'_{ct}(x).
\]
These fused maps, taken at both tapped depths, are then fed into the shared DeepLab decoder head [2204.10266].

The paper interprets the two weighting schemes as complementary. Confidence weighting handles modality reliability, for example “thermal at night, RGB by day,” while correlation weighting handles spatial misalignment [2204.10266]. This decomposition is central to the architecture’s intended robustness in day/night and imperfectly aligned multimodal settings.

## 4. Optimization and training protocol

DooDLeNet is trained with the standard pixel-wise cross-entropy applied only to the final shared-decoder output [2204.10266]. The loss is
\[
L_{CE} = - \frac{1}{N}\sum_{i=1}^{N}\sum_{k=1}^{K} \mathbf{1}[y_i=k]\cdot \log p_{i,k},
\]
where
\[
p_{i,k} = \operatorname{softmax}(\text{final\_logits}_i)_k
\]
and \(N=H \cdot W\). The paper explicitly states that it does not introduce additional auxiliary losses on the intermediate decoders.

The optimization recipe uses SGD with momentum \(0.9\) and weight decay \(5 \times 10^{-4}\). The learning rate is initialized at \(0.01\) and decayed by a factor of \(0.95\) each epoch for 50 epochs. The batch size is 8 image pairs. Data augmentation consists of random horizontal flip and random crop with size \(512 \times 512\) [2204.10266].

This training setup is notable for its restraint. The architecture includes intermediate modality-specific logits, but these are used for fusion rather than for auxiliary supervision. A plausible implication is that the design emphasizes the final fused prediction as the sole optimization target, keeping the supervision pathway simpler than in many deeply supervised segmentation systems.

## 5. Evaluation on the MF dataset

The empirical evaluation is conducted on the MF dataset, which contains 1569 RGB–thermal pairs, with 820 daytime and 749 nighttime samples [2204.10266]. The dataset split is 784 training, 392 validation, and 392 test images. It defines 9 semantic categories:
\[
\{\text{background, car, person, bike, curve, carstop, guardrail, cone, bump}\}.
\]
Image resolution is \(640 \times 512\), and the modality pairs have approximate physical alignment with known minor parallax.

The paper compares DooDLeNet to ten previous RGBT segmentation methods and reports class-wise IoU as well as overall mIoU and mAcc. In the summary columns shown, RTFNet-152 attains an mIoU of \(53.2\), FuseSeg-161 \(54.5\), FEANet \(55.3\), and DooDLeNet \(57.3\) [2204.10266]. For selected classes, DooDLeNet reports Car \(86.7\), Person \(72.2\), Bike \(62.5\), and Curve \(46.7\). The paper states that DooDLeNet achieves “state-of-the-art mIoU = 57.3% (±2 points over FEANet), notably improving ‘person’ (+1.1%) and ‘bike’ (+1.4%).”

| Method | Selected results |
|---|---|
| RTFNet-152 | Car 87.4, Person 70.3, Bike 62.7, Curve 45.3, mIoU 53.2 |
| FuseSeg-161 | Car 87.9, Person 71.7, Bike 64.6, Curve 44.8, mIoU 54.5 |
| FEANet | Car 87.8, Person 71.1, Bike 61.1, Curve 46.5, mIoU 55.3 |
| DooDLeNet | Car 86.7, Person 72.2, Bike 62.5, Curve 46.7, mIoU 57.3 |

These results situate DooDLeNet within the RGBT segmentation literature as a method that emphasizes adaptive feature fusion rather than single-stream multimodal encoding. The strongest gains highlighted in the paper occur on “person” and “bike,” which suggests that fusion quality may be especially consequential for smaller or safety-critical foreground categories. That interpretation remains inferential; the explicit reported fact is the class-wise improvement on those categories.

## 6. Ablations, interpretation, and limitations

The ablation study isolates the contribution of each fusion component using mIoU on the full test split [2204.10266]. The reported variants are: only RGB branch, \(50.7\); only thermal branch, \(50.1\); stacked input (4-ch), \(51.1\); unweighted mid-level fusion, \(54.7\); confidence-weight only, \(55.2\); and confidence plus correlation, \(57.3\). The paper concludes that confidence weighting alone yields \(+0.5\) mIoU over naïve fusion, and adding correlation yields a further \(+2.1\) mIoU.

| Variant | mIoU |
|---|---|
| Only RGB branch | 50.7 |
| Only thermal branch | 50.1 |
| Stacked input (4-ch) | 51.1 |
| Unweighted mid-level fusion | 54.7 |
| Confidence-weight only | 55.2 |
| Confidence + Correlation (Ours) | 57.3 |

These ablations clarify the model’s internal logic. First, separate modality streams are already advantageous relative to single-stream early fusion. Second, unweighted mid-level fusion is useful but insufficient. Third, the two adaptive weighting mechanisms are complementary rather than redundant. The paper states this directly: “confidence handles modality reliability (e.g. day/night), correlation handles spatial misalignment” [2204.10266].

The limitations are equally explicit. The two-stream design doubles encoder cost and therefore increases memory and GPU footprint. The correlation computation scales with \(N^2\), which requires compression. The paper identifies several future directions: lighter backbones such as MobileNet, multi-scale fusion at all decoder stages, joint training of intermediate segmentation heads, and cross-modal transformers for richer feature interaction [2204.10266]. It also notes that more efficient alignment modules such as deformable convolution or attention could replace the correlation computation.

A common misconception is that DooDLeNet’s performance derives solely from using both RGB and thermal inputs. The ablation evidence indicates a more specific conclusion: multimodality alone does not explain the result, because the 4-channel stacked-input variant remains far below the full model. What distinguishes DooDLeNet is the combination of modality specialization, confidence-aware reweighting, and correlation-aware suppression of poorly aligned features.

## 7. Position within multimodal semantic segmentation

Within RGB–thermal semantic segmentation, DooDLeNet occupies the design space of mid-level fusion with explicit modality-aware gating [2204.10266]. It inherits the encoder–decoder and ASPP machinery of DeepLab V3+ while modifying the fusion pathway to account for two multimodal failure modes: uncertain modality-specific predictions and imperfect intermodal correspondence. In this sense, the model is best understood as a structured extension of DeepLab rather than as an entirely new segmentation backbone.

The paper’s “key insights” summarize this position in three terms. First, “Double-DeepLab specialization” allows each modality to learn its own feature extractors, distinguishing “color textures” from “thermal radiometric cues.” Second, “Adaptive weighting” lets the system trust the modality showing more certain segmentation and suppress features from regions where correspondence is weak. Third, the paper frames efficiency and alignment modeling as the principal next challenges [2204.10266].

This suggests a broader methodological significance. DooDLeNet exemplifies a class of multimodal segmentation systems in which the fusion operator is conditioned on intermediate task predictions rather than solely on raw features. Such a design treats segmentation confidence and cross-modal compatibility as first-class signals in the architecture. The paper’s reported results on the MF dataset indicate that this strategy is effective in the presence of day/night variability and minor RGB–thermal parallax, which are recurrent conditions in driving perception.

Source: https://www.emergentmind.com/topics/doodlenet