---
title: 'UniDetector: Unified Object Detector'
url: https://www.emergentmind.com/topics/unidetector
type: topic
---

# UniDetector: Unified Object Detector

Searching arXiv for the specified paper and closely related context.
arXiv search query: 2207.01071 Unified Object Detector for Different Modalities based on Vision Transformers
UniDetector, termed the **Unified Object Detector for Different Modalities (UODDM)** in "Unified Object Detector for Different Modalities based on Vision Transformers" [2207.01071], is a single object detector designed to operate across **RGB images**, **depth/point-cloud-derived pseudo images**, and **inter-modality mixed images** without changing the model architecture or weights. The system targets robotic perception settings in which sensing conditions vary across lighting regimes, such as RGB-only or RGB+depth in well-lit environments and depth-only at night. Architecturally, it is built on **Vision Transformers with Swin backbones** and **Mask R-CNN** style detection heads, and it combines **cross-modality transfer learning** with **inter-modality transfer learning** to obtain comparable or superior detection performance across modalities [2207.01071].

## 1. Definition and operating scope

UniDetector addresses a practical limitation of traditional multimodal perception pipelines: separate detectors are typically trained for different modalities, such as an RGB detector for daytime imagery and a depth or point-cloud detector for low-light settings. In contrast, UODDM is intended to accept three types of input within a single detector: raw RGB images, 3-channel pseudo images converted from structured depth or point-cloud data, and mixed RGB+DHS images. The stated objective is that a robot can switch among these sensing modes seamlessly, using only a change in input and preprocessing, with **no model architecture or weight updates** [2207.01071].

The modalities are unified by representing each as a standard 2D 3-channel image. RGB images are used directly. Depth maps or structured point clouds are converted into **DHS pseudo images**, where the three channels correspond to **Depth**, **Height**, and **Signed Angle**. Inter-modality mixed images are then constructed from RGB and DHS using dedicated mixing schemes. Because all modalities are mapped into the same image-like representation, the detector processes them through the same backbone and detection head.

This design suggests a modality-agnostic detection framework in which unification is achieved primarily through representation and training strategy rather than through explicit modality-specific branches or fusion modules.

## 2. Architecture and representation pipeline

The detector uses a **Swin Transformer** backbone, specifically **Swin-Tiny (Swin-T)** or **Swin-Small (Swin-S)**, with a **Mask R-CNN** detection framework implemented via MMDetection [2207.01071]. The processing sequence is: input image, Swin Transformer backbone, **Feature Pyramid Network (FPN)**-style multi-scale features, **Region Proposal Network (RPN)**, and **ROI head** for classification and bounding-box regression. The mask branch is kept but not fine-tuned.

For Swin, the paper specifies the standard attention formulation
\[
\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V,
\]
with **window size** \(M = 7\), **head dimension** \(d = 32\), and **MLP expansion ratio** \(\alpha = 4\). The backbone configurations are:
- **Swin-T**: base channels \(C = 96\), layers per stage \(\{2, 2, 6, 2\}\)
- **Swin-S**: \(C = 96\), layers per stage \(\{2, 2, 18, 2\}\)

These backbones are pretrained on **ImageNet**, then on **COCO** detection, and then fine-tuned on **SUN RGB-D** for unified multimodal detection [2207.01071].

The unification step is representational. RGB is already a 3-channel image. Depth and point clouds are converted into DHS pseudo images, with channels normalized to \([0,1]\). Each pixel in a DHS image corresponds to a point in the structured point cloud. Once converted, the Swin backbone applies its standard patch embedding without any additional modality embedding or positional mechanism for DHS relative to RGB. The image is divided into patches, each patch is linearly embedded into a token, and positional information is handled implicitly by the windowed architecture.

A central architectural property is therefore that **all modalities go through the same backbone and head, using exactly the same weights** [2207.01071].

## 3. Cross-modality and inter-modality learning

The paper distinguishes two related training concepts. **Cross-modality transfer learning** refers to using knowledge learned from one modality, such as RGB, to improve another modality, such as DHS. **Inter-modality transfer learning** refers to knowledge transfer through joint training on multiple modalities so that the network learns representations that are robust to modality changes and can exploit complementary information [2207.01071].

The training pipeline has two stages. First, the Swin backbone is pretrained on ImageNet for classification, and the full Mask R-CNN with Swin backbone is pretrained on COCO for detection and instance segmentation. Second, the model is fine-tuned on the **SUN RGB-D train-only subset (2666 images)** and evaluated on **2619 validation images**, following prior work. For UODDM without mixing, the fine-tuning set includes both RGB and DHS images. For UODDM with mixing, mixed RGB+DHS images are additionally included. During fine-tuning, the mask head is frozen, and only detection heads are trained [2207.01071].

The augmentation regime mirrors COCO training: random horizontal flip with probability 0.5, random resizing to width 1333 and height in \([480,800]\), and random cropping to \(384 \times 600\) followed by resizing. Training is performed on a single NVIDIA Titan-X with 12 GB memory, with a typical schedule of **133K iterations**, approximately **100 epochs**, and **batch size 2**. Reported training times include about **29 hours** for Swin-T UODDM training on RGB-only and about **2 days** for UODDM with both RGB and DHS [2207.01071].

The loss follows standard Mask R-CNN form:
\[
\mathcal{L} = \lambda_{cls} \, \mathcal{L}_{cls} + \lambda_{box} \, \mathcal{L}_{box} + \lambda_{mask} \, \mathcal{L}_{mask}.
\]
During fine-tuning, \(\lambda_{mask}=0\), so only classification and box regression remain active. The same loss and the same shared weights are used for RGB, DHS, and mixed inputs. In this formulation, inter-modality transfer occurs because gradients from all modalities update a common backbone and detection head [2207.01071].

## 4. Inter-modality mixing and the CPPM/SFFM schemes

The main methodological novelty beyond joint multimodal training is an **inter-modality mixing method** intended to produce composite images that remain standard 3-channel inputs while interleaving RGB and DHS content spatially [2207.01071]. The paper introduces two such strategies.

### Chessboard Per Patch Mixing

**Chessboard Per Patch Mixing (CPPM)** divides the image into square patches of size \(P \times P\). Each patch is assigned entirely to either RGB or DHS in a chessboard-like alternation. Given paired RGB and DHS images \(I^{rgb}\) and \(I^{dhs}\), the output \(I^{mix}\) is defined patchwise as:
\[
I^{mix}[x,y] =
\begin{cases}
I^{rgb}[x,y] & \text{if the patch containing } (x,y) \text{ is assigned to RGB} \\
I^{dhs}[x,y] & \text{otherwise}.
\end{cases}
\]
This is an early-fusion mechanism, but it does not interpolate or average modalities; each pixel comes strictly from one modality. The paper reports that CPPM yields **stable training**, preserves modality identity locally, and produces the **best overall mAP50** when tested on CPPM images [2207.01071].

### Stochastic Flood Fill Mixing

**Stochastic Flood Fill Mixing (SFFM)** creates irregular RGB and DHS regions through a probabilistic flood-fill process. Starting from a random seed pixel, the algorithm grows RGB or DHS regions to 4-neighbor pixels with probabilities \(p_{rgb}\) and \(p_{dhs}\). The resulting image again assigns each pixel either RGB or DHS without interpolation:
\[
I^{mix}[x,y] =
\begin{cases}
I^{rgb}[x,y] & \text{if mask}[x,y] = \text{RGB} \\
I^{dhs}[x,y] & \text{if mask}[x,y] = \text{DHS}.
\end{cases}
\]
Empirically, the paper states that SFFM generates many highly random mixed images and causes performance degradation on pure RGB and DHS modes due to excessive variance and unnatural transitions [2207.01071].

The rationale offered for mixing is that, unlike interpolation-based fusion, the method preserves each modality’s local signal and relies on the transformer’s self-attention to integrate cues across patches. This suggests that the backbone learns both modality-agnostic features and modality-aware higher-level patterns without requiring explicit fusion blocks.

## 5. Dataset, evaluation protocol, and reported performance

Experiments are conducted on **SUN RGB-D**, an indoor scene dataset providing RGB images, depth maps from multiple sensors, and point clouds derived from depth. The train split contains **5285 images**, the test split **5050**, and the train split is further divided into **2666 train-only** and **2619 validation** images. The evaluation follows prior work and reports results on the validation set for category subsets **SUNRGBD10**, **SUNRGBD16**, **SUNRGBD66**, and **SUNRGBD79** [2207.01071].

The principal metric is **AP50 (mAP@IoU = 0.5)**, accompanied by **AP75**, **COCO-style AP** across IoU thresholds \([0.50:0.05:0.95]\), and scale-specific AP values \(AP_S\), \(AP_M\), and \(AP_L\). AP is defined from the precision-recall curve as
\[
AP = \int_0^1 P(r)\,dr.
\]

For UODDM **without inter-modality mixing**, the paper reports the following **mAP50** with **Swin-T** [2207.01071]:

| Model | Test on | SUNRGBD10 | SUNRGBD16 | SUNRGBD66 | SUNRGBD79 |
|---|---:|---:|---:|---:|---:|
| RGB only (ours) | RGB | 54.2 | 52.3 | 29.3 | 25.2 |
| UODDM | RGB | 53.9 | 52.5 | 28.7 | 24.7 |
| DHS only (simCrossTrans) | DHS | 55.8 | 52.7 | 26.1 | 22.1 |
| UODDM | DHS | 56.6 | 53.4 | 27.7 | 23.5 |

These results show that UODDM is nearly identical to RGB-only training on RGB input while improving over the DHS-only baseline on DHS input. The paper interprets this as evidence of **inter-modality transfer** [2207.01071].

For UODDM **with mixing**, again with **Swin-T**, the reported **mAP50** values are [2207.01071]:

| Model | Test on | SUNRGBD10 | SUNRGBD16 | SUNRGBD66 | SUNRGBD79 |
|---|---:|---:|---:|---:|---:|
| UODDM | RGB | 53.9 | 52.5 | 28.7 | 24.7 |
| UODDM + SFFM | RGB | 24.6 | 17.5 | 19.2 | 20.1 |
| UODDM + CPPM | RGB | 54.2 | 51.9 | 27.7 | 23.7 |
| UODDM | DHS | 56.6 | 53.4 | 27.7 | 23.5 |
| UODDM + SFFM | DHS | 25.6 | 18.7 | 20.0 | 21.3 |
| UODDM + CPPM | DHS | 55.8 | 52.8 | 26.3 | 22.4 |
| UODDM + CPPM | CPPM | 58.1 | 55.8 | 29.5 | 25.2 |

The result pattern is unambiguous: **SFFM significantly hurts performance**, while **CPPM preserves performance on RGB and DHS** and delivers the strongest numbers when both modalities are available in mixed form [2207.01071].

The paper also compares UniDetector to prior state-of-the-art methods. On **point-cloud-only (DHS) input**, UODDM improves over **simCrossTrans** on **SUNRGBD10** and **SUNRGBD16**. On **RGB+Depth (CPPM mixed input)**, UODDM+CPPM reaches **58.1 mAP50** on **SUNRGBD10** and **55.8** on **SUNRGBD16**, compared with **44.2 mAP50** on SUNRGBD10 for **RGB-D RCNN** [2207.01071]. On RGB-only input, it remains slightly below **Frustum PointNets** at SUNRGBD10 but is described as comparable.

## 6. Operational modes, robotics implications, and limitations

A defining feature of UniDetector is support for multiple sensing modes with the same network parameters. In **RGB-only mode**, a standard RGB image is passed directly into UODDM or UODDM+CPPM. In **depth-only or point-cloud-only mode**, structured point clouds or depth maps are converted into DHS pseudo images and then processed identically. In **RGB+Depth mode**, RGB and DHS are combined into a CPPM image and fed into the same detector. The mode switch is therefore limited to changing the input and preprocessing pipeline; there are **no modality-specific branches**, **no gating mechanism**, and **no weight reloading** [2207.01071].

The intended application scenario is indoor robotics, such as a mobile robot operating in warehouses or homes. During daytime or in well-lit environments, RGB-only or RGB+depth operation is possible; in nighttime or low-light conditions, the system can rely on depth-only sensing. The paper emphasizes that this enables **eco-friendly operation** because a robot can turn off illumination at night and still use the same detector through DHS input [2207.01071].

The reported computational characteristics are:
- **Swin-T UODDM**: about **48M parameters**, about **476.5 GFLOPs**, about **105 ms per frame** on Titan-X, approximately **9.5 FPS**
- **Swin-S UODDM**: about **69M parameters**, about **419.7 GFLOPs**, about **148 ms per frame**, approximately **6.8 FPS** [2207.01071]

These numbers are presented as comparable to prior ViT-based detectors and consistent with deployment on many robotic platforms.

The paper also identifies several limitations. Experiments are limited to **SUN RGB-D**, which is an indoor dataset of relatively small size, so generalization to outdoor or automotive settings is not established. The supported modalities are presently **RGB**, **depth/point clouds via DHS**, and **RGB+DHS mixing**; thermal, radar, event cameras, and other modalities are not integrated. Fusion is restricted to **early fusion at image level**, and explicit mid-level or late-fusion mechanisms are not explored. There is also a slight RGB-only trade-off relative to the strongest RGB-only models, and SFFM shows that mixing strategies can harm performance if designed poorly [2207.01071].

A plausible implication is that UniDetector should be understood as a representation- and training-centered multimodal detector rather than as a general-purpose multimodal fusion architecture. Its main contribution lies in showing that a single Swin-Transformer-based Mask R-CNN can be trained to function across RGB, DHS, and mixed RGB+DHS inputs with a shared parameterization and competitive detection accuracy [2207.01071].

Source: https://www.emergentmind.com/topics/unidetector