UniDetector: Unified Object Detector

• UniDetector is a unified object detector that processes RGB images, DHS pseudo images, and mixed inputs by converting all modalities into standard 3-channel images.
• It leverages Vision Transformers with Swin backbones and a Mask R-CNN detection head, employing cross-modality and inter-modality transfer learning.
• Results on the SUN RGB-D dataset demonstrate comparable or superior detection performance across modalities, enhancing robotic perception in varying lighting conditions.

Searching arXiv for the specified paper and closely related context. arXiv search query: (Shen et al., 2022) Unified Object Detector for Different Modalities based on Vision Transformers UniDetector, termed the Unified Object Detector for Different Modalities (UODDM) in "Unified Object Detector for Different Modalities based on Vision Transformers" (Shen et al., 2022), is a single object detector designed to operate across RGB images, depth/point-cloud-derived pseudo images, and inter-modality mixed images without changing the model architecture or weights. The system targets robotic perception settings in which sensing conditions vary across lighting regimes, such as RGB-only or RGB+depth in well-lit environments and depth-only at night. Architecturally, it is built on Vision Transformers with Swin backbones and Mask R-CNN style detection heads, and it combines cross-modality transfer learning with inter-modality transfer learning to obtain comparable or superior detection performance across modalities (Shen et al., 2022).

1. Definition and operating scope

UniDetector addresses a practical limitation of traditional multimodal perception pipelines: separate detectors are typically trained for different modalities, such as an RGB detector for daytime imagery and a depth or point-cloud detector for low-light settings. In contrast, UODDM is intended to accept three types of input within a single detector: raw RGB images, 3-channel pseudo images converted from structured depth or point-cloud data, and mixed RGB+DHS images. The stated objective is that a robot can switch among these sensing modes seamlessly, using only a change in input and preprocessing, with no model architecture or weight updates (Shen et al., 2022).

The modalities are unified by representing each as a standard 2D 3-channel image. RGB images are used directly. Depth maps or structured point clouds are converted into DHS pseudo images, where the three channels correspond to Depth, Height, and Signed Angle. Inter-modality mixed images are then constructed from RGB and DHS using dedicated mixing schemes. Because all modalities are mapped into the same image-like representation, the detector processes them through the same backbone and detection head.

This design suggests a modality-agnostic detection framework in which unification is achieved primarily through representation and training strategy rather than through explicit modality-specific branches or fusion modules.

2. Architecture and representation pipeline

The detector uses a Swin Transformer backbone, specifically Swin-Tiny (Swin-T) or Swin-Small (Swin-S), with a Mask R-CNN detection framework implemented via MMDetection (Shen et al., 2022). The processing sequence is: input image, Swin Transformer backbone, Feature Pyramid Network (FPN)-style multi-scale features, Region Proposal Network (RPN), and ROI head for classification and bounding-box regression. The mask branch is kept but not fine-tuned.

For Swin, the paper specifies the standard attention formulation

$$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d_k}}\right) V,$$

with window size $M = 7$, head dimension $d = 32$, and MLP expansion ratio $\alpha = 4$. The backbone configurations are:

• Swin-T: base channels $C = 96$, layers per stage $\{2, 2, 6, 2\}$
• Swin-S: $C = 96$, layers per stage $\{2, 2, 18, 2\}$

These backbones are pretrained on ImageNet, then on COCO detection, and then fine-tuned on SUN RGB-D for unified multimodal detection (Shen et al., 2022).

The unification step is representational. RGB is already a 3-channel image. Depth and point clouds are converted into DHS pseudo images, with channels normalized to $[0,1]$. Each pixel in a DHS image corresponds to a point in the structured point cloud. Once converted, the Swin backbone applies its standard patch embedding without any additional modality embedding or positional mechanism for DHS relative to RGB. The image is divided into patches, each patch is linearly embedded into a token, and positional information is handled implicitly by the windowed architecture.

A central architectural property is therefore that all modalities go through the same backbone and head, using exactly the same weights (Shen et al., 2022).

3. Cross-modality and inter-modality learning

The paper distinguishes two related training concepts. Cross-modality transfer learning refers to using knowledge learned from one modality, such as RGB, to improve another modality, such as DHS. Inter-modality transfer learning refers to knowledge transfer through joint training on multiple modalities so that the network learns representations that are robust to modality changes and can exploit complementary information (Shen et al., 2022).

The training pipeline has two stages. First, the Swin backbone is pretrained on ImageNet for classification, and the full Mask R-CNN with Swin backbone is pretrained on COCO for detection and instance segmentation. Second, the model is fine-tuned on the SUN RGB-D train-only subset (2666 images) and evaluated on 2619 validation images, following prior work. For UODDM without mixing, the fine-tuning set includes both RGB and DHS images. For UODDM with mixing, mixed RGB+DHS images are additionally included. During fine-tuning, the mask head is frozen, and only detection heads are trained (Shen et al., 2022).

The augmentation regime mirrors COCO training: random horizontal flip with probability 0.5, random resizing to width 1333 and height in $[480,800]$, and random cropping to $M = 7$0 followed by resizing. Training is performed on a single NVIDIA Titan-X with 12 GB memory, with a typical schedule of 133K iterations, approximately 100 epochs, and batch size 2. Reported training times include about 29 hours for Swin-T UODDM training on RGB-only and about 2 days for UODDM with both RGB and DHS (Shen et al., 2022).

The loss follows standard Mask R-CNN form: $M = 7$1 During fine-tuning, $M = 7$2, so only classification and box regression remain active. The same loss and the same shared weights are used for RGB, DHS, and mixed inputs. In this formulation, inter-modality transfer occurs because gradients from all modalities update a common backbone and detection head (Shen et al., 2022).

4. Inter-modality mixing and the CPPM/SFFM schemes

The main methodological novelty beyond joint multimodal training is an inter-modality mixing method intended to produce composite images that remain standard 3-channel inputs while interleaving RGB and DHS content spatially (Shen et al., 2022). The paper introduces two such strategies.

Chessboard Per Patch Mixing

Chessboard Per Patch Mixing (CPPM) divides the image into square patches of size $M = 7$3. Each patch is assigned entirely to either RGB or DHS in a chessboard-like alternation. Given paired RGB and DHS images $M = 7$4 and $M = 7$5, the output $M = 7$6 is defined patchwise as: $M = 7$7 This is an early-fusion mechanism, but it does not interpolate or average modalities; each pixel comes strictly from one modality. The paper reports that CPPM yields stable training, preserves modality identity locally, and produces the best overall mAP50 when tested on CPPM images (Shen et al., 2022).

Stochastic Flood Fill Mixing

Stochastic Flood Fill Mixing (SFFM) creates irregular RGB and DHS regions through a probabilistic flood-fill process. Starting from a random seed pixel, the algorithm grows RGB or DHS regions to 4-neighbor pixels with probabilities $M = 7$8 and $M = 7$9. The resulting image again assigns each pixel either RGB or DHS without interpolation: $d = 32$0 Empirically, the paper states that SFFM generates many highly random mixed images and causes performance degradation on pure RGB and DHS modes due to excessive variance and unnatural transitions (Shen et al., 2022).

The rationale offered for mixing is that, unlike interpolation-based fusion, the method preserves each modality’s local signal and relies on the transformer’s self-attention to integrate cues across patches. This suggests that the backbone learns both modality-agnostic features and modality-aware higher-level patterns without requiring explicit fusion blocks.

5. Dataset, evaluation protocol, and reported performance

Experiments are conducted on SUN RGB-D, an indoor scene dataset providing RGB images, depth maps from multiple sensors, and point clouds derived from depth. The train split contains 5285 images, the test split 5050, and the train split is further divided into 2666 train-only and 2619 validation images. The evaluation follows prior work and reports results on the validation set for category subsets SUNRGBD10, SUNRGBD16, SUNRGBD66, and SUNRGBD79 (Shen et al., 2022).

The principal metric is AP50 (mAP@IoU = 0.5), accompanied by AP75, COCO-style AP across IoU thresholds $d = 32$1, and scale-specific AP values $d = 32$2, $d = 32$3, and $d = 32$4. AP is defined from the precision-recall curve as

$d = 32$5

For UODDM without inter-modality mixing, the paper reports the following mAP50 with Swin-T (Shen et al., 2022):

Model	Test on	SUNRGBD10	SUNRGBD16	SUNRGBD66	SUNRGBD79
RGB only (ours)	RGB	54.2	52.3	29.3	25.2
UODDM	RGB	53.9	52.5	28.7	24.7
DHS only (simCrossTrans)	DHS	55.8	52.7	26.1	22.1
UODDM	DHS	56.6	53.4	27.7	23.5

These results show that UODDM is nearly identical to RGB-only training on RGB input while improving over the DHS-only baseline on DHS input. The paper interprets this as evidence of inter-modality transfer (Shen et al., 2022).

For UODDM with mixing, again with Swin-T, the reported mAP50 values are (Shen et al., 2022):

Model	Test on	SUNRGBD10	SUNRGBD16	SUNRGBD66	SUNRGBD79
UODDM	RGB	53.9	52.5	28.7	24.7
UODDM + SFFM	RGB	24.6	17.5	19.2	20.1
UODDM + CPPM	RGB	54.2	51.9	27.7	23.7
UODDM	DHS	56.6	53.4	27.7	23.5
UODDM + SFFM	DHS	25.6	18.7	20.0	21.3
UODDM + CPPM	DHS	55.8	52.8	26.3	22.4
UODDM + CPPM	CPPM	58.1	55.8	29.5	25.2

The result pattern is unambiguous: SFFM significantly hurts performance, while CPPM preserves performance on RGB and DHS and delivers the strongest numbers when both modalities are available in mixed form (Shen et al., 2022).

The paper also compares UniDetector to prior state-of-the-art methods. On point-cloud-only (DHS) input, UODDM improves over simCrossTrans on SUNRGBD10 and SUNRGBD16. On RGB+Depth (CPPM mixed input), UODDM+CPPM reaches 58.1 mAP50 on SUNRGBD10 and 55.8 on SUNRGBD16, compared with 44.2 mAP50 on SUNRGBD10 for RGB-D RCNN (Shen et al., 2022). On RGB-only input, it remains slightly below Frustum PointNets at SUNRGBD10 but is described as comparable.

6. Operational modes, robotics implications, and limitations

A defining feature of UniDetector is support for multiple sensing modes with the same network parameters. In RGB-only mode, a standard RGB image is passed directly into UODDM or UODDM+CPPM. In depth-only or point-cloud-only mode, structured point clouds or depth maps are converted into DHS pseudo images and then processed identically. In RGB+Depth mode, RGB and DHS are combined into a CPPM image and fed into the same detector. The mode switch is therefore limited to changing the input and preprocessing pipeline; there are no modality-specific branches, no gating mechanism, and no weight reloading (Shen et al., 2022).

The intended application scenario is indoor robotics, such as a mobile robot operating in warehouses or homes. During daytime or in well-lit environments, RGB-only or RGB+depth operation is possible; in nighttime or low-light conditions, the system can rely on depth-only sensing. The paper emphasizes that this enables eco-friendly operation because a robot can turn off illumination at night and still use the same detector through DHS input (Shen et al., 2022).

The reported computational characteristics are:

• Swin-T UODDM: about 48M parameters, about 476.5 GFLOPs, about 105 ms per frame on Titan-X, approximately 9.5 FPS
• Swin-S UODDM: about 69M parameters, about 419.7 GFLOPs, about 148 ms per frame, approximately 6.8 FPS (Shen et al., 2022)

These numbers are presented as comparable to prior ViT-based detectors and consistent with deployment on many robotic platforms.

The paper also identifies several limitations. Experiments are limited to SUN RGB-D, which is an indoor dataset of relatively small size, so generalization to outdoor or automotive settings is not established. The supported modalities are presently RGB, depth/point clouds via DHS, and RGB+DHS mixing; thermal, radar, event cameras, and other modalities are not integrated. Fusion is restricted to early fusion at image level, and explicit mid-level or late-fusion mechanisms are not explored. There is also a slight RGB-only trade-off relative to the strongest RGB-only models, and SFFM shows that mixing strategies can harm performance if designed poorly (Shen et al., 2022).

A plausible implication is that UniDetector should be understood as a representation- and training-centered multimodal detector rather than as a general-purpose multimodal fusion architecture. Its main contribution lies in showing that a single Swin-Transformer-based Mask R-CNN can be trained to function across RGB, DHS, and mixed RGB+DHS inputs with a shared parameterization and competitive detection accuracy (Shen et al., 2022).