SpatialRGBT-Bench: RGB–Thermal SOD Benchmark

• SpatialRGBT-Bench is a benchmark platform for multimodal salient object detection, integrating spatially aligned RGB–thermal datasets with detailed challenge annotations.
• The platform facilitates robust evaluation using metrics like precision, recall, F-measure, and MAE, ensuring reproducible and challenge-sensitive performance analysis.
• Researchers leverage baseline algorithms, including ADFNet and multi-task manifold ranking, to demonstrate the benefits of adaptive multimodal fusion and to address challenges like small objects and blurred boundaries.

SpatialRGBT-Bench defines a rigorous platform for the assessment and development of salient object detection (SOD) algorithms utilizing both RGB and thermal (T) modalities. This multimodal paradigm addresses limitations inherent in single-modality SOD, such as diminished performance under low-light, adverse weather, and background clutter. The benchmark centers on spatially aligned RGB–thermal image pairs, providing standardized datasets (notably VT5000 and VT821) annotated with explicit challenge categories and ground-truth masks. SpatialRGBT-Bench underpins algorithmic analysis, comparison, and advancement in RGBT SOD, furnishing metrics and protocols that facilitate reproducible research and comprehensive challenge-sensitive evaluation (Tu et al., 2020, Li et al., 2017).

1. Dataset Construction and Challenge Annotation

SpatialRGBT-Bench comprises multiple spatially registered RGB–thermal datasets—most prominently VT5000 (Tu et al., 2020) and VT821 (Li et al., 2017):

• Acquisition and Alignment: VT5000 utilizes FLIR T640 and T610 sensors with matched intrinsic parameters, resulting in native pixel-wise registration without manual warping. VT821 achieves sub-pixel alignment via homographic mapping from manually selected correspondence points.
• Saliency Annotation: Ground-truth masks $G(x,y)\in\{0,1\}$ are created through manual annotation, with selection protocols ensuring consensus among human annotators regarding salient regions.
• Challenge Categories: Each sample is tagged with up to two of eleven attributes defined analytically to isolate factors affecting SOD difficulty—such as small/big salient objects, low illumination, cross-boundary objects, similar appearance (RGB indistinguishability), thermal crossover, and image clutter.

Challenge	Abbreviation	Definition Example
Big Salient Object	BSO	Area $>$ 26% of image
Small Salient Object	SSO	Area $<$ 5% of image
Multiple Objects	MSO	$>$1 disjoint salient regions
Low Illumination	LI	Night/cloudy capture
Center Bias	CB	Centroid far from center
Cross-Border	CIB	Object intersects image borders
Similar Appearance	SA	RGB: foreground $\approx$ background
Thermal Crossover	TC	Thermal: foreground $\approx$ background
Image Clutter	IC	High background texture
Out of Focus	OF	Significant optical blur
Bad Weather	BW	Rain, mist, snow

This robust attribute taxonomy enables granular challenge-based analysis of algorithmic robustness.

2. Evaluation Metrics

SpatialRGBT-Bench mandates three quantitative metrics for binary saliency maps $S(x,y)\in[0,1]$:

• Precision ($P$) and Recall ($R$):

$$P = \frac{TP}{TP + FP},\quad R = \frac{TP}{TP + FN}$$

where $TP$, $FP$, and $FN$ are computed by thresholding $S$ against ground truth $G$.

• F-measure ($F_\beta$):

$$F_\beta = \frac{(1+\beta^2) P R}{\beta^2 P + R},\quad \beta^2 = 0.3$$

Emphasizes precision in aggregate evaluation.

• Mean Absolute Error (MAE):

$$MAE = \frac{1}{WH} \sum_{x=1}^W \sum_{y=1}^H |S(x,y)-G(x,y)|$$

Quantifies pixelwise deviation from ground truth.

These metrics are uniformly applied across challenge subsets and datasets for comparative benchmarking.

3. Baseline Algorithms and Network Designs

SpatialRGBT-Bench evaluates both classical and deep learning approaches:

• VT5000 Baseline—ADFNet (Tu et al., 2020): End-to-end two-stream CNN built on VGG16, processing RGB and thermal modalities independently before multi-modal fusion.
  • Feature Extraction: Multi-level features $X_i^R$, $X_i^T$.
  • Attention Modules (CBAM): Channel- and spatial-wise attention applied per modality.
  • Channel attention: $$M_i^{C_R} = \sigma(\text{Conv}_{1\times1}(\text{AvgPool}(X_i^R)) + \text{Conv}_{1\times1}(\text{MaxPool}(X_i^R))) \odot X_i^R$$
  • Spatial attention analogously defined.
  • Fusion: $$F_1 = M_1^{S_R} + M_1^{S_T};\quad F_i = \text{Conv}_{3\times3}(F_{i-1}) + M_i^{S_R} + M_i^{S_T}$$
  • Pyramid Pooling and Aggregation: Multi-scale context refinement via PPM and FAM.
  • Loss: Combined cross-entropy on masks/edges.
  • $L = L_C + L_E$, where $L_C$ is standard mask cross-entropy; $L_E$ is edge supervision using learned Laplacian maps.
• VT821 Baseline—Multi-task Manifold Ranking (Li et al., 2017): Graph-based ranking of RGB–T superpixel features with adaptive reliability weights $r^k$ and cross-modality consistency.
  • Objective function:

  $$\min_{\{s^k\},\{r^k\}} \sum_{k=1}^K \frac{(r^k)^2}{2}\,(s^k)^\top L^k\,s^k + \mu\sum_{k=1}^K \|s^k - y\|_2^2 + \|\Gamma\circ(\mathbf{1}-r)\|_2^2 + \lambda\sum_{k=2}^K \|s^k - s^{k-1}\|_2^2$$

  Spin-off advantages include robustness to noisy modalities and convergence in 5–10 iterations.

4. Benchmark Results and Challenge-Sensitive Performance

Comprehensive evaluation demonstrates substantive improvements over prior methods by fusing RGB and thermal cues.

• VT5000 Benchmarking (Tu et al., 2020):

  • ADFNet leads in Fβ and MAE across all challenge subsets, with Fβ = 0.863 and MAE = 0.049 (whole test set).
  • Per-challenge Fβ: ADFNet outperforms next-best rivals in all subsets; notable scores include BSO (0.880), SSO (0.806), LI (0.868), TC (0.841).
• VT821 Results (Li et al., 2017):
  • Multi-task manifold ranking surpasses all baselines in overall F-measure (0.680) and MAE (0.107), excelling in 9/11 challenge attributes (highest for SSO, MSO, LI, CB, etc.).
  • Table: Per-challenge F-measure, (Li et al., 2017):

Challenge	Best RGB–T Baseline	Ours (Multi-task)
SSO	0.53	0.58
MSO	0.60	0.66
LI	0.55	0.63
...	...	...

• Failure Modes and Debilitating Challenges: Models struggle on SSO (small objects), CIB (cross-boundary), and extreme TC (thermal-crossover). Fusion mechanisms must adaptively down-weight unreliable modalities. Edge detection and attention mechanisms are limited in recovering micro-structures and blurred boundaries.

5. Insights, Limitations, and Future Research Directions

Research on SpatialRGBT-Bench converges on several foundational insights:

• Multimodal Fusion: Integration of thermal with RGB channels consistently improves SOD, especially under illumination, appearance, or thermal ambiguity. Early naïve fusion may perform suboptimally; adaptive weighting and cross-modality consistency are essential for robust performance.
• Failure Cases: Severe blur in both modalities and perfect crossover (object indistinguishable from background in thermal or RGB) remain unsolved by current approaches. Very thin structures are difficult for edge and attention-based losses.
• Methodological Recommendations:

1. Modality-aware networks for per-frame reliability calibration.
2. Attribute-driven architectures for specialized challenge-handling.
3. Weakly/unsupervised learning to leverage annotation cost savings.
4. Alignment-free detection for non-coincident sensor rigs.
5. Temporal modeling for video-based RGBT saliency detection.

Further dataset expansion and enhanced graph/deep learning architectures are proposed to address extreme conditions and scalability (Tu et al., 2020, Li et al., 2017). This suggests future directions will likely emphasize dynamic modality reliability modeling and cross-domain generalization.

6. Context and Availability

SpatialRGBT-Bench datasets, including VT5000 and VT821, are publicly available for academic use. VT5000 can be accessed from authors of (Tu et al., 2020); VT821, with full protocol and baseline reproductions, is available at http://chenglongli.cn/people/lcl/journals.html (Li et al., 2017). The comprehensive challenge annotation, precise alignment, and multimodal depth position SpatialRGBT-Bench as a benchmark of record for RGB–thermal SOD research, enabling standardization and reproducibility while driving algorithmic advances and modality-adaptive approaches.