A Dual-Stream Transformer Architecture for Illumination-Invariant TIR-LiDAR Person Tracking

Abstract: Robust person tracking is a critical capability for autonomous mobile robots operating in diverse and unpredictable environments. While RGB-D tracking has shown high precision, its performance severely degrades under challenging illumination conditions, such as total darkness or intense backlighting. To achieve all-weather robustness, this paper proposes a novel Thermal-Infrared and Depth (TIR-D) tracking architecture that leverages the standard sensor suite of SLAM-capable robots, namely LiDAR and TIR cameras. A major challenge in TIR-D tracking is the scarcity of annotated multi-modal datasets. To address this, we introduce a sequential knowledge transfer strategy that evolves structural priors from a large-scale thermal-trained model into the TIR-D domain. By employing a differential learning rate strategy -- referred to as ``Fine-grained Differential Learning Rate Strategy'' -- we effectively preserve pre-trained feature extraction capabilities while enabling rapid adaptation to geometric depth cues. Experimental results demonstrate that our proposed TIR-D tracker achieves superior performance, with an Average Overlap (AO) of 0.700 and a Success Rate (SR) of 58.7\%, significantly outperforming conventional RGB-transfer and single-modality baselines. Our approach provides a practical and resource-efficient solution for robust human-following in all-weather robotics applications.

• The paper introduces a novel dual-stream transformer that fuses TIR and LiDAR data for robust person tracking under variable illumination.
• It leverages thermal pre-training and continual learning with differential learning rates to effectively bridge the modality gap.
• Experimental results show improved performance with an AO of 0.700 and SR of 58.7%, outperforming both RGB-based and single-modality trackers.

Dual-Stream Transformer Architectures for Illumination-Invariant TIR-LiDAR Person Tracking

Introduction

This paper introduces a person tracking architecture targeting deployment in SLAM-capable autonomous mobile robots operating in variable illumination. The focus is on fusing Thermal-Infrared (TIR) imagery, capturing heat signatures, with LiDAR-derived depth (D) information to address limitations faced by RGB(-D) trackers under poor or changing lighting. The design is motivated by the ubiquity of TIR and LiDAR sensors in robotics, the paucity of annotated TIR-D datasets, and the need for robust, all-weather tracking. The authors leverage knowledge transfer—starting from a model pre-trained on thermal data—and design a dual-stream Transformer framework with careful optimization strategies. Key contributions include cross-domain continual learning, dual-modality feature extraction, and discriminative domain adaptation.

Problem Setting and Motivation

Conventional SOT for robots is formulated on RGB-D or RGB input, with performance heavily tied to illumination conditions. This frequently results in failures in darkness, backlit, or adverse weather scenarios. By contrast, TIR cameras are immune to visible spectrum limitations, providing reliable person signatures where RGB-based trackers fail. However, TIR signals can be ambiguous in cluttered scenes or when human poses are highly variable. To counteract this, LiDAR-derived depth maps are added, providing orthogonal geometric cues. The central challenge lies in robustly fusing these modalities given data scarcity and bridging the domain gap with limited annotation.

Figure 1: The experimental setup for TIR-D person tracking, showing a real classroom environment and the SLAM-relevant sensor suite comprising a TIR camera and an integrated LiDAR system for aligned depth.

Related Work

Prior research in LiDAR-based tracking addresses geometric fidelity but omits thermal features, while RGB-T approaches boost nighttime tracking robustness but lack 3D context. Thermal-only trackers achieve robust detection under low illumination but can falter in cluttered environments. Recent advances involve cross-modal transformers and continual learning frameworks, but annotated TIR-D datasets remain extremely limited. The approach here synthesizes these advances by initializing both TIR and D branches from thermal pretraining, rather than random or RGB-initialized backbones, and controls adaptation with a fine-grained differential learning rate schedule. This strategic cross-domain initialization is justified by the structural and shape priors shared between thermal imagery and corresponding depth projections, as opposed to RGB, which carries more texture than shape.

Model Architecture

The architecture adopts a dual-stream backbone, each beginning with a channel adaptation layer permitting 1-channel TIR or D data to exploit 3-channel pre-trained CNN weights. The adaptation employs convolutional bottlenecks and non-linearities to project each single-channel input into a pseudo-RGB latent space. This is immediately followed by parallel ResNet-based feature extraction, producing modality-specific embeddings.

Figure 2: Example of a paired TIR (heat intensity) and aligned LiDAR depth frame, both processed via the learned Thermal-Depth Adaptation Layer.

The outputs from TIR and D backbones are joined and processed through a Transformer-based encoder–decoder, with cross-attention enabling adaptive weighting depending on modality reliability. The regressor head is a multi-layer FCN, projecting fused token representations to bounding box predictions via corner-probability regression, trained with a weighted blend of $\ell_1$ and GIoU loss. The architecture therefore not only fuses heterogeneous cues but preserves the representational strengths of pre-training while adapting rapidly to novel, under-annotated domains.

Figure 3: Depiction of the Thermal-Depth Adaptation Layer, which bridges single-channel raw modalities to multi-channel CNN backbones.

Knowledge Transfer and Optimization

A core innovation is the method of knowledge transfer, operationalized as sequential continual learning: models are pre-trained on large-scale thermal datasets, creating thermal-weighted backbones capturing robust person cues. These weights are used to initialize both TIR and D branches in the dual-stream TIR-D model, based on the hypothesis that depth maps, especially derived from the same sensor suite, encode physical structure analogous to thermal contours and limb proportions.

To prevent catastrophic forgetting, the optimization process freeze-tunes most of the backbone parameters via a learning rate one order lower than the fusion and adaptation layers. Early training includes a linear warm-up phase to stabilize randomly initialized layers, followed by cosine annealing. This targeted adaptation maintains upstream domain priors—critical for both convergence speed and final task accuracy on small TIR-D datasets.

Experimental Analysis

The evaluation was conducted with a HIKMICRO Pocket2 TIR camera and a Velodyne VLP-16 LiDAR, spatially co-located (Figure 1). Performance was primarily assessed by Average Overlap (AO), Success Rate (SR, IoU $>0.85$), and Precision. The method was compared against both RGB-D and single-modality (thermal-only, depth-only) baselines, and a variant using RGB pre-training for weight transfer rather than thermal.

Figure 4: Qualitative results in varied indoor conditions, illustrating bounding box predictions that maintain robustness to person pose, inconsistent thermal backgrounds, and overlapping heat sources.

Notably, the TIR-D tracker achieved an AO of 0.700 and an SR of 58.7%, outperforming both pure thermal (AO: 0.690, SR: 55.1%) and depth-only (AO: 0.370, SR: 14.1%) baselines, and very substantially outperforming RGB-D tracking. The TIR-D model initialized from thermal weights slightly trailed RGB-initialized models on AO but provided drastically higher reliability in low-light/dark conditions where RGB fails. The SR of 0.9966 underscores the efficacy of the differential learning rate schedule for stability-aware cross-modal adaptation.

Qualitative analysis demonstrated that in scenarios with ambiguous or overlapping thermal emissions, geometric regularization from depth enables reliable person discrimination. The results also indicate that pre-training on domain-proximate data (thermal, not RGB) is essential for effective TIR-D fusion.

Implications and Future Directions

This work establishes the advantage of sensor fusion in SLAM-robotics, showing that TIR-D dual-stream transformer architectures can realize all-weather, illumination-invariant SOT. The approach balances knowledge preservation with rapid domain adaptation via controlled continual learning. Practically, these findings suggest that TIR-LiDAR fusion can supersede RGB-based trackers in surveillance, social robotics, or disaster response.

Theoretically, the demonstration that cross-modal knowledge transfer across thermal and geometric domains is more effective than texture-based RGB transfer motivates further exploration of structural prior alignment and cross-domain meta-learning. Unexplored avenues include full integration into navigation/mapping stacks, online adaptation in dynamically varying lighting, and extension to multitarget or crowded tracking regimes.

Conclusion

The presented dual-stream transformer framework for TIR-LiDAR person tracking robustly addresses the limitations of visible-light-based approaches in mobile robotics. By leveraging thermal pre-training and careful adaptation, the system achieves superior tracking accuracy and reliability under all lighting conditions, validated both quantitatively and qualitatively. This work highlights the critical role of domain-aware knowledge transfer and optimization scheduling in multi-modal sensor fusion for robust, real-world robotics deployments.