- The paper introduces MDTrack, which combines modality-aware Mixture-of-Experts fusion with separate RGB and auxiliary-modality Mamba-style state-space models to prevent cross-modal interference.
- MDTrack achieves leading results across five benchmarks, including 93.0 MPR and 70.5 MSR on RGBT234, 67.9 F-score on DepthTrack, and 82.2 precision on VisEvent.
- The ablations show that decoupled temporal propagation provides the largest individual gain, while adaptive fusion complements it and helps unified training approach single-modality-pair performance.
Motivation and problem statement
MDTrack addresses two structural weaknesses in current multi-modal object trackers. First, state-of-the-art frameworks such as STTrack and SUTrack apply a single, uniform fusion module regardless of whether the auxiliary modality is infrared (IR), event, or depth, ignoring differences in signal characteristics, noise patterns, and semantics. Second, temporal information is propagated through mixed tokens shared across modalities, which entangles heterogeneous dynamics — RGB streams encode appearance and texture change while X-modal streams capture thermal stability, polarity events, or geometric consistency — producing confounded representations. The paper argues that both weaknesses are amplified under unified-modality training, where one model must handle all modality pairs.
Architecture
MDTrack is built on a HiViT backbone with N stacked blocks, each paired with a temporal module, followed by a modality-aware fusion module and a standard three-branch prediction head (classification via focal loss, box regression via ℓ1​ + GIoU loss). Two training regimes are supported: MDTrack-S trains on one modality pair at a time; MDTrack-U merges all modality datasets for joint training.
Decoupled temporal propagation. Rather than propagating mixed tokens, MDTrack maintains two independent Mamba-style State Space Models (SSMs), one for the RGB stream and one for the X-modal stream. Each SSM updates its own hidden state hit​ through the discretized linear dynamical system with zero-order hold, so RGB and X-modal temporal context evolve without interference. Before entering their respective SSMs, the two streams exchange information implicitly via bidirectional cross-attention on their input features; after SSM update, temporally enriched search tokens interact twice with backbone features through cross-attention, injecting temporal context into the representations used by the tracking head.
Modality-aware fusion. Fusion uses a Mixture of Experts (MoE) design with a dedicated expert library {ERGB​,ET​,EE​,ED​} and gating weight library. A router computes top-K (K=2) gating weights over concatenated RGB-X features, with learnable noise added to the routing logits. Each selected expert produces a spatial importance map Fi​∈RH×W×1 via global average pooling of sigmoid-scaled expert outputs, and the fused representation is a weighted sum of element-wise products of these maps with the per-modality features (fixed weights λ=0.5). A load-balancing loss regularizes routing.
Empirical results
Both variants achieve best or second-best results across five benchmarks, evaluated with a single run per dataset:
| Benchmark |
Metric |
Best prior |
MDTrack-S |
MDTrack-U |
| LasHeR |
Precision / AUC |
76.0 / 60.3 (STTrack) |
76.5 / 61.4 |
76.3 / 61.1 |
| RGBT234 |
MPR / MSR |
89.8 / 66.7 (STTrack) |
93.0 / 70.5 |
92.6 / 70.6 |
| DepthTrack |
F-score |
63.2 (STTrack) |
67.5 |
67.9 |
| VOT-RGBD2022 |
EAO |
77.6 (STTrack) |
79.7 |
80.0 |
| VisEvent |
Precision / Success |
78.6 / 61.9 (STTrack) |
82.2 / 65.3 |
81.3 / 63.9 |
The most substantial margins appear on RGBT234 (+3.2 MPR, +3.8 MSR over STTrack), VisEvent (+3.6 Precision, +3.4 Success), and DepthTrack (+4.3 F-score). Notably, MDTrack-U performs nearly on par with MDTrack-S, whereas Un-Track shows a clear gap between its two training modes — evidence that modality-aware fusion mitigates the interference typically caused by unified training. Inference runs at approximately 25 FPS on an RTX 4090.
Ablation analysis
Ablations on LasHeR, DepthTrack, and VisEvent isolate the contribution of each component against a baseline (mean metrics: 62.2 baseline → 64.3 full model, +2.1):
- Decoupled vs. mixed temporal propagation: token-based propagation yields only +0.5 mean gain and a single mixed SSM +0.6, which the authors attribute to inter-modality feature entanglement. The decoupled dual-SSM design achieves +1.5, the largest single-module gain.
- Modality-aware vs. uniform fusion: a unified-expert fusion adds only +0.2, while MoE-based adaptive selection adds +0.7.
- The two modules are complementary: combined they yield +2.6 AUC on LasHeR, exceeding the sum of individual gains relative to some baselines, confirming joint benefit.
These ablations directly support the paper's central claim that temporal decoupling and modality-specific fusion address distinct failure modes of uniform designs.
Limitations and open questions
The paper does not report parameter counts or FLOPs comparisons against STTrack/SUTrack, so the computational cost of dual SSMs plus four-expert MoE relative to simpler architectures remains unquantified beyond the reported FPS. The fixed fusion coefficients (λRGB​=λX​=0.5) are not ablated, leaving open whether learned or dynamic weighting would help when one modality degrades severely. Expert specialization is enforced only through the load-balancing loss; no analysis verifies that experts actually specialize per modality rather than collapsing to redundant functions. Finally, evaluation covers RGB paired with thermal, depth, and event data but not other sensor types, so the generality of the expert-library design to additional modalities is untested.
Conclusion
MDTrack demonstrates that replacing uniform fusion with an MoE-based modality-aware mechanism and mixed-token temporal propagation with dual decoupled SSMs yields consistent state-of-the-art results across RGB-T, RGB-D, and RGB-E tracking, with particularly strong gains under unified-modality training. The ablation evidence cleanly attributes improvements to the two proposed components, though cost analysis and verification of expert specialization remain open questions.