Unified Modal Training

• Unified Modal Training is an approach that utilizes a single transformer backbone to process diverse data types through shared parameters and cross-modal fusion.
• It integrates multiple modalities using adaptive gating, conditional fusion, and token-level mechanisms to enhance multi-task performance and efficiency.
• The methodology achieves state-of-the-art results on benchmarks by employing one-shot joint training, temporal token propagation, and balanced loss optimization.

Unified Modal Training

Unified modal training refers to a class of architectures, objectives, and optimization regimes designed to train a single model—usually a shared transformer backbone—across multiple data modalities so that the resulting system supports multi-modal understanding, generation, and inference with a unified set of parameters, weights, and forward passes. This paradigm eliminates the need for modality-specific architectures or separate training for each modality/task, instead achieving genericity and task transfer through shared representations, flexible fusion, and parameter sharing mechanisms. Unified modal training is a foundational approach across computer vision, language, audio, robotics, and multi-modal video understanding.

1. Principles and Goals of Unified Modal Training

Unified modal training pursues three foundational objectives:

1. Modality-Invariant Architecture: Single backbone or core model supports any combination of target modalities (e.g., RGB, depth, thermal, event, text, audio) with either identical weights or lightweight modality adapters, and without per-task architectural changes or finetuning (Zheng et al., 27 Jul 2025).
2. Parameter Sharing and Multi-Task Generalization: The entire set of model parameters is updated with signals from all modalities and tasks. No branch or task requires isolated optimization or dedicated weights; all loss functions and gradients propagate through the same backbone (Li et al., 2020, Zeng et al., 2022, Zheng et al., 27 Jul 2025).
3. Cross-Modal Fusion and Alignment: Unified training objectives are designed to align modalities into a shared space, either through cross-modal contrastive learning, mutual information maximization, or joint attention/fusion modules operating directly across modalities (Li et al., 2020, Zeng et al., 2022, Zheng et al., 27 Jul 2025).

Benefits include:

• Reduced computational and deployment cost.
• Maximal transfer and synergy across tasks/modality pairs.
• Robustness to missing or noisy modalities.

2. Core Architectural Components and Design Strategies

The implementation of unified modal training requires architectural modules that enable cross-modal fusion and modality invariance:

Tokenization and Modal Encoders

• Modality Tokenizers: A single 2D convolutional layer applied to any input frame (RGB, TIR, depth, event) yields a consistent token grid for all visual streams (Zheng et al., 27 Jul 2025).
• Shared Transformer Backbones: All modalities flow into a shared vision transformer or encoder stack; text and audio modalities use similar Transformer or BERT/ViT stacks, sometimes with per-modality pre-nets for adaptation (Zheng et al., 27 Jul 2025, Li et al., 2020, Ao et al., 2021).

Gated and Conditional Fusion Modules

• Conditional Gates: Residual fusion modules (parameter-sharing 2-layer MLPs with gated activations) inserted between backbone layers inject auxiliary modality signals with channel alignment, enabling on-the-fly fusion without duplication (Zheng et al., 27 Jul 2025).
• Cross-Modal Attention: Gated Modal-Scalable Perceivers (GMP) or related cross-attention modules aggregate features and temporal tokens from all active modalities, compressing them for downstream heads (Zheng et al., 27 Jul 2025, Li et al., 2020).

Temporal and Sequential Aggregation

• Online Temporal Token Propagation: Trainable temporal tokens are associated with each time step (frame). These tokens are updated via attention and carried forward as a compressed trajectory/appearance prompt for subsequent predictions (Zheng et al., 27 Jul 2025).
• Temporal Prompting: During inference, these tokens are recursively updated and serve as “temporal prompts,” embedding history and memory into the current frame prediction pipeline (Zheng et al., 27 Jul 2025).

3. Training Methodology and Loss Functions

Unified modal training involves specialized batch and loss formulation to mix modalities and enforce alignment:

• One-Shot Joint Training Scheme: All modalities are concatenated into unified training batches. Loss is computed across all present modalities/tasks and is simultaneously backpropagated into the shared parameter set (Zheng et al., 27 Jul 2025). Empirically, this is found to outperform separate per-task or independent modality training by 1–1.5% on average, with additional efficiency by obviating the need for finetuning.
• Per-Frame Loss Composition: A total loss $\mathcal{L}_{\text{total}}$ per frame is a weighted sum of classification (focal loss), $\ell_1$ regression, and GIoU losses, e.g.,

$$\mathcal{L}_{\text{total}} = \mathcal{L}_{\text{cls}} + \lambda_1\mathcal{L}_1 + \lambda_2\mathcal{L}_{\text{GIoU}}, \quad \lambda_1=5,\,\lambda_2=2$$

• Token-Level and Cross-Modal Losses: In other unified frameworks, objectives include masked region modeling, cross-modal contrastive loss (mutual information maximization), and cross-modal matching to enforce alignment of learned representations (Li et al., 2020, Zeng et al., 2022).

4. Temporal Propagation and Inference Paradigm

Unified modal trackers and sequential models leverage token propagation and memory mechanisms for efficient streaming inference:

• Temporal Token Update and Propagation: At each transformer layer and time $t$, the temporal token $T_t$ is updated (auto-regressively incorporating new “empty” slots at $t+1$) and participates in attention alongside sampled reference and search frames. This propagation process ensures that cues regarding appearance and motion are efficiently summarized and available for future inference (Zheng et al., 27 Jul 2025).
• Inference Loop: For each frame $t$ in a sequence, the temporal tokens are advanced, fusing current reference frames, search frame, and historical information into the forward pass. At test time, the loop maintains state (memory, temporal tokens) for a single pass per video, using past information to prompt the current prediction (Zheng et al., 27 Jul 2025).

5. Modality-Scalable Fusion and One-Shot Multi-Modal Generalization

A defining feature is robust performance across arbitrary combinations of input modalities, enabled by parameter sharing and adaptive gating:

• Parameter Scalability Across Modalities: Instead of fine-tuning for each modality combination, the same architecture and weight set are used without modification for RGB, RGB+TIR, RGB+D, RGB+E, or mixtures thereof (Zheng et al., 27 Jul 2025).
• Cross-Modal Adaptation via Gated Perceivers: Adaptive multi-head attention with learned gating fuses visible and auxiliary streams at each layer—crucially, these fusion weights are also shared across modality pairs (Zheng et al., 27 Jul 2025).
• Unified Generalization: Models trained via unified modal protocols empirically match or surpass state-of-the-art per-modal trackers on benchmarks (LaSOT, GOT10k, TrackingNet, LasHeR, DepthTrack, VisEvent) for both single-modal and multi-modal tasks. Example result: UM-ODTrack 384 achieves AUC gains of +1.5% (LaSOT), +1.2% (TrackingNet), SOTA on GOT10k, and new SOTA on RGB-T (LasHeR: 71.0%), RGB-D (DepthTrack: 0.69), and RGB-E (VisEvent: 62%) (Zheng et al., 27 Jul 2025).

6. Comparative Performance and Empirical Insights

Ablation studies and benchmark evaluations demonstrate efficacy and provide engineering guidelines:

Component/Strategy	Effect on AUC or SOTA Gain	Complexity / Param Change
Temporal Token Propagation	+1.2–1.8% AUC (vs. frame-pair)	No param increase
Gated Perceivers	+1% improvement (multi-modal)	Minimal param overhead
One-Shot Joint Training	1–1.5% AUC > per-task training	Streamlines storage, no finetune

• Ablation Evidence: Removing temporal token propagation or unified gating reduces accuracy 1.2–1.8% AUC; one-shot training further improves compared to isolated per-modality models (Zheng et al., 27 Jul 2025).
• Scalability and Efficiency: No architectural modifications or storage expansion is required to add modalities; joint batches and a single forward path minimize training and inference burden.
• Benchmark Results: Unified models set new SOTA on multi-modal and visible-only tracking challenges, consistently outperforming specialized or separately trained networks (Zheng et al., 27 Jul 2025).

7. Extensions, Limitations, and Future Work

• Extensibility to Additional Modalities and Tasks: The conditional gate and cross-attention design patterns extend naturally to further sensory streams beyond vision (e.g., event, language, or audio cues).
• Possible Limitations: The success of one-shot unified modal training depends on strong batch sampling and balanced loss propagation across modalities and temporal segments.
• Future Directions: Research directions include dynamically adaptive gating for missing modalities, expansion to arbitrary numbers of input channels, and efficiency improvements for large-scale sequential inference (Zheng et al., 27 Jul 2025).

---

Unified modal training, as exemplified by UM-ODTrack (Zheng et al., 27 Jul 2025), provides a rigorous, principled methodology for achieving universal, parameter-efficient, and high-accuracy multi-modal models, with well-characterized mechanisms for cross-modal fusion, temporal propagation, and multi-task scalability. This paradigm is now foundational in unified perception and tracking across several computer vision and robotics domains.