MODS: Dynamic Primary Modality Selection

• MODS is a dynamic multimodal framework that uses entropy-based and submodular optimization to select and weight modality subsets.
• It employs differentiable gating and adaptive scheduling with representation disentanglement to reduce redundancy and computational overhead.
• Empirical evaluations show robust performance in tasks like classification, segmentation, and tracking, with significant resource savings.

Modality Optimization and Dynamic Primary Modality Selection Framework (MODS) encompasses a family of algorithmic principles and neural architectures for instance- or batch-level selection, weighting, and routing of multimodal information streams (e.g., visual, linguistic, acoustic, biomedical, sensor, or derived expert modalities) under resource, reliability, and semantic constraints. MODS systems aim to maximize downstream performance (classification, regression, fusion, segmentation, tracking) while minimizing redundancy, computational overhead, and reliance on any single modality. Core ingredients include submodular or entropy-based optimization, differentiable gating or scheduler networks, representation disentanglement, and dynamic fusion/selection modules adapted to sample, context, or task.

1. Mathematical Foundations: Utility and Selection Criteria

MODS frameworks are generally built on formal criteria that quantify the informativeness and complementarity of modality subsets. The canonical formulation (as in greedy selection via approximate submodular maximization (Cheng et al., 2022)) defines a utility function $U(S)$ for any modality subset $S \subseteq V$: $$U(S) = \inf_{c \in \mathcal{Y}} \mathbb{E}\left[\ell(Y, c)\right] - \inf_{h \in \mathcal{H}}\mathbb{E}\left[\ell(Y, h(S))\right]$$ For cross-entropy binary classification, this reduces to mutual information $U(S) = I(S; Y) = H(Y) - H(Y|S)$. The optimization problem seeks

$$\max_{S \subseteq V} U(S)\quad\text{s.t.}\quad |S| \leq q$$

or, more generally, under a knapsack cost constraint.

Submodularity (diminishing returns) and monotonicity enable greedy algorithms to achieve near-optimal approximation guarantees, e.g.,

$U(S_q) \geq (1-1/e) U(S^*) - q\epsilon$

when $U$ is $\epsilon$-approximately submodular. These conditions can be generalized to entropy-based selection (STORM (Kamboj et al., 3 Dec 2024)), which uses class-wise entropy-imbalance gain, and resource-aware selection (DeepSuM (Gao et al., 3 Mar 2025)), which folds modality costs into the scoring function and regularizer.

2. Dynamic Selection Mechanisms and Gating Models

MODS designs employ dynamic selection of a primary modality (or set of modalities) on a per-sample, per-batch, or per-epoch basis. Mechanisms include:

• Greedy, submodular maximization (as above), with online/adaptive extensions for streaming or nonstationary input (Cheng et al., 2022).
• Entropy-imbalance gain maximization (STORM), with cascading partitioning and Gini impurity thresholds to isolate rare classes (Kamboj et al., 3 Dec 2024).
• Differentiable scheduling via modality significance networks (DeepSuM), softmax/gumbel-softmax gating, and temperature annealing for exploration/commitment (Gao et al., 3 Mar 2025).
• Sample-adaptive selectors (MSelector in MSA MODS (Yang et al., 9 Nov 2025)), where concatenated unimodal vector embeddings are scored via MLP and primary index assigned per instance.
• Soft weighting via signal-based schedulers aggregating confidence (predictive entropy), uncertainty (Monte Carlo dropout), and semantic alignment (cosine similarity) (Tanaka et al., 15 Jun 2025): $$\omega_m(x) = \frac{\exp(\alpha c_m - \beta u_m + \gamma s_m)}{\sum_{j=1}^M \exp(\alpha c_j - \beta u_j + \gamma s_j)}$$
• Auto-selector modules (DAS in UASTrack), which classify modality type from global-pooled auxiliary features to route further processing (Wang et al., 25 Feb 2025).

Key empirical findings indicate that dynamic, adaptive selection outperforms static fusion or fixed primary strategies, especially under modality imbalance, sensor noise, or dropout.

3. Representation Compression, Redundancy Reduction, and Feature Alignment

MODS architectures typically incorporate module(s) for unimodal feature compression, redundancy filtering, and cross-modal alignment. Examples include:

• Graph-based Dynamic Sequence Compressor (GDC) (Yang et al., 9 Nov 2025), using capsule networks and graph convolution to map acoustic/visual sequences to compact graphs, reducing sequential redundancy.
• Sufficient, independent, and normal encoders (DeepSuM MODS (Gao et al., 3 Mar 2025)) trained with distance-covariance penalties for representation independence and f-divergence for normality.
• Multi-modal Aggregation Module (MAM, MAGIC (Zheng et al., 16 Jul 2024)), treating modalities symmetrically via shared backbones and pooling-based aggregation.
• Multi-scale feature extraction and hierarchical selection in MAGIC++ (Zheng et al., 22 Dec 2024), aggregating features at each scale and using cosine-similarity to benchmark robustness and fragility.

These modules facilitate robust, compact fusion while avoiding representation collapse and excessive computational cost.

4. Cross-modal Interaction and Fusion Strategies

Fusing information from selected modalities is handled via attention, cross-attention, or aggregation modules:

• Primary-modality-centric cross-attention (PCCA (Yang et al., 9 Nov 2025)), where the dominant modality is the center for bi-directional fusion, with auxiliary modalities contributing via query-key attention blocks and skip connections.
• Multi-modal Interaction Module (MIM, MAGIC++), employing channel-wise and spatial-wise attention to promote cross-modal complementarity across feature scales (Zheng et al., 22 Dec 2024).
• Scheduled soft fusion by instance-dependent weights (blended fusion head), ensuring the fused embedding remains close to unimodal features as weighted by the scheduler (Tanaka et al., 15 Jun 2025).
• Task-customized optimization adapters (TCOA, UASTrack) which are applied conditioned on dynamically selected primary modality (Wang et al., 25 Feb 2025).

Losses may include task prediction terms (classification, regression, segmentation), cross-modal consistency (InfoNCE, KL or cosine regularization), selection regularizers, and modality importance or cost penalties.

5. Algorithmic Schemes: Selection, Training, and Inference

The algorithmic workflow typically comprises:

• For each batch/sample, extract representations for all modalities.
• Score or rank modalities via utility, entropy gain, confidence-uncertainty-alignment, or learned significance head.
• Select (hard) or weight (soft) modalities for fusion/processing.
• Aggregate/fuse representations with attention, pooling, or learned interaction blocks.
• Apply loss functions including task, regularization, cost/importance, and consistency terms.
• Backpropagate across entire MODS pipeline; use annealing strategies, or beam search over joint modality subsets.
• During inference, MODS modules dynamically adapt to available modalities, sensor failures, or domain shifts. Many designs are permutation-invariant; selection works with arbitrary modality subsets (MAGIC/MAGIC++).

Pseudocode for scheduling MODS steps is provided in sources such as (Tanaka et al., 15 Jun 2025, Kamboj et al., 3 Dec 2024), and (Zheng et al., 22 Dec 2024).

6. Resource Optimization, Scalability, and Empirical Performance

MODS frameworks explicitly address resource efficiency by minimizing the number and cost-weighted contribution of modalities:

• Resource-aware selection regularizers (DeepSuM MODS), direct cost subtraction in gating, and Pareto-efficient tradeoff tuning (Gao et al., 3 Mar 2025).
• Empirical results reveal that MODS pipelines achieve comparable or superior accuracy/fusion performance with reduced compute: e.g., DeepSuM MODS uses on average 1.3 modalities (vs. K), reducing compute by ~60% (Gao et al., 3 Mar 2025); MAGICS attains a 19–20 % mIoU improvement in modality-agnostic segmentation with 42% of parameters (Zheng et al., 16 Jul 2024); UASTrack adds only 1.87 M parameters yet closes the performance gap to full fine-tuning across five tracking benchmarks (Wang et al., 25 Feb 2025).
• Robustness to sensor failures, modality corruption, and domain shift is consistently improved, as in MAGIC++, DMS/MODS for BLIP-2/LLaVA models, and entropy-driven MODS for rare biomedical tasks.

7. Blueprint Extensions and Generalization

The MODS paradigm is instantiated in diverse domains: sentiment analysis (Yang et al., 9 Nov 2025), rare event detection (Kamboj et al., 3 Dec 2024), semantic segmentation (Zheng et al., 16 Jul 2024, Zheng et al., 22 Dec 2024), large multimodal transformers (Tanaka et al., 15 Jun 2025), and multimodal tracking (Wang et al., 25 Feb 2025). Commonalities include symmetric treatment of modalities, instance-level selection, cost-robust tradeoff optimization, and flexible adaption to arbitrary modality sets.

Generalizable mechanisms include cosine-based benchmarking at multiple levels of representation (Zheng et al., 22 Dec 2024), entropy-regularized selection and fusion, soft scheduling via confidence-uncertainty-semantic alignment signals (Tanaka et al., 15 Jun 2025), hierarchical selection and multi-scale fusion, and unified parameter optimization across tasks. These approaches collectively define a robust, scalable framework for next-generation multimodal AI systems in dynamic, resource-constrained, and noisy environments.

---

In summary, Modality Optimization and Dynamic Primary Modality Selection (MODS) encompasses a suite of theoretically principled, empirically validated architectures and protocols for adaptive modality ranking, routing, fusion, and resource-efficient multimodal processing, generalizable across segmentation, classification, retrieval, tracking, and generative tasks.