Selective Modality Shifting (SMS)

• Selective Modality Shifting (SMS) is a set of techniques that selectively modifies specific modality pathways instead of uniformly fusing all inputs.
• SMS methods use counterfactual perturbations, stage-conditioned gating, and selective routing to diagnose and control modality contributions in complex tasks.
• These approaches are applied across domains—from clinical vision-language diagnostics to robotics—improving performance and robustness by emphasizing targeted multimodal control.

Searching arXiv for papers on selective modality shifting and closely related modality-shifting mechanisms. Selective Modality Shifting (SMS) denotes a family of selective operations in which a system alters, emphasizes, reroutes, or stress-tests only a subset of modalities or modality-conditioned pathways, rather than uniformly modifying the entire multimodal stack. In current literature, the label is used explicitly for a perturbation-based diagnostic of text/image reliance in clinical vision-LLMs (Restrepo et al., 31 Jul 2025). Closely related work studies stage-dependent modality weighting in multimodal video question answering (Kim et al., 2020), context-driven shifts in multimodal sentiment analysis (Kim et al., 2022), neuron- and head-level modality arbitration in speech-text and multimodal LLMs (Nakai et al., 24 Jan 2026, Zhang et al., 3 Feb 2026), switching between language and structured spatial representations (Rajpal et al., 30 Jun 2026), selective modality communication in federated learning (Yuan et al., 2023), adaptive operation under sensor failure (Wen et al., 9 Mar 2026), and timestep-wise modality selection for robotic manipulation (Jiang et al., 20 Apr 2025). An acronymal ambiguity also exists: in integrated photonics, SMS denotes “Selective Mode Splitting,” a boundary-engineering method for shifting one chosen cavity resonance while leaving the rest of the spectrum essentially intact (Lu et al., 2014).

1. Nomenclature, scope, and the selective-local design principle

A recurring property across these works is locality of intervention. In the photonics usage, SMS introduces a sinusoidal radius modulation

$r = r_0 + \alpha \cos(2n\phi)$

to couple the clockwise and counter-clockwise whispering-gallery waves only when the azimuthal mode number matches the modulation order, yielding

$$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$

The selectivity condition is explicit: the targeted resonance is split, while modes with $m \neq n$ remain essentially unchanged (Lu et al., 2014). In nonlinear optics, this resolves frequency matching without re-engineering the whole dispersion landscape.

In multimodal machine learning, the same selective-local motif appears in different guises. Sometimes the selected object is an input modality; sometimes it is a subtask-specific weighting, a sparse internal pathway, a communication channel, or a fallback representation. The central contrast is with global fusion or global invariance assumptions: instead of assuming that all modalities should contribute uniformly, SMS-like methods ask which modality, component, or representation should dominate at a particular stage, depth, or operating condition.

Setting	What is selectively shifted or chosen	Representative mechanism
Integrated photonics	One cavity resonance	Sinusoidal boundary modulation and mode splitting
Clinical VLM diagnosis	Image or text contribution	Counterfactual modality swaps
Multimodal neural architectures	Stage-, layer-, token-, neuron-, or head-level influence	Gating, attention, and causal interventions
Systems under constraints	Uploaded modalities, surviving sensors, or reasoning medium	Priority scores, adaptive fusion, threshold routing

This suggests a useful cross-domain abstraction: SMS is best treated not as a single architecture, but as an Editor’s term for selective multimodal control.

2. SMS as a perturbation-based diagnostic in clinical vision-LLMs

The most explicit machine-learning definition of SMS appears in multimodal clinical AI, where it is introduced as a perturbation-based diagnostic for binary classification over image-text pairs of the form

$(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$

The method evaluates both the unperturbed input and two counterfactual settings constructed from samples with opposing labels: a text swap

$$(I,T',y),\qquad \hat y_I^{(T')} = f_{\mathrm{VLM}}(I,T'),$$

and an image swap

$$(I',T,y),\qquad \hat y_T^{(I')} = f_{\mathrm{VLM}}(I',T).$$

Because the replacement modality comes from the opposite class, the perturbation creates deliberate modality conflict and exposes whether the model follows image evidence or text cues (Restrepo et al., 31 Jul 2025).

The framework is designed to diagnose over-reliance on text, underuse of image evidence, shortcut learning, and poor multimodal integration. Standard metrics are reported in the unperturbed and perturbed settings, together with “Only Text” and “Only Image” ablations. The paper further measures Negative Flip Rate,

$\mathrm{NFR}=\frac{1}{N}\sum_{i=1}^{N}\mathds{1}\bigl(\hat y_i^{(\mathrm{shift})}\neq y_i,\;\hat y_i=y_i\bigr),$

to quantify how often a previously correct prediction becomes wrong after a modality shift, and Expected Calibration Error with 10 bins,

$$\mathrm{ECE}=\sum_{m=1}^{M}\frac{|B_m|}{n}\left|\mathrm{acc}(B_m)-\mathrm{conf}(B_m)\right|.$$

The empirical setting consists of zero-shot evaluation on MIMIC-CXR and FairVLMed. The paper uses 10k random test images from MIMIC-CXR and the full 2k-image test set from FairVLMed, and evaluates six open-source VLMs, including LLaVA 1.5, LLaVA variants, Qwen2-VL, Llama 3.2 10B, Janus-Pro, Med-LLaVA, and MedGemma, on a single A100 GPU with fp16 precision, temperature $=0$, and no sampling (Restrepo et al., 31 Jul 2025).

The reported pattern is strong textual bias. Under Text Shift, several models show large performance drops; Qwen-2 VL and LLaVA 1.5 are reported with NFR values above 0.60. Image swapping often causes much smaller degradation, and many models remain relatively functional with text alone but degrade severely with image alone. Calibration also worsens under text shift, and for some models becomes “inverted,” so confidence no longer tracks correctness. The attention-based analysis decomposes token attention as

$$A_t=[A_t^{(T)},A_t^{(I)}],\qquad A_t(\mathrm{BOS})=0,$$

and the qualitative finding is that image-token attention remains relatively stable, whereas text-token attention changes more dynamically during decoding and generated tokens attend much more strongly to textual input. Within this usage, SMS is therefore not a fusion module but a causal-style stress test for modality reliance.

3. Stage-conditioned and context-conditioned shifting mechanisms

A different line of work implements modality shifting directly inside the model. In multimodal video question answering, the Modality Shifting Attention Network (MSAN) formalizes the idea that temporal localization and answer prediction may require different modalities. Its architecture separates the task into a Moment Proposal Network (MPN) for temporal moment localization and a Heterogeneous Reasoning Network (HRN) for answer prediction, with Modality Importance Modulation (MIM) supplying question-dependent modality weights at both stages (Kim et al., 2020).

For localization, the model computes

$$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$0

then modulates the video and subtitle moment scores as

$$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$1

For answer prediction, it computes

$$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$2

The modulation functions include additive, multiplicative, and residual forms. On TVQA, the full model with BERT, visual concepts, and action concepts reports 71.13% test accuracy, and ablations show that MIM improves both localization and answer prediction (Kim et al., 2020).

In multimodal sentiment analysis, CMSBERT-CLR extends the shifting idea from stage selection to context-conditioned token representation. Its Context-driven Modality Shifting (CMS) mechanism computes visual and acoustic gates

$$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$3

builds a non-verbal contribution

$$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$4

and then averages over the whole utterance,

$$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$5

The shifted vector enters the self-attention computation through the query, key, and value streams: $$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$6 The design is explicitly proposed as a generalized version of MAG-BERT, replacing same-timestep non-verbal shifting with whole-utterance context, and combined with contrastive learning across linguistic, visual, and acoustic representations. The reported results on CMU-MOSI and CMU-MOSEI are state of the art for the compared baselines (Kim et al., 2022).

In contact-rich robotics, cross-modality attention (CMA) plays a similar role but at the level of action generation and skill structure. The model encodes third-person camera, gripper camera, proprioception/state, and tactile data into a shared 128-dimensional latent space, stacks observations from two consecutive timesteps, applies CMA with 8 attention heads and 2 transformer layers, and uses the fused output as the conditioning signal for a 1D conditional U-Net diffusion policy (Jiang et al., 20 Apr 2025). The attention weights of the last transformer layer differ across six manually defined primitives—Reach Base, Grip and Move Base, Reach Leg, Grip and Move Leg, Insert, and Screw—and these patterns are used to segment expert demonstrations into primitive skills. Here, selectivity is soft and time-varying: modalities are not dropped discretely, but their contribution changes with the primitive phase.

4. Internal circuitry of modality arbitration and incomplete invariance

Mechanistic studies of multimodal transformers increasingly show that modality shifting is localized rather than uniformly distributed. In the neuron-level analysis of SeamlessM4T v2, language- and modality-selective neurons are identified by average-precision ranking after mean-pooling intermediate activations,

$$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$7

The analysis distinguishes unimodal language-specific neurons, multimodal language-specific neurons, modality-specific neurons, and language-modality-specific neurons, with the modality-involving categories defined only for the shared decoder (Nakai et al., 24 Jan 2026).

The causal intervention is median replacement: selected neurons have their activations replaced by the median of their activation distributions, and the effect is compared with matched random-neuron controls. Across intervention sizes $$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$8 to $$\omega_m^\pm = \omega_m \pm \beta_m,\qquad \beta_m = g\alpha\delta_{mn}.$$9, unimodal language neurons, multimodal language neurons, and language-modality neurons typically do not catastrophically damage performance relative to random ablations, but modality-specific neurons do. The strongest disruption appears in text-to-text translation, where ablating modality-specific neurons causes collapse at all intervention sizes; speech-to-text also degrades strongly at larger $m \neq n$0. The same study reports that modality-specific neurons are concentrated almost entirely in cross-attention, especially $m \neq n$1-proj and $m \neq n$2-proj, and gives Gini coefficients of 0.900 for speech-to-text modality-specific neurons and 0.980 for text-to-text modality-specific neurons, compared with 0.774 and 0.607 for other neuron types. Speech-conditioned decoding and non-dominant scripts, especially Chinese and Japanese, show higher activation concentration. The resulting picture is partial modality invariance with a sharply localized decoder bottleneck (Nakai et al., 24 Jan 2026).

A complementary head-level analysis in multimodal LLMs describes modality following as causal arbitration under cross-modal conflict. The residual stream is written as

$m \neq n$3

and the claim is that instruction tokens act as structural anchors for modality arbitration. Shallow attention layers perform largely non-selective transfer into these anchors as a latent buffer; deep attention layers resolve the competition according to instruction intent; and MLP layers exhibit semantic inertia, acting as an adversarial force (Zhang et al., 3 Feb 2026).

This work identifies a sparse set of deep attention heads—modality-specific experts plus modality-shared hubs—by ranking their contribution to the modality arbitration margin. Causal interventions are then performed by blocking heads, $m \neq n$4, or amplifying them. Blocking the top 40 heads, about 5% of the total, causes about a 60% absolute drop in modality-following ratio, while amplifying the same sparse set restores modality following by about 60% on failure samples. Shared heads alone are insufficient; they must operate together with modality-specific heads. The implication is that SMS in MLLMs is not an opaque output-layer switch, but a sparse, instruction-centered attention process (Zhang et al., 3 Feb 2026).

Taken together, these studies reject any strong equation of SMS with fully distributed modality invariance. The evidence instead favors incomplete invariance, sparse arbitration loci, and localized failure modes.

5. Routing under complexity, communication, and sensor failure

Another class of SMS-like methods moves the selectivity decision outside the core fusion block and turns it into explicit routing. In spatial reasoning, a training-free framework decides whether to remain in natural language or switch to a grid-based representation. The switching metric combines trustworthiness

$m \neq n$5

with complexity $m \neq n$6, and the rule is to stay in language if

$m \neq n$7

or equivalently $m \neq n$8 and $m \neq n$9; otherwise the system routes to grid reasoning (Rajpal et al., 30 Jun 2026). Faithfulness is defined from sufficiency and necessity, plausibility from paraphrase stability and flip consistency, and complexity from Support Burden, Chain Length, Selection Difficulty, Hard Language, Diagonal Burden, Entity Load, and Coreference Difficulty. Thresholds are tuned on validation data, and short-circuit computation avoids unnecessary score estimation. The headline empirical result is that switching from natural language reasoning to a grid-based representation improves performance by up to 42%, and switching can recover up to 82.4% of text-based reasoning failures in some cases (Rajpal et al., 30 Jun 2026).

In federated multimodal learning, selectivity becomes a communication policy. FedMFS assumes that each client $(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$0 holds a local multimodal dataset

$(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$1

and corresponding modality-specific models

$(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$2

with $(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$3 varying across clients. Instead of uploading all modality models, each client computes a normalized Shapley-based importance score and a normalized model-size penalty, then forms the priority

$(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$4

and uploads only the top-$(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$5 modalities (Yuan et al., 2023). On ActionSense, the highlighted configuration $(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$6, $(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$7, $(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$8 achieves 97.34% accuracy with 0.72 MB communication overhead, while the abstract reports over 4x communication reduction relative to baselines (Yuan et al., 2023). Here, the “shift” is neither representational nor inferential: it is selective modality communication under resource constraints.

In collaborative perception, selectivity is required for continued operation when a sensor disappears. SiMO addresses the failure of standard multimodal fusion under missing modalities by aligning features into a shared semantic space before fusion and using Length-Adaptive Multi-Modal Fusion (LAMMA), whose query length adapts to the available modalities. When one modality is absent, the mechanism automatically downgrades to self-attention on the remaining modality rather than changing the architecture or requiring explicit failure detection (Wen et al., 9 Mar 2026). The method is trained with a Pretrain-Align-Fuse-RD schedule: independent unimodal pretraining, alignment with frozen extractors, shared-module fusion with frozen branches, and Randomly Modality Drop fine-tuning. The paper further reports that Procrustes disparity between camera and LiDAR features drops from 0.6747 to 0.0472 after LAMMA, and that existing multimodal baselines fail when LiDAR is absent whereas SiMO remains operable with $(I,T,y),\qquad \hat y = f_{\mathrm{VLM}}(I,T).$9, $$(I,T',y),\qquad \hat y_I^{(T')} = f_{\mathrm{VLM}}(I,T'),$$0 only, or $$(I,T',y),\qquad \hat y_I^{(T')} = f_{\mathrm{VLM}}(I,T'),$$1 only (Wen et al., 9 Mar 2026). In this setting, SMS is best understood as single-modality operability with preserved downstream semantics.

6. Common patterns, limitations, and significance

Several misconceptions are clarified by this literature. First, SMS is not synonymous with full modality invariance. The speech-text analysis of SeamlessM4T v2 finds incomplete modality invariance and localized decoder dependence on cross-attention $$(I,T',y),\qquad \hat y_I^{(T')} = f_{\mathrm{VLM}}(I,T'),$$2- and $$(I,T',y),\qquad \hat y_I^{(T')} = f_{\mathrm{VLM}}(I,T'),$$3-projections, while the MLLM arbitration study locates decision finalization in instruction anchors and deep attention heads rather than in a modality-agnostic residual stream (Nakai et al., 24 Jan 2026, Zhang et al., 3 Feb 2026). Second, SMS is not necessarily a hard switch. The field includes hard counterfactual swaps in clinical diagnostics, threshold routing in spatial reasoning, top-$$(I,T',y),\qquad \hat y_I^{(T')} = f_{\mathrm{VLM}}(I,T'),$$4 communication in federated learning, and modal dropout in collaborative perception, but also continuous gating and attention-based weighting in sentiment analysis, video QA, and robotics (Restrepo et al., 31 Jul 2025, Rajpal et al., 30 Jun 2026, Yuan et al., 2023, Wen et al., 9 Mar 2026, Kim et al., 2022, Kim et al., 2020, Jiang et al., 20 Apr 2025).

Third, selectivity does not eliminate brittleness. In clinical VLMs, text shifts can invert calibration and produce confidently wrong predictions (Restrepo et al., 31 Jul 2025). In speech-text models, modality-selective neurons are highly concentrated, which the authors connect to brittleness across modalities and languages (Nakai et al., 24 Jan 2026). In MLLMs, manipulating a sparse set of arbitration heads strongly changes modality-following behavior, which simultaneously reveals controllability and exposes a structural bottleneck (Zhang et al., 3 Feb 2026). In spatial reasoning, gains from switching depend on reliable relation extraction and grid construction, and the framework explicitly notes computational overhead and downstream propagation of extraction errors (Rajpal et al., 30 Jun 2026). In collaborative perception, random modality drop without the full training strategy can hurt performance, showing that selective robustness is not an automatic byproduct of fusion (Wen et al., 9 Mar 2026).

A broader significance nonetheless emerges. Across optical parametric resonators, clinical VLMs, MVQA, sentiment models, speech-text systems, MLLMs, federated learning, spatial reasoning, robotics, and collaborative perception, the same high-level engineering preference reappears: alter only the component that must move, preserve the rest of the system as much as possible, and make modality dependence explicit rather than implicit. In photonics this is expressed by the exact selectivity condition $$(I,T',y),\qquad \hat y_I^{(T')} = f_{\mathrm{VLM}}(I,T'),$$5 for one chosen resonance (Lu et al., 2014). In multimodal AI, the corresponding selectivity may be expressed as counterfactual perturbation, question-conditioned weighting, sparse head intervention, priority-based communication, threshold-based representation switching, or single-modality-operable fusion. The literature therefore supports a unified view of SMS as selective multimodal control, while also showing that the concrete object of selection—input, pathway, representation, communication channel, or sensor branch—varies substantially by domain.