Dark Side of Modalities (DSoM) in Multimodal Systems

• DSoM is a cluster of multimodal failure phenomena where added modalities become harmful by introducing misleading, exploitable, and non-transferable signals.
• Research shows that fusing multiple modalities can trigger vulnerabilities such as modality sabotage, latent space attacks, and breakdowns in robustness and calibration.
• The framework advocates treating each modality as a potential adversarial source to enhance system security and reliability in tasks like multimodal knowledge graph reasoning and forensics.

Searching arXiv for papers on "Dark Side of Modalities" and closely related multimodal failure modes. Dark Side of Modalities (DSoM) denotes a cluster of multimodal failure phenomena in which modalities cease to be purely complementary and instead become harmful, misleading, exploitable, or non-transferable. In recent work, the term has been used for unhelpful modalities in multimodal knowledge graph reasoning, overconfident unimodal streams that sabotage fusion, shared-latent-space vulnerabilities in connector-based multimodal LLMs, controlled failures under cross-modal dissonance, and generalization breakdown on unseen “dark modalities” in multimodal forensics (Zhao et al., 28 Jul 2025, Zhang et al., 4 Nov 2025, Wang et al., 8 May 2026, Nazi et al., 28 Mar 2026, Dou et al., 9 Apr 2026). Earlier adversarial evaluation of multimodal classifiers already established a security precursor to this line of thought by showing that attacking multiple modalities can induce errors in up to 73% of cases, with unimodal image attacks averaging 45% induced errors and character-based text augmentation attacks averaging 30% (Evtimov et al., 2020).

1. Research Scope and Core Meanings

Across the literature, DSoM is not a single formalism but a recurring diagnosis: adding or coupling modalities can introduce distinct liabilities that are not visible in unimodal evaluation. In multimodal knowledge graph reasoning, the term refers simultaneously to “dark knowledge from non-target entities” and to the “dark side in unhelpful modalities,” where irrelevant or misleading information should be excluded rather than merely down-weighted (Zhao et al., 28 Jul 2025). In multimodal reasoning audits, the same family of concerns appears as “modality sabotage,” where a high-confidence unimodal error overrides other evidence and misleads the fused result (Zhang et al., 4 Nov 2025). In multimodal security, DSoM refers to vulnerabilities that arise when modalities share a latent space, so that a compromise in one modality’s connector can propagate malicious intent across all modalities (Wang et al., 8 May 2026). In robustness benchmarks, the term names failures that emerge when modality consensus is deliberately broken through corruption, conflict, or abstention stress tests (Nazi et al., 28 Mar 2026). In multimodal forensics, “dark modalities” are previously unseen or even unalignable channels that expose the modality-binding bottleneck of artifact-driven detectors (Dou et al., 9 Apr 2026).

Setting	DSoM formulation	Representative source
MKGR	Unhelpful modalities and dark knowledge from non-target entities	(Zhao et al., 28 Jul 2025)
Multimodal reasoning audit	Modality sabotage by an overconfident wrong stream	(Zhang et al., 4 Nov 2025)
MLLM security	Shared-latent-space vulnerability via a connector	(Wang et al., 8 May 2026)
Omni-modal robustness	Broken modality consensus under controlled corruption	(Nazi et al., 28 Mar 2026)
Multimodal forensics	Failure on unseen “dark modalities”	(Dou et al., 9 Apr 2026)

Taken together, these usages describe a broad shift away from the assumption that more modalities monotonically improve performance. A plausible implication is that multimodal system design must treat each modality as a potential source of adversarial leverage, calibration error, or distributional mismatch, not merely as an auxiliary signal.

2. Diagnostic and Attributional Formalisms

One of the clearest DSoM diagnostics is the “modality-as-agent” layer for multimodal emotion recognition. Each modality $m \in \{T,A,V,TAV\}$ produces candidate labels with raw confidence scores $S_m(y)\in[0,100]$ and a self-assessment score $q_m\in[0,1]$. These are normalized as

$$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$

For threshold $\tau$ with default $\tau=0.70$, potential sabotage is defined by

$$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$

and successful sabotage additionally requires that the fused decision satisfies $\hat y = y_m$. Fusion is computed by unweighted vote summation,

$$\tilde s(y)=\sum_{m\in\{T,A,V,TAV\}} w_m S_m(y), \quad w_m=1,$$

followed by normalization and Top-1 prediction; an ablation with $w_m=q_m$ was possible, but plain confidence fusion gave the best Top-1 accuracy (Zhang et al., 4 Nov 2025).

Other DSoM work formalizes failure through intervention and attribution rather than sabotage thresholds. In OMD-Bench, every sample begins with congruent tri-modal content and is then corrupted by patterns $S_m(y)\in[0,100]$0, yielding metrics for Abstention Alignment Score,

$S_m(y)\in[0,100]$1

Expected Calibration Error,

$S_m(y)\in[0,100]$2

and robustness under corruption $S_m(y)\in[0,100]$3 (Nazi et al., 28 Mar 2026). In multimodal diffusion backdoors, modality dominance is quantified by Trigger Modality Attribution (TMA), defined as a Shapley value $S_m(y)\in[0,100]$4 over coalition values $S_m(y)\in[0,100]$5, and Cross-Trigger Interaction (CTI),

$S_m(y)\in[0,100]$6

with $S_m(y)\in[0,100]$7 indicating interference or redundancy rather than synergy (Wang et al., 6 Mar 2026).

These formalisms share a common methodological commitment: DSoM is studied by decomposing multimodal behavior into per-modality contributions, then asking whether fusion, consensus, or joint triggering genuinely adds value or merely hides a dominant or corrupted stream.

3. Adversarial and Backdoor Manifestations

The adversarial lineage of DSoM begins with realistic gray-box assumptions for multimodal classifiers. Under partial model knowledge and access, multimodal attack methodologies were developed for the Hateful Memes Challenge classification task, and attacking multiple modalities yielded stronger attacks than unimodal attacks alone, inducing errors in up to 73% of cases. Within the explored setting, unimodal image attacks on multimodal classifiers were stronger than character-based text augmentation attacks, inducing errors on average in 45% and 30% of cases, respectively (Evtimov et al., 2020).

In connector-based multimodal LLMs, DSoM is tied to the shared interface between a frozen encoder $S_m(y)\in[0,100]$8, a lightweight connector $S_m(y)\in[0,100]$9, and a frozen LLM head $q_m\in[0,1]$0. The attacker chooses a target output $q_m\in[0,1]$1 and a latent backdoor region

$q_m\in[0,1]$2

then poisons only the connector so that $q_m\in[0,1]$3 for a seed sample $q_m\in[0,1]$4. The poisoning loss is

$q_m\in[0,1]$5

and cross-modal activation is obtained by extracting a malicious centroid

$q_m\in[0,1]$6

and optimizing perturbations $q_m\in[0,1]$7 so that unseen inputs from other modalities move toward $q_m\in[0,1]$8. On PandaGPT and NExT-GPT, the attack achieved up to 99.9% attack success rate in same-modality settings, most cross-modal settings exceeded 95.0% ASR under bounded perturbations, and weight-cosine similarity remained above 0.97 relative to benign connectors (Wang et al., 8 May 2026).

A different security form of DSoM appears in multimodal diffusion models as Backdoor Modality Collapse. Even when triggers are injected into every modality during training, the activated backdoor can collapse to a strict subset $q_m\in[0,1]$9, producing a “winner-takes-all” dynamic rather than synergistic cross-modal vulnerability. In experiments with InstructPix2Pix fine-tuned via LoRA on CelebA, TMA and CTI showed pronounced text dominance: $$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$0–$$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$1, $$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$2–$$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$3, and $$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$4 throughout. A unimodal-only sanity check showed that each trigger alone was functional, with image-only and text-only $$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$5, so the collapse was not due to a broken trigger (Wang et al., 6 Mar 2026).

These studies establish DSoM as a security property of multimodal architectures. Shared latent spaces, multimodal fusion, and multi-trigger training do not guarantee robust composition; they can instead concentrate vulnerability into a compact interface or a single dominant modality.

4. DSoM as a Training Principle in Multimodal Knowledge Graph Reasoning

In multimodal knowledge graph reasoning (MKGR), DSoM is framed constructively rather than purely diagnostically. The task predicts missing facts in incomplete multimodal knowledge graphs using auxiliary images and descriptions of entities. The motivating claim is twofold: single-target objectives neglect probabilistic correlations among entity labels, especially in non-target entities, and prior methods that incorporate all modalities statically or adaptively overlook the negative impacts of irrelevant or misleading information in incompetent modalities (Zhao et al., 28 Jul 2025).

The resulting framework uses one pre-trained multimodal teacher per modality $$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$6 and a unimodal student KGR model. Distillation begins from teacher soft labels

$$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$7

with vanilla KD

$$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$8

To preserve different kinds of entity correlation, logits are decoupled into neighbor entities $$p_m(y)=\frac{S_m(y)}{\sum_{y'} S_m(y')}, \qquad y_m=\arg\max_y p_m(y), \qquad c_m=\max_y p_m(y).$$9 and non-neighbor entities $\tau$0, yielding

$\tau$1

Teacher selection is then posed as reinforcement learning over the seven nonempty subsets of $\tau$2. The state is $\tau$3, the policy is $\tau$4, and the reward is $\tau$5 if the selected teachers’ cross-entropy is below the student’s and $\tau$6 otherwise. REINFORCE updates the policy, with the reward of the all-teachers ensemble used as a baseline.

Empirically, this DSoM framework was evaluated on DB15K, MKG-W, MKG-Y, FB15K-237, and WN18 using Mean Reciprocal Rank, Mean Rank, and Hits@1/3/10 under the filtered setting. On DB15K, MKG-W, and MKG-Y it yielded relative MRR gains of +13.1%, +13.3%, and +3.8% over the previous best. Removing the reinforced selection dropped MRR by $\tau$7–$\tau$8 points, and removing neighbor-decoupling degraded performance by $\tau$9–$\tau=0.70$0 MRR. A qualitative example involving the triple $\tau=0.70$1 showed the visual teacher boosting Samsung because of an image with an iPhone, while the RL agent selected $\tau=0.70$2 and excluded the harmful visual teacher (Zhao et al., 28 Jul 2025).

This formulation treats DSoM not as an exceptional pathology but as a standard training-time consideration: multimodal supervision is valuable precisely when harmful modalities can be removed on a per-instance basis.

5. Robustness, Calibration, and Safety Under Cross-Modal Conflict

Controlled corruption benchmarks make DSoM measurable as a robustness and calibration problem. OMD-Bench begins from tri-modal samples in which video, audio, and text all depict the same anchor and then systematically replaces one or more modalities with distractor anchors. The benchmark contains 4,080 instances spanning 27 anchors across eight corruption conditions. Human recognition exceeds 93% per modality, which is intended to eliminate the confound that naturally co-occurring modalities carry correlated yet unequal information. Under two corrupted modalities ($\tau=0.70$3), human annotators abstain on only $\tau=0.70$4 of cases, but models abstain far more often: 19.5% on average in zero-shot and 21.1% under chain-of-thought. Under full corruption ($\tau=0.70$5), humans abstain on $\tau=0.70$6 of cases, yet even the best abstaining model reaches only 59.6% in zero-shot and 51.9% under chain-of-thought, while many models remain below 20%; mean predictive confidence often stays between 96% and 99% (Nazi et al., 28 Mar 2026).

The “Mirage of Multimodality” studies a related failure mode in which slower reasoning models become more, not less, distortion-prone under ambiguous or misleading visual input. On a 5,000-sample hierarchical prompt dataset annotated by 50 human participants, each image is paired with Level 1, Level 2, and Level 3 prompts of increasing deceptive complexity. The Correctness Attenuation Index is defined as

$\tau=0.70$7

and the reported mean accuracies over 50+ models are 81.85% at Level 1, 55.37% at Level 2, and 44.96% at Level 3. Chat models cluster at low CAI and low ECE, whereas reasoning models shift toward higher CAI and higher ECE; reasoning models also exhibit an inverse scaling law in which larger parameter counts worsen Level 3 accuracy, ECE, and CAI (Ji et al., 26 May 2025).

Safety red-teaming further shows that DSoM cannot be reduced to a simple claim that multimodal prompts are always more dangerous. In one benchmark with 726 adversarial prompts, 2,904 model outputs, and 47,408 annotations, text-only prompts achieved an overall attack success rate of 35% versus 31% for multimodal prompts; Pixtral 12B was the most vulnerable at approximately 62% harmful responses, while Claude Sonnet 3.5 was the most resistant at approximately 10% (Doren et al., 18 Sep 2025). In a cross-lingual study over 363 jailbreak scenarios and 52,272 harm ratings, Bayesian mixed-effects analysis yielded a language main effect $\tau=0.70$8 with 95% credible interval $\tau=0.70$9, but safety rankings were not preserved across languages: in en-US, Pixtral was the most vulnerable, whereas in es-MX Qwen Omni overtook Pixtral (Ford et al., 22 May 2026). This suggests that DSoM includes not only harmful modality content but also modality-by-language interaction effects that are invisible in English-only, text-only evaluation.

6. Forensic Generalization and Unseen “Dark Modalities”

In multimodal forensics, DSoM appears as a generalization bottleneck. A “dark modality” $$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$0 is defined as a signal channel for which no pre-trained semantic embedding $$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$1 exists that can map it into the shared feature space $$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$2 used for training on known modalities. The claimed failure of prior detectors is a “modality-binding bottleneck”: they overfit to modality-specific style cues $$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$3, such as blending seams or frequency anomalies, and therefore fail catastrophically when an unseen modality does not express those artifacts in the same way (Dou et al., 9 Apr 2026).

The proposed Modality-Agnostic Forgery framework introduces modality-specific perceptors $$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$4 that produce

$$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$5

followed by a shared detector $$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$6. Training combines a classification loss

$$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$7

with a style-disentanglement regularizer, either by pairwise KL matching

$$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$8

or by an IRM-style gradient penalty, under the total objective

$$\text{potential sabotage}_m \iff (c_m \ge \tau)\wedge (y_m \ne y^*),$$9

The framework distinguishes Weak MAF, where an unseen modality can still be embedded into the training space, from Strong MAF, where it cannot and only an isolated self-supervised perceptor is learned for the new modality.

DeepModal-Bench evaluates these settings on LAV-DF, FakeAVCeleb, ASVspoof5, and Celeb-DF++. Under Weak MAF, domain-generalization variants such as IRM, Mixup, and CDANN outperform conventional multimodal fusion by 6–10 AUC points on average. Under Strong MAF, absolute AUC drops, but the method still achieves approximately 59–61% AUC versus approximately 50% random. The cross-modal KL divergence of forgery features falls by up to 90% in the forensic space, and the PCA $\hat y = y_m$0 drops from $\hat y = y_m$1 to 2–50 principal components across audio, image, and video. The central claim is therefore that “universal forgery traces” exist beneath modality-specific appearances (Dou et al., 9 Apr 2026).

7. Related Epistemic Use in Unreliable Communication

A distinct line of work uses the “dark side” idea outside machine learning, in epistemic analysis of distributed systems with unreliable communication. There the problem is not multimodal fusion in the modern ML sense, but the failure of standard knowledge and belief modalities when agents may lie. In fully Byzantine systems, classical factive knowledge $\hat y = y_m$2 becomes unattainable because fake messages and fabricated events destroy the assumption that incoming messages are trustworthy. To model the informational content of communication without taking it at face value, two additional modalities are introduced: hope $\hat y = y_m$3 and creed $\hat y = y_m$4 (Kuznets, 2024).

For $\hat y = y_m$5, the semantics uses a frame $\hat y = y_m$6 in which each $\hat y = y_m$7 is an equivalence relation, each $\hat y = y_m$8 is a partial equivalence relation with $\hat y = y_m$9, and a strong coherence condition links $$\tilde s(y)=\sum_{m\in\{T,A,V,TAV\}} w_m S_m(y), \quad w_m=1,$$0 to $$\tilde s(y)=\sum_{m\in\{T,A,V,TAV\}} w_m S_m(y), \quad w_m=1,$$1. The bridge axiom is

$$\tilde s(y)=\sum_{m\in\{T,A,V,TAV\}} w_m S_m(y), \quad w_m=1,$$2

Creed further indexes interpretation by listener and speaker types. In a system with at most $$\tilde s(y)=\sum_{m\in\{T,A,V,TAV\}} w_m S_m(y), \quad w_m=1,$$3 Byzantine agents, mutual hope over any set $$\tilde s(y)=\sum_{m\in\{T,A,V,TAV\}} w_m S_m(y), \quad w_m=1,$$4 of size $$\tilde s(y)=\sum_{m\in\{T,A,V,TAV\}} w_m S_m(y), \quad w_m=1,$$5,

$$\tilde s(y)=\sum_{m\in\{T,A,V,TAV\}} w_m S_m(y), \quad w_m=1,$$6

entails $$\tilde s(y)=\sum_{m\in\{T,A,V,TAV\}} w_m S_m(y), \quad w_m=1,$$7. When specialized to two types, correct and faulty, creed generalizes hope. This is a separate research tradition, but it shares the same underlying intuition as multimodal DSoM: informational channels that are usually modeled as helpful or transparent can become structurally deceptive, and the formalism must represent that possibility explicitly (Kuznets, 2024).