Generalizing robustness across semantic and modality perturbations in multimodal LLMs

Develop mechanisms for Multimodal Large Language Models that generalize across both semantic perturbations (e.g., misleading or irrelevant content in text or images) and modality perturbations (e.g., input- or embedding-level noise), so that robustness is maintained without relying on task-specific or attack-specific defenses.

Background

The paper diagnoses modality interference in Multimodal LLMs and proposes a fine-tuning framework combining heuristic and adversarial perturbations with consistency regularization. While this approach improves robustness, the authors emphasize that PGD-based adversarial training is a partial solution confined to a bounded embedding-level perturbation space.

In the Limitations section, the authors highlight the need for defenses that can uniformly handle both semantic perturbations (e.g., misleading descriptions or irrelevant images) and modality perturbations, acknowledging that designing perturbations is inherently open-ended and that current methods do not comprehensively address all forms of interference.

References

Developing mechanisms that generalize across both semantic and modality perturbations remains an open and challenging direction.

— Diagnosing and Mitigating Modality Interference in Multimodal Large Language Models  (2505.19616 - Cai et al., 26 May 2025) in Appendix, Section Limitations

Our conjecture is that isotropic perturbations in the embedding space, uniformly spread across all token dimensions, are more desirable, whereas pixel-level perturbations incur bias.

— Isotropic Embedding Perturbations for Robust Vision Language Encoders  (2609.10292 - Choi et al., 9 Sep 2026) in Section 3, subsection “Theoretical backup: why at the embedding space?”

Our evaluation focuses primarily on object hallucination. The applicability of ResOT to other forms of multimodal error remains to be verified.

— From Suppression to Repair: Mitigating Object Hallucination in Large Vision-Language Models via Localized Distribution Alignment  (2610.11826 - Zhao et al., 8 Oct 2026) in Section E, “Limitation”

It remains unclear whether the “semantic reasoning” proficiency of detectors such as GPT-5.2 would maintain this performance under true forensic settings.

— Beyond Generation: Using Multimodal LLMs as Detectors of LLM-Generated Deepfake Facial Images  (2610.21760 - Salamon et al., 15 Oct 2026) in Section 5.4, Limitations of the Study