MultiTrust-X: A Benchmark for MLLM Trustworthiness

• MultiTrust-X is a comprehensive benchmark that evaluates trustworthiness aspects of multimodal LLMs, including truthfulness, robustness, safety, fairness, and privacy.
• It differentiates between multimodal risks and cross-modal impacts, addressing challenges unique to image-text interactions and visual context effects.
• The benchmark integrates extensive datasets, 32 tasks, and mitigation strategies, notably showcasing the RESA method for state-of-the-art safety alignment.

MultiTrust-X is a comprehensive benchmark for evaluating, analyzing, and mitigating the trustworthiness issues of multimodal LLMs (MLLMs). It is organized around a three-dimensional framework that spans five trustworthiness aspects—truthfulness, robustness, safety, fairness, and privacy—two multimodality-specific risk types—multimodal risks and cross-modal impacts—and mitigation strategies from the perspectives of data, model architecture, training, and inference algorithms. The benchmark includes 32 tasks, 28 curated datasets, about 23.0K image-text pairs, evaluation of over 30 open-source and proprietary MLLMs, and analysis with 8 representative mitigation methods; it also introduces Reasoning-Enhanced Safety Alignment (RESA), a reasoning-based mitigation method reported to achieve state-of-the-art results among open-source MLLMs (Zhang et al., 21 Aug 2025).

1. Historical positioning and problem definition

MultiTrust-X is best understood in relation to the earlier MultiTrust benchmark, which introduced the first comprehensive and unified benchmark for MLLM trustworthiness across the same five primary aspects and 32 tasks, and evaluated 21 modern MLLMs (Zhang et al., 2024). Relative to that earlier benchmark, MultiTrust-X extends the agenda from evaluation alone to evaluation, analysis, and mitigation, while broadening the benchmark to 28 curated datasets, over 30 MLLMs, and 8 representative mitigation methods (Zhang et al., 21 Aug 2025).

The motivating problem is that MLLMs have advanced rapidly in general multimodal capability, but their trustworthiness remains fragile and poorly understood. The benchmark is explicitly designed around two shortcomings of prior work. First, prior evaluation was too narrow, often isolating a single issue such as hallucination, jailbreaking, privacy leakage, or robustness rather than providing a unified view across multiple trust dimensions. Second, prior evaluation underexplored risks introduced by multimodality itself. MultiTrust-X therefore distinguishes between multimodal risks, which arise in genuinely image-text settings, and cross-modal impacts, where adding visual input changes behavior even on tasks that are fundamentally text-centric (Zhang et al., 21 Aug 2025).

This distinction is central to the benchmark’s conception of trustworthy multimodal systems. The paper argues that current models do not merely inherit text-only LLM risks; multimodal training and multimodal inference can amplify those risks. A recurring empirical theme is that semantically relevant images, irrelevant natural images, random noise images, and color blocks can all alter answers, refusal behavior, privacy behavior, safety, or bias (Zhang et al., 21 Aug 2025).

2. Three-dimensional framework

The first dimension of MultiTrust-X is the five-aspect trustworthiness taxonomy. Each aspect is subdivided into two sub-aspects, yielding a ten-part structure.

Aspect	Sub-aspects
Truthfulness	Inherent deficiency; Misguided mistakes
Robustness	OOD robustness; Adversarial robustness
Safety	Toxicity; Jailbreaking
Fairness	Stereotype; Bias preference
Privacy	Privacy awareness; Privacy leakage

Truthfulness measures errors caused either by internal limitations or by external misguidance. Robustness measures stability under distribution shifts and adversarial perturbations. Safety measures harmful generation, unsafe compliance, and jailbreaking. Fairness measures stereotype and bias preference effects. Privacy measures both recognition of privacy-sensitive content and avoidance of disclosure (Zhang et al., 21 Aug 2025).

The second dimension is the pair of multimodality-specific risk types. Multimodal risks include misleading images in VQA, adversarial image perturbations, typographic image jailbreaks, NSFW image prompting, and image-based privacy leakage. Cross-modal impacts refer to behavioral changes induced by the mere presence of images, including relevant images, irrelevant natural images, random noise images, and color blocks. To isolate these effects, the benchmark uses text-only conditions, text plus relevant image conditions, and text plus irrelevant image conditions. For irrelevant-image evaluations, it uses random noise, color blocks, and sampled ImageNet natural images; for each irrelevant-image condition, three images are sampled and results are averaged (Zhang et al., 21 Aug 2025).

The third dimension classifies mitigation methods by intervention locus: data, model architecture, training algorithm, and inference algorithm. This makes mitigation analysis commensurable across otherwise heterogeneous methods such as VLGuard, SPA-VL, RLAIF-V, FARECLIP, SimCLIP, VCD, ECSO, and ETA (Zhang et al., 21 Aug 2025).

3. Benchmark composition, datasets, and evaluation protocol

MultiTrust-X organizes its 32 tasks into five aspect-specific blocks. Truthfulness comprises seven tasks: T.1 Basic world understanding, T.2 Advanced cognitive inference, T.3 Guided factual reasoning via VQA, T.4 Factual QA with visual assistance, T.5 Misleading textual prompts, T.6 Visually deceptive content / illusions, and T.7 Faulty images in factual QA. Robustness comprises six tasks: R.1 Artistic/stylized image robustness, R.2 Sensor-derived imagery robustness, R.3 Cross-modal impact on sentiment analysis, R.4 Untargeted adversarial image attacks for captioning, R.5 Targeted adversarial image attacks, and R.6 Textual adversarial attacks with visual pairing. Safety comprises six tasks: S.1 NSFW description, S.2 Risk identification, S.3 Toxicity variation under diverse visual stimuli, S.4 Typographic jailbreaking, S.5 Optimized multimodal jailbreaking, and S.6 Cross-modal influence on text jailbreaks. Fairness comprises seven tasks: F.1 Stereotype in generated responses, F.2–F.4 agreement/classification/sensitivity to stereotypes, F.5 Visual preference testing, F.6 Competence bias judgment, and F.7 Influence of visual context on text-only preferences. Privacy comprises six tasks: P.1 Identify private content in images, P.2 Determine whether questions probe sensitive details, P.3 Visual context influence on privacy expectations, P.4 PII disclosure from celebrity images, P.5 PII leakage from visual content, and P.6 Leakage of prior conversational privacy with image presence (Zhang et al., 21 Aug 2025).

The dataset layer combines 20 adapted datasets and 8 newly created datasets. Construction methods include prompt engineering, adapting image-text pairs, synthetic generation with Stable Diffusion, generation or scoring with GPT-4V and GPT-4o, manual annotation, and online image sourcing. The paper explicitly mentions source resources such as COCO, VizWiz, AdvGLUE, AdvGLUE++, ImageNet, VLGuard, SPA-VL, and LLaVA-NeXT data (Zhang et al., 21 Aug 2025).

Evaluation covers 30 MLLMs in the visible leaderboard, including 7 proprietary and 23 open-source systems. Proprietary models include GPT-4-Vision, Claude3.5-Sonnet, GPT-4o, Claude3-Sonnet, Qwen-VL-Plus, Gemini1.0-Pro, and Hunyuan-V. Open-source coverage includes LLaVA, InternVL, InternLM-XComposer, Qwen, Phi, MiniGPT, mPLUG, Cambrian, CogVLM, DeepSeek-VL, ShareGPT4V, LVIS-Instruct4V, InstructBLIP, and Otter variants (Zhang et al., 21 Aug 2025).

The benchmark uses task-specific metrics, including Accuracy, Pearson Correlation, p-value, Attack Success Rate (ASR), Cure Rate, Toxicity Score, Refuse-to-Answer (RtA) Rate, Containing Rate, and GPT-Score. The basic accuracy definition is given as

$Acc=\frac{\sum_{i=1}^N\mathbb{I}(r_i = y_i)}{N}.$

Task scores are scaled to 0–100 and then averaged over sub-aspects and overall. For open-ended tasks, the benchmark relies on judge-based components such as Moderation API, Perspective API, Longformer, and GPT-4; the reported GPT-4-based scoring correlates with human ratings at 0.91 (Zhang et al., 21 Aug 2025).

4. Empirical findings

The benchmark reports a clear separation between general capability and trustworthiness. The correlation between general benchmark performance from MME and MMBench and MultiTrust-X trustworthiness is described as moderate, with Pearson’s $r = 0.69$. This is stronger than no relationship, but weaker than the common assumption that better general models are automatically more trustworthy. The leaderboard places GPT-4-Vision at 78.28, Claude3.5-Sonnet at 76.70, GPT-4o at 76.61, and Claude3-Sonnet at 72.76, while the best pre-RESA open-source baseline is Phi-3.5-Vision at 66.29 (Zhang et al., 21 Aug 2025).

Several aspect-specific findings recur across the evaluation. In truthfulness, coarse visual perception often appears strong, with object recognition and scene understanding often exceeding 80%, but fine-grained grounding remains weak: grounding falls to 32% for InternLM-XC2 and 8% for Gemini-Pro. The paper also notes that models perform better on commonsense reasoning that can be handled by the base LLM than on tasks requiring genuine visual reasoning (Zhang et al., 21 Aug 2025).

In robustness, the most severe weaknesses arise under adversarial attack rather than ordinary out-of-distribution shift. Under untargeted image attacks, many models drop from above 90% accuracy to below 20%. Under targeted attacks, multiple models exceed 50% attack success rate. The paper highlights CogVLM and InternVL-Chat as comparatively robust (Zhang et al., 21 Aug 2025).

Safety is one of the strongest failure modes. On NSFW description, GPT-4V records an RtA rate of 59.80% and toxicity score 0.26, whereas mPLUG-Owl2 records RtA 0.00% and toxicity score 0.62. Typographic jailbreaking yields ASR 0.17% for GPT-4V, 1.50% for MiniGPT-4-L2, 34.50% for mPLUG-Owl2, and 13.67% for InternLM-XC2. More strikingly, some models can be compromised by image-only harmful attacks: InternLM-XC2 in 71% and LLaVA-1.5 in 80% (Zhang et al., 21 Aug 2025).

Fairness results are more mixed. Average stereotype rejection is reported as 93.79%, but this high refusal rate conceals topic-dependent variation. The paper reports that age-related stereotypes are harder for models than some gender, race, or religion cases, and that visual context can shift preference expression and permissiveness (Zhang et al., 21 Aug 2025).

Privacy findings are similarly bifurcated between awareness and leakage. Average model accuracy on identifying private image content is 72.30%, but more reasoning-heavy privacy awareness tasks are close to random for many models. The strongest examples concern cross-modal privacy degradation. In task P.6, Qwen-VL-Plus drops from RtA 38.00 in text-only mode to 27.87 with irrelevant images and 13.00 with relevant images; LLaVA-NeXT drops from 51.00 to 25.67 to 1.50; MiniGPT-4-L2 drops from 100.00 to 66.78 to 24.00 (Zhang et al., 21 Aug 2025).

These results support two broader claims emphasized by the benchmark. Proprietary models remain much more trustworthy than most open-source models, and multimodality can amplify inherited LLM risks rather than simply extending capability (Zhang et al., 21 Aug 2025).

5. Mitigation analysis and RESA

A distinctive feature of MultiTrust-X is that it evaluates mitigation methods under the same benchmark rather than treating evaluation and defense as separate problems. Using LLaVA-1.5-7B as the common base model, the paper studies eight methods: RLAIF-V, VCD, FARECLIP, SimCLIP, VLGuard, SPA-VL, ECSO, and ETA. Their overall scores illustrate the benchmark’s central trade-off: narrow improvements do not reliably transfer to holistic trustworthiness.

Method	Category	Overall score
LLaVA-1.5-7B	Base model	48.45
RLAIF-7B	Data / Training	55.36
VCD	Inference	45.53
FARECLIP	Architecture	50.24
SimCLIP	Architecture	50.06
VLGuard	Data / Training	67.40
SPA-VL	Data / Training	66.03
ECSO	Inference	50.92
ETA	Inference	55.49

The mitigation study identifies several stable patterns. Hallucination-focused methods do not reliably improve overall trustworthiness: RLAIF-7B improves some truthfulness tasks, such as lifting T.1 from 66.9 to 74.6, but does not generalize broadly, while VCD reduces overall score below the base model. Robust encoders such as FARECLIP and SimCLIP dramatically improve adversarial robustness—FARECLIP raises adversarial robustness to 75.00—but incur the usual robustness–accuracy trade-off, degrading some truthfulness performance (Zhang et al., 21 Aug 2025).

Safety alignment methods are the strongest overall. VLGuard, trained on about 2k safety samples, and SPA-VL, trained on over 90k samples, produce the highest holistic gains among non-RESA methods. Yet the paper’s controlled analysis finds that refusal-heavy data creates over-refusal and utility loss. Increasing safety data improves refusal behavior—such as Toxic Content Generation from 64.21 to 77.68 to 81.85 and Textual Jailbreaking from 25.00 to 88.75 to 96.25—but degrades some utility tasks, including Advanced Cognitive Inference and Visual Confusion VQA. Adding helpfulness data partially repairs this trade-off but does not fully restore original utility (Zhang et al., 21 Aug 2025).

The paper also reports that chain-of-thought formatting improves the safety–utility balance. Converting VLGuard into CoT form raises Basic World Understanding from 66.05 to 70.03, Advanced Cognitive Inference from 50.79 to 54.97, and Instruction Assisted VQA from 7.00 to 12.75, while maintaining near-saturated jailbreak refusal. It further reports that Direct Preference Optimization preserves utility better than supervised fine-tuning in safety alignment settings, and that inference-time methods such as ECSO and ETA improve safety with limited utility damage but remain fundamentally bounded by the base model’s own harmfulness detection and benign-response generation capacity (Zhang et al., 21 Aug 2025).

These observations motivate RESA (Reasoning-Enhanced Safety Alignment). RESA starts from LLaVA-1.5-7B, converts VLGuard examples into reasoning-augmented form using GPT-4o, wraps reasoning in > ... tags, and retains the original answer after the reasoning chain. A second variant adds 10k sampled LLaVA-NeXT helpfulness examples converted into the same CoT format. Training uses supervised fine-tuning with global batch size 128, learning rate $1e^{-5}$, and 3 epochs. A robustness-enhanced variant, RESA-R, replaces the visual encoder with FARECLIP after RESA fine-tuning (Zhang et al., 21 Aug 2025).

The headline result is that RESA-R-7B raises LLaVA-1.5-7B from 48.45 to 69.66, surpassing Phi-3.5-Vision at 66.29 and establishing the best open-source result on the benchmark. The paper reports this as state of the art for open-source MLLMs on MultiTrust-X (Zhang et al., 21 Aug 2025).

6. Significance, limitations, and research implications

MultiTrust-X is significant because it recasts MLLM trustworthiness as a multidimensional and multimodality-specific research problem rather than a collection of isolated failures. It does so in three linked ways: by unifying five trustworthiness aspects, by separating multimodal risks from cross-modal impacts, and by evaluating mitigation strategies within the same framework that exposes model failures (Zhang et al., 21 Aug 2025).

Several misconceptions are directly challenged by the benchmark. One is that general multimodal capability can stand in for trustworthiness; the reported $r = 0.69$ relation is only moderate. Another is that adding vision primarily adds capability; the benchmark repeatedly finds that multimodal training and inference can amplify base-LLM risks. A third is that successful mitigation on one axis transfers broadly; the mitigation study shows that some methods improve specific aspects while few effectively address overall trustworthiness, and many introduce unexpected trade-offs that compromise model utility (Zhang et al., 21 Aug 2025).

The benchmark also has stated limitations. Its scope is mainly vision-plus-language rather than broader modality sets such as audio or video. Some evaluations rely on external judges such as GPT-4, Longformer, Moderation API, and Perspective API. Parts of the dataset are synthetic or adapted using Stable Diffusion, GPT-4V, GPT-4o, and prompt engineering. The mitigation study covers eight representative methods rather than the full design space. The paper therefore does not claim to exhaust real-world deployment risk, long-context multimodal agent behavior, or all future architectural variants (Zhang et al., 21 Aug 2025).

Taken together, these findings suggest that trustworthy MLLM development cannot be reduced to scaling, narrow hallucination control, or refusal-only alignment. The benchmark’s strongest practical implication is that multimodal trustworthiness should be treated as a holistic systems property in which truthfulness, robustness, safety, fairness, privacy, and model utility are jointly constrained. Within that framing, MultiTrust-X serves both as a benchmark and as an analysis framework for identifying where current MLLMs fail, why multimodal inputs destabilize alignment, and which mitigation strategies remain viable under full-spectrum evaluation (Zhang et al., 21 Aug 2025).