MTMCS-Bench: Multimodal Safety Benchmark

• MTMCS-Bench is a multi-turn, multimodal safety benchmark that assesses LLMs using paired safe and unsafe image-dialogue samples to capture escalation and context-switch risks.
• It employs a controlled dataset with over 30,000 samples and metrics like contextual intent recognition accuracy, safety-awareness, and helpfulness scores to compare model performance.
• Empirical results reveal safety–utility trade-offs where models may under-refuse escalating risks or over-reject benign inputs, emphasizing the need for refined guardrail mechanisms.

The Multi-Turn Multimodal Contextual Safety Benchmark (MTMCS-Bench) provides a comprehensive large-scale framework for evaluating contextual safety in multimodal LLMs (MLLMs) interacting through both text and images in multi-turn dialogues. Unlike prior safety benchmarks that focus on single-turn or single-modality scenarios, MTMCS-Bench systematically characterizes risks emerging from extended, visually grounded conversations by constructing elaborate paired safe and unsafe dialogue corpora, precisely defined risk types, and multi-faceted metrics. This enables rigorous comparative analysis of MLLMs’ safe intent recognition, refusal behavior, and helpfulness under complex, evolving safety threats (Liu et al., 11 Jan 2026).

1. Motivation and Scope

Modern MLLMs, increasingly deployed as general-purpose assistants, frequently mediate user requests referencing both images and multi-turn dialogue histories. Real-world adversarial threats often exploit two phenomena overlooked by previous benchmarks: escalation risk (benign-seeming turns that incrementally aggregate into unsafe requests) and context-switch risk (explicit initial harms obscured by subsequent innocuous context). MTMCS-Bench is designed to diagnose models' capacity to (i) track user intent conditioned on joint image-dialogue context, (ii) refuse/redirect when risk emerges, and (iii) maintain utility for non-harmful inquiries (Liu et al., 11 Jan 2026).

2. Dataset Composition and Scenario Design

MTMCS-Bench contains over 30,000 samples generated from a combinatorial construction process:

• Base Imagery: 752 COCO-style images, each with three automatically generated variants (Qwen-Image-Edit), resulting in 2,256 distinct images.
• Dialogue Structure: Each scenario is instantiated under two complementary risk types:
  • Type A (Escalation Risk): R₁ and R₂ are identical in both safe and unsafe; R₃ diverges to encode risk only in the unsafe dialogue.
  • Type B (Context-Switch Risk): R₂ and R₃ are identical across variants; R₁ varies between safe/unsafe intent.
• Modalities: Every dialogue is rendered as both multimodal (image + dialogue) and unimodal (text-only, scene description).
• Scale: 752 scenarios × 2 risk types × 2 intent labels × 2 modalities yield 12,032 dialogues (36,096 turns); with 18,048 intent evaluation questions, this totals 30,080 samples.

This controlled pairing of safe/unsafe, multi-modal/unimodal dialogues effectively isolates the effects of risk emergence and model access to visual cues. Each dialogue comprises three user–assistant exchanges (R₁, R₂, R₃), ensuring each risk trajectory is explicitly codified (Liu et al., 11 Jan 2026).

3. Formal Metrics and Evaluation Protocol

MTMCS-Bench reports three complementary axes of model safety-utility interaction:

• Contextual Intent Recognition Accuracy: MCQ and TF questions, formulated over full three-turn (plus image) context, asking, “What is the user’s intent?” Metrics:

$$\textrm{Acc}_{\textrm{MCQ}} = 100 \cdot \frac{1}{|\mathcal{Q}_{\textrm{MCQ}}|}\sum_{j\in \mathcal{Q}_{\textrm{MCQ}}} \mathbb{I}[\hat{a}_j = a_j]$$

$$\textrm{Acc}_{\textrm{TF}} = 100 \cdot \frac{1}{|\mathcal{Q}_{\textrm{TF}}|}\sum_{k\in \mathcal{Q}_{\textrm{TF}}} \mathbb{I}[\hat{y}_k = y_k]$$

• Safety-Awareness Score (SA): On unsafe dialogues, a judge (GPT-5-mini) assigns $s_i \in \{1, \dots, 5\}$, measuring detection and refusal of risk, with aggregate

$$\textrm{SA} = \frac{1}{|\mathcal{U}|} \sum_{i \in \mathcal{U}} s_i$$

• Helpfulness Score (HS): On safe dialogues, helpfulness $h_j \in \{1, \dots, 5\}$ is assigned, with aggregate

$$\textrm{HS} = \frac{1}{|\mathcal{B}|} \sum_{j \in \mathcal{B}} h_j$$

These metrics are reported both globally and per-risk-type, capturing fine-grained distinctions between safe/unsafe context interpretation, refusal strategies, and maintenance of benign utility (Liu et al., 11 Jan 2026).

Evaluation protocol treats each dialogue as a unit for intent recognition; open-generation responses to R₃ are additionally judged for SA or HS. Models are evaluated on both multi-modal and unimodal settings to quantify the incremental liability introduced by vision encoders.

4. Empirical Analysis and Comparative Findings

Model Classes Tested

• Open-Source: LLaVA-1.6-7B, LLaVA-Next-72B, Idefics-3 8B, Qwen3-VL-8B, Qwen3-VL-32B, LLaMA 3.2-11B/90B, InstructBLIP-7B.
• Proprietary: GPT-4.1, GPT-5-mini, GPT-5.2, GPT-o4-mini, Claude Haiku 4.5, Claude Sonnet 4.5, Claude Opus 4.5.

Key trends:

• Open-source models reach $\sim$75% MCQ accuracy (Qwen3-VL-32B) but attain low SA in Type A ($\approx$2.2/5), indicating deficiency in tracking gradual multi-turn escalation.
• Proprietary models (e.g., GPT-5.2) achieve best joint performance: $\sim$84% MCQ, SA$\approx$3.4, HS$\approx$3.8 on Type A, but still under-detect nuanced unsafe intent and over-refuse some benign queries.
• Type B risks are consistently more tractable for all models: MCQ, TF, and SA rise by 5–20 percentage points relative to Type A, while HS remains stable.
• Image access benefits strong models (higher intent and SA), but can spur over-sensitivity and utility loss in weaker ones.

Persistent safety–utility trade-offs are observed—models either “under-refuse” dangerous context or “over-refuse” benign contexts, highlighting the nuanced calibration required for sustained safety in practical deployments (Liu et al., 11 Jan 2026).

5. Guardrail Defenses: Techniques and Limitations

MTMCS-Bench systematically evaluates five representative defenses using Qwen3-VL-8B:

Defense	Principal Mechanism	Net Effects
Defensive Prompt Patch	Genetic search for refusal suffix	SA improvement, but MCQ on unsafe drops; style more conservative
AdaShield-Adaptive	Adaptive pre-prompt shielding	SA boost, minor HS drop, MCQ on Type B unsafe down
Self-Examination	Post-hoc output critique	Mild SA gain, generally preserves utility, misses subtle multi-turn risk
Immune (RM-Guided Dec.)	Reward-model biases decoding	Strong SA gain, HS decrease (over-safety bias)
Chain-of-Thought+Agg	Reasoning plus aggregation	Modest SA improvement, MCQ and HS drop (performance/utility loss)

No single method simultaneously optimizes intent accuracy, SA, and HS. Prompt-based approaches trend toward over-cautious refusal, reward-model steering can suppress utility, and self-critique or reasoning-based methods struggle with subtle or distributed risks (Liu et al., 11 Jan 2026).

6. Significance, Related Efforts, and Extensions

MTMCS-Bench formalizes multi-turn, multi-modal safety evaluation with rigor exceeding prior benchmarks such as SafeMT (Zhu et al., 14 Oct 2025), MMDS (LLaVAShield) (Huang et al., 30 Sep 2025), and methods emphasizing reward-model alignment (AM$^3$Safety (Zhu et al., 8 Jan 2026)). These related frameworks reinforce the central observation: single-turn safety interventions are insufficient—gradual, multi-turn intent reconstruction remains a key vulnerability. Benchmarks such as MMDS provide fine-grained policy annotation, while SafeMT/AM$^3$Safety further analyze multi-length escalations and optimized RLHF variants.

The benchmark’s design directly influences the construction and calibration of future safety-critical MLLMs and guides the creation of dynamic policy, sequence-level detection, and red-teaming strategies. The structured scale and broad context modeling of MTMCS-Bench make it a cornerstone for ongoing empirical and methodological safety studies across the field.

7. Outlook and Recommendations

The inability of current guardrails—even in advanced models—to fully resolve contextual multi-turn risks motivates further research into:

• Explicit joint modeling of dialogue and visual context to preserve evolving risk estimates.
• Fine-grained reward and policy models sensitive to incremental risk trajectories.
• Per-turn adaptive defense mechanisms, tightening refusal as intent thresholds are crossed.
• Expansion into audio-visual and video-modalities for broader dialogue settings.

MTMCS-Bench provides a reproducible, extensible foundation to drive these advances. Best practices inferred from the data include emphasizing multi-turn dialogue accumulation, scenario diversity, robust annotation, and integration with automated, adversarial red-teaming workflows (Liu et al., 11 Jan 2026).