---
title: 'Yuvion VL RiskEval: Multimodal Safety Benchmark'
url: https://www.emergentmind.com/topics/yuvion-vl-riskeval-yvre
type: topic
---

# Yuvion VL RiskEval: Multimodal Safety Benchmark

Searching arXiv for the primary Yuvion VL paper and a few related benchmark papers mentioned in the provided data.
{"query":"arXiv:2606.25034"}
I’ll look up the primary paper by arXiv ID and then a small set of related benchmark papers referenced in the provided material.
arXiv search query: 2606.25034
Yuvion VL RiskEval (YVRE) is a purpose-built, three-level benchmark suite for systematically measuring multimodal content and AI safety capabilities—particularly adversarial robustness and real-world deployment readiness—of vision–language models. It is introduced alongside Yuvion VL, a family of multimodal large language models built for content and AI safety, and is designed around the premise that content and AI safety is inherently adversarial and multimodal: malicious actors may embed tiny watermarks, mutate logos, compose allegorical imagery, or generate synthetic media to evade detection. Within that framing, YVRE evaluates capability preservation, safety competence, and operational readiness while emphasizing adversarial robustness and real-world capability requirements [2606.25034].

## 1. Conceptual basis and evaluation objectives

YVRE is motivated by the claim that real-world safety assessment for vision–language models cannot be reduced to generic multimodal evaluation. The benchmark specification states that models must integrate visual and textual evidence, domain knowledge such as regulations, symbols, and policies, and multi-step reasoning, while resisting evolving evasion tactics. This design directly reflects the surrounding Yuvion VL training pipeline, which treats safety as an inherently adversarial and multimodal problem and includes continued pretraining, instruct post-training, reasoning post-training, and Confuse-then-Contrast Fine-Tuning (C2FT) for explicit discrimination of fine-grained visual-semantic elements [2606.25034].

The benchmark’s primary goals are organized into three levels. Level 1, Capability Preservation, is intended to ensure that safety-specialization does not degrade core vision–language understanding by more than a small budget, stated as \( \le 3 \) percentage points aggregate. Level 2, Safety Competence, evaluates open-domain and fine-grained safety detection, adversarial robustness to real-world evasion, and domain-specific governance, including e-commerce logos, brands, product categories, and price inference. Level 3, Operational Readiness, stress-tests production tasks such as knowledge probes, fine-grained perception, instruction-following under dynamic policy, AI-generated image detection, adversarial dialog robustness, and end-to-end moderation pipelines [2606.25034].

A plausible implication is that YVRE is not intended as a single-task safety leaderboard. Instead, it encodes a staged evaluation logic in which generic multimodal competence is retained, specialized safety skill is measured, and deployment-facing failure surfaces are exposed separately.

## 2. Three-level benchmark organization

YVRE comprises 58 distinct benchmarks, organized into three progressive evaluation levels. The suite’s reported structure is summarized below [2606.25034].

| Level | Task family | Reported emphasis |
|---|---|---|
| 1 | General Multimodal | Capability Preservation |
| 2a | Public Safety | Safety Competence |
| 2b | E-commerce Governance | Safety Competence |
| 3 | In-House Business & Safety Tasks | Operational Readiness |

Level 1 covers general multimodal benchmarks. The listed task types include visual question answering, chart/document parsing, 2D grounding, STEM puzzles, multi-image reasoning, hallucination robustness, Chinese/English text understanding, and math reasoning. The named datasets include MMMU, MathVista-mini, AI2D, MMBench, MME, MMStar, SEEDBench, ScienceQA, A-Bench, Q-Bench, HallusionBench, BLINK, CountBenchQA, C3, CLUEWSC, BoolQ, WinoGrande, TAL-SCQ5K-CN, APE210k, GSM8K-ZH, and Xiezhi-EN. The stated protocol is zero-shot inference via OpenCompass standard prompts, deterministic low-temperature decoding, and answer formats constrained to benchmark-specific outputs such as option labels or short text [2606.25034].

Level 2 measures multimodal content and AI safety. Its public safety component lists Hateful Memes, MMHS, HOD, LlavaGuard, ProGuard, MM-SafetyBench, UnsafeBench, and EVADE-Bench, with two evaluation settings: original protocols where applicable, and unified binary safe/unsafe classification under a shared prompt and parsing pipeline. Its e-commerce governance component consists of four self-constructed benchmarks—LogoRecognition, BrandRecognition, ProdCategoryRecognition, and ProdPriceRecognition—each containing several thousand real-world listings with adversarial modifications such as occlusion, stylization, and counterfeit cues. The protocol uses zero-shot classification prompts with one-word outputs, “safe” or “unsafe,” or category labels; e-commerce tasks additionally permit multi-choice or open-ended price-range inference [2606.25034].

Level 3 targets in-house capability and business benchmarks. The task families are Domain Knowledge, Visual Perception, AI-Generated Image Detection, Instruction Following under Dynamic Policy, Adversarial Robustness, and End-to-End Safety Tasks. The specification describes 20+ datasets, while the composite averages section reports 21 internal tasks; both formulations appear in the source description. Production-mirroring prompts are used, including structured JSON, evidence citation, and free-text reasoning, and many tasks employ constrained decoding and automated parsing [2606.25034].

## 3. Task content and operational protocols

The operational logic of YVRE lies in the transition from broad capability probes to domain-specific governance and then to production-facing workflows. At Level 1, the benchmark uses conventional public multimodal tasks to test whether a safety-specialized model has preserved broad competence. The example difficulties reported in the summary table include multi-image illusions in HallusionBench and fine-grained counting in CountBenchQA [2606.25034].

At Level 2, the suite centers on multimodal safety classifications and adversarial governance settings. Public safety tasks include subtle hate-speech memes and multimodal jailbreaks. The e-commerce tasks are explicitly framed as governance problems involving partially occluded logos and price inference from visual cues. This pairing is significant because it moves beyond open-domain harmful-content detection toward cases in which safety judgments depend on brand identity, counterfeit indicators, or product semantics [2606.25034].

At Level 3, YVRE shifts from benchmark-style classification to deployment-oriented work units. Domain Knowledge covers multiple-choice and true/false questions on politically sensitive imagery, pornography regulations, and a content-safety knowledge base. Visual Perception includes sensitive face ID, flag and symbol detection, multi-scene OCR, logo and brand recognition, and product category and price reading. AI-Generated Image Detection is described as an internal AIGC detector covering diffusion-, GAN-, and transformer-based outputs under various post-processings. Instruction Following under Dynamic Policy uses bilingual image-text and text-only prompts with injected rule revisions. Adversarial Robustness includes coded language and metaphor identification, interactive high-risk content prevention through multi-turn jailbreak, and semantic disguise probes. End-to-End Safety Tasks evaluate full moderation pipelines with chain-of-thought, evidence attribution, and structured JSON outputs for live traffic [2606.25034].

This composition suggests that YVRE is designed to probe not only classification accuracy but also whether a model can satisfy production constraints such as structured outputs, policy-conditioned instruction following, and auditable evidence attribution.

## 4. Metrics and scoring rules

YVRE specifies a compact metric set intended to support cross-level aggregation while preserving task-appropriate scoring. Let \(N\) be the total number of test examples and \(C\) the set of classes [2606.25034].

Accuracy is defined as
$$
\mathrm{Accuracy} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}(\hat y_i = y_i).
$$

For imbalanced classification tasks, YVRE reports Macro F1. For each class \(c\),
$$
\mathrm{Precision}_c = \frac{\mathit{TP}_c}{\mathit{TP}_c + \mathit{FP}_c},\quad
\mathrm{Recall}_c = \frac{\mathit{TP}_c}{\mathit{TP}_c + \mathit{FN}_c},
$$
and
$$
F1_c = \frac{2\,\mathrm{Precision}_c\,\mathrm{Recall}_c}{\mathrm{Precision}_c + \mathrm{Recall}_c},
\qquad
\mathrm{Macro\ F1} = \frac{1}{|C|}\sum_{c\in C} F1_c.
$$

For adversarial perturbation sets \( \{(x_i,\delta_i,y_i)\}_{i=1}^N \), the benchmark description gives an illustrative Adversarial Robustness Score:
$$
\mathrm{AR} = \frac{1}{N}\sum_{i=1}^N \mathbf{1}\bigl(f(x_i + \delta_i)=y_i\bigr).
$$

Composite averages are defined for the Level 1 general multimodal average over 23 benchmarks, the Level 2 safety average over 12 public + 4 e-commerce benchmarks, and the Level 3 in-house average over 21 internal tasks. In the summary table, Level 3 metrics are reported as Acc/Macro F1/Struct. Corr., reflecting the fact that some production-style tasks require structured-correctness evaluation in addition to standard classification metrics [2606.25034].

A notable point is that YVRE emphasizes robustness metrics but does not yet present a fully standardized adversarial robustness protocol. This is acknowledged directly in the future recommendations, which call for standardizing adversarial robustness metrics such as AR and Attack Success Rate with shared perturbation protocols for fair cross-model comparisons [2606.25034].

## 5. Reported empirical results for Yuvion VL-32B

The reported results are zero-shot unless otherwise noted. On Level 1 general multimodal evaluation, averaged over 23 benchmarks, Qwen3-VL-32B scores 83.6 in Instruct mode and 84.9 in Thinking mode, while Yuvion VL-32B scores 81.0 in Instruct mode and 82.3 in Reasoning mode. Ultra-large baselines are reported as GPT-5.4 at 83.8, Opus-4.6 at 84.6, K2.5 at 85.8, and Qwen3.5-Plus at 86.4. The interpretation given in the benchmark summary is that YV-32B preserves core VLM competence within a 2–3 percentage point margin [2606.25034].

On Level 2 multimodal safety, reported over 12 public + 4 e-commerce benchmarks, Qwen3-VL-32B scores 65.7 in Instruct mode and 65.2 in Thinking mode, while YV-32B scores 76.9 in Instruct mode and 76.7 in Reasoning mode. The listed baselines are GPT-5.4 at 70.1, Opus-4.6 at 69.3, K2.5 at 69.8, and Q3.5-Plus at 71.4. The source summary states that YV-32B outperforms comparably sized open-source models by approximately 11 percentage points and the best closed-source baseline by approximately 6 percentage points on safety tasks [2606.25034].

On Level 3 in-house business benchmarks, averaged over 21 tasks, Qwen3-VL-32B scores 74.2 in Instruct mode and 71.3 in Thinking mode, whereas YV-32B scores 82.8 in Instruct mode and 82.6 in Reasoning mode. The listed baselines are GPT-5.4 at 75.9, GLM-5 at 58.7, K2.5 at 64.1, and Q3.5-Plus at 74.9. The benchmark summary states that YV-32B leads by 7–11 percentage points over ultra-large models on production-critical metrics [2606.25034].

A separate result is reported for AI-Generated Image Detection using Macro F1: Qwen3-VL-32B scores 60.4, YV-32B scores 74.1, GPT-5.4 scores 75.1, Q3.5-Plus scores 69.6, and K2.5 scores 65.8. The corresponding interpretation is that YV-32B nearly matches GPT-5.4 while exceeding other closed-source MLLMs [2606.25034].

| Evaluation | Key reported scores (%) | Stated interpretation |
|---|---|---|
| Level 1 general multimodal | YV-32B 81.0 / 82.3; Qwen3-VL-32B 83.6 / 84.9 | Core competence preserved within 2–3 pp |
| Level 2 multimodal safety | YV-32B 76.9 / 76.7; Qwen3-VL-32B 65.7 / 65.2 | Approx. 11 pp over open-source and 6 pp over best closed-source |
| Level 3 in-house business | YV-32B 82.8 / 82.6; GPT-5.4 75.9 | Leads by 7–11 pp on production-critical metrics |
| AIGC detection Macro F1 | YV-32B 74.1; GPT-5.4 75.1 | Nearly matches GPT-5.4 |

## 6. Qualitative findings, limitations, and future development

The benchmark description identifies several strengths. Fine-grained adversarial contrast through C2FT is said to dramatically improve disambiguation of visually similar but semantically opposing cases, such as mutated logos versus legitimate symbols. Chain-of-Thought and reasoning post-training are reported to enable auditable multi-step decisions in high-stakes domains, including legal and policy rationale. The adversarial-aware data flywheel is described as ensuring continuous coverage of novel evasion strategies, including small-text watermarks, semantic disguises, and coded language [2606.25034].

The reported failure modes are equally specific. Residual degradation remains on some abstract commonsense VQA tasks, including BLINK and MMStar, due to domain over-specialization. The model shows occasional false positives on benign content in metaphorical or allegorical scenarios when domain rules conflict. Price estimation is also identified as challenging for highly stylized or context-devoid product images [2606.25034].

The future recommendations indicate the authors’ view of what is still missing from multimodal safety evaluation. They propose extending YVRE with dynamic adversarial dialog suites that combine image- and text-based jailbreak attempts, introducing continuous-monitoring benchmarks that replay real production failure logs to close the data flywheel loop, standardizing adversarial robustness metrics with shared perturbation protocols, and expanding multilingual coverage because policies and visual symbols vary drastically across regions [2606.25034].

A plausible implication is that YVRE is intended not as a fixed benchmark suite but as an evolving evaluation infrastructure tied to a continuous adversarial adaptation cycle.

## 7. Position within adversarial risk-evaluation research

YVRE belongs to a broader family of adversarially aware evaluation frameworks, but its scope is specifically multimodal and deployment-oriented. A closely related text-centric framework is Yuvion LLM RiskEval (YLRE), introduced for Yuvion LLM as a collection of 93 benchmarks across four evaluation categories, with a focus on safety, adversarial robustness, and real-world capability requirements [2606.27632]. The presence of both suites indicates a shared design philosophy across vision–language and language-only settings: general capability must be preserved, adversarial safety must be stress-tested explicitly, and in-house workflows must be evaluated separately rather than inferred from public benchmarks alone.

A further related design pattern appears in medical multimodal security evaluation. VSF-Med is described as an end-to-end vulnerability-scoring framework for medical vision-language models that combines a library of text-prompt attack templates, imperceptible visual perturbations calibrated by Structural Similarity Index thresholds, and an eight-dimensional rubric consolidated into a composite risk metric [2507.00052]. This suggests a broader methodological convergence around adversarial variant generation, domain-specific scoring, and composite risk summaries, although YVRE differs in target domain, benchmark breadth, and explicit production-readiness emphasis.

Within that landscape, Yuvion VL RiskEval is distinguished by its three-level structure, its combination of public and internal evaluations, and its attempt to quantify both capability preservation and adversarial safety gains in the same framework. The benchmark’s reported results present Yuvion VL-32B as achieving strong multimodal safety performance while maintaining comparable general capabilities, and the suite itself as an adversarially aware framework for evaluating content and AI safety in vision–language systems [2606.25034].

Source: https://www.emergentmind.com/topics/yuvion-vl-riskeval-yvre