---
title: 'MultiVis-Bench: Multi-Modal Viz Benchmark'
url: https://www.emergentmind.com/topics/multivis-bench
type: topic
---

# MultiVis-Bench: Multi-Modal Viz Benchmark

MultiVis-Bench is a large-scale benchmark designed to rigorously evaluate multi-modal, end-to-end data visualization generation systems across a spectrum of realistic analytical scenarios, with a particular focus on cross-modal and iterative workflows. It provides 1,202 expert-validated cases covering four orthogonal visualization generation scenarios—including basic text-to-visualization, image-referenced, code-referenced, and iterative refinement tasks—aimed at supporting the development and assessment of reliable, logic rule-enhanced LLM frameworks for data visualization. Unlike prior efforts restricted to text-plus-table translation or intermediate representations, MultiVis-Bench enables direct evaluation of systems generating executable Python (Altair) code with built-in protocol mechanisms for comprehensive quality and reliability measurement [2601.18320].

## 1. Motivation and Scope

MultiVis-Bench was conceived in response to two central limitations of existing Text-to-Vis benchmarks: their coverage is restricted to single-shot, text-plus-table input and they generate non-executable intermediate representations (e.g., VQL, Vega-Lite), which do not reflect the complexity of real-world analytical workflows. In practice, analysts require multi-modal input channels (spanning reference imagery and code), iterative solution refinement, and outputs that are directly executable for seamless analysis and visualization. MultiVis-Bench explicitly addresses these gaps by formalizing a family of data-to-visualization mappings and providing a unified, expert-curated dataset for end-to-end system evaluation.

## 2. Multi-Scenario Structure and Data Organization

MultiVis-Bench is organized into four scenario classes, each corresponding to a distinct family of cross-modal visualization tasks. The formal mapping for each scenario is denoted $f_S: \mathcal{X} \rightarrow V$, where $\mathcal{X}$ represents the tuple of input modalities and $V$ is the Altair visualization code.

| Scenario | Inputs $\mathcal{X}$                       | Cases | Key Modalities                                         |
|----------|-------------------------------------------|-------|--------------------------------------------------------|
| A—Basic Generation (BG)        | $(Q, D)$                                     | 306   | Natural language + database schema/data                |
| B—Image-Referenced Generation (IRG)   | $(Q, D, I_{ref})$                            | 109   | Text + database + reference image                      |
| C—Code-Referenced Generation (CRG)   | $(Q, D, C_{ref})$                            | 233   | Text + database + reference code (Matplotlib/Altair)   |
| D—Iterative Refinement (IR)          | $(Q, D, V_{old})$                            | 554   | Text + database + previous Altair code                 |

The 1,202 cases are distributed to sample diverse analytical needs: 127 Altair chart templates are utilized on 141 SQLite databases (from Spider, filtered for schema and attribute diversity). Each example includes high-fidelity input-output pairs, supporting multi-modal evaluation with real database constraints and direct execution of generated Python code.

## 3. Construction and Quality Assurance Protocols

The assembly of MultiVis-Bench followed a hybrid human-in-the-loop pipeline:

- **Example Generation:** Draft samples are proposed by LLMs (Gemini-2.0-pro-exp), conditioned on schema, template, and scenario specifications.
- **Human Validation:** Each case undergoes an average of 2.5 expert review rounds for technical correctness, semantic faithfulness, and perceptual effectiveness.
- **Automatic Checks:** Python code is formatted via Black and executed on the associated SQLite database; any samples failing syntactic or logical execution, or failing checklist-based semantic/visual quality standards, are rejected or corrected manually.

Dataset releases do not prescribe explicit train/dev/test splits, but users are encouraged to apply stratified sampling (e.g., 70/15/15) by scenario for robust generalization benchmarking [2601.18320].

## 4. Evaluation Metrics and Scoring Functions

MultiVis-Bench adopts a dual-layer scoring strategy for visualization quality, supplemented by reliability metrics tailored to the unique failure modes of LLM-based agents.

- **Low-Level Structural Score:** $S_{low} = \sum_{i=1}^6 w_i M_i$, covering chart type, data mapping, encoding, interaction, configuration, and transformation. Each $M_i \in [0, 1]$ is normalized, and $\sum_i w_i = 1$.
- **High-Level Perceptual Score:** $S_{high} = \sum_{j=1}^6 v_j P_j$, with $P_j$ (VLM-based) quantifying type appropriateness, layout, text, data representation, style, and clarity.
- **Combined Visualization Score:** $S_{vis} = \alpha S_{low} + (1-\alpha) S_{high}$, $\alpha = 0.5$ by default.

**Reliability Metrics:**
- **Task Completion Rate:** $(\#\text{tasks passed}) / (\#\text{tasks})$
- **Code Execution Success:** $(\#\text{snippets executed without error}) / (\#\text{snippets})$

This framework ensures both fine-grained assessment of visualization correctness and perceptual quality, as well as robustness to LLM agent failure cases.

## 5. Logic Rule Framework and Benchmarking Robustness

Although structurally static, MultiVis-Bench is intrinsically designed to facilitate evaluation under logic rule-enhanced agent frameworks (e.g., MultiVis-Agent). The evaluation protocol is underpinned by a four-layer logic rule system that regulates:

- **Coordination (CR):** Scenario classification by $\mathcal{F}_T(I) = \arg\max_{t \in \{A,B,C,D\}} \pi(t, I)$, with preference ordering $D \succ C \succ B \succ A$.
- **Tool Execution (TE):** Parameter boundary constraints $\phi(p, v)$.
- **Error Handling (EH):** Automatic LLM output error classification $\epsilon(x)$.
- **ReAct Control (RC):** Iteration termination $\tau(i, r)$, including hard cap $T_{max}=10$.

These rules guarantee that benchmarks avoid infinite loops, boundary violations, or error cascades, thus enabling repeatable, trustworthy system comparisons.

## 6. Empirical Results and Comparative Analysis

Experimental outcomes with MultiVis-Agent on MultiVis-Bench demonstrate substantial performance improvements attributable to the logic rule-backed approach:

- **Visualization Quality:** Average $S_{vis}$ of 74.18% across scenarios A–C, surpassing the strongest baseline by approximately 10 percentage points; in scenario B (IRG) MultiVis-Agent achieves 75.63% versus 62.79% (LLM Workflow) and 57.54% (Instructing LLM).
- **Reliability Gains:**
  - **Basic Generation:** Completion 98.68% (+20.46 pp), ExecSuccess 95.71% (+32.53 pp)
  - **Image-Referenced Generation:** Completion 99.58% (+25.10 pp), ExecSuccess 94.56% (+29.46 pp)
  - **Code-Referenced Generation:** Completion 99.81% (+9.25 pp), ExecSuccess 96.32% (+15.20 pp)
  - **Iterative Refinement:** Completion 100.00% (+7.97 pp), ExecSuccess 97.10% (+13.77 pp)

A plausible implication is that these logic rule constraints are necessary for both high-level output quality and operational stability in LLM-based, multi-modal visualization agents [2601.18320].

## 7. Significance and Future Directions

MultiVis-Bench represents a step-change in the benchmarking of automated analytical visualization agents by marrying scenario diversity, rigorous ground-truth curation, and executable output specification. It sets a new methodological standard for evaluating cross-modal, iterative, and agent-centric systems, especially those relying on large language models combined with logic control. While MultiVis-Bench currently operates as a static dataset, its logic rule-aware design anticipates broader adoption of programmatic reliability mechanisms in autonomous data analytics frameworks. Future expansions might incorporate new modalities or domain-specific constraints but will likely retain the dual emphasis on end-to-end, executable evaluation and formalized system robustness [2601.18320].

Source: https://www.emergentmind.com/topics/multivis-bench