DV-World: Benchmarking Data Visualization Agents in Real-World Scenarios

Abstract: Real-world data visualization (DV) requires native environmental grounding, cross-platform evolution, and proactive intent alignment. Yet, existing benchmarks often suffer from code-sandbox confinement, single-language creation-only tasks, and assumption of perfect intent. To bridge these gaps, we introduce DV-World, a benchmark of 260 tasks designed to evaluate DV agents across real-world professional lifecycles. DV-World spans three domains: DV-Sheet for native spreadsheet manipulation including chart and dashboard creation as well as diagnostic repair; DV-Evolution for adapting and restructuring reference visual artifacts to fit new data across diverse programming paradigms and DV-Interact for proactive intent alignment with a user simulator that mimics real-world ambiguous requirements. Our hybrid evaluation framework integrates Table-value Alignment for numerical precision and MLLM-as-a-Judge with rubrics for semantic-visual assessment. Experiments reveal that state-of-the-art models achieve less than 50% overall performance, exposing critical deficits in handling the complex challenges of real-world data visualization. DV-World provides a realistic testbed to steer development toward the versatile expertise required in enterprise workflows. Our data and code are available at \href{https://github.com/DA-Open/DV-World}{this project page}.

• The paper introduces DV-World, a comprehensive benchmark evaluating DV agents across spreadsheet manipulation, logic evolution, and interactive disambiguation in native software environments.
• It employs a hybrid evaluation framework combining quantitative table-value alignment and qualitative multimodal rubric-based assessments, revealing state-of-the-art models achieving below 52% overall performance.
• Findings reveal that current generative models struggle with data accuracy, layout aesthetics, and robust multi-turn interaction, underscoring the need for advanced research in DV-agent development.

DV-World: A Comprehensive Benchmark for Real-World Data Visualization Agents

Motivation and Benchmark Design

DV-World (2604.25914) presents a systematic benchmark targeting the end-to-end lifecycle of data visualization (DV) agents in professional, real-world scenarios. Unlike prior benchmarks limited to code-sandboxed or single-turn chart creation, DV-World covers native software environments, cross-modal logic evolution, and proactive interactive disambiguation. It deconstructs practical DV workflows into three domains:

• DV-Sheet (native spreadsheet manipulation: creation, repair, dashboards)
• DV-Evolution (cross-paradigm artifact adaptation across visualization frameworks)
• DV-Interact (proactive alignment with ambiguous user requirements via dialogue-driven clarification)

  Figure 1: DV-World aims to evaluate data visualization agents across the full lifecycle of native manipulation in real software environments (DV-Sheet), cross-modal logic evolution (DV-Evolution), and proactive iterative interaction (DV-Interact) scenarios.

The benchmark introduces 260 expert-annotated tasks sampled from real data analysis communities (e.g., Kaggle, ExcelForum, Chandoo.org) and rigorously anonymized/perturbed to preserve realism while mitigating information leakage. The annotation workflow enforces stringent object model compliance, multi-stage intent modeling, and generation of gold-standard artifacts for deterministic validation.

Evaluation Protocols and Metrics

DV-World employs a hybrid evaluation framework integrating both quantitative and qualitative metrics. For tasks yielding concrete artifacts (DVSheet-Crea, DV-Evol), it combines:

• Table-value Alignment: stringent cell-wise correspondence between generated outputs and reference tables (no tolerance for semantic/metric drift).
• MLLM-as-a-Judge with Rubrics: rubric-based semantic/visual assessment using strong multimodal LLMs, validated against human expert ratings for rubric soundness and scoring stability.
• Domain-Specific Criteria: e.g., binary success in repair (DVSheet-Fix), dashboard insightfulness, multi-turn interaction success (DV-Interact ISR metric).

Figure 2 demonstrates the hybrid rubric + data-fidelity evaluation pipeline.

Figure 2: Hybrid evaluation approach for quantifying both structural and visual/semantic fidelity in DVSheet-Crea and DV-Evol tasks.

The evaluation protocol enforces compliance with dynamic data binding, preservation of provenance, reproducibility, and professional visual standards—criteria not adequately captured by execution-only or image-similarity metrics alone.

Experimental Results and Error Analysis

State-of-the-art LLMs and agents (Gemini-3-Pro, GPT-5.2, DeepSeek-V3.2, Qwen3-VL, SheetCopilot, OpenHands) are comprehensively evaluated on all DV-World domains. Across the board, no model attains even 52% overall performance, a strong empirical claim that underscores fundamental gaps in current generative and agentic approaches.

DV-Sheet

Spreadsheet-native chart generation and repair remain challenging. The best performance in dashboard creation or repair tasks does not surpass 40.5%. Extensive error analysis further decomposes failure cases:

• Data Accuracy dominates (>50% of errors in chart creation, >69% in fixes). Agents frequently misbind cell ranges, misaggregate, or fail to preserve structural provenance.
• Layout/Aesthetics are consistently suboptimal, manifesting as unreadable layouts and noncompliance with visual conventions.

Figure 3: (a) DVSheet-Dash performance drops with increasing table scale; (b) framework-specific disparities in DV-Evol logic evolution.

DV-Evolution

Agents are evaluated on their capability to migrate chart logic and style across Python, Apache ECharts, Vega-Lite, D3.js, and Plotly.js—each posing varying demands in API verbosity, abstraction, and declarativity. Highest aggregate scores remain at 51.4% (Gemini-3-Pro).

• Performance decays steeply with artifact complexity (increasing LOC)—see Figure 4—particularly for low-level APIs such as D3.js, indicating limitations in long-horizon code reasoning and semantic carryover.

  Figure 4: Agent performance sharply decays as the required Lines of Code increase for evolving visualizations across frameworks.

• The load_image tool is shown to be essential for semantic fidelity; ablations result in drops up to 7.7% for D3.js, confirming the necessity of multimodal grounding rather than unimodal code synthesis.

DV-Interact

Interactive DV tasks highlight the inability of models to robustly identify and resolve user ambiguity. The highest observed scores just exceed 40%, indicating:

• Cognitive Execution Gaps: agents may identify intent constraints but fail to execute consistent, data-faithful code.
• Interactive Avoidance: agents overconfidently predict visualizations without adequate clarification (cf. GLM-4.7, GPT-4.1).

Simulator and Judge Alignment

DV-World additionally validates its user simulator (for dialogue-based tasks) and rubric-MLLM judge protocol:

• The GPT-5-mini simulator achieves 88.7% faithfulness and 0.86 Pearson correlation with human user responses (Figure 5), maintaining high fidelity in intent modeling and refusal logic, while constraining privileged data leakage.

  Figure 5: Agent capability is strongly tied to simulator intelligence; GPT-5-mini provides high faithfulness with lower operational cost.

• Multi-model rubrics and judges show high human-model agreement, with ICC(A,1) ≥ 0.85 and Kendall’s τ_b ≥ 0.97 for leaderboard rankings, confirming the stability of scoring.

Implications and Future Directions

Strong empirical evidence from DV-World highlights that even the most advanced LLMs and DV agents are not yet robust or reliable enough for complex, production-grade DV workflows. Integrating dynamic environmental mastery, semantic portability across paradigms, and high-bandwidth, dialogic user alignment is a nontrivial multi-agent or multi-modal challenge unsolvable by current one-shot or unimodal approaches.

• Practically, this implies that deployment of agentic DV systems in enterprise analytics remains premature. Models exhibit brittle behaviors under realistic ambiguity, cross-framework translation, and large-scale object model manipulation, which are typical of real-world business environments.
• Theoretically, DV-World exposes the urgent need for research into multi-turn semantic recovery, robust multimodal grounding, advanced repair/diffusion mechanisms, and iterative feedback integration for trusted dashboarding and analytics automation.
• Future LLM/DV-agent approaches must move beyond code generation, incorporating model-based planning, visual reasoning, intent subgoal clarification, structured auditability, and complex error recovery grounded in true software environments.

Conclusion

DV-World establishes a high-fidelity, multifaceted benchmark for the DV agent community, grounded in real-world software ecosystems, cross-modal logic, and ambiguous user dialogue. The strong, consistent performance ceiling (<52%) across state-of-the-art models constitutes a substantive, technically-supported indictment of current methods’ limitations. The benchmark’s hybrid evaluation rubric, extensive error taxonomies, and validated judge/simulator components set a new standard for rigorous DV agent assessment. It is positioned as a necessary testbed to catalyze research in truly agentic data visualization and to facilitate longitudinal progress tracking toward industrial-grade, reliable DV systems.