SciVis Agentic Evaluation Benchmark

• SciVis Agentic Evaluation Benchmark is a framework that assesses AI agents’ ability to execute complex visualization workflows with natural-language intent and tool interaction.
• It structures evaluation over diverse dimensions such as application domain, data type, complexity, and visualization operations to ensure holistic performance.
• The benchmark combines multimodal LLM-based judging, deterministic verifiers, and human-in-the-loop studies to measure both technical output quality and trustworthiness.

A SciVis Agentic Evaluation Benchmark is a benchmarking framework for scientific visualization agents: systems that interpret natural-language intent, interact with visualization pipelines and tools, and produce visual outputs aligned with analysis goals. In current work, the topic spans outcome-centric evaluation of real SciVis workflows, multimodal judging of rendered outputs, deterministic verification of tool states and scripts, human-in-the-loop studies of mixed-initiative interaction, and broader trustworthiness assessment over representative socio-technical scenario distributions rather than isolated benchmark items (Ai et al., 31 Mar 2026, Ai et al., 4 Jun 2026, Ai et al., 25 Jun 2026, Qi et al., 16 Mar 2026).

1. Conceptual basis and rationale

The benchmarking problem arises because agentic systems differ from static question-answering models in both capability and risk profile. Agentic AI systems are described as planning multi-step workflows, calling tools, maintaining internal state and memory, and executing actions with real-world side effects; their failures include goal drift, unsafe or unauthorized tool use, resource misuse, and social harms. In this view, trustworthiness is a deployment-level property shaped by environment, tools, policies, and social context, not only by model weights (Qi et al., 16 Mar 2026).

Within scientific visualization, the same shift appears as agents move from chart generation or code completion toward full workflow execution in tools such as ParaView, napari, VMD, and TTK. HiLSVA defines agentic SciVis as “AI agents that interpret user intent expressed in natural language and autonomously interact with visualization pipelines to produce outputs aligned with user-specified analysis goals,” but argues that the central challenge is to balance human analytical control with machine autonomy rather than maximizing autonomy alone (Ai et al., 25 Jun 2026). The evaluation-centric position paper makes a closely related claim: SciVis agents should be designed around evaluation from the start, because benchmarks become specifications for what these agents must be able to do in realistic scientific settings (Ai et al., 18 Sep 2025).

A major motivation for a dedicated benchmark is the broader “measurement imbalance” in agentic AI evaluation. A systematic review of 84 papers reported technical metrics in 83% of papers, human-centered metrics in 30%, economic metrics in 30%, safety/governance metrics in 53%, both technical and human-centered metrics in only 15%, and any longitudinal or temporal dimension in only 5% (Meimandi et al., 1 Jun 2025). For SciVis agents, this implies that a benchmark limited to output correctness or latency will likely overestimate deployment value and underestimate workflow, trust, and safety failure modes.

2. Task space, benchmark content, and scenario structure

The most explicit benchmark instantiation is SciVisAgentBench, which organizes 108 expert-crafted cases using a structured taxonomy over four dimensions: application domain, data type, complexity level, and visualization operation. The benchmark covers eight domains, five data types, three conceptual complexity levels, and fifteen visualization operations, and includes suites for ParaView visualization, molecular visualization, bioimage visualization, topology visualization, and object identification (Ai et al., 31 Mar 2026, Ai et al., 4 Jun 2026).

Its task space reflects realistic SciVis practice rather than isolated plotting primitives. Representative cases span volume rendering, isosurface extraction, flow visualization, molecular trajectories, napari-based multi-channel image analysis, topological analysis with TTK, and ParaView-based object identification. The complexity distinction is especially important: the benchmark reports task- and workflow-level cases, while single operations are treated as compositional building blocks (Ai et al., 31 Mar 2026).

A complementary human-centered decomposition appears in HiLSVA, which structures evaluation around three SciVis stages: data exploration, data visualization, and scientific insight or scientific analysis. Its user-study tasks include hurricane visualization with slices and contours, foot volume and isosurface rendering, tornado vector-field streamlines and glyphs, combustion time-varying multivariate volume rendering and isosurface animation, and a half-cylinder case with Q-criterion and $\lambda_2$ vortex identification (Ai et al., 25 Jun 2026). This stage-based framing is not identical to SciVisAgentBench’s taxonomy, but it suggests that a comprehensive benchmark should cover both low-level operations and higher-level scientific analysis tasks.

Earlier LLM-based scientific visualization benchmarking focused on Matplotlib-style tasks. MatPlotBench introduced 100 human-verified test cases drawn from Matplotlib Gallery and OriginLab GraphGallery, with diverse types including line plots, bar charts, scatter plots, contour plots, polar plots, Sankey diagrams, sunburst charts, radial plots, chord diagrams, and streamplots (Yang et al., 2024). A plausible implication is that current SciVis agentic benchmarking has evolved from static plotting tasks toward broader, tool-grounded, long-horizon workflow evaluation.

3. Evaluation methodologies and core metrics

Current benchmarks use a hybrid evaluation stack rather than a single scoring mechanism. SciVisAgentBench combines multimodal LLM-based judging with deterministic evaluators, including image-based metrics, code checkers, rule-based verifiers, and case-specific evaluators. For controlled-view ParaView tasks, it reports scaled image metrics: $$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$

$$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$

$$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$

where $r = N_{\text{pass}} / N_{\text{total}}$ is completion rate. For topology tasks, it uses case-specific evaluators such as Wasserstein distance for persistence diagrams and Dice-based matching for segmentation-like outputs (Ai et al., 31 Mar 2026).

SciVisAgentSkills evaluates agent performance on SciVisAgentBench with overall score, completion rate, and consistency-oriented pass metrics. With task-level success indicators $r_{i,j} \in \{0,1\}$ for task $i$ and run $j$: $$\text{pass@k} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\left( \sum_{j=1}^{k} r_{i,j} \ge 1 \right)$$

$$\text{pass}^{k} = \frac{1}{N} \sum_{i=1}^{N} \mathbf{1}\left( \sum_{j=1}^{k} r_{i,j} = k \right)$$

These metrics distinguish solve-once capability from repeated reliability, which is crucial for long-horizon agentic workflows (Ai et al., 4 Jun 2026).

MatPlotBench uses GPT-4V as an automatic visual evaluator by comparing a model-generated plot against a ground-truth figure and assigning a score from 0 to 100. The resulting automatic scores show strong correlation with human evaluation: for GPT-4-generated plots, Pearson $$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$0 with $$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$1; for GPT-3.5-generated plots, $$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$2 with $$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$3 (Yang et al., 2024). By contrast, $$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$4-Rec criticizes “LLM-as-a-judge” evaluations for subjectivity, cost, and inconsistency, and proposes verifiable rewards plus pass$$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$5 reliability for conversational recommender agents (Narasimhan et al., 8 Jun 2026). The current SciVis literature therefore treats LLM judging as useful but insufficient on its own: it is typically paired with deterministic checks rather than used as a standalone ground truth mechanism (Ai et al., 31 Mar 2026).

Long-horizon scientific tool-use benchmarks outside visualization reinforce this direction. SciAgentBench defines Success Rate,

$$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$6

and Success weighted by Path Length,

$$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$7

to distinguish correctness from workflow efficiency in multi-step scientific environments (Shen et al., 13 Feb 2026). This suggests that SciVis benchmarks benefit from path-aware metrics whenever multiple tool-call trajectories can yield acceptable outputs.

4. Human-centered, mixed-initiative, and trustworthiness dimensions

A purely technical benchmark does not capture the full evaluation problem. HiLSVA makes human oversight, provenance, and autonomy control central benchmark dimensions. Its system uses a plan-first multi-agent architecture in which the orchestrator synthesizes an explicit, editable stepwise plan, execution begins only after user approval, and users can intervene at any time through natural-language instructions or direct interaction with the visualization environment. It also provides prospective and retrospective provenance, rollback, branching, and sandboxed execution in isolated Docker containers (Ai et al., 25 Jun 2026).

HiLSVA evaluates three autonomy modes: full-autonomous (FA), half-autonomous (HA) without learn-at-test-time, and mixed-initiative (MI) with learn-at-test-time. In a controlled user study with 12 participants across three expertise levels, all participants completed the tasks successfully. Mean execution times for the first three tasks were $$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$8 minutes for FA, $$\mathrm{PSNR}_{\text{scaled}} = r \times \overline{\mathrm{PSNR}}$$9 for HA, and $$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$0 for MI, with a Friedman test of $$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$1, $$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$2, and Kendall’s $$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$3. Average interpretation accuracy was $$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$4 out of 12, with novices reaching near-expert performance. Questionnaire scores were especially high for review and approval of actions, autonomy adjustment, clarity of system feedback, stepwise provenance, and revisiting previous steps (Ai et al., 25 Jun 2026). The benchmark implication is that SciVis agents should be assessed not only for output quality but also for user control, transparency, collaboration quality, and the tradeoff between execution efficiency and oversight.

The broader trustworthiness literature generalizes this point beyond human factors. HAAF argues that evaluation should move from “benchmark islands” toward representative socio-technical scenario distributions and models trustworthiness over a space $$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$5 of scenarios: $$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$6 It proposes four benchmark components—static cognitive and policy analysis, interactive sandbox simulation, social-ethical alignment assessment, and a distribution-aware representative sampling engine—linked by an iterative Trustworthy Optimization Factory (Qi et al., 16 Mar 2026). For scenario sets $$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$7, it defines: $$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$8 It also uses a structured failure taxonomy including Prompt Injection, Goal Drift, Unauthorized Action, Hallucinated Tool-use, Recovery Failure, Operational Failure, Policy Leak, and Social Harm (Qi et al., 16 Mar 2026). In a pilot with 24 scenarios, interventions reduced violations from 4/24 to 1/24, lowering VR from 16.7% to 4.2% and RWF from 0.210 to 0.062 (Qi et al., 16 Mar 2026).

This broadens the meaning of a SciVis benchmark. A plausible implication is that future SciVis agentic evaluation will need to assess not only whether a visualization is correct, but whether the agent behaves safely and transparently under realistic operational, social, and organizational pressures. That interpretation aligns with the four-axis model of Technical, Human-centered, Temporal, and Contextual evaluation: $$\mathrm{SSIM}_{\text{scaled}} = r \times \overline{\mathrm{SSIM}}$$9 and the weighted effectiveness score

$$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$0

which was proposed precisely to counteract measurement imbalance in agentic AI evaluation (Meimandi et al., 1 Jun 2025).

5. Empirical findings from current benchmark instantiations

Existing benchmark results show that SciVis agents remain uneven across tools, workflow types, and harnesses. On SciVisAgentBench, general-purpose coding agents often outperform specialized SciVis agents in overall outcome quality, especially in ParaView and topology suites. In the ParaView visualization suite, Claude Code with Sonnet-4.5 scored about 62.6 with completion around 98%, and Codex with GPT-5.2 scored about 60.2 with completion around 95%, while ChatVis and ParaView-MCP variants scored substantially lower. In the topology suite, Codex reached about 76.4 with completion around 85.2%, well above TopoPilot variants near 31–32 (Ai et al., 31 Mar 2026).

At the same time, specialized agents can be more efficient or stronger in narrower domains. BioImage-Agent outperformed general-purpose coding agents in bioimage tasks for a given token budget, and specialized MCP-style pipelines used fewer tokens because their tool interfaces were more constrained (Ai et al., 31 Mar 2026). The literature therefore points to a recurring tradeoff between broad adaptability and tool-specific efficiency.

SciVisAgentSkills studies this tradeoff by adding structured procedural knowledge modules to Claude Code and Codex. Across the five SciVisAgentBench suites, skills improved mean scores in every case. Examples include Claude Code on ParaView visualization from $$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$1 to $$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$2, Codex on molecular visualization from $$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$3 to $$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$4, Claude Code on topology from $$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$5 to $$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$6, and Claude Code on object identification from $$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$7 to $$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$8 (Ai et al., 4 Jun 2026). The study also shows that token-efficiency benefits depend on the harness that loads and applies the skills, so skills cannot be evaluated independently of the execution environment (Ai et al., 4 Jun 2026).

MatPlotAgent provides a simpler but earlier demonstration of the same pattern. On MatPlotBench, GPT-4 improved from 48.86 under direct decoding to 61.16 with MatPlotAgent, GPT-3.5 improved from 38.03 to 47.51, and Magicoder-S-DS-6.7B improved from 38.49 to 51.70 (Yang et al., 2024). Visual feedback is a substantial contributor: for GPT-4, removing the visual feedback mechanism reduced MatPlotAgent performance from 61.16 to 53.44 (Yang et al., 2024). Across these studies, the dominant empirical conclusion is that agentic scaffolding—tool interfaces, planning structure, self-debugging, visual feedback, and reusable procedural skills—matters as much as the backbone model.

6. Open problems and design directions

The literature converges on several unresolved design problems. First is representativeness. HAAF argues that benchmark scores are unreliable when $$\mathrm{LPIPS}_{\text{scaled}} = 1 - r \times (1 - \overline{\mathrm{LPIPS}})$$9, especially for rare, high-consequence events, and therefore reframes evaluation as distribution design over a scenario manifold rather than metric design over isolated tasks (Qi et al., 16 Mar 2026). A plausible implication is that a mature SciVis benchmark will need scenario sampling over task family, tool interface, interaction horizon, operational stress, social sensitivity, and consequence severity, rather than a fixed menu of canonical tasks.

Second is real tool ecosystems and long horizons. SciAgentGym and SciAgentBench show that scientific agents degrade sharply as interaction horizons grow, with GPT-5 dropping from 60.6% success at L1 to 30.9% at L3, and identify long-horizon workflow execution as the central bottleneck (Shen et al., 13 Feb 2026). MCPVerse makes a related point for large action spaces by exposing more than 550 real-world executable tools and showing that most models degrade when confronted with larger tool sets, while a few agentic models can use expanded exploration spaces to improve accuracy (Lei et al., 22 Aug 2025). For SciVis, these results support benchmarks that use real executable visualization tools, large tool catalogs, and explicit stress tests for error recovery and workflow efficiency.

Third is verifiability and automation of evaluation itself. A2Eval treats benchmark curation and evaluation as a multi-agent optimization problem, compressing embodied evaluation suites by 85%, reducing overall computational costs by 77%, and achieving a 4.6$r = N_{\text{pass}} / N_{\text{total}}$0 speedup while maintaining ranking fidelity and improving human alignment to Spearman’s $r = N_{\text{pass}} / N_{\text{total}}$1 (Zhang et al., 2 Feb 2026). $r = N_{\text{pass}} / N_{\text{total}}$2-Rec contributes a complementary lesson: reliability cliffs become visible only when benchmarks use verifiable rewards and repeated-run pass$r = N_{\text{pass}} / N_{\text{total}}$3 metrics rather than single-run subjective judgments (Narasimhan et al., 8 Jun 2026). This suggests that future SciVis benchmarks may need automated dimension induction, balanced suite construction, and more verifiable rewards than current LLM-judge-heavy protocols provide.

Fourth is multi-turn interaction, memory, and task dependency modeling. ATOD formalizes advanced task-oriented dialogue systems around multi-goal coordination, interleaving, long-horizon memory, asynchronous execution, and proactivity, and evaluates them with dependency-aware goal completion and memory recall metrics (Zhang et al., 17 Jan 2026). Toward evaluation frameworks for scientific multi-agent systems makes a similar case for multi-turn scientific dialogue, contamination-resistant tasks, and tool-augmented reasoning that is distinguishable from retrieval (Abram, 18 Mar 2026). In SciVis, this points toward benchmarks that model iterative analysis sessions, not only one-shot rendering tasks.

Finally, benchmark construction itself is becoming more framework-based and assumption-aware. The PHM paper-to-benchmark work argues that paper reproduction should be treated as framework integration through a slot-binding interface that explicitly records unresolved assumptions, enabling executable, comparable benchmark implementations rather than isolated code artifacts (Theiler et al., 27 May 2026). SciHorizon-DataEVA adds a multi-dimensional evaluation model for scientific datasets—Governance Trustworthiness, Data Quality, AI Compatibility, and Scientific Adaptability—operationalized through hierarchical multi-agent evaluation (Liu et al., 29 Apr 2026). Applied to SciVis, these strands suggest benchmarks that do not merely test agents on tasks, but also evaluate whether the underlying data, workflows, and assumptions are fit for AI-mediated scientific visualization.

Taken together, the current literature presents the SciVis Agentic Evaluation Benchmark not as a single leaderboard but as an emerging benchmark class: tool-grounded, multi-step, multimodal, human-aware, and increasingly distribution-aware. Its central problem is no longer whether an agent can generate a visualization, but whether it can do so reliably, transparently, and comparably across realistic scientific workflows (Ai et al., 18 Sep 2025).