- The paper benchmarks six multimodal large language models on the 49-item SVLAT against 485 human participants, finding Gemini-3.1-Pro leads with 88.6% overall accuracy versus 75.6% for humans.
- Models perform unevenly across visualization techniques and tasks, with strong results on scientific illustrations and rendering but major weaknesses in quantitative estimation, dense flow fields, and encoding interpretation.
- The findings show that high accuracy can conceal shallow visual grounding, supporting SciVis-native benchmarks as an essential complement to chart-focused evaluations of multimodal AI.
Motivation and positioning
Scientific visualization (SciVis) differs from information visualization in that it encodes data with intrinsic spatial or physical structure, requiring viewers to reason about geometry, depth, topology, motion, and relationships among scalar, vector, and tensor fields. Existing evaluations of multimodal LLMs (MLLMs) on visualization understanding remain largely chart-centric—built on benchmarks such as FigureQA, DVQA, PlotQA, ChartQA, CharXiv, and SPIQA—and prior literacy-focused studies of GPT-4 and VLAT-style instruments do not address SciVis-native representations. This paper addresses that gap by benchmarking MLLMs on the Scientific Visualization Literacy Assessment Test (SVLAT), a psychometrically grounded instrument comprising 49 items over 18 scientific visualizations and illustrations, spanning 8 SciVis techniques and 11 task types under a closed-world design in which each item is answerable from the visualization and caption alone.
The central claim is that current MLLMs are selectively capable rather than broadly SciVis literate, and that SciVis literacy should be treated as a necessary benchmark dimension for multimodal AI systems—a claim supported by both quantitative comparisons against 485 human participants and qualitative error analysis of model rationales.
Evaluation methodology
Six models were evaluated: three closed-source systems accessed via OpenRouter (GPT-5.4, Claude-Opus-4.6, Gemini-3.1-Pro-Preview) and three open-source models (Qwen3.5-9B, InternVL3.5-8B, LLaVA-OneVision-1.5-8B-Instruct). All models used temperature 0, max_tokens 300, and a single standardized prompt that frames the model as a SciVis expert, restricts it to the provided visualization and caption, requires structured JSON output with a rationale, and includes a "Not sure" option to discourage unsupported guessing. Each item was run 10 times to mitigate residual backend nondeterminism, with averaged accuracy reported. Because GPT and Claude APIs did not support direct video input at evaluation time, animation items were handled via frame extraction at one frame per second—an important protocol caveat when interpreting their animation results. Human baselines come from the original SVLAT tryout with 485 non-expert participants.
Gemini is the strongest model by a wide margin, exceeding human performance on every subset:
| Model |
Image (%) |
Animation (%) |
All (%) |
| Gemini-3.1-Pro-Preview |
90.9 ± 4.3 |
82.9 ± 8.5 |
88.6 ± 3.9 |
| Claude-Opus-4.6 |
81.7 ± 5.8 |
59.3 ± 11.7 |
75.3 ± 5.5 |
| GPT-5.4 |
75.7 ± 5.5 |
73.6 ± 10.4 |
75.1 ± 4.9 |
| Qwen3.5-9B |
69.4 ± 6.0 |
67.1 ± 9.4 |
68.8 ± 5.0 |
| InternVL3.5-8B |
60.9 ± 6.7 |
72.9 ± 9.9 |
64.3 ± 5.6 |
| LLaVA-OneVision-1.5-8B |
60.3 ± 6.9 |
73.6 ± 9.7 |
64.1 ± 5.8 |
| Human |
76.2 ± 2.4 |
73.9 ± 4.7 |
75.6 ± 2.2 |
Two results stand out. First, Gemini's overall accuracy of 88.6% exceeds the human mean of roughly 76%, making it the only model above the human baseline; the three open-source models all fall below it. Second, Claude's animation performance (59.3%) is the lowest of all models and humans, despite competitive image-item performance—a drop the authors attribute partly to the frame-extraction protocol, though this remains an assumption rather than an established cause. Notably, the open-source models InternVL and LLaVA-OneVision perform better on animation items than on static images, and MLLMs exhibit substantially greater performance heterogeneity across item types (SDs of 3.9–11.7) than humans (SDs of 2.2–4.7), indicating less consistent literacy across formats.
Humans show the most balanced profile across techniques (68.9%–88.8%), while models fluctuate widely. Scientific Illustration is the most accessible category for everyone: humans reach 79.5% and all models range from 81.4% to 95.7%, with Gemini achieving 95.7%. Gemini also attains perfect scores on Surface Rendering and Volume Rendering (100.0%), 95.0% on Mixed Rendering, and 91.8% on Glyph/Mesh/Plot, exceeding humans on each. Closed-source models consistently outperform open-source ones on Surface Rendering, Mixed Rendering, and Glyph/Mesh/Plot, with the gap most pronounced on the latter.
The clearest human–model gaps appear on Texture-Based Visualization (humans 82.0%) and Integration-Based Visualization (humans 88.8%). LLaVA-OneVision is a notable exception on texture-based items (90.0%), outperforming all other models there, while on integration-based items LLaVA-OneVision (82.5%), GPT (80.0%), and Gemini (75%) remain competitive with humans. These uneven profiles indicate that model competence does not transfer uniformly across SciVis encoding families.
Search and Spatial Understanding tasks are the most accessible, whereas Quantitative Estimation–Relative Estimation (Quantitative) is the hardest for every model and for humans, remaining below 50% throughout—including a human mean of only 27.0% on Item 71. Humans excel on Spatial Understanding–Absolute (88.1%), Spatial Understanding–Intersection (87.7%), and Search–Counting (85.6%), and models broadly follow this ordering. Gemini leads on nearly every task but drops sharply on Pattern Recognition–Repetition (50.0%) and Quantitative Estimation–Quantitative (45.0%). On Spatial Understanding–Absolute, all three closed-source models saturate while open-source models lag considerably—for example, LLaVA-OneVision scores 30.0% on Item 10 where closed-source models reach 100%.
Error analysis
Qualitative examination of rationales identifies three recurring failure modes:
Fine-grained quantitative misestimation. Models often identify the correct measurement strategy but fail at precise estimation. In a contour-map item, Gemini correctly identified the 40-foot contour interval and located waypoint D between contours, yet misread waypoint A as lying exactly on the 7200 ft index contour when it lies just above it. In a scale-bar item asking for the radius of a lipid envelope, Gemini estimated "approximately five times" the 20 nm scale bar (radius 50 nm) instead of four times (radius ~40 nm); Claude produced the same overestimate in all 10 runs, and GPT in 3 of 10, suggesting a systematic bias in closed-source models that the open-source models did not share.
Flow-direction interpretation errors. On a texture-based wind map, Gemini and Qwen concluded that winds near Chicago and Houston flow in different directions when both flow roughly northwest to southeast. Even models answering correctly showed degraded grounding: InternVL misread the direction as west-to-east, and LLaVA avoided committing to any direction. Since most models correctly identified hurricane direction in another item, the authors localize the difficulty to decoding local motion in dense texture-based fields—possibly confusing streak orientation with actual flow direction—rather than directional reasoning generally.
Encoding-mapping errors during higher-level interpretation. Models sometimes reach plausible-sounding conclusions via unsupported visual mappings. In a brain fiber-bundle item, Gemini inferred fiber density from opacity and color solidity; Claude reached the same wrong answer in all 10 runs by importing neuroanatomical priors about the splenium of the corpus callosum rather than grounding its judgment in fiber concentration visible in the image. Both patterns violate SVLAT's closed-world principle and echo earlier findings that MLLM answers may not rest on genuine visual evidence.
Limitations
The authors acknowledge several constraints. The GPT and Claude animation results depend on frame extraction rather than native video input, so cross-model animation comparisons are confounded by differing input pipelines. The evaluation rests on a single benchmark (SVLAT), a single standardized prompt format, and lightweight open-source models (8–9B parameters), leaving open whether larger open-weight models would close the gap with closed-source systems. Additionally, some tasks are represented by only one or two items, limiting the reliability of per-task estimates and potentially under-sampling difficulty variation within tasks. The error analysis is also qualitative and based on selected exemplars rather than a systematic taxonomy applied to all failures.
Conclusion
This study provides the first systematic assessment of MLLM performance on a standardized SciVis literacy instrument, comparing six models against 485 human participants. Its principal findings are that Gemini exceeds human-level SciVis literacy overall (88.6% vs. ~75.6%), that open-source models remain below the human baseline, and that capability is highly uneven across techniques and tasks—with particular weaknesses in texture-based and integration-based visualizations, quantitative estimation, dense flow-field decoding, and encoding-grounded reasoning. The recurring failure modes documented here suggest that high aggregate accuracy can mask shallow visual grounding, reinforcing the paper's argument that SciVis-native literacy assessment constitutes a necessary complement to chart-centric evaluation of multimodal AI systems.