QCalEval: Benchmarking Vision-Language Models for Quantum Calibration Plot Understanding

Abstract: Quantum computing calibration depends on interpreting experimental data, and calibration plots provide the most universal human-readable representation for this task, yet no systematic evaluation exists of how well vision-LLMs (VLMs) interpret them. We introduce QCalEval, the first VLM benchmark for quantum calibration plots: 243 samples across 87 scenario types from 22 experiment families, spanning superconducting qubits and neutral atoms, evaluated on six question types in both zero-shot and in-context learning settings. The best general-purpose zero-shot model reaches a mean score of 72.3, and many open-weight models degrade under multi-image in-context learning, whereas frontier closed models improve substantially. A supervised fine-tuning ablation at the 9-billion-parameter scale shows that SFT improves zero-shot performance but cannot close the multimodal in-context learning gap. As a reference case study, we release NVIDIA Ising Calibration 1, an open-weight model based on Qwen3.5-35B-A3B that reaches 74.7 zero-shot average score.

• The paper introduces QCalEval, a benchmark that rigorously evaluates vision-language models on the interpretation of quantum calibration plots.
• It details a multidimensional evaluation protocol with six task axes and tests both zero-shot and in-context learning performance across diverse calibration scenarios.
• Results reveal that while VLMs excel at visual feature detection, they struggle with mapping features to operational states, underscoring the need for domain-tuned models.

QCalEval: A Formal Evaluation Suite for Vision-LLMs on Quantum Calibration Plots

Introduction

Quantum hardware calibration relies fundamentally on interpreting a diverse array of calibration plots, which encode the operational state and failure modes of quantum devices. While vision-LLMs (VLMs) have demonstrated strong capabilities in natural image understanding and general visual reasoning, their utility as autonomous scientific observers in the quantum domain has remained unexplored due to the absence of robust, domain-centric evaluation resources. The "QCalEval: Benchmarking Vision-LLMs for Quantum Calibration Plot Understanding" (2604.25884) addresses this gap by introducing QCalEval, a comprehensive multimodal benchmark specifically tailored to assess VLM proficiency on quantum hardware calibration tasks.

This essay provides a detailed technical synthesis of the methodology, experimental design, and key findings of the QCalEval study. Discussion focuses on its impact for quantum automation, vision-LLM development, and future directions in autonomous scientific experimentation.

Benchmark Structure and Methodology

QCalEval establishes a fine-grained evaluation protocol for VLMs, emulating genuine quantum laboratory calibration workflows. The benchmark consists of 243 samples covering 87 scenario types from 22 experiment families, encompassing both superconducting qubits and neutral atom platforms. Each sample pairs one or more calibration plot images with ground-truth annotations across six task axes, explicitly designed to probe both visual and domain-specific reasoning.

Representative calibration plot examples in QCalEval span 1D traces, 2D maps, histograms, and spatially-structured images. The evaluation is uniquely challenging: models cannot rely on object identity or text recognition, but must interpret scientific geometry—peak locations, fringe spacing, linewidths, clustering features, and fit quality—to assess experimental reliability.

Figure 1: The calibration plots in QCalEval are visually heterogeneous; scientific geometry and feature localization are critical for assessing reliability—not object identity.

The benchmark's task taxonomy is anchored in the operational loop of quantum experimentalists. Specifically, six question types (Q1–Q6) isolate complementary visual and cognitive skills: structured technical description (JSON schema), coarse outcome classification, free-text scientific significance reasoning, fit validity, machine-readable parameter extraction, and family-specific calibration diagnosis. Task prompts are systematically diversified by experiment family and scenario type to enforce true generalization. Each sample is deployed in both zero-shot (single plot, no examples) and multimodal in-context learning (MM-ICL, with demonstration examples) settings.

The format of in-context learning closely replicates practical scenario-based reasoning, where reference plots with expert annotations are supplied as demonstrations for each scenario type. For performance quantification, Q2, Q4, Q5, and Q6 are programmatically scored; Q1 and Q3 (requiring nuanced, open-ended judgments) are evaluated using dual large model judges to mitigate scoring bias.

Figure 2: Task illustration from qubit spectroscopy—models are evaluated in both zero-shot and in-context learning regimes; each evaluation is a stateless conversation.

QCalEval's multidimensional question schema is illustrated (Figure 3) using a DRAG calibration case, underscoring how each task axis isolates a different reasoning failure and how family-specific context is injected.

Figure 3: Six core question types in QCalEval; only three axes—scientific analysis (Q3), parameter extraction (Q5), and status diagnosis (Q6)—admit transfer via in-context demonstrations.

Model Evaluation and Key Results

Eighteen representative VLMs are assessed, spanning leading closed-source APIs (e.g., Gemini-3.1-Pro, Claude Opus 4.6, GPT-5.4), competitive open-weight models (Gemma-4-31B-IT, Qwen3.5 variants, InternVL3), and one domain-tuned case study (NVIDIA Ising Calibration 1).

Zero-Shot Performance

Across all models, visual feature detection (Q1) attains high scores (65-91%), but performance drops sharply on outcome classification and diagnosis (Q2: 32-67%, Q6: 37-75%). The limiting factor is not the perception of geometry, but the mapping of feature constellations to domain-specific operational states. Systematic optimistic bias is observed: in 60.7% of "suboptimal" cases, models default to "expected behavior," highlighting an inability to resolve subtle, expert-level failure modes.

Fit reliability (Q4) serves as an independent perception axis; models diverge in their ability to resolve fit/data consistency and to recognize "no fit" conditions. Principal failure modes are persistent across architectures and scales.

The best general-purpose zero-shot model (Gemini-3.1-Pro) reaches a mean score of 72.3, with domain-tuned Ising-Cal-1 outperforming on classification and diagnosis (mean 74.7).

In-Context Learning and MM-ICL Gap

Closed-source models and Gemma-4-31B-IT benefit strongly from multi-example in-context learning on transferable axes (Q3, Q5, Q6): diagnosis gains are dramatic (e.g., Claude Opus 4.6: Q6 from 60.5 to 89.4, Gemma-4-31B-IT: 62.1 to 86.0). This demonstrates that the MM-ICL mechanism successfully injects operational knowledge and label vocabulary via demonstrations for these families.

In contrast, Qwen3.5, Kimi-VL, and MiniCPM-o families experience consistent degradation in MM-ICL settings—performance drops on multi-image prompts, and only recovers for single-shot extraction tasks. Extensive $N$-way studies exclude simple image-overload as the root cause, indicating broader architectural or training limitations in utilization of visual demonstrations. This MM-ICL gap constitutes a primary limitation for practical, extensible scientific reasoning in open-source VLMs.

SFT Ablation and Domain-Tuned Models

Ablation studies at 9B scale using Qwen3.5 variants demonstrate that SFT using synthetic, curriculum-structured data robustly improves zero-shot classification and diagnosis (Q6 61.1→70.6). Two-phase and blended recipes target further increases and demonstrate that training order matters for transfer. Critically, MM-ICL gains cannot be closed by SFT alone; free-text scientific reasoning (Q3) remains below baseline even with carefully structured SFT, pointing to limits in fine-tuning and the need for fundamentally more powerful reasoning or feedback mechanisms.

Based on these findings, NVIDIA Ising Calibration 1—a Qwen3.5-35B-A3B domain-tuned via sequential SFT (ICL→zero-shot)—is released as an open-weight reference. It achieves 74.7 average zero-shot on QCalEval and consistently outperforms generic VLMs on structured classification and fit judgment, but fails to overcome MM-ICL deficits.

Analysis of Failure Modes

Detailed error analysis identifies the key limitations of current VLMs on quantum calibration:

• Visual-to-operational mapping deficiency: VLMs can describe scientific geometry accurately yet misclassify operational states, particularly in patterns such as beating, failure to excite, or fit/model mismatch.
• Optimistic bias: Models overwhelmingly predict success in ambiguous or noisy regimes. This reflects both a lack of conservative (robust-to-failure) reasoning and an over-reliance on presence of regularity or fits.
• Demonstration utilization bottleneck in open models: Open-weights largely fail to utilize multi-shot visual context, peaking at one demonstration before degrading.
• Limited abstraction in free-text reasoning: No examined approach can provoke robust, generalizable free-text scientific inference using SFT or existing demonstrations.

Implications and Future Directions

Practically, QCalEval reveals that autonomous quantum agent architectures relying on generic VLMs for calibration plot interpretation will routinely propagate undetected critical failures, unless paired with extensive domain tuning and/or curated prompt engineering. Agentic calibration loop closure—enabling true closed-loop quantum device operation—remains bottlenecked by robust scientific visualization comprehension.

Theoretically, the MM-ICL gap revealed for open-weight VLMs, and the inability of SFT to bridge reasoning in high-fidelity scientific regimes, sets a program of research for the VLM field: transfer learning protocols, demonstration encoding strategies, and training data composition for scientific images remain open and urgent questions. There is a clear need for specialized architectures and learning signals tailored for geometric, not object-based, scientific vision tasks.

QCalEval and NVIDIA Ising Calibration 1 are released as open benchmarks and models, establishing a baseline for future model pretraining, instruction tuning, and RLHF refinement for quantum hardware operations. Their extensibility to other scientific domains (beyond quantum calibration) is plausible given similar visual and operational reasoning demands across scientific instrumentation.

Conclusion

QCalEval represents a methodologically rigorous standard for benchmarking VLMs on quantum calibration plot understanding (2604.25884). The resource exposes the dual challenge of robust scientific geometry perception and correct operational outcome mapping in high-stakes laboratory automation. Baseline results indicate that while leading VLMs demonstrate basic visual competence, their ability to reason about, classify, and recommend actions based on quantum calibration plots is fundamentally imperfect outside the closed/model-tuned regime—and MM-ICL remains a critical bottleneck for open-source models. Systematic SFT improves zero-shot performance but does not confer transferable, expert-level scientific reasoning.

The QCalEval suite will accelerate targeted VLM development for quantum and scientific automation, catalyzing work on task-driven architectures, domain-driven instruction tuning, and robust utilization of demonstrations in complex visual reasoning pipelines.