SciVQA 2025 Shared Task Overview

• SciVQA 2025 Shared Task is a scientific visual QA challenge that tests models on interpreting figures like charts and graphs from scholarly articles.
• The dataset features 3,000 real-world figures with seven diverse, annotated QA pairs per figure, including binary, multiple-choice, and open-ended questions.
• Evaluations rely on strict answer formatting and metrics such as ROUGE and BERTScore, highlighting the importance of chain-of-thought prompting and confidence-informed ensembling.

Searching arXiv for the shared task and system papers to ground the article in current literature. SciVQA 2025 Shared Task is a scientific visual question answering task centered on figures from scholarly articles, including charts, diagrams, graphs, and related semi-structured visual artifacts drawn from ACL Anthology and arXiv. In the participant reports currently available, the task is characterized as requiring answers to natural-language questions grounded in scientific figures, with access to figure metadata such as caption, figure type, and compound-figure information under an “oracle-style” setup. The task combines visual and non-visual reasoning, finite and infinite answer sets, and explicit unanswerable cases, thereby stressing precise numerical interpretation, multi-step reasoning, and strict answer formatting rather than generic image captioning or open-ended multimodal dialogue (Movva et al., 8 Jul 2025, Jaumann et al., 3 Jul 2025).

1. Task definition and scope

SciVQA 2025 evaluates visual question answering over scientific figures from scholarly articles. Each figure is paired with seven question–answer pairs. The reported question taxonomy spans visual versus non-visual questions, finite versus infinite answer sets, and multiple reasoning difficulties, including numeric extraction, descriptive interpretation, and multi-step reasoning. Visual questions target visual attributes such as color, shape, size, height, and axes or scale readings, whereas non-visual questions require reasoning beyond visual attributes, for example by synthesizing figure content with caption or context (Movva et al., 8 Jul 2025).

A complementary participant description specifies the seven-question structure more explicitly: two binary questions, one visual and one non-visual; two multiple-choice questions with four options, one visual and one non-visual; two open-ended questions with an infinite answer set, one visual and one non-visual; and one unanswerable question (Jaumann et al., 3 Jul 2025). Taken together, these descriptions indicate that the shared task was designed to test both figure-grounded observation and controlled answer production under heterogeneous supervision regimes.

The scope of the task is broader than chart QA in the narrow sense. Reported figure types include line charts, bar charts, scatter plots, confusion matrices, architecture diagrams, trees, graphs, pie charts, neural networks, box plots, histograms, tables, vector plots, and illustrative diagrams (Movva et al., 8 Jul 2025, Jaumann et al., 3 Jul 2025). This suggests that SciVQA 2025 occupies an intermediate position between document understanding, multimodal reasoning, and structured visual interpretation.

2. Dataset composition and annotation structure

The dataset comprises 3,000 real-world figures from scientific publications, with seven annotated questions per figure. One report gives the sample-level split sizes as Train: 15,120 samples, Validation: 1,680 samples, and Test: 4,200 samples, which is consistent with the seven-questions-per-figure design (Movva et al., 8 Jul 2025). The dataset is unevenly distributed across figure types, and line charts dominate. In the train split, line chart accounts for 10,007 samples, approximately 66%, while additional reported counts include tree 924, scatter plot 735, graph 553, bar chart 525, architecture diagram 504, pie chart 497, neural networks 462, confusion matrix 427, box plot 133, histogram 77, and other 77 (Movva et al., 8 Jul 2025). A second report similarly states that line charts dominate the development set at 65% (Jaumann et al., 3 Jul 2025).

Split	Samples	Reported source
Train	15,120	(Movva et al., 8 Jul 2025)
Validation	1,680	(Movva et al., 8 Jul 2025)
Test	4,200	(Movva et al., 8 Jul 2025)

The annotation scheme is also skewed by question type. In the train split, reported counts are infinite-set visual 1,079; infinite-set non-visual 2,172; finite-set binary visual 1,124; finite-set binary non-visual 3,219; finite-set non-binary visual 1,751; finite-set non-binary non-visual 3,615; and unanswerable 2,160 (Movva et al., 8 Jul 2025). One participant paper explicitly notes that the dataset is skewed toward line charts and non-visual questions (Movva et al., 8 Jul 2025).

Inputs reported for the task include the scientific figure itself and figure metadata such as caption, figure ID, figure type, and QA pair type; another report adds a flag indicating whether the figure is compound, meaning that it contains subfigures (Movva et al., 8 Jul 2025, Jaumann et al., 3 Jul 2025). The answer space is tightly constrained: binary questions require explicit “Yes” or “No,” multiple-choice questions require answer letters such as A, B, C, and open-ended questions require concise answers such as a raw numerical value or a single word or phrase (Movva et al., 8 Jul 2025). This answer normalization is central to the task because formatting mismatches can affect automatic metrics even when semantic content is correct.

3. Evaluation protocol and official metrics

Systems were evaluated with ROUGE-1, ROUGE-L, and BERTScore, each reported with Precision, Recall, and F1. One participant report states that leaderboard rank is by the average of the three F1 scores (Jaumann et al., 3 Jul 2025). The metrics are defined explicitly in the submitted systems.

For ROUGE-1, if $P$ denotes unigram precision and $R$ denotes unigram recall between system output and reference, then

$F1 = \frac{2PR}{P + R}.$

For ROUGE-L, based on the longest common subsequence, one report gives

$$P_{LCS} = \frac{LCS(sys, ref)}{|sys|}, \quad R_{LCS} = \frac{LCS(sys, ref)}{|ref|},$$

and with $\beta = 1$,

$$F1_{LCS} = \frac{(1 + \beta^2) P_{LCS} R_{LCS}}{R_{LCS} + \beta^2 P_{LCS}}.$$

A second report presents the same idea as

$$P = \frac{LCS(X, Y)}{|X|}, \quad R = \frac{LCS(X, Y)}{|Y|}, \quad F1 = \frac{2PR}{P + R}.$$

For BERTScore, both reports describe contextual embedding alignment with precision and recall computed via maximum cosine-similarity matches, and F1 as the harmonic mean (Movva et al., 8 Jul 2025, Jaumann et al., 3 Jul 2025).

The evaluation setup places unusual pressure on output formatting. Participant reports emphasize that prompts were engineered to force “only the raw numerical value or single word/phrase,” to standardize unanswerable outputs as “It is not possible to answer this question based only on the provided data,” and to restrict multiple-choice answers to letters only (Movva et al., 8 Jul 2025, Jaumann et al., 3 Jul 2025). A plausible implication is that SciVQA 2025 simultaneously measures visual reasoning and adherence to exact response schemas.

4. Reported methodological patterns in submissions

The available system papers reveal two distinct but related methodological trends: prompt-centered multimodal reasoning and ensemble-based routing.

One submission evaluates VLMs in the 5B to 8B range—InternVL3-8B, Qwen2.5-VL-7B Instruct, Bespoke MiniChart 7B, and Phi-4 Multimodal Instruct—and reports that its main results come from zero-shot or instruction-following inference plus prompt optimization and chain-of-thought prompting. The system uses figure-type guidance, compound-image navigation prompts, binary and choice-specific templates, and a two-stage chain-of-thought with “Rethink,” where Step 1 extracts and verifies visual evidence inside <reasoning> and Step 2 emits the final answer inside <answer> (Movva et al., 8 Jul 2025). The same paper states that the approach does not perform explicit figure cropping or segmentation, dedicated OCR, or chart derendering, but instead relies on the VLMs’ built-in vision encoders and instruction prompts (Movva et al., 8 Jul 2025).

A second submission uses an ensemble of two open-weights MLLMs, InternVL3-78B and Pixtral-Large-Instruct-2411, together with few-shot example retrieval and confidence-informed ensembling. It evaluates question-only similarity with SBERT embeddings, question+image similarity with CLIP, and BLIP‑2 image-question embeddings, using cosine similarity

$$s(q, e_i) = \frac{v_q \cdot v_{e_i}}{\|v_q\| \|v_{e_i}\|}.$$

The final system first accepts predictions from InternVL3-78B with 1-shot, question+image retrieval based on BLIP‑2 whenever confidence exceeds a threshold, then routes remaining instances by question type to a small set of fixed configurations (Jaumann et al., 3 Jul 2025).

The reported confidence function is

$$conf_m(y) = \exp\left(\frac{1}{T}\sum_{t=1}^{T}\log p_m(y_t \mid x, y_{<t})\right),$$

that is, the per-token geometric mean probability under model $m$ (Jaumann et al., 3 Jul 2025). No additional calibration such as temperature scaling is applied.

Across both submissions, several common design choices recur: no OCR or external tools, no explicit region serialization or cropping, reliance on provided metadata, deterministic decoding with temperature $R$0 in the few-shot system, and routing or prompt specialization based on figure type, question type, or both (Movva et al., 8 Jul 2025, Jaumann et al., 3 Jul 2025). This suggests that the shared task rewarded disciplined prompt engineering and calibrated answer selection at least as much as explicit document parsing.

5. Performance and leaderboard results

The participant reports provide partial but informative visibility into the leaderboard. Coling-UniA ranked 3rd of 7 on the blind test data with average F1 = 85.12 across ROUGE-1, ROUGE-L, and BERTS, specifically R1-F1 78.62, RL-F1 78.56, and BERTS-F1 98.17 (Jaumann et al., 3 Jul 2025). The same report lists the top three by average F1 as ExpertNeurons 86.47, THAii_LAB 85.43, and Coling-UniA 85.12 (Jaumann et al., 3 Jul 2025).

The Infyn submission reports that its submitted ensemble ranked 5th with ROUGE-1 F1 0.735, ROUGE-L F1 0.734, and BERTScore F1 0.979 (Movva et al., 8 Jul 2025). Within that submission, the strongest standalone model was InternVL3-8B, which achieved ROUGE-1 and ROUGE-L F1 scores of 0.740 and a BERTScore of 0.983 on the SciVQA test split (Movva et al., 8 Jul 2025). Its router ensemble improved over most individual models but not over InternVL3 alone (Movva et al., 8 Jul 2025).

Ablation results in the same report emphasize the importance of answer-format constraints. Baseline 1, described as a generic prompt, achieved ROUGE-1 F1 0.180, whereas Baseline 2, described as explicit format, achieved ROUGE-1 F1 0.700, for a reported gain of +0.520 (Movva et al., 8 Jul 2025). The paper also reports test-time ROUGE-1 gains from chain-of-thought prompting for every evaluated model: InternVL3 +0.013, Qwen2.5-VL +0.062, Bespoke +0.057, Phi-4 +0.018, and Ensemble +0.030 (Movva et al., 8 Jul 2025).

The Coling-UniA report similarly finds that few-shot prompting improves over zero-shot on average, and that confidence-informed ensembling outperforms both single best runs and question/figure-type routing alone on test (Jaumann et al., 3 Jul 2025). For example, its best single configuration on test is InternVL3-78B with R1-F1 77.2, RL-F1 77.2, and BERTS-F1 98.1; the Question/Figure-Type Ensemble reaches R1-F1 77.7, RL-F1 77.6, and BERTS-F1 98.1; and the Confidence-Informed Ensemble reaches R1-F1 78.6, RL-F1 78.6, and BERTS-F1 98.2 (Jaumann et al., 3 Jul 2025).

6. Failure modes, limitations, and emerging directions

The reported error analyses converge on several persistent difficulties. One submission identifies visual misinterpretations arising from comparing subplots with different scales, misunderstanding axis origins, overlapping text, and low-resolution artifacts; it also reports numerical misalignments such as small errors in reading or rounding scales or ticks, as well as flawed reasoning in which observations are correct but the logical progression or output format is wrong (Movva et al., 8 Jul 2025). The other submission reports that infinite answer-set questions about visual features remain the most difficult category, and notes that no explicit figure parsing such as axis reading or multi-panel segmentation is used (Jaumann et al., 3 Jul 2025).

Both reports stress that unanswerable handling is nontrivial. One normalizes all unanswerable outputs to a single canonical string (Movva et al., 8 Jul 2025). The other reports strong performance on the category but cautions that dataset unanswerables may follow specific patterns and may not transfer to real-world unanswerables with different distributions (Jaumann et al., 3 Jul 2025). This directly addresses a common misconception that high benchmark performance on unanswerable detection necessarily reflects broad robustness.

The limitations reported by participants are also revealing. Compute constraints limited one system to approximately 5B to 8B models, with no full-scale fine-tuning for InternVL3, Qwen, or Phi-4 and only limited LoRA SFT on Bespoke MiniChart 7B with 5,000 training samples (Movva et al., 8 Jul 2025). The other system used Nvidia A100 (80 GB) GPUs, up to 4 GPUs in parallel, and approximately 3600 GPU hours total, but still performed no supervised fine-tuning or instruction tuning (Jaumann et al., 3 Jul 2025). Both papers explicitly state that OCR and dedicated chart parsing pipelines were not used (Movva et al., 8 Jul 2025, Jaumann et al., 3 Jul 2025).

The future directions named in the reports are consistent with these bottlenecks: advanced prompting such as Tree-of-Thought, Mixture-of-Experts routing, expert–critic ensembles for re-ranking, larger models, more extensive fine-tuning including GRPO-style methods, better confidence calibration, stronger figure parsing, and improved data standardization (Movva et al., 8 Jul 2025, Jaumann et al., 3 Jul 2025). This suggests that SciVQA 2025 exposed a boundary condition for current multimodal LLMs: general-purpose vision encoders and prompting can be competitive, but scientific figure QA remains constrained by numeric precision, panel-aware grounding, and output-level exactness.