SIN-Bench: Tracing Native Evidence Chains in Long-Context Multimodal Scientific Interleaved Literature

Abstract: Evaluating whether multimodal LLMs truly understand long-form scientific papers remains challenging: answer-only metrics and synthetic "Needle-In-A-Haystack" tests often reward answer matching without requiring a causal, evidence-linked reasoning trace in the document. We propose the "Fish-in-the-Ocean" (FITO) paradigm, which requires models to construct explicit cross-modal evidence chains within native scientific documents. To operationalize FITO, we build SIN-Data, a scientific interleaved corpus that preserves the native interleaving of text and figures. On top of it, we construct SIN-Bench with four progressive tasks covering evidence discovery (SIN-Find), hypothesis verification (SIN-Verify), grounded QA (SIN-QA), and evidence-anchored synthesis (SIN-Summary). We further introduce "No Evidence, No Score", scoring predictions when grounded to verifiable anchors and diagnosing evidence quality via matching, relevance, and logic. Experiments on eight MLLMs show that grounding is the primary bottleneck: Gemini-3-pro achieves the best average overall score (0.573), while GPT-5 attains the highest SIN-QA answer accuracy (0.767) but underperforms on evidence-aligned overall scores, exposing a gap between correctness and traceable support.

• The paper introduces SIN-Bench, a novel benchmark that emphasizes constructing natural evidence chains for multimodal long-context scientific reasoning.
• It employs the 'Fish-in-the-Ocean' (FITO) paradigm to link interleaved text and figures through tasks like SIN-Find, SIN-QA, and SIN-Summary.
• Empirical evaluations show models such as Gemini-3-pro excel in evidence grounding, highlighting the need for verifiable, logical reasoning in AI.

Evidence-Chain-Centric Evaluation of Multimodal LLMs: An Expert Perspectives on SIN-Bench

Introduction and Paradigm Shift in Long-Context Evaluation

SIN-Bench presents a rigorous benchmark designed to assess the evidence-grounded reasoning capabilities of Multimodal LLMs (MLLMs) when tasked with expert-level comprehension of long-form, interleaved scientific literature. The benchmark introduces the "Fish-in-the-Ocean" (FITO) evaluation paradigm, contrasting sharply with prevailing "Needle-In-A-Haystack" (NIAH) approaches. Whereas NIAH centers on the retrieval of artificially inserted facts amidst irrelevant noise, FITO demands the aggregation of natural, interconnected knowledge units embedded across multimodal contexts in authentic scientific works. This shift from surface-level retrieval to evidence-chain construction more closely aligns evaluation with genuine scientific cognition demands.

Figure 1: Comparison of NIAH (synthetic retrieval) with FITO's native document reasoning and evidence chain tracing.

SIN-Bench: Corpus, Task Design, Metrics, and Construction Pipeline

SIN-Data Infrastructure

The authors devised a comprehensive preprocessing pipeline unifying both arXiv and PubMed Central literature into a Scientific INterleaved (SIN) format, preserving the logical coupling between text and figures independent of spatial layout. This corpus spans 12 primary domains and 80+ subfields, with strict quality filtering ensuring high multimodal density and reliable evidence linking.

Hierarchical Task Suite

SIN-Bench operationalizes four evidence-centric tasks abstracting scientific inquiry workflows:

• SIN-Find: Evidence pathfinding for complex queries, requiring cross-modal and cross-sectional retrieval and ordered chain construction.
• SIN-Verify: Hypothesis verification via binary classification on evidence sufficiency, designed to discriminate between valid and near-miss evidence.
• SIN-QA: Joint answer and provenance generation, enforcing answer derivation strictly from evidence traceable to document anchors.
• SIN-Summary: Evidence-anchored synthesis of document claims, integrating multi-claim generation with explicit citation of supporting evidence chains.

Ground-truth instances comprise documents, queries, answers, and ordered evidence anchors; evaluation uses unified formats for cross-task comparability.

Figure 2: SIN-Bench pipeline, from raw source parsing to multi-model synthesis, evidence-chain annotation, cross-validation, and human audit.

Metrics: "No Evidence, No Score"

To disincentivize parametric hallucinations and enforce answer traceability, SIN-Bench uses the "No Evidence, No Score" protocol. For evidence-requiring tasks, predictions are credited only when grounded to verifiable anchors. Metrics span three dimensions:

• Matching: Anchor-wise semantic consistency, adjudicated by an LLM judge (Qwen3-8B), normalized to [0,1].
• Relevance: F1-based precision and recall on anchor correctness, thresholded for semantic fidelity.
• Logic: Kendall–Tau order similarity, assessing correct logical sequencing of evidence units.

For SIN-QA, answer accuracy (semantic scoring) is reported independently but does not elevate task score in absence of valid evidence. SIN-Verify employs standard accuracy.

Empirical Evaluation of MLLMs

Model Benchmarks

SIN-Bench includes five proprietary models (Gemini-3-pro, Gemini-2.5-pro, GPT-5, Claude-sonnet-4.5, Grok-4) and three open-weight Qwen3-VL variants. Task-level and overall scores are reported, facilitating granular diagnosis of reasoning abilities.

Figure 3: Task-level heatmap; darker shading indicates higher model scores across SIN-Bench tasks.

Strong Numerical Results and Notable Findings

• Gemini-3-pro attains the highest aggregate score (0.566), evidencing robust multimodal evidence grounding.
• GPT-5 yields the best SIN-QA answer accuracy (0.767) but lower scores on evidence-linked tasks, confirming a reliance on parametric knowledge rather than explicit document rationalization.
• Qwen3-VL-8B outperforms the larger MoE variant, indicating the preeminence of reasoning-oriented fine-tuning over raw parameter count for evidence-centric scientific tasks.

Task-Level Breakdown and Contradictory Behaviors

• SIN-Find: Gemini-2.5-pro, while inferior in anchor precision, excels in logic preservation, illustrating a trade-off between correct anchor retrieval and sequential reasoning.
• SIN-Verify: Models saturate on "easy negatives" (irrelevant evidence), but performance collapses on "hard negatives" (ambiguous/near-miss chains), e.g., GPT-5 drops from 1.000 to 0.208 accuracy—highlighting weak logical discrimination.
• SIN-QA: Gemini-3-pro achieves strongest overall evidence-grounded score (0.567), while GPT-5's gap between answer accuracy and evidence quality persists.
• SIN-Summary: GPT-5 shows superior logical synthesis, likely reflecting advanced learning of document flow.

Analysis: Evidence Grounding, Multimodal Structure, and Error Modes

Native Interleaving Advantage

The preservation of interleaving between text and figures led to substantial performance gains in Gemini-3-pro over separated modalities, elucidating the importance of format fidelity for reasoning tasks.

Figure 4: SIN-QA and SIN-Summary scores as functions of input modality encoding for Gemini-3-pro.

Evidence Chain Enforced Decoding

Mandating explicit evidence-chain output improves answer performance, further establishing evidence traceability as a de facto multimodal chain-of-thought mechanism.

Input Length Robustness

Gemini-3-pro and GPT-5 exhibit stable performance across prompts exceeding 19k tokens, whereas Qwen3-VL-2B demonstrates volatility and frequent collapse, attributed less to context length than to deficiencies in evidence retrieval and alignment.

Figure 5: Score density distributions for SIN-QA and SIN-Summary by token length.

Error Analysis

Qualitative error review surfaces two modes: (1) Information Deficiency—omission of crucial reasoning steps; (2) Spurious Reasoning—irrelevant citations ("shotgun evidence") that degrade precision. Evidence-based QA prompts consistently mitigate these effects relative to answer-only setups.

Figure 6: Examples of reasoning failures in SIN-Find and SIN-QA for Gemini-3-pro.

Cross-Domain and Verification Stress Testing

Performance varies substantially across disciplines, with highest domain scores in Economics/Finance and Medicine/Health, and lowest in Mathematics, marking symbolic manipulation as an enduring challenge. Adversarial verification on near-miss evidence reveals weak rigor in evidence sufficiency assessment for all major models.

Figure 7: Domain-wise radar plot of overall scores, exposing strong domain-dependent variability.

Figure 8: SIN-Verify accuracy for easy versus hard negatives—performance drops precipitously for fine-grained logical discrimination.

Qualitative Task Examples

SIN-Bench provides curated "golden" instances illustrating document, query, answer, and evidence-chain annotation, supporting transparency in ground-truth construction.

Figure 9: Golden samples for SIN-Find and SIN-QA—explicit demarcation of document, evidence, and answer.

Figure 10: Golden samples for SIN-Verify and SIN-Summary—evidence chain auditing in binary verification and synthesis tasks.

Implications and Future Directions

The evidence-chain paradigm exposes critical limitations of current MLLMs—namely, the propensity for parametric hallucination absent document-grounded reasoning and systemic vulnerabilities in evidence verification and structured output generation (especially for open-weight models). SIN-Bench's comprehensive, scalable pipeline enables continual extension; future directions include integrating discipline-specific expert models, enhancing support for ultra-long contexts, and deploying evidence-chain tracing for fraud detection or provenance auditing in scientific publishing. The established metrics and formats further advance the development and evaluation of next-generation MLLMs toward verifiable, interpretable scientific understanding.

Conclusion

SIN-Bench advances multimodal LLM evaluation from answer-centric to evidence-centric reasoning, leveraging explicit chain-of-evidence construction over native, interleaved scientific literature. Its robust metrics and task suite provide fine-grained diagnosis of grounding, support, and logical integration, revealing both the strengths and deficiencies of contemporary frontier models. The benchmark sets a rigorous precedent for future assessment and development of AI systems intended for expert-level scientific cognition and reasoning traceability.