EpiBench: Benchmarking Multi-turn Research Workflows for Multimodal Agents

Abstract: Scientific research follows multi-turn, multi-step workflows that require proactively searching the literature, consulting figures and tables, and integrating evidence across papers to align experimental settings and support reproducible conclusions. This joint capability is not systematically assessed in existing benchmarks, which largely under-evaluate proactive search, multi-evidence integration and sustained evidence use over time. In this work, we introduce EpiBench, an episodic multi-turn multimodal benchmark that instantiates short research workflows. Given a research task, agents must navigate across papers over multiple turns, align evidence from figures and tables, and use the accumulated evidence in the memory to answer objective questions that require cross paper comparisons and multi-figure integration. EpiBench introduces a process-level evaluation framework for fine-grained testing and diagnosis of research agents. Our experiments show that even the leading model achieves an accuracy of only 29.23% on the hard split, indicating substantial room for improvement in multi-turn, multi-evidence research workflows, providing an evaluation platform for verifiable and reproducible research agents.

• The paper introduces 102 episodic benchmark tasks that require agents to identify papers from indirect cues, combine evidence across figures and tables, and reuse that evidence across multiple turns.
• GPT-5.2 achieves the strongest model performance but only 29.23% episode success on the hard split, compared with 81.36% for Ph.D. experts, while final evidence-fusion accuracy drops sharply after strong early-turn results.
• The process evaluation shows that evidence selection and multimodal memory reuse—not additional tool steps alone—are the main bottlenecks, with evidence correctness, tool minimality, and trace-based errors exposing failures hidden by answer accuracy.

EpiBench is an episodic, multi-turn, multimodal benchmark designed to evaluate research agents on short scientific research workflows rather than isolated question-answering tasks (2604.05557). The benchmark's central premise is that existing evaluations of paper-centric agents systematically under-assess three capabilities that real research demands: proactive literature search from indirect cues, integration of evidence across multiple figures and tables from multiple papers, and sustained reuse of accumulated evidence over successive turns. The authors pair the task design with a process-level evaluation protocol that scores not only final answers but also evidence grounding, tool efficiency, and memory reuse, and they show that even frontier models fall far short of expert performance on this combined workload.

Motivation and positioning

The paper identifies two structural mismatches between current benchmarks and realistic research workflows. First, most benchmarks are not workflow-faithful: tasks typically begin from an explicitly specified target paper or identifier, questions are frequently answerable from a single figure or table within one document, and multi-turn episodes in which evidence must be accumulated and reused are rare. Second, evaluation protocols are incomplete: because agents are usually allowed to re-browse documents freely, benchmarks rarely measure whether previously acquired evidence is reused correctly, and intermediate evidence is seldom checked for correct grounding. This makes attribution errors and integration failures difficult to diagnose.

Against this backdrop, EpiBench is positioned relative to two families of prior work. Paper-centric benchmarks such as CharXiv, SPIQA, SciVQA, LiveXiv, SIN-Bench, and PaperArena largely operate in single-document settings or evaluate tool-augmented navigation without requiring joint fusion of multiple figures and tables. Multi-turn multimodal benchmarks such as MMDU, MultiVerse, MMCR, VisChainBench, and Memory-QA extend reasoning horizons but are generally not paper-centric and do not require cross-paper multimodal evidence integration. EpiBench claims to be the first benchmark to jointly require proactive search, cross-paper multimodal grounding, multi-turn coherence, and evidence memory under process supervision.

Task formulation and construction

Each episode is a sequence of turns $\{(q_t, \mathcal{I}_t)\}_{t=1}^{T}$, where $q_t$ is an objective question and $\mathcal{I}_t$ optionally provides a seed cue such as an image snippet or bibliographic hint. Crucially, task descriptions never name a target paper directly; agents must resolve paper identities from citation cues or indirect hints using an external toolkit (web search, PDF extraction, retrieval over parsed text, and figure/table extraction). Later turns depend on evidence acquired earlier, and the final turn requires fusing all previously collected evidence — with tool access disabled to enforce memory-based reuse rather than re-browsing.

Construction proceeds through a pipeline of expert extraction, expansion, and curation. A seed set of 68 classic papers across six areas of computer vision and machine learning is expanded via Connected Papers into a 485-paper source corpus. GPT-5.2 drafts episodic question chains along with per-turn checklists of required tool calls specifying target papers and figures/tables. Five Ph.D.-level annotators then refine, validate, and consolidate the drafts. The appendix makes clear that human intervention is substantial: only 8% of episodes are kept as drafted and 6% receive light edits, while 27% require rewriting the final question, 33% require rewriting all questions, and 26% require rebuilding the entire episode; on the hard split, 94% of episodes undergo substantial rewriting or rebuilding. This mitigates concerns that the benchmark inherits GPT-5.2-specific phrasing or reasoning patterns.

The resulting benchmark contains 102 episodes split into Easy (39) and Hard (63), stratified by turn count, degree of cross-paper and multimodal integration, and extent of evidence reuse. Episodes span constraint-style identification, cross-paper comparison, and aggregation task types, and search initiation ranges from direct titles to citation-based cues to purely indirect hints.

Evaluation metrics

Beyond episode success rate (ESR, where all turns must be answered correctly), accuracy on the final turn (Acc$_{\mathrm{final}}$), and average accuracy on non-final turns (Acc$_{\mathrm{pre}}$), the paper introduces two process-level diagnostics:

• Evidence correctness (EC): the fraction of annotated required evidence units — indexed by paper and figure/table identifiers — actually accessed by the agent on correctly answered turns, computed from tool traces.
• Minimality gap (MG): the ratio of executed tool calls to the minimal viable checklist path, reported on successful episodes; higher values indicate redundant exploration.

Failures are additionally attributed to their earliest point of deviation in the trace, yielding five categories: Retrieve, Perception, Reasoning, Reading Memory, and Others (max-step terminations, runtime errors, parsing failures).

Agent framework

Experiments use a ReAct-like workflow built on smolagents, coupling an LMM backbone with four tools: Web Search (Google and DuckDuckGo backends), PDF Extractor (MinerU-based parsing to markdown), PDF Extractor RAG (span-level retrieval with provenance), and Content Extractor for figures and tables. Memory is an episode-level chronological store of retrieved content, which enables post hoc computation of reuse and efficiency metrics. Agents operate under a default step budget of 10 per turn, with one fallback opportunity to answer after budget exhaustion.

Main results

Eight LMM backbones were evaluated — Qwen3-VL-235B (Instruct and Thinking), GLM-4.5V, Kimi-K2.5, GPT-5-Mini, GPT-5.2, Grok-4.1, and Gemini-2.5-Pro — alongside a human baseline of two computer-science Ph.D. experts solving the same episodes under the same tool interface. Answers are scored by GPT-5.2 as an LLM judge under a fixed rubric; a reliability study with Gemini-2.5-Pro and independent human judges shows near-perfect agreement (turn-level Cohen's $\kappa$ of 0.97–1.00), suggesting results are not tied to a single judge model.

Model	Avg ESR (%)	Hard ESR (%)	Avg Acc$_{\mathrm{final}}$ (%)
GPT-5.2	39.60	29.23	59.41
Kimi-K2.5	35.35	26.15	57.58
GPT-5-Mini	24.51	21.54	54.90
Gemini-2.5-Pro	14.71	7.69	52.94
Qwen3-VL-235B-Thinking	13.73	3.23	51.96
Qwen3-VL-235B-Instruct	12.75	8.16	46.08
Grok-4.1	9.80	6.35	50.00
GLM-4.5V	2.94	0.00	38.24
Ph.D. experts	85.11	81.36	95.74

GPT-5.2 achieves the best overall performance but only 29.23% ESR on the hard split — the headline result of the paper. The most striking pattern is the persistent gap between Acc$_{\mathrm{pre}}$ and Acc$_{\mathrm{final}}$: several closed-source models reach high non-final accuracy (e.g., GPT-5.2 at 87.63% average) yet drop sharply on the final fusion turn, indicating that failures arise after initial evidence acquisition, when agents must organize, recall, and align evidence under constraints. The expert baseline of 85.11% average ESR underscores how large the remaining gap is.

Process-level diagnostics

EC and MG reveal behaviors invisible to accuracy alone. GPT-5.2 (86.10% EC) and Kimi-K2.5 (84.13%) achieve consistently high evidence correctness, comparable to the expert baseline (86.88%), whereas other models more frequently produce correct-looking answers unsupported by accessed evidence. MG varies widely and is weakly coupled with EC; notably, redundancy is often higher on Easy than Hard episodes, consistent with over-exploration on simpler tasks. Grok-4.1 exhibits the worst efficiency (average MG of 3.07).

Error attribution shows a systematic shift in failure locus across model strength. Weaker models such as Gemini-2.5-Pro and Qwen3-VL-Thinking are dominated by Retrieve errors — failing before multi-evidence fusion becomes decisive. Stronger reasoning-oriented models shift mass toward later-stage Perception and Reasoning errors, indicating that once retrieval succeeds, the bottleneck lies in reliable multimodal alignment and integration. GPT-5.2 also shows elevated Reading Memory error rates, which the authors attribute to it reaching memory-reuse turns more often due to stronger early acquisition — still exposing substantial weakness in multimodal evidence reuse from memory.

Ablations on memory and step budgets

Two ablations probe the memory bottleneck directly. Allowing tool use on the final turn produces a large recovery in ESR and Acc$_{\mathrm{final}}$, confirming that multimodal evidence cached across turns is often insufficient for reliable final fusion under the memory-only protocol. Removing the PDF text RAG tool has smaller, model-dependent effects, suggesting text retrieval is not the limiting factor; the primary challenge lies in multimodal evidence selection, indexing, and alignment within memory.

Step-budget analysis ($q_t$0) shows consistent ESR improvement from 5 to 10 steps but saturation beyond that, with max-steps errors dropping sharply from 5 to 10 and only marginally from 10 to 15. The implication is that residual failures stem from systematic issues in evidence grounding, cross-evidence alignment, and memory reuse that additional tool calls cannot resolve.

Efficiency considerations

Aggregate runtime statistics add a practical dimension: the Ph.D. expert baseline requires roughly $q_t$1 seconds to complete the benchmark, versus $q_t$2–$q_t$3 seconds for model agents. Although agents remain far less reliable, they already execute these workflows at substantially lower time cost, so even moderate reliability gains could translate into meaningful labor savings. Token costs vary widely; Qwen3-VL-235B-Instruct consumes $q_t$4 total tokens despite low success, highlighting that cost-performance balance remains unresolved.

Limitations and open questions

Several limitations deserve plain statement. The benchmark is small — 102 episodes — and confined to computer vision and machine learning literature, so generalization to other scientific domains is untested. Episode generation relies on GPT-5.2 drafting followed by heavy human revision; while revision statistics argue against model-specific leakage, the pipeline remains expensive to scale. Judge agreement was validated on a 30% sample, leaving full-corpus judge reliability unverified. The memory-only final-turn protocol enforces reuse but does not distinguish between failures of memory writing versus memory reading at finer granularity. Finally, EC is computed only on correctly answered turns, so evidence behavior on failed turns is not scored. Open questions include whether explicit memory-indexing mechanisms for multimodal evidence can close the Acc$q_t$5-to-Acc$q_t$6 gap, and whether the observed saturation in step budgets holds for longer-horizon workflows.

Conclusion

EpiBench contributes a workflow-faithful, process-supervised evaluation setting for research agents, combining proactive citation-guided search, cross-paper multimodal evidence integration, and enforced memory reuse with fine-grained diagnostics of evidence correctness, tool minimality, and trace-based error attribution. Its empirical findings are sobering: the strongest available model completes fewer than a third of hard episodes, while experts exceed 80%, and ablations localize the dominant bottleneck to multimodal evidence selection and reuse from memory rather than retrieval or step capacity. The benchmark offers a concrete platform for measuring progress toward verifiable and reproducible research agents.