ResearchPulse-Bench: Citation-Aware Inference

Updated 10 July 2026

ResearchPulse-Bench is a citation-aware benchmark that supports multi-document scientific inference by reconstructing research development chains from aligned paper clusters.
It standardizes extraction of key motivations, methodologies, and experimental results using structured annotations and chronological ordering.
The benchmark employs rigorous evaluation metrics and quality assurance processes to validate both method tracking and experimental analysis tasks.

ResearchPulse-Bench is a citation-aware, cluster-level benchmark introduced to support the task of multi-document scientific inference, namely the reconstruction of “research development chains” across related papers by extracting and aligning motivation, methodology, and experimental results over time (Chen et al., 3 Sep 2025). It underpins the ResearchPulse framework and supervises two complementary subtasks—Method-Tracking and Experimental-Analysis—with annotations designed to build and evaluate explicit method–experiment chains rather than single-document summaries or high-level thematic surveys.

1. Definition and task scope

ResearchPulse-Bench is built around the formal setting of a related document set

$\mathcal{C} = \{D_1, D_2, \dots, D_N\},$

where each $D_i$ belongs to a research thread or citation lineage. Within this setting, the benchmark operationalizes multi-document scientific inference as structured cross-document reasoning over scientific elements that must be aligned temporally, structurally, and empirically (Chen et al., 3 Sep 2025).

The first subtask, Method-Tracking, requires extraction of each paper’s core motivation $m_i$ and methodological description $r_i$ , primarily from the abstract and introduction, followed by chronological alignment by publication timestamp $t_i$ . The expected output is a structured representation $\mathcal{M}_{\text{chain}}$ in hierarchical markdown, later rendered as a mind-map–style visualization. The second subtask, Experimental-Analysis, requires extraction of the main experimental table $T_i$ , model names $\mathcal{M}_i$ , evaluation metrics $\mathcal{E}_i$ , and cited baseline years $\mathcal{Y}_i$ , followed by organization of metric values over time into a structured summary $D_i$ 0 and rendering as a line chart through auto-generated Python code.

The benchmark is explicitly designed to supervise extraction of motivations and methods per paper, normalization of experimental tables per paper, chronological and citation-aware alignment of these elements, and generation of mind maps and line charts that describe research progression. The paper positions this task as distinct from both scientific summarization and automatic survey generation: prior work such as BERTSum, LoBART, HEGEL, HAESum, BigSurvey, CHIME, AutoSurvey, and SurveyX is described as focusing on high-level thematic abstraction, while lacking fine-grained modeling of the scientific logic “motivation $D_i$ 1 method $D_i$ 2 experiment” and lacking supervision for temporal alignment of method descriptions or standardization of heterogeneous experimental tables (Chen et al., 3 Sep 2025).

2. Corpus construction and scale

ResearchPulse-Bench is organized as 100 citation-aware document clusters, each curated for either the Method-Tracking or Experimental-Analysis task (Chen et al., 3 Sep 2025). Citation awareness is established through citation graphs built from forward and backward references among papers retrieved from arXiv and OpenReview. Cluster formation combines citation connectivity, semantic similarity, and temporal coherence: abstracts are encoded with Sentence-BERT, grouped with K-Means, labeled using high-frequency terms, and reviewed manually.

The source corpus is restricted to AI-related papers from the last five years, selected through AI-related keywords including “Machine Learning”, “Computer Vision”, and “Natural Language Processing”. The benchmark split is at the cluster level, with 80 clusters for training and 20 for testing. Table 3 reports cluster statistics as follows: Minimum Papers: 6 for train, 8 for test, and 14 in total; Maximum Papers: 33 for train, 31 for test, and 38 in total; Average Papers: 19.19 for train, 19.9 for test, and 24.93 in total (Chen et al., 3 Sep 2025).

Aspect	Value
Document clusters	100
Train/test clusters	80 / 20
Method-Tracking samples	1,958 train / 491 test
Experimental-Analysis samples	1,550 train / 320 test

At the sample level, the two tasks differ sharply in context length. For Method-Tracking, the benchmark contains 1,958 train samples and 491 test samples; query length ranges from 55 to 22,326 tokens in train and from 78 to 24,031 in test, with averages of 1,161.16 and 1,210.78 tokens respectively. Answer length averages 344.54 tokens in train and 355.6 in test. For Experimental-Analysis, the benchmark contains 1,550 train samples and 320 test samples; query length averages 14,402.73 tokens in train and 16,545.45 in test, with maxima of 46,877 and 32,971 tokens. Answer length averages 885.98 tokens in train and 928 in test (Chen et al., 3 Sep 2025). These figures make long-context processing a central benchmark property, especially for experimental extraction.

3. Annotation schema and benchmark representations

The Method-Tracking annotations attach two principal fields to each document $D_i$ 3: Motivation $D_i$ 4 and Methodology $D_i$ 5 (Chen et al., 3 Sep 2025). Motivation includes problem formulations, gaps in prior work, key hypotheses, and research questions. Methodology includes main models, algorithms, architectures, procedures, and key design choices or innovations, such as a new loss, architecture module, or training scheme. These fields are extracted primarily from abstract and introduction sections via DeepSeek-R1 or GPT-4o and then manually checked.

All method tuples within a cluster are sorted by publication timestamp and stored as a hierarchical markdown representation. This representation can encode structures such as top-level research topic, subproblems, and methods per year, and is later converted into mind map diagrams. For each sample, the benchmark therefore provides long markdown-ized paper content from a cluster as input and a markdown-formatted research chain listing $D_i$ 6 and $D_i$ 7 for each paper in temporal order as output.

The Experimental-Analysis annotations attach four fields to each document: the main experimental table $D_i$ 8, model names $D_i$ 9, evaluation metrics $m_i$ 0, and baseline publication years $m_i$ 1. The schema captures both focal models and baselines, metric names and values, table structure and numeric entries, and the temporal metadata needed for cross-paper comparison. The aligned output is an experiment chain that reorders and normalizes metrics across papers to form benchmark trajectories suitable for line-chart visualization. The outputs are stored as structured text or code templates for charts, alongside markdown for method chains (Chen et al., 3 Sep 2025).

A central feature of the benchmark is the pairing of these two views within the same citation-aware cluster. Each cluster can therefore expose a method chain—a temporally ordered list of motivation–method pairs—and an experiment chain—a temporally and citation-aligned list of tables and performance metrics. This dual annotation structure is what enables explicit reconstruction of method–experiment chains rather than isolated summaries.

4. Temporal alignment and citation structure

Temporal and structural alignment are the organizing principles of ResearchPulse-Bench (Chen et al., 3 Sep 2025). In Method-Tracking, the benchmark enforces chronological ordering by publication timestamp $m_i$ 2, so that motivations and methods are not merely collected but sequenced into an explicit developmental chain. In Experimental-Analysis, the alignment function incorporates both temporal information and citation-resolved baseline years $m_i$ 3, allowing cross-paper metric trajectories such as performance trends across a benchmark from 2019 to 2024.

Citation information serves two distinct roles. First, citation graphs are used to build the clusters themselves by identifying papers that belong to the same research “series”. Second, the benchmark uses citation resolution to associate baselines in experimental tables with publication years, which supports statements of the form “this method improves over a 2021 baseline” at the representation level. The benchmark does not introduce a full symbolic relation language with explicit labels such as “extension” or “improvement”. Instead, relations are implicitly encoded through chronology, citation structure, and co-presence within the same cluster.

This design resolves a common misunderstanding about citation-aware benchmarks. ResearchPulse-Bench is not a citation graph benchmark in the narrow sense of labeled edge prediction, nor is it a survey-writing corpus. Its central object is a structured research chain in which temporality, citation lineage, and experimental normalization jointly support inference over scientific development. A plausible implication is that the benchmark evaluates whether a model can reconstruct scientific progression from loosely structured prose and heterogeneous tables, rather than merely detect topical relatedness.

5. Evaluation protocol, metrics, and benchmark findings

ResearchPulse-Bench supports evaluation of both core subtasks with task-specific metrics and human inspection procedures (Chen et al., 3 Sep 2025). For Method-Tracking, the input is a long textual representation of one or more papers in a cluster plus a method instruction; the output is a structured markdown chain containing temporally ordered $m_i$ 4 pairs. Ground truth consists of human-validated motivation and method summaries organized in temporal sequence. The reported metrics are METEOR, BERTScore (Precision, Recall, F1), and GPT-Score, which the paper defines as a GPT-4o-based rating between 0 and 1 across fluency, relevance, accuracy, creativity, and overall quality.

For Experimental-Analysis, the input is long experimental sections and tables plus an experiment instruction. The output comprises structured extraction $m_i$ 5 per paper, an aligned performance trajectory $m_i$ 6, and Python code generating a line chart. Ground truth consists of LLM-assisted and human-verified tables, model names, metric values, baseline years, and reference visualizations. Evaluation uses pass@1 for code executability and a suite of image or visualization metrics: Inception Score (IS), Fréchet Inception Distance (FID), Kernel Inception Distance (KID), CLIP-FID, SSIM, MS-SSIM, PSNR, LPIPS, and CMMD.

The paper also describes a multi-stage human inspection process, including manual review of cluster topical coherence, motivation–method chronological consistency, table extraction fidelity, citation year resolution, metric consistency, and structural completeness and semantic fidelity of mind maps and line charts. It explicitly notes that inter-annotator agreement coefficients are not reported. Feedback from annotators is used to refine clustering and LLM prompts in what the paper calls a “closed-loop quality assurance process”.

Empirically, ResearchPulse uses the benchmark both for training and for test-time comparison against strong baselines. On Method-Tracking, Mmap-Agent (7B) is reported as tied-best on METEOR at 46.14 with InternLM3-8B, best on BERTScore F1 at 90.39, and best on GPT-Score overall quality at 85.59. On Experimental-Analysis, Lchart-Agent reaches pass@1 = 97.50, exceeding GPT-4o’s 96.25, and posts best or near-best chart metrics including FID 6.73, IS 2.65, LPIPS 8.49, CMMD 4.06, SSIM 55.56, PSNR 12.49, and MS-SSIM 36.55. The paper further reports that ablation of planning and composition components can reduce Method-Tracking BERTScore F1 from 90.39 to as low as 82.83, and identifies characteristic failure modes including misaligned attribution, missing extraction, factual deviations in experiments, and partial referencing (Chen et al., 3 Sep 2025).

ResearchPulse-Bench is publicly available at https://huggingface.co/datasets/ResearchPulse/ResearchPulse-Bench and is derived from arXiv and OpenReview while “complying with their open-access terms” (Chen et al., 3 Sep 2025). The paper describes the construction pipeline in enough detail—Google Scholar retrieval, citation graph construction, PDF-to-markdown conversion, Sentence-BERT plus K-Means clustering, LLM-assisted extraction, and human inspection—that extension to additional clusters or adjacent domains is technically plausible, although the paper does not specify an explicit JSON schema.

Within the broader benchmark landscape, ResearchPulse-Bench occupies a specialized position. Bench-CoE formulates a benchmark-driven framework for routing queries among heterogeneous experts using query-level and subject-level labels derived from benchmark evaluations, emphasizing orchestration over expert models rather than scientific chain reconstruction (Wang et al., 2024). BenchBench, by contrast, benchmarks automated benchmark generation through domain cards, quota-controlled generation, quality filtering, and designer–answerer matrices, focusing on the meta-capability of benchmark design rather than citation-aware scientific inference (Zheng et al., 21 Mar 2026). The “continuous benchmarking” framework frames benchmarking as a CI-like automated pipeline with hierarchical configurations and templates, targeting evolving software and hardware ecosystems rather than document clusters and scientific alignment (Vogelsang et al., 17 Apr 2026). PBench addresses benchmark construction from real execution statistics through multi-objective optimization, timestamp assignment, and LLM-based augmentation of workload components, again in a different domain centered on cloud analytics traces (Zhou et al., 19 Jun 2025). LLF-Bench evaluates interactive learning from natural language feedback via a unified Gym interface across bandits, navigation, recommendation, and control, making language feedback central but not addressing multi-document scientific reasoning (Cheng et al., 2023).

Against these neighboring paradigms, ResearchPulse-Bench is distinguished by its explicit supervision of paper-level motivations, methods, tables, metrics, and citation years within temporally ordered clusters. Its significance lies in turning the evolution of a research area into a benchmarkable structure: not only what a paper says, but how a sequence of related papers motivates new methods, compares against prior baselines, and accumulates experimental evidence over time (Chen et al., 3 Sep 2025).