---
title: 'CiteME Benchmark: Citation Attribution'
url: https://www.emergentmind.com/topics/citeme-benchmark
type: topic
---

# CiteME Benchmark: Citation Attribution

Searching arXiv for CiteME benchmark and closely related citation-attribution work.
CiteME is a benchmark for open-ended citation attribution in recent machine learning literature. It evaluates whether a language model, given a text excerpt from a paper in which exactly one citation is masked, can identify the specific referenced paper [2407.12861]. The benchmark was introduced to test claim attribution under conditions intended to be attributable, unambiguous, and non-trivial, in contrast to prior citation datasets described as automatically curated and often dominated by ambiguous, unattributable, or trivial excerpts [2407.12861]. In subsequent work, CiteME is used as the primary benchmark for measuring faithful citation attribution, including evaluation of retrieval-augmented validation systems such as CiteGuard [2510.17853].

## 1. Concept and research question

CiteME is centered on the question: given a text excerpt referencing a paper, could a language model act as a research assistant and correctly identify the referenced paper [2407.12861]. The benchmark therefore operationalizes citation attribution as an open-ended identification problem rather than a closed-set retrieval task over a small candidate pool. The output is the referenced paper, with the paper title used for scoring [2407.12861].

The benchmark’s motivating premise is that scientific information overload makes it difficult to verify and correctly attribute claims, and that practical scientific assistance requires more than fluent text generation. In this framing, citation attribution is not merely recommendation of a thematically related work, but recovery of the exact source implied by a specific scientific claim or description [2407.12861]. This emphasis distinguishes CiteME from local citation recommendation benchmarks that rank candidate papers for masked citation contexts under fixed retrieval settings and metrics such as Recall@10 and MRR@10 [2412.07713].

A central design objective was to avoid easy cases. CiteME was constructed to contain excerpts that clearly reference exactly one prior paper and are solvable by humans, while excluding trivial lexical cues such as author names or acronymized titles that would permit memorization-based matching [2407.12861]. This suggests that the benchmark is intended to measure a compound capability involving semantic understanding of the claim, literature search, and source disambiguation rather than template completion.

## 2. Construction and dataset composition

CiteME consists of text excerpts from recent machine learning papers, each referencing a single other paper [2407.12861]. The domain focus is explicitly recent machine learning literature, and the distribution is described as strongly skewed to recent years [2407.12861]. Tags inferred by GPT-4 suggest standard ML subareas, while overly broad tags such as “Machine Learning” were excluded [2407.12861].

The benchmark was collected by four ML graduate students described as experts [2407.12861]. Candidate excerpts were included only if they satisfied three criteria: they referenced a single paper, included enough context to be solvable with minimal background knowledge, and were non-trivial in the sense that they did not reveal the answer through obvious author-name or title-like cues [2407.12861]. Each candidate excerpt was checked for reasonableness, and only those approved by at least two experts were retained; minor edits were allowed to ensure reasonableness [2407.12861].

A further filtering step removed all instances that GPT-4o, without any tools, could answer in five runs. This removed 124 instances and yielded a final dataset size of 130 instances [2407.12861]. The benchmark is presented as a held-out evaluation set; no train/dev/test splits are described [2407.12861].

Each instance includes a text excerpt with a single masked citation and a gold target paper title [2407.12861]. Excerpts are anonymized for citations, for example with the token “[CITATION],” and are vetted for clarity [2407.12861]. The paper states that evaluation via tools uses Semantic Scholar IDs during retrieval, but does not explicitly enumerate all metadata fields stored in the released dataset [2407.12861].

| Aspect | CiteME specification |
|---|---|
| Domain | Recent machine learning papers |
| Instance structure | Text excerpt with one masked citation |
| Target | Gold target paper title |
| Final size | 130 instances |
| Splits | No train/dev/test splits described |
| Access | Dataset and code at citeme.ai |

The authors justify the modest dataset size by comparison to other LM benchmarks with approximately 100–200 items, arguing that such benchmark scales can still be informative [2407.12861]. A plausible implication is that the benchmark prioritizes curation quality and attribution difficulty over scale.

## 3. Task formalization and evaluation protocol

The task is open-ended citation attribution from an excerpt [2407.12861]. The input is a text excerpt $t$ from a recent ML paper in which exactly one citation is masked, and the output is the referenced paper $p^*$ [2407.12861]. The candidate set is open-world: there is no fixed retrieval pool, and a system may need to query an external index and inspect paper content in order to identify the correct source [2407.12861].

The primary evaluation metric is accuracy. The paper formalizes scoring as

$$
\mathrm{Acc} = \frac{1}{N} \sum_{i=1}^{N} 1[\hat{p}_i = y_i]
$$

where $\hat{p}_i$ is the system prediction and $y_i$ is the gold answer for item $i$ [2407.12861]. The benchmark does not report Recall@k, MRR, or nDCG [2407.12861].

The evaluation protocol includes several model settings. Retrieval baselines use SPECTER and SPECTER2 with embedding kNN over the Semantic Scholar corpus using titles and abstracts and exact cosine similarity nearest neighbor [2407.12861]. Language-model baselines without tools, called “No Commands,” require the model to produce the paper title directly from the excerpt [2407.12861]. The agentic setting introduces CiteAgent, which enables search over Semantic Scholar and reading of PDFs, with up to 15 tool actions per instance [2407.12861].

For the agent, search is implemented via Selenium on Semantic Scholar, returning the top-10 results per query with paper ID, title, abstract, and citation count [2407.12861]. Results published after the source paper and the source paper itself are filtered out to avoid leakage [2407.12861]. A run terminates when the model issues a `select(ID)` command or reaches the action limit; there is forced selection at action 14 [2407.12861]. Reading is performed on open-access PDFs via PyPDF2, and the entire extracted text is made available to the agent; for LLaMA-3-70B, read is disabled because of 8K context constraints [2407.12861].

This protocol makes CiteME closer to an end-to-end research-assistant task than to standard local citation recommendation benchmarks. In a local citation recommendation benchmark such as the context-aware benchmark proposed in [2412.07713], the input is a masked citing sentence and the output is a ranked list from a large but still defined candidate pool, evaluated with top-$k$ metrics. CiteME instead requires exact attribution in an open-world setting and scores only whether the final identified paper is correct [2407.12861].

## 4. Empirical performance and benchmark difficulty

CiteME reveals a substantial gap between language models and human experts [2407.12861]. Frontier LMs without tools achieve only 4.2–18.5% accuracy, whereas human experts reach 69.7% accuracy on a random subset of 100 benchmark items under a two-minute cap per item [2407.12861]. The average human completion time is 38.2 seconds per item, and the reported human score is described as a lower bound because of the time cap [2407.12861].

The benchmark’s best-performing system in the original study is CiteAgent built on GPT-4o with Search+Read and a demonstration trajectory, which achieves 35.3% accuracy [2407.12861]. Retrieval-only embedding baselines perform extremely poorly: both SPECTER and SPECTER2 achieve 0% accuracy on CiteME [2407.12861].

The paper reports the following performance ranges and configurations [2407.12861]:

| System/setting | Accuracy |
|---|---|
| Humans | 69.7% |
| CiteAgent (GPT-4o, Search+Read, w/ demonstration) | 35.3% |
| Frontier LMs, No Commands | 4.2–18.5% |
| SPECTER | 0% |
| SPECTER2 | 0% |

More detailed LM results underscore the benchmark’s difficulty. In “No Commands,” GPT-4o scores 0 without demonstration and 7.6 with demonstration; LLaMA-3-70B scores 4.2 and 5.9; Claude 3 Opus scores 15.1 and 18.5 [2407.12861]. In “Search Only,” GPT-4o reaches 26.1 without demonstration and 29.4 with demonstration, Claude 3 Opus reaches 26.1 and 27.7, and LLaMA-3-70B reaches 21.0 without demonstration but drops to 2.5 with demonstration because of context-length issues [2407.12861]. In “Search and Read,” GPT-4o scores 22.7 without demonstration and 35.3 with demonstration, while Claude 3 Opus scores 27.7 without demonstration and 26.1 with demonstration [2407.12861].

These results indicate that access to search tools is necessary but insufficient, and that full-text reading can help when integrated effectively. They also suggest that open-ended claim attribution is more demanding than embedding retrieval over titles and abstracts.

## 5. CiteAgent and agentic attribution

CiteAgent is the autonomous system introduced alongside CiteME to narrow the gap between model and human performance [2407.12861]. It can be built on GPT-4o, Claude 3 Opus, or LLaMA-3-70B and is equipped with three commands: `search(query, sort)`, `read(ID)`, and `select(ID)` [2407.12861]. Search returns the top-10 papers with ID, title, citations, and abstract; read returns full-text content including title, authors, abstract, and in-paper text; select finalizes the prediction [2407.12861].

The system prompt includes tool descriptions, a task description, and a demonstration trajectory, and experiments use temperature $0.95$ [2407.12861]. Search results are filtered by publication date and by exclusion of the source paper to reduce leakage [2407.12861]. The use of Selenium rather than the public API is justified in the paper by better reranking and compatibility with human trajectory annotation [2407.12861].

Usage patterns reported in the paper indicate that GPT-4o benefits from both search and read, often refining queries, changing sort orders, and reading multiple papers [2407.12861]. Claude 3 Opus benefits less from read, and LLaMA-3-70B is constrained by context window limits [2407.12861]. Correct predictions typically occur within approximately five actions [2407.12861].

The benchmark therefore functions not only as a scorecard but also as a systems test for agentic scientific search. This interpretation is reinforced by later work on CiteGuard, which reframes evaluation as citation attribution alignment and uses CiteME as the primary benchmark for faithful citation attribution [2510.17853]. In that later framing, the benchmark is used to assess whether generated citations genuinely support claims rather than merely appearing plausible [2510.17853]. However, the available description of CiteGuard does not reproduce CiteME’s internal formal definitions or protocols beyond this role.

## 6. Error profile, distinctiveness, and subsequent use

The original CiteME study includes an error analysis on 50 random failures from the best GPT-4o configuration [2407.12861]. Three error types are reported. Type 1, “Misunderstands excerpt,” accounts for 50% of failures and involves focusing on irrelevant parts of the text [2407.12861]. Type 2, “Understands excerpt but stops early,” accounts for 32% and involves locating a roughly matching paper but not the exact one; in 12.5% of these cases, the correct paper was present in the search results but not selected [2407.12861]. Type 3, “Finds the correct citation inside a paper but stops with the citing paper,” accounts for 18% and involves reading a paper that cites the correct target without then identifying the true target [2407.12861]. Technical issues are also reported at 5.8%, including JSON formatting errors and occasional missing or incorrect open-access PDF links in Semantic Scholar [2407.12861].

These failure modes clarify that benchmark difficulty does not arise only from retrieval coverage. A large fraction of errors originate in excerpt understanding and in multi-step reasoning over retrieved evidence. This suggests that exact attribution requires claim decomposition and source tracing, not only lexical or embedding similarity.

CiteME is differentiated from prior citation datasets on several grounds [2407.12861]. The paper argues that many earlier datasets are auto-curated, skew older, and often contain ambiguous or unattributable contexts, while some restrict evaluation to titles and abstracts or to closed retrieval settings [2407.12861]. Human experts labeled four popular datasets with high rates of ambiguity, unattributability, or triviality: FullTextPeerRead had 0% reasonable excerpts, ACL-200 30%, RefSeer 40%, and the arXiv dataset of Gu et al. 2022 had 10%, with an average across the four of 20% reasonable, 37.5% ambiguous, 25% unattributable, and 17.5% trivial [2407.12861].

Later work has used CiteME as a benchmark for faithful citation validation rather than only source identification. CiteGuard explicitly situates its evaluation within CiteME and describes the benchmark as the primary testbed for whether models can provide faithful citation attribution for scientific claims [2510.17853]. The available description states that CiteGuard improves a prior baseline by 12.3% and achieves up to 65.4% accuracy on CiteME, approaching the reported human-level performance of 69.7%, while also identifying alternative but valid citations [2510.17853]. Because the cited manuscript excerpt does not include the full experimental subfiles, detailed task protocol, metrics, and decision rules for that later use are not reproduced in the available source [2510.17853].

## 7. Limitations and broader significance

CiteME has several stated limitations [2407.12861]. It contains only 130 items and is focused on the machine learning domain, so generalization to other fields is untested [2407.12861]. The dataset intentionally excludes “easy” items that tool-free GPT-4o could solve, making the benchmark biased toward challenging cases [2407.12861]. No explicit license is specified in the paper text, and users are directed to review dataset licensing on citeme.ai [2407.12861].

The benchmark also foregrounds risks of automation. Systems may still misattribute claims, so human oversight remains necessary to prevent erroneous citations [2407.12861]. This caution is consistent with later work on citation faithfulness and bibliographic hallucinations, which treats claim attribution, citation validation, and metadata correctness as distinct but complementary reliability problems [2510.17853; 2604.03159]. A plausible implication is that trustworthy scientific writing assistance requires separate evaluation of at least three stages: identifying the correct source, validating that the source supports the claim, and ensuring the bibliographic record itself is correct.

Within that broader landscape, CiteME occupies the claim-attribution layer. It asks whether a masked citation in a realistic scientific excerpt can be resolved to the exact referenced paper in an open-world environment [2407.12861]. Its principal significance lies in demonstrating that this task remains difficult for frontier LMs even when augmented with search and document-reading tools, while remaining tractable for human experts [2407.12861]. As a result, it serves as a high-difficulty benchmark for research on source-grounded scientific assistance, citation validation, and agentic literature search.

Source: https://www.emergentmind.com/topics/citeme-benchmark