CiteME Benchmark: Citation Attribution
- CiteME is a benchmark for open-ended citation attribution in recent machine learning literature, requiring exact identification of masked referenced papers.
- It utilizes a carefully curated dataset of 130 non-trivial paper excerpts that avoid obvious lexical cues, emphasizing semantic understanding and source disambiguation.
- Empirical results reveal a significant performance gap between LMs and human experts, highlighting limitations in current retrieval and multi-step reasoning methods.
Searching arXiv for CiteME benchmark and closely related citation-attribution work. CiteME is a benchmark for open-ended citation attribution in recent machine learning literature. It evaluates whether a LLM, given a text excerpt from a paper in which exactly one citation is masked, can identify the specific referenced paper (Press et al., 2024). The benchmark was introduced to test claim attribution under conditions intended to be attributable, unambiguous, and non-trivial, in contrast to prior citation datasets described as automatically curated and often dominated by ambiguous, unattributable, or trivial excerpts (Press et al., 2024). In subsequent work, CiteME is used as the primary benchmark for measuring faithful citation attribution, including evaluation of retrieval-augmented validation systems such as CiteGuard (Choi et al., 15 Oct 2025).
1. Concept and research question
CiteME is centered on the question: given a text excerpt referencing a paper, could a LLM act as a research assistant and correctly identify the referenced paper (Press et al., 2024). The benchmark therefore operationalizes citation attribution as an open-ended identification problem rather than a closed-set retrieval task over a small candidate pool. The output is the referenced paper, with the paper title used for scoring (Press et al., 2024).
The benchmark’s motivating premise is that scientific information overload makes it difficult to verify and correctly attribute claims, and that practical scientific assistance requires more than fluent text generation. In this framing, citation attribution is not merely recommendation of a thematically related work, but recovery of the exact source implied by a specific scientific claim or description (Press et al., 2024). This emphasis distinguishes CiteME from local citation recommendation benchmarks that rank candidate papers for masked citation contexts under fixed retrieval settings and metrics such as Recall@10 and MRR@10 (Maharjan, 2024).
A central design objective was to avoid easy cases. CiteME was constructed to contain excerpts that clearly reference exactly one prior paper and are solvable by humans, while excluding trivial lexical cues such as author names or acronymized titles that would permit memorization-based matching (Press et al., 2024). This suggests that the benchmark is intended to measure a compound capability involving semantic understanding of the claim, literature search, and source disambiguation rather than template completion.
2. Construction and dataset composition
CiteME consists of text excerpts from recent machine learning papers, each referencing a single other paper (Press et al., 2024). The domain focus is explicitly recent machine learning literature, and the distribution is described as strongly skewed to recent years (Press et al., 2024). Tags inferred by GPT-4 suggest standard ML subareas, while overly broad tags such as “Machine Learning” were excluded (Press et al., 2024).
The benchmark was collected by four ML graduate students described as experts (Press et al., 2024). Candidate excerpts were included only if they satisfied three criteria: they referenced a single paper, included enough context to be solvable with minimal background knowledge, and were non-trivial in the sense that they did not reveal the answer through obvious author-name or title-like cues (Press et al., 2024). Each candidate excerpt was checked for reasonableness, and only those approved by at least two experts were retained; minor edits were allowed to ensure reasonableness (Press et al., 2024).
A further filtering step removed all instances that GPT-4o, without any tools, could answer in five runs. This removed 124 instances and yielded a final dataset size of 130 instances (Press et al., 2024). The benchmark is presented as a held-out evaluation set; no train/dev/test splits are described (Press et al., 2024).
Each instance includes a text excerpt with a single masked citation and a gold target paper title (Press et al., 2024). Excerpts are anonymized for citations, for example with the token “[CITATION],” and are vetted for clarity (Press et al., 2024). The paper states that evaluation via tools uses Semantic Scholar IDs during retrieval, but does not explicitly enumerate all metadata fields stored in the released dataset (Press et al., 2024).
| Aspect | CiteME specification |
|---|---|
| Domain | Recent machine learning papers |
| Instance structure | Text excerpt with one masked citation |
| Target | Gold target paper title |
| Final size | 130 instances |
| Splits | No train/dev/test splits described |
| Access | Dataset and code at citeme.ai |
The authors justify the modest dataset size by comparison to other LM benchmarks with approximately 100–200 items, arguing that such benchmark scales can still be informative (Press et al., 2024). A plausible implication is that the benchmark prioritizes curation quality and attribution difficulty over scale.
3. Task formalization and evaluation protocol
The task is open-ended citation attribution from an excerpt (Press et al., 2024). The input is a text excerpt from a recent ML paper in which exactly one citation is masked, and the output is the referenced paper (Press et al., 2024). The candidate set is open-world: there is no fixed retrieval pool, and a system may need to query an external index and inspect paper content in order to identify the correct source (Press et al., 2024).
The primary evaluation metric is accuracy. The paper formalizes scoring as
where is the system prediction and is the gold answer for item (Press et al., 2024). The benchmark does not report Recall@k, MRR, or nDCG (Press et al., 2024).
The evaluation protocol includes several model settings. Retrieval baselines use SPECTER and SPECTER2 with embedding kNN over the Semantic Scholar corpus using titles and abstracts and exact cosine similarity nearest neighbor (Press et al., 2024). Language-model baselines without tools, called “No Commands,” require the model to produce the paper title directly from the excerpt (Press et al., 2024). The agentic setting introduces CiteAgent, which enables search over Semantic Scholar and reading of PDFs, with up to 15 tool actions per instance (Press et al., 2024).
For the agent, search is implemented via Selenium on Semantic Scholar, returning the top-10 results per query with paper ID, title, abstract, and citation count (Press et al., 2024). Results published after the source paper and the source paper itself are filtered out to avoid leakage (Press et al., 2024). A run terminates when the model issues a select(ID) command or reaches the action limit; there is forced selection at action 14 (Press et al., 2024). Reading is performed on open-access PDFs via PyPDF2, and the entire extracted text is made available to the agent; for LLaMA-3-70B, read is disabled because of 8K context constraints (Press et al., 2024).
This protocol makes CiteME closer to an end-to-end research-assistant task than to standard local citation recommendation benchmarks. In a local citation recommendation benchmark such as the context-aware benchmark proposed in (Maharjan, 2024), the input is a masked citing sentence and the output is a ranked list from a large but still defined candidate pool, evaluated with top- metrics. CiteME instead requires exact attribution in an open-world setting and scores only whether the final identified paper is correct (Press et al., 2024).
4. Empirical performance and benchmark difficulty
CiteME reveals a substantial gap between LLMs and human experts (Press et al., 2024). Frontier LMs without tools achieve only 4.2–18.5% accuracy, whereas human experts reach 69.7% accuracy on a random subset of 100 benchmark items under a two-minute cap per item (Press et al., 2024). The average human completion time is 38.2 seconds per item, and the reported human score is described as a lower bound because of the time cap (Press et al., 2024).
The benchmark’s best-performing system in the original study is CiteAgent built on GPT-4o with Search+Read and a demonstration trajectory, which achieves 35.3% accuracy (Press et al., 2024). Retrieval-only embedding baselines perform extremely poorly: both SPECTER and SPECTER2 achieve 0% accuracy on CiteME (Press et al., 2024).
The paper reports the following performance ranges and configurations (Press et al., 2024):
| System/setting | Accuracy |
|---|---|
| Humans | 69.7% |
| CiteAgent (GPT-4o, Search+Read, w/ demonstration) | 35.3% |
| Frontier LMs, No Commands | 4.2–18.5% |
| SPECTER | 0% |
| SPECTER2 | 0% |
More detailed LM results underscore the benchmark’s difficulty. In “No Commands,” GPT-4o scores 0 without demonstration and 7.6 with demonstration; LLaMA-3-70B scores 4.2 and 5.9; Claude 3 Opus scores 15.1 and 18.5 (Press et al., 2024). In “Search Only,” GPT-4o reaches 26.1 without demonstration and 29.4 with demonstration, Claude 3 Opus reaches 26.1 and 27.7, and LLaMA-3-70B reaches 21.0 without demonstration but drops to 2.5 with demonstration because of context-length issues (Press et al., 2024). In “Search and Read,” GPT-4o scores 22.7 without demonstration and 35.3 with demonstration, while Claude 3 Opus scores 27.7 without demonstration and 26.1 with demonstration (Press et al., 2024).
These results indicate that access to search tools is necessary but insufficient, and that full-text reading can help when integrated effectively. They also suggest that open-ended claim attribution is more demanding than embedding retrieval over titles and abstracts.
5. CiteAgent and agentic attribution
CiteAgent is the autonomous system introduced alongside CiteME to narrow the gap between model and human performance (Press et al., 2024). It can be built on GPT-4o, Claude 3 Opus, or LLaMA-3-70B and is equipped with three commands: search(query, sort), read(ID), and select(ID) (Press et al., 2024). Search returns the top-10 papers with ID, title, citations, and abstract; read returns full-text content including title, authors, abstract, and in-paper text; select finalizes the prediction (Press et al., 2024).
The system prompt includes tool descriptions, a task description, and a demonstration trajectory, and experiments use temperature $0.95$ (Press et al., 2024). Search results are filtered by publication date and by exclusion of the source paper to reduce leakage (Press et al., 2024). The use of Selenium rather than the public API is justified in the paper by better reranking and compatibility with human trajectory annotation (Press et al., 2024).
Usage patterns reported in the paper indicate that GPT-4o benefits from both search and read, often refining queries, changing sort orders, and reading multiple papers (Press et al., 2024). Claude 3 Opus benefits less from read, and LLaMA-3-70B is constrained by context window limits (Press et al., 2024). Correct predictions typically occur within approximately five actions (Press et al., 2024).
The benchmark therefore functions not only as a scorecard but also as a systems test for agentic scientific search. This interpretation is reinforced by later work on CiteGuard, which reframes evaluation as citation attribution alignment and uses CiteME as the primary benchmark for faithful citation attribution (Choi et al., 15 Oct 2025). In that later framing, the benchmark is used to assess whether generated citations genuinely support claims rather than merely appearing plausible (Choi et al., 15 Oct 2025). However, the available description of CiteGuard does not reproduce CiteME’s internal formal definitions or protocols beyond this role.
6. Error profile, distinctiveness, and subsequent use
The original CiteME study includes an error analysis on 50 random failures from the best GPT-4o configuration (Press et al., 2024). Three error types are reported. Type 1, “Misunderstands excerpt,” accounts for 50% of failures and involves focusing on irrelevant parts of the text (Press et al., 2024). Type 2, “Understands excerpt but stops early,” accounts for 32% and involves locating a roughly matching paper but not the exact one; in 12.5% of these cases, the correct paper was present in the search results but not selected (Press et al., 2024). Type 3, “Finds the correct citation inside a paper but stops with the citing paper,” accounts for 18% and involves reading a paper that cites the correct target without then identifying the true target (Press et al., 2024). Technical issues are also reported at 5.8%, including JSON formatting errors and occasional missing or incorrect open-access PDF links in Semantic Scholar (Press et al., 2024).
These failure modes clarify that benchmark difficulty does not arise only from retrieval coverage. A large fraction of errors originate in excerpt understanding and in multi-step reasoning over retrieved evidence. This suggests that exact attribution requires claim decomposition and source tracing, not only lexical or embedding similarity.
CiteME is differentiated from prior citation datasets on several grounds (Press et al., 2024). The paper argues that many earlier datasets are auto-curated, skew older, and often contain ambiguous or unattributable contexts, while some restrict evaluation to titles and abstracts or to closed retrieval settings (Press et al., 2024). Human experts labeled four popular datasets with high rates of ambiguity, unattributability, or triviality: FullTextPeerRead had 0% reasonable excerpts, ACL-200 30%, RefSeer 40%, and the arXiv dataset of Gu et al. 2022 had 10%, with an average across the four of 20% reasonable, 37.5% ambiguous, 25% unattributable, and 17.5% trivial (Press et al., 2024).
Later work has used CiteME as a benchmark for faithful citation validation rather than only source identification. CiteGuard explicitly situates its evaluation within CiteME and describes the benchmark as the primary testbed for whether models can provide faithful citation attribution for scientific claims (Choi et al., 15 Oct 2025). The available description states that CiteGuard improves a prior baseline by 12.3% and achieves up to 65.4% accuracy on CiteME, approaching the reported human-level performance of 69.7%, while also identifying alternative but valid citations (Choi et al., 15 Oct 2025). Because the cited manuscript excerpt does not include the full experimental subfiles, detailed task protocol, metrics, and decision rules for that later use are not reproduced in the available source (Choi et al., 15 Oct 2025).
7. Limitations and broader significance
CiteME has several stated limitations (Press et al., 2024). It contains only 130 items and is focused on the machine learning domain, so generalization to other fields is untested (Press et al., 2024). The dataset intentionally excludes “easy” items that tool-free GPT-4o could solve, making the benchmark biased toward challenging cases (Press et al., 2024). No explicit license is specified in the paper text, and users are directed to review dataset licensing on citeme.ai (Press et al., 2024).
The benchmark also foregrounds risks of automation. Systems may still misattribute claims, so human oversight remains necessary to prevent erroneous citations (Press et al., 2024). This caution is consistent with later work on citation faithfulness and bibliographic hallucinations, which treats claim attribution, citation validation, and metadata correctness as distinct but complementary reliability problems (Choi et al., 15 Oct 2025, Rao et al., 3 Apr 2026). A plausible implication is that trustworthy scientific writing assistance requires separate evaluation of at least three stages: identifying the correct source, validating that the source supports the claim, and ensuring the bibliographic record itself is correct.
Within that broader landscape, CiteME occupies the claim-attribution layer. It asks whether a masked citation in a realistic scientific excerpt can be resolved to the exact referenced paper in an open-world environment (Press et al., 2024). Its principal significance lies in demonstrating that this task remains difficult for frontier LMs even when augmented with search and document-reading tools, while remaining tractable for human experts (Press et al., 2024). As a result, it serves as a high-difficulty benchmark for research on source-grounded scientific assistance, citation validation, and agentic literature search.