---
title: Citation Level Noise in Scholarly Works
url: https://www.emergentmind.com/topics/citation-level-noise
type: topic
---

# Citation Level Noise in Scholarly Works

Citation level noise denotes error, distortion, or unwanted variability at the level of individual citations rather than at the level of aggregate citation counts. In recent literature, the term is used in several related senses: as the between-author variance in erroneous citation decisions within a citation system, as information change between a cited claim and the sentence that cites it, as field-specific corruption inside structured bibliographic metadata, and as the rise of non-existent references in LLM-assisted scientific writing [2508.12735][2502.20581][2604.18880][2605.07723]. Across these uses, the common concern is that citations are often treated as measurements of knowledge flow, yet the citation itself may be noisy even when the surrounding document appears formally correct.

## 1. Operational meanings of citation level noise

The expression does not have a single universal definition. Recent work operationalizes it at different granularities, from author behavior to sentence fidelity to reference metadata.

| Source | Unit of analysis | What counts as noise |
|---|---|---|
| [2508.12735] | Citing authors in a citation system | Between-author variance in erroneous citation decisions |
| [2502.20581] | Citing sentence vs. cited-paper claim | Distortion, compression, loose paraphrase, or departure from the original claim |
| [2604.18880] | Bibliographic fields inside a generated reference | One or more wrong fields among title, authors, venue, year, DOI |
| [2605.07723] | Reference existence | Excess unmatched citations above a pre-LLM baseline |
| [1507.02099] | Citation indicator construction | Instability from database choice, selection rules, normalization, and aggregation |

These senses are not interchangeable. In the formal bibliometric framework, citation level noise concerns variability in author-specific error propensity. In the citation-fidelity framework, it concerns information change from source claim to citation sentence. In the LLM literature, it often concerns fabricated or partially fabricated references. This suggests that “citation level noise” functions less as a single settled metric than as a family of error models centered on the unreliability of citations as carriers of scientific information.

## 2. Citation level noise as between-author variance in citation error

A formal definition is provided in "Citation accuracy, citation noise, and citation bias: A foundation of citation analysis" [2508.12735]. That work treats citations as a measurement instrument for knowledge flow from a cited paper to a citing paper. An **accurate citation** is one made **only if** knowledge flow has happened. An **erroneous citation** is one made when no such knowledge flow exists, or when a necessary citation is omitted. Within this framework, **citation level noise** is defined as the variability in the author-specific tendency to make citation errors across different citing authors.

The paper denotes the person-specific average error rate of citing author $i$ by $PE_i$, and it defines citation level noise as the weighted standard deviation of these author-specific error rates around the system-wide error rate $PE$, weighted by the number of papers authored by that citing author, $n_i$. This is distinct from **citation pattern noise**, which is the variability in citation decisions of one and the same author across different citation occasions. The paper gives the author-specific pattern-noise formula as
$$
OPN_i = \sqrt{\frac{1}{n_i}\sum_{j=1}^{n_i}(PE_i - PE_{ij})^2},
$$
and the overall citation system noise as
$$
OSYS = \sqrt{(OLN)^2 + (OPN)^2}.
$$

The distinction from **citation bias** is central. Bias is defined as a systematic directional deviation from accurate citation decisions, caused by factors unrelated to knowledge flow. Level noise, by contrast, is scatter rather than direction: some authors tend to cite too much, some too little, some more carefully, some less carefully. In the paper’s fictitious system, overall citation accuracy is $PA = 0.54$, overall error proportion is $PE = 0.46$, citation level noise is $\ØLN = 0.06$, citation pattern noise is $OPN = 0.17$, and total citation system noise is $OSYS = 0.18$. The substantive implication is that citation counts can be contaminated even when no single directional bias dominates, because author-specific citation habits differ in their error propensity [2508.12735].

## 3. Citation fidelity and the noisy path from source to citation

A second line of work measures citation-level noise as information change between the cited source and the citing sentence. "The Noisy Path from Source to Citation: Measuring How Scholars Engage with Past Research" introduces **citation fidelity** as the degree to which a citing sentence preserves the original scientific claim [2502.20581]. In this framing, high citation fidelity means low noise, while low citation fidelity means high noise.

The paper builds a computational pipeline on **S2ORC**, which contains metadata for **136 million** scholarly articles; after filtering for English and full-text availability, about **42 million papers** are retained. It focuses on **reporting citations**, isolated through **single-source citations** and **background citation classification**. Background citations are detected with a **SciBERT** classifier trained on **27,052 instances**, with **11,635 (43%)** positive background labels and **F1 = 0.81**. Candidate claim sentences in cited papers are extracted with a **RoBERTa** sentence classifier trained on **200,000 PubMed abstracts**, with **F1 = 0.92** on a held-out 10% set; about **30%** of full-text sentences are classified as results or conclusions. For a citing sentence $c$ and cited-paper sentence $s$, the paper uses an upper-bound matching strategy,
$$
\text{Fidelity}(c, P) = \max_{s \in \text{Claims}(P)} \text{Score}(c,s),
$$
and then scores sentence pairs on a **1–5 ordinal similarity score**, where **5** means identical scientific information and **1** means completely different.

At scale, this yields approximately **13 million citation sentence pairs**. The fidelity scores are reported to roughly follow a normal distribution centered around about **3.5**, spanning **1.0 to 5.0**. Higher fidelity is associated with cited papers that are more recent and intellectually close, with **within-field** citations, **self-citations**, **open-access** works, lower first-author H-index, and **medium-sized teams**; fidelity is also higher in **biology and medicine** than in physics and computer science, highest for **review articles**, and lowest for **conference papers**. The paper’s quasi-experimental “telephone effect” identifies about **50,000 paper pairs** and shows that the treatment group’s fidelity to the original paper is **0.06 lower** than the control group’s when an intermediary paper sits in the citation path. Here, citation level noise is not fabrication of reference strings but degradation of scientific content as claims move across citation chains [2502.20581].

## 4. Structured-reference hallucination and non-existent citations

In LLM-generated references, citation-level noise is often treated as corruption inside the bibliographic record itself. "Where Fake Citations Are Made: Tracing Field-Level Hallucination to Specific Neurons in LLMs" decomposes each generated reference into **title, authors, venue, year, DOI** and verifies each field independently against OpenAlex plus web-grounded verification when needed [2604.18880]. A citation can therefore be **Supported**, **Partial**, or **Unsupported**. The benchmark covers **9 models**, **50 topics**, **8 citation styles**, and **3 generation volumes** with $N \in \{5,10,15\}$, producing about **108,000 generated references**. The strongest finding is that **author names are by far the most error-prone field across all models and settings**, with the difficulty hierarchy
$$
\text{Author} \;<\; \text{Venue} \;<\; \text{Title} \approx \text{Year},
$$
while **DOI** is highly model-dependent. Citation style has **no measurable effect**: across **APA, MLA, Chicago, Harvard, Vancouver, IEEE, ACM, AMA**, the maximum difference between styles is **less than 0.04**, and Kruskal–Wallis tests are not significant for any field. Probes trained to detect hallucination are highly field-specific, with **in-field AUC = 0.812–0.922** and **cross-field AUC = 0.46–0.59**. Using CETT features on **Qwen2.5-32B-Instruct**, the paper identifies sparse **field-specific hallucination neurons (FH-neurons)** via elastic-net regression and stability selection, retaining **224** title neurons, **78** author neurons, **129** year neurons, **51** venue neurons, and **30** DOI neurons out of **1,769,472 candidates per field**. Causal intervention shows that amplifying these neurons worsens hallucination, while suppressing them improves performance on most fields.

A complementary corpus-scale perspective is provided by "LLM hallucinations in the wild: Large-scale evidence from non-existent citations" [2605.07723]. That paper audits **111 million references across 2.5 million papers** in **arXiv, bioRxiv, SSRN, and PubMed Central** and defines hallucinated citations as the estimated excess of unmatched citations above a pre-LLM baseline. The verification pipeline matches titles against **Semantic Scholar** and **OpenAlex**, applies an **LLM-based cleaning step** using **GPT-4o-mini**, and then queries **Google Scholar** for remaining cases. The initial title-match pipeline matches **95.1%** of references; after cleaning and re-parsing, the unmatched rate drops to **2.33%**, then to **1.54%**. Unmatched rates remain relatively stable until **late 2022**, then rise sharply in **2023**, with the steepest growth beginning in **mid-2024**. By **August 2025**, estimated hallucination rates are **0.39%** for arXiv, **0.21%** for bioRxiv, **1.91%** for SSRN, and **0.27%** for PMC, implying a conservative lower bound of **146,932** hallucinated citations in **2025** across the four corpora. The contamination is described as diffuse rather than concentrated in a few papers, is especially prominent in **social sciences** and **computer science**, is positively associated with estimated LLM use in arXiv abstracts with $r = 0.441,\ P < 0.001$, and is more common among **small** and **less experienced** author teams.

## 5. Measurement conditions, audits, and the broader noise framework

A broader scientometric literature shows that citation counts themselves are produced by many measurement choices. "A review of the literature on citation impact indicators" does not formalize “citation level noise” as a single parameter, but it identifies recurrent sources of instability and distortion in citation-based indicators [1507.02099]. These include differences in **Web of Science**, **Scopus**, and **Google Scholar** coverage; incorrect citation relations, document-type misclassification, citation matching errors, duplicate records, DOI assignment errors, omitted citations, and manipulation; selection choices regarding **document type**, **language**, **national vs. international journals**, **self-citations**, and **citation windows**; field-normalization uncertainty; counting choices such as **full counting** versus **fractional counting**; and skewed journal-level citation distributions. In this perspective, citation noise is inseparable from database architecture and indicator design.

A more general vocabulary for noise comes from "Precision in the Face of Noise -- Lessons from Kahneman, Siboney, and Sunstein for Radiation Oncology", which defines noise as **unwanted variability in judgments that should ideally be the same** and states
$$
\text{mean squared error} = \text{bias}^2 + \text{noise}^2,
$$
with system noise further decomposed as
$$
\text{system noise}^2 = \text{level noise}^2 + \text{pattern noise}^2.
$$
Although the paper is about radiation oncology, it explicitly presents transferable ideas for citation-based evaluation, including a **noise audit** as a structured method to quantify unwanted variability and **decision hygiene** through standardization, structured procedures, aggregation of multiple judgments, checklists, and AI where appropriate [2412.02724]. This mapping is conceptually close to the bibliometric distinction between citation level noise and citation pattern noise.

## 6. Implications for research evaluation and mitigation

Across these literatures, a common conclusion is that raw citation counts are an incomplete representation of scientific influence. One paper states the point directly: **counts measure attention; fidelity measures the quality of knowledge flow** [2502.20581]. Another argues that citation-based indicators are valid only insofar as citation decisions are mostly accurate, and that the lower the level of aggregation, the more damaging citation noise becomes, because individual papers and researchers are less able to average out random error [2508.12735]. A further implication is that a citation may be formally fluent while still being only **Partial** rather than **Supported**, or may be entirely non-existent despite appearing plausible [2604.18880][2605.07723].

Several misconceptions are rejected by the empirical record. Citation style does not fix hallucination: in the field-level reference benchmark, citation style has no measurable effect on error rates [2604.18880]. Editorial safeguards also do not remove most LLM-era citation errors: in arXiv, rejected manuscripts reach a hallucination rate of **2.2% by August 2025**, or **4.5×** the rate among accepted manuscripts, yet **78.8%** of non-existent citations are estimated to pass moderation; among **2,241 bioRxiv preprints** with unmatched references traced to PMC, **85.3%** of hallucinations persist into the published record [2605.07723]. The problem is therefore not restricted to isolated outliers or purely preprint-stage artifacts.

Mitigation proposals differ by operationalization. In the formal bibliometric framework, proposed interventions include aggregation of citation decisions, **citation decision hygiene**, stronger guidelines for accurate citation, **citation justification tables**, training of researchers, post-publication correction of citation errors, and **AI-assisted citation checking** that flags possible omissions or erroneous insertions without autonomously deciding when a citation is warranted [2508.12735]. In the sentence-fidelity framework, the principal intervention is to evaluate not only whether a citation exists but how faithfully it represents the source [2502.20581]. In the LLM-reference framework, the results suggest a lightweight approach to hallucination detection and mitigation using internal model signals alone, since suppression of field-specific hallucination neurons improves performance on most fields [2604.18880].

Taken together, these results suggest that citation level noise is not a marginal defect but a structural property of modern citation systems. It appears in author-specific error tendencies, in the semantic transformation of claims, in the internal fields of generated references, in database-dependent citation indicators, and in corpus-scale LLM hallucinations. The central research implication is that the validity of citation-based evaluation depends not only on how many citations are observed, but on how accurately, faithfully, and verifiably those citations encode knowledge flow.

Source: https://www.emergentmind.com/topics/citation-level-noise