Citation Level Noise in Scholarly Works

• Citation level noise is the unwanted variability and distortion in individual citations, reflecting errors in author decisions, paraphrasing, and metadata integrity.
• Research operationalizes this noise through measures such as between-author variance, sentence fidelity degradation, and LLM-induced hallucinations in reference fields.
• The implications call for improved citation practices and AI-driven checking mechanisms to enhance the reliability of citation-based evaluations.

Citation level noise denotes error, distortion, or unwanted variability at the level of individual citations rather than at the level of aggregate citation counts. In recent literature, the term is used in several related senses: as the between-author variance in erroneous citation decisions within a citation system, as information change between a cited claim and the sentence that cites it, as field-specific corruption inside structured bibliographic metadata, and as the rise of non-existent references in LLM-assisted scientific writing (Bornmann et al., 18 Aug 2025, Chen et al., 27 Feb 2025, Chen et al., 20 Apr 2026, Zhao et al., 8 May 2026). Across these uses, the common concern is that citations are often treated as measurements of knowledge flow, yet the citation itself may be noisy even when the surrounding document appears formally correct.

1. Operational meanings of citation level noise

The expression does not have a single universal definition. Recent work operationalizes it at different granularities, from author behavior to sentence fidelity to reference metadata.

Source	Unit of analysis	What counts as noise
(Bornmann et al., 18 Aug 2025)	Citing authors in a citation system	Between-author variance in erroneous citation decisions
(Chen et al., 27 Feb 2025)	Citing sentence vs. cited-paper claim	Distortion, compression, loose paraphrase, or departure from the original claim
(Chen et al., 20 Apr 2026)	Bibliographic fields inside a generated reference	One or more wrong fields among title, authors, venue, year, DOI
(Zhao et al., 8 May 2026)	Reference existence	Excess unmatched citations above a pre-LLM baseline
(Waltman, 2015)	Citation indicator construction	Instability from database choice, selection rules, normalization, and aggregation

These senses are not interchangeable. In the formal bibliometric framework, citation level noise concerns variability in author-specific error propensity. In the citation-fidelity framework, it concerns information change from source claim to citation sentence. In the LLM literature, it often concerns fabricated or partially fabricated references. This suggests that “citation level noise” functions less as a single settled metric than as a family of error models centered on the unreliability of citations as carriers of scientific information.

2. Citation level noise as between-author variance in citation error

A formal definition is provided in "Citation accuracy, citation noise, and citation bias: A foundation of citation analysis" (Bornmann et al., 18 Aug 2025). That work treats citations as a measurement instrument for knowledge flow from a cited paper to a citing paper. An accurate citation is one made only if knowledge flow has happened. An erroneous citation is one made when no such knowledge flow exists, or when a necessary citation is omitted. Within this framework, citation level noise is defined as the variability in the author-specific tendency to make citation errors across different citing authors.

The paper denotes the person-specific average error rate of citing author $i$ by $PE_i$, and it defines citation level noise as the weighted standard deviation of these author-specific error rates around the system-wide error rate $PE$, weighted by the number of papers authored by that citing author, $n_i$. This is distinct from citation pattern noise, which is the variability in citation decisions of one and the same author across different citation occasions. The paper gives the author-specific pattern-noise formula as

$$OPN_i = \sqrt{\frac{1}{n_i}\sum_{j=1}^{n_i}(PE_i - PE_{ij})^2},$$

and the overall citation system noise as

$OSYS = \sqrt{(OLN)^2 + (OPN)^2}.$

The distinction from citation bias is central. Bias is defined as a systematic directional deviation from accurate citation decisions, caused by factors unrelated to knowledge flow. Level noise, by contrast, is scatter rather than direction: some authors tend to cite too much, some too little, some more carefully, some less carefully. In the paper’s fictitious system, overall citation accuracy is $PA = 0.54$, overall error proportion is $PE = 0.46$, citation level noise is $\ØLN = 0.06$, citation pattern noise is $OPN = 0.17$, and total citation system noise is $PE_i$0. The substantive implication is that citation counts can be contaminated even when no single directional bias dominates, because author-specific citation habits differ in their error propensity (Bornmann et al., 18 Aug 2025).

3. Citation fidelity and the noisy path from source to citation

A second line of work measures citation-level noise as information change between the cited source and the citing sentence. "The Noisy Path from Source to Citation: Measuring How Scholars Engage with Past Research" introduces citation fidelity as the degree to which a citing sentence preserves the original scientific claim (Chen et al., 27 Feb 2025). In this framing, high citation fidelity means low noise, while low citation fidelity means high noise.

The paper builds a computational pipeline on S2ORC, which contains metadata for 136 million scholarly articles; after filtering for English and full-text availability, about 42 million papers are retained. It focuses on reporting citations, isolated through single-source citations and background citation classification. Background citations are detected with a SciBERT classifier trained on 27,052 instances, with 11,635 (43%) positive background labels and F1 = 0.81. Candidate claim sentences in cited papers are extracted with a RoBERTa sentence classifier trained on 200,000 PubMed abstracts, with F1 = 0.92 on a held-out 10% set; about 30% of full-text sentences are classified as results or conclusions. For a citing sentence $PE_i$1 and cited-paper sentence $PE_i$2, the paper uses an upper-bound matching strategy,

$PE_i$3

and then scores sentence pairs on a 1–5 ordinal similarity score, where 5 means identical scientific information and 1 means completely different.

At scale, this yields approximately 13 million citation sentence pairs. The fidelity scores are reported to roughly follow a normal distribution centered around about 3.5, spanning 1.0 to 5.0. Higher fidelity is associated with cited papers that are more recent and intellectually close, with within-field citations, self-citations, open-access works, lower first-author H-index, and medium-sized teams; fidelity is also higher in biology and medicine than in physics and computer science, highest for review articles, and lowest for conference papers. The paper’s quasi-experimental “telephone effect” identifies about 50,000 paper pairs and shows that the treatment group’s fidelity to the original paper is 0.06 lower than the control group’s when an intermediary paper sits in the citation path. Here, citation level noise is not fabrication of reference strings but degradation of scientific content as claims move across citation chains (Chen et al., 27 Feb 2025).

4. Structured-reference hallucination and non-existent citations

In LLM-generated references, citation-level noise is often treated as corruption inside the bibliographic record itself. "Where Fake Citations Are Made: Tracing Field-Level Hallucination to Specific Neurons in LLMs" decomposes each generated reference into title, authors, venue, year, DOI and verifies each field independently against OpenAlex plus web-grounded verification when needed (Chen et al., 20 Apr 2026). A citation can therefore be Supported, Partial, or Unsupported. The benchmark covers 9 models, 50 topics, 8 citation styles, and 3 generation volumes with $PE_i$4, producing about 108,000 generated references. The strongest finding is that author names are by far the most error-prone field across all models and settings, with the difficulty hierarchy

$PE_i$5

while DOI is highly model-dependent. Citation style has no measurable effect: across APA, MLA, Chicago, Harvard, Vancouver, IEEE, ACM, AMA, the maximum difference between styles is less than 0.04, and Kruskal–Wallis tests are not significant for any field. Probes trained to detect hallucination are highly field-specific, with in-field AUC = 0.812–0.922 and cross-field AUC = 0.46–0.59. Using CETT features on Qwen2.5-32B-Instruct, the paper identifies sparse field-specific hallucination neurons (FH-neurons) via elastic-net regression and stability selection, retaining 224 title neurons, 78 author neurons, 129 year neurons, 51 venue neurons, and 30 DOI neurons out of 1,769,472 candidates per field. Causal intervention shows that amplifying these neurons worsens hallucination, while suppressing them improves performance on most fields.

A complementary corpus-scale perspective is provided by "LLM hallucinations in the wild: Large-scale evidence from non-existent citations" (Zhao et al., 8 May 2026). That paper audits 111 million references across 2.5 million papers in arXiv, bioRxiv, SSRN, and PubMed Central and defines hallucinated citations as the estimated excess of unmatched citations above a pre-LLM baseline. The verification pipeline matches titles against Semantic Scholar and OpenAlex, applies an LLM-based cleaning step using GPT-4o-mini, and then queries Google Scholar for remaining cases. The initial title-match pipeline matches 95.1% of references; after cleaning and re-parsing, the unmatched rate drops to 2.33%, then to 1.54%. Unmatched rates remain relatively stable until late 2022, then rise sharply in 2023, with the steepest growth beginning in mid-2024. By August 2025, estimated hallucination rates are 0.39% for arXiv, 0.21% for bioRxiv, 1.91% for SSRN, and 0.27% for PMC, implying a conservative lower bound of 146,932 hallucinated citations in 2025 across the four corpora. The contamination is described as diffuse rather than concentrated in a few papers, is especially prominent in social sciences and computer science, is positively associated with estimated LLM use in arXiv abstracts with $PE_i$6, and is more common among small and less experienced author teams.

5. Measurement conditions, audits, and the broader noise framework

A broader scientometric literature shows that citation counts themselves are produced by many measurement choices. "A review of the literature on citation impact indicators" does not formalize “citation level noise” as a single parameter, but it identifies recurrent sources of instability and distortion in citation-based indicators (Waltman, 2015). These include differences in Web of Science, Scopus, and Google Scholar coverage; incorrect citation relations, document-type misclassification, citation matching errors, duplicate records, DOI assignment errors, omitted citations, and manipulation; selection choices regarding document type, language, national vs. international journals, self-citations, and citation windows; field-normalization uncertainty; counting choices such as full counting versus fractional counting; and skewed journal-level citation distributions. In this perspective, citation noise is inseparable from database architecture and indicator design.

A more general vocabulary for noise comes from "Precision in the Face of Noise -- Lessons from Kahneman, Siboney, and Sunstein for Radiation Oncology", which defines noise as unwanted variability in judgments that should ideally be the same and states

$PE_i$7

with system noise further decomposed as

$PE_i$8

Although the paper is about radiation oncology, it explicitly presents transferable ideas for citation-based evaluation, including a noise audit as a structured method to quantify unwanted variability and decision hygiene through standardization, structured procedures, aggregation of multiple judgments, checklists, and AI where appropriate (Wahid et al., 2024). This mapping is conceptually close to the bibliometric distinction between citation level noise and citation pattern noise.

6. Implications for research evaluation and mitigation

Across these literatures, a common conclusion is that raw citation counts are an incomplete representation of scientific influence. One paper states the point directly: counts measure attention; fidelity measures the quality of knowledge flow (Chen et al., 27 Feb 2025). Another argues that citation-based indicators are valid only insofar as citation decisions are mostly accurate, and that the lower the level of aggregation, the more damaging citation noise becomes, because individual papers and researchers are less able to average out random error (Bornmann et al., 18 Aug 2025). A further implication is that a citation may be formally fluent while still being only Partial rather than Supported, or may be entirely non-existent despite appearing plausible (Chen et al., 20 Apr 2026, Zhao et al., 8 May 2026).

Several misconceptions are rejected by the empirical record. Citation style does not fix hallucination: in the field-level reference benchmark, citation style has no measurable effect on error rates (Chen et al., 20 Apr 2026). Editorial safeguards also do not remove most LLM-era citation errors: in arXiv, rejected manuscripts reach a hallucination rate of 2.2% by August 2025, or 4.5× the rate among accepted manuscripts, yet 78.8% of non-existent citations are estimated to pass moderation; among 2,241 bioRxiv preprints with unmatched references traced to PMC, 85.3% of hallucinations persist into the published record (Zhao et al., 8 May 2026). The problem is therefore not restricted to isolated outliers or purely preprint-stage artifacts.

Mitigation proposals differ by operationalization. In the formal bibliometric framework, proposed interventions include aggregation of citation decisions, citation decision hygiene, stronger guidelines for accurate citation, citation justification tables, training of researchers, post-publication correction of citation errors, and AI-assisted citation checking that flags possible omissions or erroneous insertions without autonomously deciding when a citation is warranted (Bornmann et al., 18 Aug 2025). In the sentence-fidelity framework, the principal intervention is to evaluate not only whether a citation exists but how faithfully it represents the source (Chen et al., 27 Feb 2025). In the LLM-reference framework, the results suggest a lightweight approach to hallucination detection and mitigation using internal model signals alone, since suppression of field-specific hallucination neurons improves performance on most fields (Chen et al., 20 Apr 2026).

Taken together, these results suggest that citation level noise is not a marginal defect but a structural property of modern citation systems. It appears in author-specific error tendencies, in the semantic transformation of claims, in the internal fields of generated references, in database-dependent citation indicators, and in corpus-scale LLM hallucinations. The central research implication is that the validity of citation-based evaluation depends not only on how many citations are observed, but on how accurately, faithfully, and verifiably those citations encode knowledge flow.