Citation Noise in Academic Research

• Citation noise is defined as unwanted distortions in academic citations, including metadata errors, fabricated entries, and misattribution of references.
• Studies show that citation noise stems from both human citation variability and LLM-generated reference hallucinations, affecting bibliometrics and scholarly credit.
• Mitigation strategies involve verification pipelines, editorial guidelines, and automated tools to enhance reference accuracy and maintain the integrity of the scientific record.

Searching arXiv for papers on citation hallucination, citation fidelity, and citation noise in academic and RAG settings. Citation noise denotes unwanted distortion in citations and citation systems. In current research, the term is used for several related phenomena: fabricated or corrupted bibliographic metadata, visible citations that are relevant yet do not warrant the strength of the attached claim, variability in citation decisions relative to true knowledge flow, and metadata-driven misassociation of references to the wrong records (Bornmann et al., 18 Aug 2025, Chen et al., 20 Apr 2026, Qian et al., 27 May 2026, Kriváchy, 3 Nov 2025). The topic has become central to both bibliometrics and LLM evaluation because citation noise can erode trust in automatically generated bibliographies, pollute the scientific record, distort citation-based assessment, and redistribute scholarly credit in ways not justified by the underlying literature (Zhao et al., 8 May 2026).

1. Conceptual scope and formal boundaries

Bornmann and Leibel define citation noise as the undesirable variability in citation decisions that remains after controlling for systematic bias (Bornmann et al., 18 Aug 2025). In their formalization, each potential citation has a realized decision $R_{(ij)} \in \{0,1\}$, an accurate decision $A_{(ij)} \in \{0,1\}$, and an error indicator $E_{(ij)} = |R_{(ij)} - A_{(ij)}|$. Within this framework, citation accuracy concerns whether $R_{(ij)} = A_{(ij)}$, citation bias concerns systematic over- or under-citation of particular papers, and citation noise concerns the variance of citation errors across authors and papers. They distinguish citation-level noise as between-author variability and citation-pattern noise as within-author variability, combining them as $OSYS = \sqrt{OLN^2 + OPN^2}$ (Bornmann et al., 18 Aug 2025).

Subsequent work uses the term in more operational settings. Naser defines a hallucinated citation as one that cannot be matched with high confidence to any real publication in CrossRef, OpenAlex, or Semantic Scholar (Naser, 7 Feb 2026). The BibTeX benchmark for scientific publishing agents defines citation noise as any deviation of a generated BibTeX field from its true, publisher-deposited value, including fabrication, substitution, omission, and partial corruption (Rao et al., 3 Apr 2026). CiteCheck adopts a three-way taxonomy: Exact, Minor, and Major, where Minor denotes “metadata drift” on a real paper and Major denotes a citation that does not resolve to any sufficiently similar real work (Khajavi et al., 26 May 2026).

These definitions are not identical, but they are structurally compatible. A plausible implication is that citation noise is best understood as a family of distortions affecting different stages of the citation pipeline: citation decision, metadata production, source attribution, and bibliometric aggregation.

2. Field-level hallucination in LLM-generated references

Chen et al. study citation noise as field-level hallucination when an LLM is asked to generate references from parametric memory alone (Chen et al., 20 Apr 2026). Across nine models and over 108,000 structured references, each serialized into Title, Authors, Venue, Year, and DOI, they find a stable field-level error hierarchy: the author field is by far the most error-prone, followed by venue, then title $\approx$ year, while DOI correctness varies by model. At $N=15$, author correctness is below 14% of cases; title and year are approximately 27–41%; and Qwen2.5-32B reaches 53.5% DOI correctness. The ordering author $<$ venue $<$ title $\approx$ year holds universally across models and settings, which the paper interprets as a structural difficulty in recalling author lists from parametric memory. Citation style has no measurable effect: across eight formats, the maximum hallucination-rate difference is below 0.04 and all $A_{(ij)} \in \{0,1\}$0. Reasoning-oriented distillation is associated with markedly worse recall; a DeepSeek-R1-distilled Qwen2.5-14B variant falls to 0.4% overall at $A_{(ij)} \in \{0,1\}$1 (Chen et al., 20 Apr 2026).

A separate cross-model audit by Naser reports 69,557 citation instances from 10 commercially deployed LLMs across four academic domains (Naser, 7 Feb 2026). Using confirmation against CrossRef, OpenAlex, and Semantic Scholar, hallucination rates span a fivefold range, from 11.4% to 56.8%, and vary strongly by model, domain, and prompt framing. “Recent and influential” prompts produce 74.1% hallucination versus 55.0% for “seminal and foundational,” and domain differences are larger than framing differences. The unprompted control yields exactly 0 citations across 3,030 responses, which the paper states seems to establish hallucination as prompt-induced rather than intrinsic. Generational improvement is not monotonic: GPT-4o-mini to GPT-5-mini improves from 45.3% to 11.4%, whereas haiku-3.5 to haiku-4.5 worsens from 48.8% to 56.8%; within families, larger-capacity variants reduce hallucination (Naser, 7 Feb 2026).

A recurrent misconception is that formatting or style is the principal source of citation error. The reported evidence points elsewhere: errors are predominantly field-specific and model-specific, and the strongest distortions arise in metadata recall rather than in the external formatting of the reference (Chen et al., 20 Apr 2026).

3. Search-enabled agents, verification pipelines, and mitigation

Citation noise persists even when LLMs are augmented with web search. In a 931-paper benchmark spanning Artificial Intelligence, Medicine, Materials Science, and Quantum Computing, search-enabled frontier models generate 2,793 BibTeX entries scored on nine fields and a six-way error taxonomy (Rao et al., 3 Apr 2026). Overall field accuracy is 83.6%, but only 50.9% of entries are fully correct. Accuracy drops by 27.7 percentage points from popular papers to recent post-cutoff papers, indicating heavy reliance on parametric memory even when search is available. Field-error co-occurrence analysis identifies two dominant failure modes: wholesale entry substitution, where identity fields fail together, and isolated field error, where a single field such as pages, year, or author list is corrupted. The open-source tool clibib, which deterministically fetches BibTeX from the Zotero Translation Server with CrossRef fallback, improves a two-stage revision pipeline from 83.6% to 91.5% field accuracy and from 50.9% to 78.3% fully correct entries, with a regression rate of 0.8% (Rao et al., 3 Apr 2026).

CiteCheck addresses the verification problem directly through a hybrid retrieval-grounded pipeline (Khajavi et al., 26 May 2026). Candidate retrieval proceeds through a waterfall over arXiv, CrossRef, Semantic Scholar, OpenAlex, and optional web search, with normalized Levenshtein title similarity and ranking heuristics. A structured LLM verifier then compares parsed citation metadata against the retrieved candidate, assigns a numeric score $A_{(ij)} \in \{0,1\}$2, and maps it to Exact, Minor, or Major using thresholds $A_{(ij)} \in \{0,1\}$3 and $A_{(ij)} \in \{0,1\}$4. On a held-out physics test set of 792 citations, CiteCheck achieves 88.7 macro-F1 and 88.9% accuracy, outperforming GPT, Claude, and Gemini baselines, including web-search and few-shot variants (Khajavi et al., 26 May 2026).

Naser reports complementary low-overhead filters for pre-screening (Naser, 7 Feb 2026). Multi-model consensus yields 95.6% accuracy when more than three LLMs cite the same work, compared with 16.5% at $A_{(ij)} \in \{0,1\}$5. Within-model repetition yields 88.9% real citations when a citation recurs at least twice across three replications, compared with 28.6% at $A_{(ij)} \in \{0,1\}$6. A gradient-boosted classifier trained only on 27 bibliographic string features reaches AUC 0.876 in 5-fold cross-validation and 0.834 in leave-one-model-out generalization, suggesting that substantial signal is available in bibliographic strings alone (Naser, 7 Feb 2026).

Taken together, these results show that web access does not remove citation noise by itself. Reliable mitigation depends on grounding generated references against authoritative sources, separating retrieval from revision, and applying calibrated decision rules rather than relying on raw model confidence (Rao et al., 3 Apr 2026, Khajavi et al., 26 May 2026).

4. Internal representation and causal localization of hallucination

Chen et al. investigate whether citation hallucination is encoded internally rather than arising purely from decoding artifacts (Chen et al., 20 Apr 2026). For each bibliographic field, they extract hidden-state vectors over the token span of that field in Qwen2.5-32B-Instruct and train $A_{(ij)} \in \{0,1\}$7-regularized logistic probes layer by layer. The resulting layer-wise trajectories differ by field: author-hallucination AUC dips in layers 6–10 and then rises to 0.935 at layer 46; title AUC grows monotonically to 0.888 at layer 64; year declines steadily, with a reported Spearman correlation $A_{(ij)} \in \{0,1\}$8 and $A_{(ij)} \in \{0,1\}$9. Cross-field transfer fails: off-diagonal AUC is 0.46–0.59, whereas in-field AUC is 0.812–0.922. The paper concludes that each bibliographic field’s hallucination status resides in a distinct linear subspace of the model’s representations (Chen et al., 20 Apr 2026).

To localize the representation further, the study uses the Causal Effect via the Token-to-Top-of-the-residual-stream Transformation metric,

$E_{(ij)} = |R_{(ij)} - A_{(ij)}|$0

computed across approximately 1.77 million neurons per token and averaged over tokens in each field span (Chen et al., 20 Apr 2026). Field-wise elastic-net logistic regression is then fitted with

$E_{(ij)} = |R_{(ij)} - A_{(ij)}|$1

using $E_{(ij)} = |R_{(ij)} - A_{(ij)}|$2, followed by 20 bootstrap resamplings with 50% subsamples; only neurons selected in at least 60% of runs and with positive weight are kept. This produces sparse field-specific hallucination-neuron sets: 224 neurons for title, 78 for authors, 129 for year, 51 for venue, and 30 for DOI. The sets occupy different layer bands, including DOI neurons concentrated 66.7% in early layers and author neurons concentrated 60.3% in middle layers (Chen et al., 20 Apr 2026).

Causal intervention strengthens the interpretation. Amplifying identified neurons with $E_{(ij)} = |R_{(ij)} - A_{(ij)}|$3 sharply degrades the targeted field, including title accuracy from 76.3% to 4.7% and DOI from 54.6% to 38.1%; a Wilcoxon test across five fields gives $E_{(ij)} = |R_{(ij)} - A_{(ij)}|$4. Suppressing the same neurons with $E_{(ij)} = |R_{(ij)} - A_{(ij)}|$5 improves several fields, including title to 82.8% and authors to 23.7%, while random ablation degrades all fields, such as title from 76.3% to 59.9% (Chen et al., 20 Apr 2026). This suggests that citation noise is not merely an external symptom of generation, but is partially traceable to sparse, field-specific internal circuitry.

5. Warrant calibration and citation fidelity beyond fabricated references

Citation noise is not limited to invented references. In cited RAG, a citation can be real and topically relevant while still failing to warrant the attached wording. “Relevant Is Not Warranted” names this failure mode citation laundering and frames it as a problem of evidence-force calibration (Qian et al., 27 May 2026). The core constraint is

$E_{(ij)} = |R_{(ij)} - A_{(ij)}|$6

evaluated along five axes: relation, modality, scope, temporal validity, and numeric specificity. FORCEBENCH operationalizes the problem as a contrastive monotonicity test. For a fixed cited passage $E_{(ij)} = |R_{(ij)} - A_{(ij)}|$7, an evaluator should assign a higher score to the evidence-calibrated claim than to a minimally stronger force-raised variant:

$E_{(ij)} = |R_{(ij)} - A_{(ij)}|$8

A monotonicity violation rate is then defined as

$E_{(ij)} = |R_{(ij)} - A_{(ij)}|$9

On the fixed 198-pair locality-filtered evaluation set, citation presence is uninformative by design with $R_{(ij)} = A_{(ij)}$0 and $R_{(ij)} = A_{(ij)}$1. Token overlap and entity overlap still violate monotonicity on 32.8–36.4% of pairs. Across four model judges, generic support prompting yields aggregate $R_{(ij)} = A_{(ij)}$2, whereas force-aware prompting lowers it to 0.245 but remains imperfect (Qian et al., 27 May 2026).

A related but distinct line of work measures how much scientific information changes from source to citation. The fidelity pipeline of “The Noisy Path from Source to Citation” defines sentence-level citation fidelity as

$R_{(ij)} = A_{(ij)}$3

where $R_{(ij)} = A_{(ij)}$4 and 1 means “completely different claims” while 5 means “identical scientific claims” (Chen et al., 27 Feb 2025). Applied to approximately 13 million citation sentence pairs from S2ORC, fidelity is higher when authors cite papers that are more recent and intellectually close, more accessible, and when the first author has a lower H-index and the team is medium-sized. The paper reports a publication-gap coefficient of approximately $R_{(ij)} = A_{(ij)}$5 per year and identifies a telephone effect: when an intermediary paper cites the original with low fidelity, future papers that cite both the intermediary and the original also have lower fidelity to the original. In a matched quasi-experiment, treatment papers that cite both the original and an intermediary have mean fidelity lower by $R_{(ij)} = A_{(ij)}$6 relative to controls, and fidelity to the original correlates positively with the intermediary’s fidelity at $R_{(ij)} = A_{(ij)}$7 (Chen et al., 27 Feb 2025).

These studies broaden the concept of citation noise from metadata failure to epistemic distortion. A visible citation is therefore not sufficient for grounding: the citation may exist, may resolve correctly, and may still misstate either the evidential force of the cited passage or the scientific claim originally reported (Qian et al., 27 May 2026, Chen et al., 27 Feb 2025).

6. Scientific-record contamination, metric distortion, and governance

At population scale, Zhao et al. operationalize citation noise as the excess fraction of unmatched references above a stable pre-LLM baseline (Zhao et al., 8 May 2026). Auditing 111 million references across 2.5 million papers in arXiv, bioRxiv, SSRN, and PubMed Central, they estimate 146,932 hallucinated citations in 2025 alone. By August 2025, excess unmatched rates reach 0.39% in arXiv, 0.21% in bioRxiv, 1.91% in SSRN, and 0.27% in PubMed Central. The distribution is diffuse across many papers, but especially pronounced in fields with rapid AI uptake, in manuscripts with linguistic signatures of AI-assisted writing, and among small and early-career author teams. The statistical association with AI-assisted writing is positive both at subfield level, with correlation 0.441 and $R_{(ij)} = A_{(ij)}$8, and at paper level, where the coefficient on AI-usage intensity is positive at $R_{(ij)} = A_{(ij)}$9 in all four corpora (Zhao et al., 8 May 2026). Moderation and publication filters capture only a fraction of the problem: 78.8% of hallucinated citations on arXiv survive moderator screening, and 85.3% of those in bioRxiv preprints persist into journal publications (Zhao et al., 8 May 2026).

Citation noise can also be introduced by publisher and aggregator infrastructure rather than by authors or models. In online-only Springer Nature journals, incorrect citation association arises when article-number metadata are mishandled so that references intended for one article resolve to Article Number 1 of the relevant volume (Kriváchy, 3 Nov 2025). The distortion is visible across Crossref, OpenCitations, Semantic Scholar, and publisher websites. After year-and-journal normalization, Article 1 z-scores typically lie between +2 and +5, with $OSYS = \sqrt{OLN^2 + OPN^2}$0 often exceeding +50 citations and in some cases +200. In both Scientific Reports and Nature Communications, 5 of the 10 top-cited articles are Article 1s, even though Article 1s comprise less than 1% of published papers in those journals. A manual audit of Nature Communications Vol. 9 No. 1 found $OSYS = \sqrt{OLN^2 + OPN^2}$1 in archived 2022 data and $OSYS = \sqrt{OLN^2 + OPN^2}$2 in 2025 data. The anomaly appears with the 2011 API era, persists through at least October 2025, and is estimated to affect approximately 450,000 articles and approximately 1.5 million unique authors in the three largest affected journals alone (Kriváchy, 3 Nov 2025).

Mitigation proposals therefore span technical, editorial, and policy layers. Zhao et al. recommend automated reference verifiers, editorial dashboards flagging papers with more than 1% unmatched references, researcher training, bibliometric infrastructure hardening, and disclosure mandates for generative AI use (Zhao et al., 8 May 2026). Bornmann and Leibel emphasize aggregation and citation-decision hygiene, including written guidelines, citation justification tables, training for early-career researchers, post-publication correction, and AI-based tools that flag missing or mismatched citations (Bornmann et al., 18 Aug 2025). Across these literatures, the shared conclusion is that citation noise is not a single bug. It is a multi-level reliability problem that links author behavior, model internals, retrieval and verification architecture, publisher metadata pipelines, and evaluation norms.