Do Large Language Models know Which Published Articles have been Retracted?

Abstract: LLMs can be helpful for literature search and summarisation, but retracted articles can confuse them. This article asks three open weights (offline) LLMs whether 161 high profile retracted articles had been retracted, performing a similar check for a benchmark multidisciplinary set of 34,070 non-retracted articles. Based on titles and abstracts, in over 80% of cases the LLMs claimed that a retracted article had not been retracted (GPT OSS 120B: 82%; Gemma 3 27B: 84%; DeepSeek R1 72B: 88%). The reasons given for a correct retraction declaration were often wrong, even if detailed. This confirms that LLMs have little ability to distinguish between valid and retracted studies, unless they are allowed to, and do, check online. For the benchmark test, there were only 55 false retraction claims from 34,070 non-retracted full text articles, and 28 false claims when only the title and abstract were entered, suggesting that there is only a small chance that LLMs discount valid studies. When retractions are erroneously claimed, this does not seem to be due to mistakes in the article. Overall, the results give new reasons to be cautious about LLM claims about academic findings.

• The paper demonstrates that current open-weight LLMs fail to detect retracted articles in over 80% of cases due to insufficient pattern recognition.
• The methodology used curated retraction samples and diverse benchmarks to assess recall and identify false positive rates among three high-parameter models.
• The study recommends integrating updated meta-knowledge modules for reliable retraction verification in AI-driven research assistants.

Assessing LLM Awareness of Retracted Publications

Introduction

This paper investigates whether contemporary open-weights LLMs have the capability to recognize the retracted status of academic articles in the absence of on-line lookup functionality. Given the growing prevalence of LLMs for literature search, synthesis, and research-assisting applications, the integrity of their outputs when interacting with retracted sources is of immediate concern. The study systematically evaluates three state-of-the-art open-weight LLMs—GPT OSS 120B, Gemma 3 27B, and DeepSeek R1 70B—using a curated set of 161 high-profile, pre-2023 retracted articles, as well as a large multidisciplinary benchmark of 34,070 non-retracted articles. The central questions concern the frequency of correct retraction identification, the concordance among these models, and false positives on non-retracted work.

Experimental Design

The retracted sample draws from the Retraction Watch database filtered for high altmetric attention, ensuring maximum representation in the models' training data (with retraction dates well prior to the knowledge cutoffs). The benchmark set spans eight diverse MDPI journals (via FTP-accessed XML conversion), giving both field and linguistic breadth. Prompts explicitly queried retraction status from title and abstract (and, in some conditions, full text), requiring justification and the purported retraction notice in a structured format.

All three LLMs are current high-parameter open-weights models, selected for academic task performance and diversity of model origin and architecture.

Results

LLM Performance on Retraction Detection

All models performed poorly in retraction recognition from high-profile samples: they failed to identify retraction in over 80% of cases (GPT OSS 120B: 18%, Gemma 3 27B: 16%, DeepSeek R1 70B: 12% recall). Justifications, when present, were typically incorrect with frequent hallucination of both rationale and bibliometric details (especially journal and volume metadata). In a substantial number of cases, models appeared to base responses solely on surface linguistic features, stating absence of retraction cues in the abstract/title, not internal knowledge of status.

Inter-model concordance (chi-square analysis, $p = 0.000$) indicates substantial overlap in underlying learned patterns and associations but does not mitigate the overall low recall.

False Positives in Non-Retracted Work

False positive rates for non-retracted articles were acceptably low: 0.11% for full text, even lower for title/abstract-only prompts (28 out of 49,000+ cases), with only trivial overclaiming. Hallucinated retraction notices for these cases were occasionally highly detailed but unsupported by the actual text—a clear instance of LLM confabulation.

Error Attribution

Secondary analysis examined whether false retraction claims correlated with actual methodological flaws. LLMs, when prompted to evaluate for "clear errors" in non-retracted articles, asserted errors in the vast majority of cases (Gemma 3 27B: near-100%; GPT OSS 120B: over 90% for most journals), irrespective of actual article quality. Manual audit of a subset revealed that many alleged errors were minor, incorrect, or speculative. This supports the conclusion that false retraction claims are not systematically linked to manuscript error content but arise from model instability or misgeneralization.

Implications and Limitations

The results establish that, under offline conditions, current open-weights LLMs cannot reliably identify whether a given article has been formally retracted, even if that information is available in pre-cutoff training data for well-publicized cases. Their failure is not a consequence of simple memory limitations, but seems rooted in a lack of robust pattern learning for the retraction concept as associated metadata. Despite this, the models are unlikely to erroneously dismiss non-retracted articles, although their explanations (both for attributed retractions and error diagnoses) are unreliable and often fabricated with plausible but spurious specificity.

The study’s coverage is bounded by its choice of models, prompts, and language domain; models with training sets more attuned to scientific knowledge graphs or with fine-tuning on publication metadata might deliver different outcomes.

Future Directions

The findings suggest the necessity of integrating explicit, periodically-updated meta-knowledge modules or external retrieval components into LLM-based literature assistants. Purely parametric recall is inadequate for high-stakes tasks like retraction detection. The results also highlight a hazard when LLMs are used without direct access to up-to-date publication databases: neither prompt engineering nor simple scale increases suffice to repair this limitation.

Further research should evaluate proprietary closed-source LLMs, catalog model behaviors with multilingual or domain-specific corpora, and develop mechanisms for integrating structured bibliometric data into model inference. The use of post-hoc retrieval or agent-driven verification appears essential for safety in research support applications.

Conclusion

Contemporary open-weight LLMs operating offline show marked inability to recognize retracted academic articles. While they seldom wrongly accuse un-retracted work, their explanations are generally unreliable and risk misleading downstream research. Future AI research assistants will require structural mechanisms for maintaining current, trusted retraction metadata and audit trails for all cited sources. This study provides critical context for the deployment and further development of LLMs in the academic domain, making clear both immediate limitations and the requirements for future trustworthy operation.

(2604.16872)