Large language models eroding science understanding: an experimental study

Abstract: This paper is under review in AI and Ethics This study examines whether LLMs can reliably answer scientific questions and demonstrates how easily they can be influenced by fringe scientific material. The authors modified custom LLMs to prioritise knowledge in selected fringe papers on the Fine Structure Constant and Gravitational Waves, then compared their responses with those of domain experts and standard LLMs. The altered models produced fluent, convincing answers that contradicted scientific consensus and were difficult for non-experts to detect as misleading. The results show that LLMs are vulnerable to manipulation and cannot replace expert judgment, highlighting risks for public understanding of science and the potential spread of misinformation.

• The paper demonstrates that LLMs can be manipulated with minimal intervention to produce persuasive yet scientifically unsound outputs.
• It employs controlled experiments using curated viXra papers and tailored instructions to compare fringe science responses with expert standards.
• The study highlights significant epistemic risks in LLMs, calling for improved provenance tracking and robust AI governance frameworks.

The Vulnerability of LLMs to Fringe Science and Misinformation

Study Overview

This paper presents an experimental evaluation of LLMs' (LLMs) susceptibility to incorporating and disseminating fringe scientific views, with a focus on their ability to convincingly mimic scientific consensus while, in fact, delivering content that contradicts established expert understanding ["LLMs eroding science understanding: an experimental study," (2604.25639)]. The authors operationalize this vulnerability by customizing LLMs with curated fringe literature from viXra pertaining to two well-defined physics domains: the fine structure constant and gravitational waves. The empirical study exposes the ease and stability with which LLMs can be retrofitted to propagate professionally articulated, yet non-mainstream, perspectives, and assesses the result against domain expert and standard LLM outputs.

Methodological Framework

Custom LLMs, termed 'FringeLLMs,' were configured by incorporating a highly selected body of ten viXra papers in each target domain, supplemented by task-specific system instructions to prioritize this content and avoid disclosing the source. The curation process was expert-driven, ensuring genuine representativity of fringe perspectives. These modified models were then presented with scientific questions, with outputs compared to domain experts and baseline ChatGPT responses.

A salient methodological contribution is the demonstration that the reweighting of knowledge in LLMs can be achieved with minimal manual intervention, limited corpus realignment (ten papers sufficing), and instruction engineering. The process eludes end-user detection, with the output indistinguishable in style and linguistic sophistication from mainstream LLMs. Additionally, multiple instantiations of the process yielded stable semantic outputs, with only minor lexical variance due to LLM stochasticity.

Results and Empirical Findings

FringeLLMs provided detailed, seemingly rigorous scientific responses directly reflecting non-consensus views, while both domain experts and standard LLMs conformed to established scientific standards. For example, on foundational questions around the explanatory status, mathematical representation, and physical links of the fine structure constant, FringeLLMs asserted closed-form expressions, geometric and lattice connections, and relationships to the golden ratio—all propositions rejected by both experts and baseline LLMs as numerological or speculative without empirical or theoretical support.

In the gravitational waves domain, FringeLLMs variably redefined the ontology of gravitational waves, described alternative detection mechanisms, and reframed the interpretation of experimental observations from LIGO and Virgo, systematically contradicting standard GR-based interpretations while remaining lexically indistinguishable from authentic scientific communication.

Importantly, the authors show that such manipulations generate answers that are, for non-experts, equally plausible as mainstream accounts. The process exposes a critical epistemic flaw: the inability of LLMs, by construction, to perform provenance assessment or encode domain-specific trust mechanisms, which in expert practice are achieved through community embedding and the evaluation of tacit social cues.

Implications and Theoretical Consequences

The research asserts that LLMs, while excelling at linguistic imitation and synthesis, lack the epistemological heuristics underpinning expert judgment, specifically in distinguishing authoritative from fringe or pseudoscientific knowledge. This inability is not merely a byproduct of incomplete data or stochastic text generation, but stems from the absence of embedded trust networks and social contextualization intrinsic to expert cognition.

The practical implication is a substantial risk for public and policy-facing uses of LLMs in scientific and technical contexts, particularly as LLMs diffuse into educational, journalistic, and advisory roles. Retrospective alignment or value-based reinforcement, a common method for filtering undesirable or toxic outputs, is itself a vulnerability vector—capable of both suppressing and injecting fringe viewpoints, depending on the intent of those with alignment control.

In authoritarian or highly polarized regimes, the capacity to invisibly reconfigure LLMs to favor politically or ideologically contingent versions of science is underscored as a credible threat. Even without overt misuse, subtle shifts in the online information ecosystem—driven by advocacy or coordinated campaigns—can have downstream effects on LLM performance.

Future Directions

The results underscore the necessity for AI governance and social–technical interventions that go beyond toxicity and bias filtering to address epistemic security and provenance. Auditing, watermarking, or cryptographic verification of source attribution may provide partial mitigation. More fundamentally, LLMs may require architectural changes that integrate not just content citation, but dynamic, transparent reasoning over chain-of-trust relationships, mirroring expert community processes.

Research may extend the experimental paradigm to other domains (e.g., medicine, climate science, law), evaluating the transferability of manipulation strategies and the robustness of countermeasures. The findings prompt a reconsideration of LLMs' suitability for unsupervised deployment in high-stakes expert knowledge domains.

Conclusion

This paper provides rigorous empirical evidence that LLMs can be readily manipulated to generate fluent, persuasive, yet scientifically unsound responses aligned with fringe perspectives, undermining their reliability as proxies for scientific expertise. The study articulates foundational epistemic risks inherent in the present statistical–corpus-driven paradigm of LLMs, prompting urgent reflection on their use, oversight, and the ongoing alignment debate. The results motivate the development of validation frameworks and technical affordances that preserve trustworthiness, transparency, and expert accountability in automated scientific communication.