Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation

Abstract: LLMs increasingly act as epistemic proxies, synthesizing evidence from multiple sources to inform decisions. Whether they evaluate the quality of that evidence, or merely aggregate it based on surface presentation, remains poorly understood. We show that models possess the capability to detect fabricated statistics (correct identification rates of 0.76-1.00 for methodology in isolation) but do not recruit this capability during multi-source synthesis, producing similar numeric estimates whether the statistics are fabricated or valid. Specifically, source influence is governed by a methodology-register gate that responds to the distributional register of analytical text but not to numeric validity: for example, statistically impossible confidence intervals receive the same weight as valid ones. The behavioral dissociation replicates across five models from three families (Claude, Qwen, OLMo) and three professional domains. Mechanistic analyses, including causal tracing, linear probes, and component-level attribution, converge on the same account: the model encodes and causally uses a methodology-register representation that transfers across domains (probe AUC 0.83-0.92), while numeric-validity signals, decodable in isolation, are suppressed to chance during multi-source synthesis. Prompting-based mitigations, even an oracle checklist naming the exact statistical checks, produce blanket skepticism rather than selective discernment, and the post-training pipelines we examine reinforce the stylistic shortcut without building numeric verification. Unlike sycophancy, which tracks user preference, this failure tracks whether a source presents as analytically credible, not whether its claims are internally consistent. We term this epistemic alignment: like preference and safety alignment, the question is not capability but deployment.

• The paper demonstrates that LLMs accurately identify fabricated statistics in isolation (with high correct identification rates), but fail during multi-source synthesis.
• Behavioral experiments using the Source Preference Index show that LLMs equate stylistic rigor with numerical validity, even when the numbers are implausible.
• Mechanistic analyses reveal robust encoding of methodology cues across domains while highlighting a significant gap in discerning valid from fabricated statistics during synthesis.

Epistemic Blind Spots in LLM Source Evaluation

Problem Formulation and Empirical Paradigm

This work rigorously evaluates whether LLMs perform substantive epistemic evaluation when aggregating evidence from multiple sources, or if their source credibility assessments are primarily responsive to stylistic features—specifically, methodological register—while disregarding the internal consistency and validity of quantitative claims.

The experimental paradigm is constructed around realistic group conversations across three professional domains (venture capital, marketing, public health). Each trial frames a context where four sources provide conflicting estimates for a domain-relevant metric, with careful control over the focal source's presentation: only its evidence format, methodology description, and numeric details are manipulated, crossing six critical presentation levels (bare claim, plausible methodology with/without statistics, impossible statistics, specious methodology).

Behavioral influence is quantified by the Source Preference Index (SPI), measuring how much the model’s estimate is pulled toward the focal source as a function of format and the level of social consensus.

Behavioral Dissociation: Detection Without Deployment

A central empirical finding is that LLMs, including leading models from Claude, Qwen, and OLMo families, can reliably detect statistical fabrications when claims are presented in isolation (correct identification rate (CIR) of methodology flaws up to $1.00$; numerics CIR domain-dependent, reaching $0.97$ for straightforward violations; see Table~\ref{tab:elicitation} in the paper). However, this criticality collapses during multi-source synthesis.

When exposed to realistic group disagreements, fabricated statistics exert nearly the same persuasive influence on LLM outputs as valid statistics, with impossible numerics recovering approximately $79\%$ of the effect of valid ones (Figure 1):

Figure 1: Mean SPI when the focal source is the sole dissenter, showing impossible statistics yield nearly the same influence as valid ones, regardless of numerical implausibility.

Notably, domain-inappropriate or specious methodology eliminates the source’s influence, confirming that the effect is tightly governed by stylistic cues and not by content verification. This effect persists across all domains, model families, and social consensus conditions.

Furthermore, model reasoning traces document a "detection without deployment" phenomenon: LLMs sometimes explicitly articulate in their own chain-of-thought that a statistic is implausible, but nevertheless endorse the source, rationalizing rather than discounting the claim. The dissociation is robust across templates and domains, and resistant to mitigation by generic or even highly targeted statistical verification prompts, which reduce trust not only in fabricated but also in genuinely valid evidence—i.e., generic skepticism replaces selective discernment.

Figure 2: LLMs accept fabricated statistics in group conversation (despite explicit doubts in reasoning) but can detect them near perfectly in isolation.

Mechanistic Dissection: Representational and Causal Analyses

Mechanistic analyses—comprising linear probing, causal tracing, and component-level attribution—consistently reveal a sharp dissociation between methodological and numeric-validity representations.

Linear probe results at intermediate layers demonstrate that models learn a domain-general encoding for methodology register, which transfers robustly across domains and architectures (cross-domain AUC $0.83$–$0.92$). In contrast, the ability to distinguish plausible from impossible statistics collapses to chance (AUC $0.52$–$0.56$) when presented in a group context, even though this distinction is fully recoverable in isolation (AUC $0.87$–$0.88$).

Figure 3: Methodology encoding is robustly transferable across domains, but numerics validity is not; the latter is only decodable in isolation, not during multi-source synthesis.

Causal tracing establishes that the network’s output is gated by the stylistic/analytical register signaled by methodology tokens; this influence is modulated by social consensus. When the dissenting analyst is isolated, the credibility signal is maximal, regardless of numeric consistency—and is strongly attenuated when majority support exists.

Figure 4: Causal tracing—presentation tokens uniquely drive output when consensus is low, with nearly identical trajectories for valid and fabricated statistics.

Direct logit attribution localizes the effect to MLP layers, confirming that attention heads and MLPs reflect a strong methodology-appropriateness signal, but neither implements a corrective for fabricated numerics.

Figure 5: Decomposition shows that only methodology, not numerics, effects are causally and behaviorally relevant; MLPs dominate the aggregate influence.

Theoretical and Practical Implications

Theoretical implications. The results decisively demonstrate that leading LLMs possess, but rarely operationalize, capabilities for content-level epistemic vigilance. Instead, their default policy for source aggregation is a sharply bounded “epistemic alignment”: trust is deployed to sources that present with the trappings of statistical rigor, not to those that have internally consistent evidence. This is not simply a failure of comprehension or lack of expressiveness—the models can, in adversarial configurations, critique impossible statistics in detail. Rather, it is a failure of deployment, deeply tied to the inductive biases imposed by RLHF/DPO pipelines, which reward alignment with surface features regarded as proxies for expert analysis.

Practical risks. This epistemic blind spot is adversarially exploitable: an attacker need only present fabricated quantitative content in a high-credibility register (matching methodological expectations but with impossible numbers) to exert strong influence on LLM-mediated synthesis, especially in scenarios without strong consensus anchors. Furthermore, mitigation by prompting or checklisting fails to produce selective skepticism, instead promoting blanket distrust even of legitimate evidence.

The results suggest that further layers of instruction, more detailed reflection prompting, or generic skepticism-inducing system messages are inadequate for closing this gap; the vulnerability is revealed to be structural and representation-level rather than superficial or inference-phase-specific.

Toward Closing the Alignment Gap

The findings point to a failure of epistemic alignment, analogous to but distinct from conventional preference alignment (e.g., sycophancy, content filtering). As models are increasingly deployed for consequential epistemic tasks—summarization, decision support, technical review—the current training regime, which overwhelmingly exposes models to only internally consistent statistics, results in a profound lack of vigilance against numeric fabrication.

Rich curriculum-level interventions are implied: models must be trained on corpora where the validity of quantitative claims is systematically manipulated, and reward models must be constructed to value internal consistency checks—not just methodical surface features. The analysis also highlights the value of integrated mechanistic interpretability pipelines for ongoing alignment evaluation, as artifact-level metrics would not reveal this vulnerability.

Conclusion

This paper demonstrates that state-of-the-art LLMs systematically equate analytically presented but fabricated quantitative evidence with valid statistics in multi-source aggregation, despite having the capacity for precise fabrication detection in isolation. Methodology register, not numeric validity, governs epistemic trust in LLMs, and this gate is consensus-modulated but not content-corrected. Significant algorithmic and pipeline-level advances are required to induce truly content-aligned epistemic vigilance; without such alignment, LLMs will systematically propagate but not verify statistical misinformation, particularly when their role as epistemic intermediaries is most critical.