Provable Failure of Language Models

• Provable Failure of Language Models are systematic limitations where LMs cannot fully capture deep linguistic properties or perform reliable reasoning.
• Empirical diagnostics show that despite meeting superficial statistical laws like Zipf and Heaps, LMs struggle with key long-range dependencies and semantic nuances.
• Theoretical analyses reveal that architectural and optimization constraints, including translation and reasoning barriers, fundamentally undermine LM performance.

Provable Failure of LLMs

Provable failure of LLMs refers to systematic and theoretically or empirically demonstrated limitations in their ability to capture essential properties of natural language, perform reliable reasoning, or transfer learned knowledge across contexts. These failures are not isolated errors but are driven by fundamental architectural, optimization, or methodological constraints, and can often be precisely quantified with mathematical tools and explicit benchmarks.

1. Scaling Properties and Memory: Quantitative Gaps in Natural LLMing

Comprehensive assessment of LLM (LM) failures requires moving beyond basic predictive metrics like perplexity to evaluate whether generated language exhibits deeper statistical regularities of human discourse. Scaling properties—statistical laws governing vocabulary population and long-range dependencies—constitute a principled framework for diagnosis (Takahashi et al., 2018).

Vocabulary Laws: Zipf and Heaps

• Zipf’s law ($f(r) \propto r^{-\alpha}$, typically $\alpha \approx 1$) and Heaps' law ($v(n) \propto n^{\beta}$, $0 < \beta < 1$) are satisfied by nearly all contemporary LMs, including $n$-grams, PCFGs, and neural models. These checks are necessary to rule out obviously unnatural vocabularies (e.g., pathological token frequency distributions) but offer little discriminative power for failure analysis.

Long Memory Properties

• Ebeling’s method (character-level fluctuation analysis), Taylor’s law (relating mean and standard deviation of word counts across segments; $\sigma \propto \mu^\zeta$), and long-range autocorrelation ($c(s) \propto s^{-\xi}$) identify whether models capture the long memory (long-range dependencies) observed in natural language.
• Empirically, neural LLMs (e.g., AWD-LSTM) partially reproduce long memory effects but with significant quantitative gaps. For Taylor’s law, natural language has $\zeta \approx 0.62$, while neural LMs are limited to $\zeta \approx 0.55–0.59$. Simpler models (e.g., $n$-grams, PCFGs, Pitman–Yor variants) perform no better than an i.i.d. baseline ($\zeta = 0.5$).
• Thus, even the most advanced LMs to date fail to fully model the global statistical structure of language, with scaling properties providing rigorous, interpretable diagnostics of these shortcomings.

Test	Natural Language	$n$-gram/PCFG	Neural LM
Zipf/Heaps laws	Yes	Yes	Yes
Taylor exponent $\zeta$	≈ 0.62	0.50	0.55–0.59
Long-range autocorr.	Sustained	Absent	Weakly present

2. Semantic and Theoretical Barriers: Limits of Meaning and Reasoning

Several works formally demonstrate that contemporary LMs are fundamentally incapable of acquiring or expressing certain semantic and logical notions, even in principle, due to inherent limitations of architecture or training paradigm (Merrill et al., 2021, Asher et al., 2023).

Ungrounded Understanding and Semantic Transparency

• Systems trained solely on ungrounded text (i.e., no extralinguistic input) cannot in general acquire “true” meaning. Formal models show that, unless languages satisfy strong semantic transparency, assertions alone (i.e., observable equivalence relationships between expressions) are insufficient for semantic emulation.
• In cases where meaning varies by context (e.g., variable binding, intensional semantics), recovering denotational semantics based solely on assertion oracles is uncomputable. Natural language, with its high degree of context sensitivity and world dependence, is especially prone to these provable failures (Merrill et al., 2021).

Universal Quantification and the Borel Hierarchy

• Formal semantics characterizes many critical properties (e.g., semantic entailment and consistency) via universal quantification over infinite sets—necessary, for example, to interpret quantifiers like “every” or logical planners (“always do X”).
• Theoretical analysis proves that LMs trained with standard maximum-likelihood methods cannot learn concepts beyond the first level of the Borel hierarchy. As a result, they systematically fail to capture entailments and deep linguistic generalizations required for robust reasoning (Asher et al., 2023).
• Empirical evaluation confirms that modern LMs (including those based on GPT-3.5, BERT, RoBERTa) cannot reliably judge sentences requiring universal quantification, and their failures scale with problem size (e.g., more colors or objects).

3. Optimization Barriers: Training Dynamics and Logical Function Learning

Even where model architectures can theoretically represent complex functions, the optimization process may block LMs from discovering them in practice (Chen et al., 7 Apr 2025).

• Transformer models belong to the class $\mathsf{TC}^0$, permitting the construction (with the right weights) of Boolean majority gates, among other logical primitives.
• However, when Transformers are trained via gradient descent—even with polynomially or exponentially many samples—the optimization dynamics are characterized by extremely low gradient variance for the relevant parameter space.
• Analytical bounds and coefficient-extraction arguments show that, for the $k$-majority function, the learning signal is exponentially small in the dimension $d$, and gradient-based updating cannot distinguish between correct and incorrect supports $S$ for the majority vote. Thus, the generalization error remains high, demonstrating a mismatch between expressivity and trainability (Chen et al., 7 Apr 2025).

4. Multilingual and Cross-Lingual Failures

LLMs display fundamental cross-lingual asymmetries in factual recall and knowledge transfer (Aggarwal et al., 25 Feb 2025, Bafna et al., 28 Jun 2025).

Language-Specific Factuality and Knowledge Transfer

• Systematic experiments over 10,000 country-related facts across 13 languages reveal that LMs encode knowledge in language-specific silos. For example, the fact “Rashed Al Shashai is from Saudi Arabia” may be correctly recalled in Arabic but missed in English or Swahili.
• Metrics such as Factual Recall Score (FRS), Knowledge Transferability Score (KTS), and Cross-Lingual Factual Knowledge Transferability (X-FaKT) quantify this failure, demonstrating persistent gaps in generalization from associative (high-resource) to non-associative (low-resource) languages (Aggarwal et al., 25 Feb 2025).

Translation Barrier Hypothesis

• Detailed layerwise analysis with the logit lens methodology reveals that, for multilingual generation, LLMs often solve tasks in an internal, language-agnostic space (often English) and then “translate” the answer to the target language only in late layers.
• For low-resource targets, the failure in the translation stage, not the task-solving step, accounts for a large portion of errors. Translation loss proportion (TLP) can reach 80–90% for certain language pairs, confirming the translation barrier as a major limit for end-to-end multilingual generation (Bafna et al., 28 Jun 2025).

5. Controlling and Explaining LM Outputs: Veracity, Truth Signals, and Instruction Hierarchies

The detection and explanation of LMs’ internal factual knowledge surfaces deficiencies affecting reliability, fact-checking, and controllability (Savcisens et al., 30 Jun 2025, Geng et al., 21 Feb 2025).

Veracity Probing and the Trilemma of Truth

• The sAwMIL methodology (Sparse Aware Multiple-Instance Learning with conformal prediction) probes the internal activations of LMs across layers to classify statements as true, false, or neither (Savcisens et al., 30 Jun 2025).
• Veracity signals are localized to the third quarter of the network’s depth and are not symmetrically distributed for truth and falsehood. Linear probes work best on chat models, whereas RLHF and knowledge distillation may necessitate nonlinear probes.
• The existence of a distinct “neither” class (neither true nor false) is critical for identifying hallucinations or unsupported statements. The ability (or inability) to reliably probe or manipulate these signals provides a statistical foundation for proving model failures in aligning internal knowledge with external factuality.

Instruction Hierarchy and Control Illusion

• When models are deployed with hierarchical instruction schemes (e.g., system vs. user constraints), they fail to consistently prioritize higher-level instructions, even in simple formatting conflicts. Metrics such as Primary Obedience Rate and Constraint Bias reveal substantial default biases and inherent prioritization failures (Geng et al., 21 Feb 2025).
• Controlled prompt engineering and fine-tuning produce marginal improvements, but do not remedy persistent inconsistencies in hierarchy enforcement, underscoring the need for architectural solutions.

6. Error Accumulation, Scaling Laws, and Reliability in Generation

Assumptions regarding error propagation and the limitations they imply require careful scrutiny. Recent evidence challenges the notion that reliability in LLM outputs collapses exponentially with sequence length (Arbuzov et al., 30 May 2025, Wang et al., 19 Dec 2024).

Key Tokens and Reliability

• Analysis of generation reveals that not all tokens contribute equally to global output correctness. Only a small fraction (“key tokens,” ~5–10%) act as semantic junctions whose accurate resolution is critical for coherent outputs. Non-key tokens are highly predictable and low-error under sufficient context (Arbuzov et al., 30 May 2025).
• Revised reliability formulas that separately account for error rates in key and non-key tokens explain the persistence of output quality in long generations, contradicting the basic exponential decay model $(1 - e)^n$.

Model Collapse under Data Scarcity

• When LLMs are iteratively retrained on their own generated outputs in contexts where real-world data is finite, mathematically inevitable collapse occurs. The recursive accumulation of non-negative estimation errors ($\alpha$ terms in the MLE) shifts the output distribution away from the original data, regardless of synthetic data volume capping (Wang et al., 19 Dec 2024).
• Only rigorous quality control, synthetic data filtering, and maintenance of authentic data anchors can mitigate this collapse.

7. Formal Detection Limits: Hardness of Distinguishing LM-Generated Text

The limits of provable detection of LM outputs point to inherent vulnerabilities in content verification (Varshney et al., 2020).

• Detection of LM-generated texts is formulated as a hypothesis test between genuine (distribution $P$) and generated (distribution $Q$) text. The optimal error exponent for such tests is the KL divergence $D(P\|Q) = H(P,Q) - H(P)$.
• As LMs improve and their perplexity $PPL(P,Q)$ decreases, the statistical “distance” between $P$ and $Q$ shrinks, reducing the efficacy of detection tests exponentially with sample size. This effect is robust to ergodic, stationary, and Markov approximations.
• Incorporation of semantic side information may improve detection, but mathematical integration of such cues remains unresolved.

Conclusion

The provable failure of LLMs is multifaceted: models fail to capture essential statistical properties of language, cannot reliably acquire or express deep semantic or logical properties, encounter optimization barriers in learning structured functions, and show systematic weaknesses in cross-lingual, factual, and controlled reasoning contexts. These failures can be sharply quantified through scaling laws, theoretical impossibility proofs, empirical evaluation of internal signals, and well-designed benchmarks. Addressing these limitations will require architectural innovation, better optimization strategies, integration of explicit reasoning or symbolic components, and principled evaluation frameworks that move far beyond surface-level accuracy.