Even GPT-5.2 Can't Count to Five: The Case for Zero-Error Horizons in Trustworthy LLMs

Abstract: We propose Zero-Error Horizon (ZEH) for trustworthy LLMs, which represents the maximum range that a model can solve without any errors. While ZEH itself is simple, we demonstrate that evaluating the ZEH of state-of-the-art LLMs yields abundant insights. For example, by evaluating the ZEH of GPT-5.2, we found that GPT-5.2 cannot even compute the parity of a short string like 11000, and GPT-5.2 cannot determine whether the parentheses in ((((()))))) are balanced. This is surprising given the excellent capabilities of GPT-5.2. The fact that LLMs make mistakes on such simple problems serves as an important lesson when applying LLMs to safety-critical domains. By applying ZEH to Qwen2.5 and conducting detailed analysis, we found that while ZEH correlates with accuracy, the detailed behaviors differ, and ZEH provides clues about the emergence of algorithmic capabilities. Finally, while computing ZEH incurs significant computational cost, we discuss how to mitigate this cost by achieving up to one order of magnitude speedup using tree structures and online softmax.

• The paper introduces the Zero-Error Horizon (ZEH) metric to determine the largest problem size for dependable LLM performance.
• It demonstrates that GPT-5.2 exhibits low ZEH values, failing on simple tasks like parity, balancing parentheses, and multiplication at critical boundaries.
• The study proposes methodological optimizations—teacher forcing, prompt cache sharing, and trie-based management—to enable scalable ZEH evaluation.

Zero-Error Horizon: A Metric for Trustworthy LLM Evaluation

Motivation and Problem Statement

LLMs are increasingly deployed in high-stakes, safety-critical domains spanning healthcare, law, finance, and scientific discovery. Despite their advanced generative and reasoning abilities, contemporary LLMs like GPT-5.2 display stark inconsistencies: models capable of synthesizing complex simulation code may simultaneously fail on elementary algorithmic tasks such as parity checking, parenthesis balancing, or basic multiplication.

Figure 1: GPT-5.2 fails on computing the parity of "11000", assessing balance in "((((( ))))))", and calculating $127 \times 82$.

This paper introduces the Zero-Error Horizon (ZEH) as an operationally interpretable metric that quantifies the maximal problem size for which an LLM performs flawlessly on all instances, given a fixed prompt and deterministic greedy decoding. This construct directly exposes critical failure modes in otherwise highly competent models and reframes model trustworthiness around safety boundaries rather than aggregate accuracy.

Defining the Zero-Error Horizon (ZEH)

Formally, given a model, task (e.g., multiplication, parity, balanced parentheses), and prompt, ZEH is defined as the maximum problem size $n$ such that the model answers every instance of size $\leq n$ correctly. The smallest instance at size $n+1$ eliciting an error is called a ZEH limiter.

In contrast to classic accuracy—which is typically measured on an arbitrarily chosen test set—ZEH guarantees exhaustive correctness over all instances within a specified range and thus provides a strongly diagnostic trust boundary. For example, GPT-5.2's ZEH is 4 for binary string parity, as it fails on 5-bit input "11000"; it is 10 for balanced parentheses, failing on 11-character strings; and it is 126 for multiplication, failing on $127 \times 82$.

Figure 2: GPT-5.2-Thinking fails to correctly classify balance in "((((( ))))))".

ZEH versus Accuracy: Range Arbitrary, Stability, and Benchmark Obsolescence

ZEH exhibits several essential properties differentiating it from standard accuracy or other aggregate measures:

• Safety and Warning Boundaries: ZEH directly identifies problem sizes where zero-error performance is guaranteed and—importantly—where errors are known to manifest.
• Evidence via ZEH Limiters: Because every failure corresponds to a concrete, minimal failing input (the ZEH limiter), these serve as indisputable evidence for the failure boundary.
• Objectivity and Range Bias: ZEH is immune to cherry-picking or arbitrary benchmark ranges. In contrast, accuracy can be artificially inflated or deflated based on test set selection (e.g., comparing $n\leq 20$ versus $n\leq 99$ for multiplication).
• Sensitivity to Model Weakness: ZEH is highly sensitive, dropping sharply if even a single error is introduced at a smaller size, making it an excellent alarm for failures in safety-critical contexts.
• Longevity as a Metric: Whereas fixed-size benchmarks saturate and become obsolete for advanced models, ZEH is inherently adaptive, growing as models improve.

  Figure 3: Accuracy for Qwen2.5 varies by range selection, but ZEH provides an objective, range-free metric.

Strong Empirical Findings

The authors apply ZEH to evaluate GPT-5.2 and a spectrum of Qwen2.5-Instruct models (from 0.5B to 72B parameters) across arithmetic and algorithmic tasks. Key observations:

• GPT-5.2 ZEH is surprisingly low on simple problems: parity (4), balanced parentheses (10), and multiplication (126). LLMs making errors on such small and in-distribution instances highlights a severe risk for safety-critical deployment.
• Qwen2.5 scaling: Both ZEH and accuracy grow with model size, but ZEH is especially diagnostic of reasoning transitions. As model size increases, reasoning emerges from memorization (with random errors) to algorithmic (with errors biased towards instances involving carries or complex calculations).

Notably, error analysis shows:

• For small Qwen2.5 models, performance correlates with instance frequency in training corpora, indicating memorization.
• As size increases, errors become increasingly structured (e.g., off by multiples of 10), implicating algorithmic execution with human-like mistakes (such as mis-handling carries), rather than random guessing.

Computational Cost and Verification Efficiency

A challenge in ZEH evaluation is the exhaustive coverage required. The authors introduce optimizations to make ZEH computation practical:

• Teacher Forcing: Converts autoregressive output checking into more efficient parallel verification.
• Prompt Cache Sharing: Reuses prefixed computation across multiple problem instances.
• Trie/Tree Structure Sharing (FlashTree): Implements trie-based management of queries with optimized attention kernels, further boosting throughput by up to one order of magnitude over naive approaches.
• Batching Across Sizes: Maximizes hardware utilization via look-ahead processing.

These innovations enable ZEH evaluation at scale, though the authors signal that further advances—potentially leveraging formal verification techniques—are needed for future frontier-scale models.

Implications and Future Directions

ZEH offers a fundamentally different lens on LLM reliability, shifting focus from distributional performance to error-free operation guarantees on all instances up to a model-determined horizon. This has direct consequences:

• Safety-Critical Applications: ZEH provides actionable deployment thresholds, demarcating domains where LLM outputs can be trusted without exhaustive downstream verification.
• Interpretability and Debugging: ZEH limiters localize concrete failure modes, informing targeted model improvement and prompt engineering.
• Theoretical Insights: Systematic analysis of ZEH limiters enables empirical study of emergent algorithmic reasoning and memorization-algorithm transitions as model capacity grows.
• Benchmark Design and Policy: ZEH suggests best practice for LLM certification in regulated contexts, moving away from aggregate metrics to horizon-based guarantees.

The ZEH concept enlightens vital open research problems: how to extend such 'guaranteed correctness' principles to open-ended reasoning (beyond finite, exhaustively testable domains), how to formally relate ZEH to model interpretability and verification under prompt variation, and how ZEH can inform the co-development of LLMs and tool-use architectures.

Conclusion

ZEH is a salient, exhaustively checkable metric that exposes critical gaps in current LLM performance undetectable by conventional accuracy. It provides both trust boundaries for deployment and sharp diagnostic insight into the emergence of algorithmic behaviors. ZEH thus stands as a foundational addition to reliable LLM evaluation and deployment in high-stakes contexts.