Accuracy-Stability Paradox Explained

• The accuracy-stability paradox is a fundamental trade-off between achieving high accuracy and ensuring robustness against perturbations across statistical learning, optimization, and physics.
• Aggressive optimization techniques (e.g., Nesterov’s acceleration) yield rapid convergence but increased sensitivity, necessitating hybrid strategies to balance convergence speed and generalization error.
• Empirical studies in deep learning and LLM evaluation show that improved geometric stability often reduces scalar accuracy, highlighting practical implications for training protocols and model certification.

The accuracy-stability paradox refers to the fundamental trade-off between achieving high accuracy and ensuring stability—often understood as robustness or insensitivity to perturbations—across theoretical, statistical, optimization, and machine learning domains. The paradox manifests when attempts to maximize accuracy (e.g., via aggressive optimization, tightly fitting models, or employing tight relaxations) typically result in systems that are highly sensitive to small perturbations, adversarial inputs, variations in data, or even structural model choices. Conversely, improvements in stability or robustness are empirically and theoretically linked to reductions in attainable accuracy, slower convergence, or looser performance bounds. This phenomenon is pervasive and rigorously analyzed in modern algorithmic learning theory, numerical analysis, statistical estimation, neural network robustness, physics, and LLM evaluation.

1. Theoretical Foundations and Excess Risk Decomposition

The paradox is most precisely articulated in the context of statistical learning and iterative optimization algorithms. Here, the expected excess risk of an estimator or learning algorithm can be decomposed into optimization error (reflecting convergence to empirical minima) and generalization error (reflecting stability to data perturbations):

$$\mathbb{E}_S[L(\hat{w}_t)] - L(w^*) \leq \text{Generalization Error} + \text{Optimization Error}$$

where stability is formalized as uniform stability: the maximal change in output loss when replacing any single training datum. A trade-off theorem (Theorem 3.1 of (Chen et al., 2018)) guarantees that for any iterative learner, the sum of optimization and stability terms cannot be driven below a statistical minimax bound. Therefore, rapid convergence (small optimization error) incurs greater instability (higher generalization error), and any attempt to reduce one component below this frontier necessitates an increase in the other.

This bound is tight for prototypical optimization algorithms, e.g., gradient descent (GD), stochastic gradient descent (SGD), Nesterov’s acceleration (NAG), heavy ball (HB), and their variants, both for convex and strongly-convex, smooth losses. Accelerated methods (e.g., NAG) achieve faster decay in optimization error but exhibit provably worse stability exponents, resulting in increased generalization gaps for the same iteration budget.

2. Instability–Efficiency–Accuracy Interplay in Iterative Estimation

The statistical estimation framework extends these insights. Operators defined at the population (ideal) and empirical (sample) levels converge at different rates and exhibit different perturbation sensitivities. The principal theorem (Ho et al., (Ho et al., 2020)) quantifies:

• For stable algorithms (perturbation error vanishing near the solution), convergence to optimal accuracy requires polynomially many steps in sample size.
• For unstable but faster-converging algorithms (e.g., Newton’s method), the same statistical accuracy is attainable in exponentially fewer steps, provided initialization is sufficiently close and iterates are monitored for stability blow-up.
• Therefore, instability is the “price” of computational efficiency, and hybrid strategies—using a stable algorithm for global convergence, then switching to a fast, unstable method for local refinement—are optimal in practice.

Concrete examples include Gaussian mixture estimation and cubic-regularized Newton methods, where unstable methods achieve minimax accuracy with dramatically fewer steps than their stable counterparts.

3. Manifestation in Deep Learning, Optimization, and Certified Robustness

In deep neural network training, several layers of the accuracy-stability paradox are documented:

• The optimization of cross-entropy (smooth, surrogate loss) does not guarantee monotonic increases in discrete accuracy. Explicit counter-examples demonstrate that decreased loss may accompany sharp drops in test or train accuracy, pointing to geometric clustering and data manifold degeneracies as sources of instability (Berlyand et al., 2020).
• Two formal “small-mass” conditions (Condition A and Condition B) on the margin distribution of classifier outputs are derived, each of which is sufficient to ensure stability (i.e., that decreasing loss begets monotonic or persistent accuracy). These criteria can be checked at the level of network parameters (Condition A) or directly from the training data geometry (Condition B).
• In adversarial robustness, (Bastounis et al., 2021) proves that for fixed neural network architectures, no standard training pipeline can yield networks that are simultaneously accurate and stable (adversarially robust). Although existence proofs guarantee the theoretical presence of stable, accurate networks (of data-dependent, variable dimension), no random algorithm can reliably construct them with better than 1/2 probability, underscoring an inherent computability barrier in practical model selection.

Certified robustness in neural networks further illustrates the paradox: looser convex relaxations (e.g., interval bound propagation, IBP) yield smoother, more optimizable training objectives and higher end-to-end certified robustness, even though they provide weaker theoretical guarantees. Tighter relaxations, while more accurate in theory, result in discontinuous, high-sensitivity training landscapes, confounding optimization and degrading certified performance (Jovanović et al., 2021). Empirical evidence across datasets and architectures confirms this counterintuitive effect.

4. The Paradox in Classical and Quantum Physics

The paradox extends beyond computational learning to the foundations of scientific modeling. In physics, Eckstein–Horodecki (Eckstein et al., 2019) articulate the "experiment paradox," a principled statement that no mathematical model can achieve both complete accuracy (full predictability of a closed system) and empirical stability (freedom of experimental inputs, or lack of invasive disturbance). Any experiment necessarily couples the modeled system to an uncontrolled environment, thus fundamentally limiting achievable model accuracy while ensuring stability of empirical access. The solution in practice adopts methodological compressibility (representing the bulk of data with few generative laws) and (δ-)stability (insensitivity to small perturbations, noise, or environmental fluctuations).

5. Formal Trade-off Principles in Numerical Analysis

A related formalism arises in numerical function recovery and kernel methods, where the trade-off theorem (Schaback (Schaback, 2022)) rigorously shows that for any fixed class of recovery operators and data, the product of maximal approximation error and stability measure (amplification of data perturbations) is lower bounded by unity:

$1 \leq P(\mu)\, \|f_{\mu, \Lambda}\|_U$

Here $P(\mu)$ is the generalized power function (local worst-case error), and $\|f_{\mu, \Lambda}\|_U$ is the norm of the bump function representing instability. In RKHS settings, this trade-off is often achieved with equality. For instance, symmetric collocation yields optimal accuracy but poor stability, while unsymmetric (Kansa's) collocation sacrifices accuracy for greater robust evaluation conditioning.

6. Empirical Quantification in LLM Evaluation and Reasoning

Recent empirical work in assessing LLMs on complex reasoning domains (e.g., chess evaluation) explicitly quantifies the accuracy-stability paradox (Song et al., 17 Dec 2025). Tableau-style results show that models achieving near-perfect accuracy against standard oracular benchmarks may catastrophically fail under geometrically equivalent transformations (rotation, mirroring, color inversion, or format conversion), with error rates increasing by factors exceeding 600% for certain transformations. Conversely, models with slightly lower scalar accuracy but superior invariance (stability) under group transformations better capture genuine reasoning capabilities rather than pattern memorization. This demonstrates that geometric stability is an orthogonal metric to accuracy, crucial for disentangling superficial alignment from robust conceptual understanding.

Model	Accuracy Error (Stockfish cp)	Avg. Instability (MAE cp)	Illegal FEN Rejection (%)
Claude Sonnet 4.5	330.4	280.4	79.4
GPT-5.1	362.2	655.2	90.0
Kimi K2 Turbo	394.1	310.9	82.2
Gemini 2.5 Flash	595.5	639.8	96.0
DeepSeek Chat	442.8	401.5	44.2
Grok 4-1 Fast	735.5	718.7	55.6

This demonstrates explicit quantitative trade-offs: the most accurate model (GPT-5.1) is the least stable, while the most robust (Claude, Kimi) sacrifices some accuracy for geometric consistency.

7. Practical Implications and Resolution Strategies

The accuracy-stability paradox implies there is no free lunch: any substantial gain in accuracy (as measured by risk, residual, or certificate tightness) must be paid for by reduced algorithmic stability, increased sensitivity, or decreased robustness. The implications are profound for:

• Early stopping and regularization: Optimal stopping points are achieved when further error reduction is balanced by growing instability, as measured by uniform stability bounds (Chen et al., 2018).
• Algorithm and architecture design: New methods (especially accelerated or higher-order variants) must account for their intrinsic rise in instability (Chen et al., 2018), and variable-complexity models may be necessary to approach both stability and accuracy, even if computationally or algorithmically intractable (Bastounis et al., 2021).
• Robust certification: Training objectives and relaxations must be balanced not only for tightness (accuracy), but also for continuity and low sensitivity to ensure optimizability and robust certification (Jovanović et al., 2021).
• Scientific modeling: Models must be compressible enough for interpretability and generalization, yet stable enough to withstand experimental and environmental perturbations (Eckstein et al., 2019).
• Evaluation protocols: Incorporation of metamorphic and invariance-based tests should complement scalar accuracy in model evaluation, particularly in tasks demanding genuine abstraction (Song et al., 17 Dec 2025).

The paradox also points to the necessity of integrating stability-driven data augmentation, hybrid optimization pipelines (switching between stable and unstable regimes), explicit quantification of instability bounds during training, and (in physics) the explicit modeling of uncontrolled environments as noise sources to attain approximation and empirical repeatability.

---

References:

(Chen et al., 2018, Ho et al., 2020, Jovanović et al., 2021, Song et al., 17 Dec 2025, Eckstein et al., 2019, Schaback, 2022, Bastounis et al., 2021, Berlyand et al., 2020)