LPDS: Evaluating LLM Robustness Through Logic-Preserving Difficulty Scaling

Abstract: As LLMs are increasingly deployed to perform tasks with minimal human oversight, it is crucial that these models operate robustly. In particular, a model that can solve a given problem should not fail simply because certain entities$\unicode{x2013}$such as names, numbers, or other contextual details$\unicode{x2013}$have changed while the underlying problem logic remains the same. Prior work suggests that current LLMs still struggle with this form of robustness: they often succeed on some variations of a problem but fail on others. However, existing evaluations often lack a systematic way to identify which logic-preserving variations are most likely to induce failure. Instead, they typically test a random subset of allowable variations, which can overstate robustness. To address this gap, we introduce logic-preserving difficulty scaling (LPDS), a framework that (i) quantifies the difficulty of a problem variation and (ii) systematically searches the space of allowable variations to find those that maximize difficulty and expose failures. We show that as difficulty increases, performance declines and errors in the models' reasoning chains become more pronounced. We further demonstrate that LPDS efficiently finds difficult problem variations for a model, resulting in performance drops up to 5 times larger compared to random sampling. Finally, we show that fine-tuning on more difficult variations leads to more consistent robustness gains than training on easier ones.

• The paper introduces LPDS, which combines response-distance metrics—especially hidden-state Mahalanobis distance—with beam search to find logic-preserving variations that are difficult for specific language models.
• LPDS reveals substantially greater brittleness than random sampling, producing accuracy drops up to five times larger and reducing median beam accuracy to 0% after six search iterations on GSM-Symbolic.
• The paper shows that difficulty-aware fine-tuning improves robustness, with high-difficulty training data achieving up to 88.18% accuracy versus 86.16% for random-mixture training, while transfer beyond symbolic templates remains open.

Motivation and problem statement

Symbolic templates such as those in GSM-Symbolic, FinChain, and EngTrace define a problem's variables, constraints, and full reasoning procedure while parameterizing names, numbers, and contextual details. Instantiating such a template yields many logic-preserving variations that share the same reasoning graph, making them a natural instrument for measuring robustness to superficial change. The standard practice, however, is to evaluate models on a randomly sampled subset of these variations. Because the space of allowable substitutions is typically enormous, random sampling can miss the variations most likely to induce failure and therefore overstate robustness. The paper introduces logic-preserving difficulty scaling (LPDS), a framework that (i) quantifies how difficult a given variation is for a specific model and (ii) searches the template's variation space via beam search to find maximally difficult instances (2605.15393).

Quantifying variation difficulty

The authors first establish metrics that predict whether a model will answer a variation correctly. They generate 1,000 variations per template across 100 templates from each of three datasets (GSM-Symbolic, FinChain, EngTrace), yielding 100,000 problems per dataset, and evaluate seven instruction-tuned models (Llama-3.2-3B/3.1-8B/70B, Qwen-2.5-7B/32B/72B, Phi-4) with five-shot chain-of-thought prompting and greedy decoding.

Two reference-based distance metrics prove most predictive:

• Minimum Levenshtein distance ($\mathrm{LD}_{\min}$): the normalized edit distance between the model's response and the closest of $N=200$ correct responses to other variations from the same template.
• Mahalanobis distance ($\mathrm{MD}_{\mathcal{H}}$): the distance between the average hidden state of the response (taken roughly two-thirds through the network) and a Gaussian fitted to hidden states of correct responses.

For Llama-3.1-8B-Instruct, $\mathrm{MD}_{\mathcal{H}}$ achieves micro-averaged AUCs of 0.80 (GSM-Symbolic), 0.74 (FinChain), and 0.78 (EngTrace), with odds ratios as low as 0.14 and 0.02 — meaning a one-standard-deviation increase in difficulty reduces the odds of a correct answer by 86% and 98%, respectively. These substantially outperform output perplexity (AUC 0.44–0.48) and self-certainty (AUC 0.54–0.56), and input-only metrics are weaker still. Qualitative analysis shows that as $\mathrm{MD}_{\mathcal{H}}$ increases, responses progress from correct solutions to minor arithmetic slips to severe misinterpretations of the problem structure, so the metric also ranks failures by severity. A practical caveat: when the reference set must come from another model or ground-truth traces rather than the evaluated model itself, predictive power drops moderately (e.g., Llama-3.1-8B's AUC falls from 0.783 to 0.700 with ground-truth references), suggesting solution strategies are partly model-specific.

Difficulty scaling via beam search

LPDS then performs beam search over slot substitutions. Starting from a base variation, each iteration expands beam nodes by replacing one variable slot at a time, scores candidates with a cheap input-embedding Mahalanobis approximation $\tilde f$, prunes with a mix of top-scoring and random candidates (exploration ratio $\rho_{\text{expl}} = 0.2$, selection ratio $\rho_{\text{sel}} = 0.4$), rescores survivors with the exact $\mathrm{MD}_{\mathcal{H}}$, and keeps the top-$w$ ($N=200$0, $N=200$1 iterations). Reference distributions are refreshed every 50 newly solved variations during search.

The search behaves as intended: beam difficulty rises monotonically while accuracy on beam members falls, reaching 0% median accuracy after only six iterations across GSM-Symbolic templates. This efficiency means LPDS exposes failures at a fraction of the cost of exhaustive evaluation.

Exposing brittleness beyond random sampling

Evaluating models on all variations explored by beam search reveals performance drops relative to base problems up to five times larger than under matched random sampling. On GSM-Symbolic versus GSM8K baselines: Llama-3.2-3B-Instruct drops 17.6% under LPDS versus 9.7% under random sampling; Qwen-2.5-72B-Instruct drops 9.0% versus 2.2%; Llama-3.1-70B-Instruct is most robust at 2.3%. Per-template error rates are higher under beam search for nearly all templates, and empirical accuracy distributions over repeated subsampling are left-shifted and broader under LPDS, indicating lower and less consistent performance than random sampling suggests.

Grouping explored variations into 20 quantile bins of increasing difficulty yields reliability curves showing monotonic accuracy decline, but with distinct robustness profiles: Qwen-2.5-7B stays stable through quantile 14 before dropping; Llama-3.1-8B degrades earlier; in the hardest bin, accuracies fall to 18.19% (Qwen-7B), 11.25% (Llama-8B), and 42.45% (Llama-70B). A normalized area-under-curve "difficulty-robustness score" summarizes these profiles (e.g., 0.90 for Llama-70B vs. 0.71 for Llama-8B on GSM-Symbolic; 0.24–0.30 on EngTrace, which proves hardest overall). Notably, similarly sized Qwen models are more robust than Llama counterparts, an observation independent of raw benchmark accuracy.

Difficulty-aware fine-tuning

Finally, the authors fine-tune Llama-3.1-8B-Instruct via SFT on ground-truth reasoning traces of initially unsolved variations, split into low/mid/high difficulty terciles. Training on harder subsets produces more consistent gains: on GSM-Symbolic test subsets, training on $N=200$2 reaches 73.68%/85.71%/85.71% on low/mid/high-difficulty tests, whereas training on $N=200$3 reaches 78.20%/60.15%/36.84%. Mixtures containing substantial high-difficulty data perform best overall (up to 88.18% on the full test set with a 20/30/50 low/mid/high mixture), exceeding random-mixture training (86.16%). The same pattern holds on FinChain. This indicates the difficulty estimate is useful not only for evaluation but also for curriculum-style data selection.

Limitations and open questions

The framework depends on symbolic templates that expose explicit slots, constraints, and ground-truth reasoning graphs; the authors state plainly that whether LPDS transfers to search spaces beyond such templates remains open. The difficulty metrics require a reference set of correct responses, which is unavailable when a model solves no variation of a template (one GSM-Symbolic and ten FinChain templates were excluded for this reason); cross-model or ground-truth references mitigate but weaken the signal. The fine-tuning experiments cover a single model and two datasets, and the robustness gains are measured on variations drawn from the same search procedure used to construct training data, so generalization to independently sampled hard variations is not directly established.

Conclusion

LPDS provides a systematic alternative to random sampling for template-based robustness evaluation: it defines validated difficulty metrics ($N=200$4, $N=200$5), uses two-stage beam search to efficiently locate failure-inducing logic-preserving variations, quantifies brittleness through difficulty-stratified reliability curves, and demonstrates that difficulty-ranked data improves fine-tuning outcomes. Its main open question is applicability beyond symbolic-template search spaces.