Revisiting Generalization Across Difficulty Levels: It's Not So Easy (2511.21692v1)

Abstract: We investigate how well LLMs generalize across different task difficulties, a key question for effective data curation and evaluation. Existing research is mixed regarding whether training on easier or harder data leads to better results, and whether those gains come on easier or harder test data. We address this question by conducting a systematic evaluation of LLMs' generalization across models, datasets, and fine-grained groups of example difficulty. We rank examples in six datasets using the outputs of thousands of different LLMs and Item Response Theory (IRT), a well-established difficulty metric in educational testing. Unlike prior work, our difficulty ratings are therefore determined solely by the abilities of many different LLMs, excluding human opinions of difficulty. With a more objective, larger-scale, and finer-grained analysis, we show that cross-difficulty generalization is often limited; training on either easy or hard data cannot achieve consistent improvements across the full range of difficulties. These results show the importance of having a range of difficulties in both training and evaluation data for LLMs, and that taking shortcuts with respect to difficulty is risky.

• The paper demonstrates that LLM generalization sharply degrades as the training and testing difficulty bins diverge.
• It adopts the Rasch IRT model across six benchmarks and 5,000 models to reveal weak correlations between human-centric metrics and actual difficulty.
• The findings advocate for difficulty-diverse datasets and adaptive evaluation protocols to better capture and enhance LLM performance.

Revisiting Generalization Across Difficulty Levels in LLMs

Motivation and Research Context

The paper "Revisiting Generalization Across Difficulty Levels: It's Not So Easy" (2511.21692) performs a rigorous investigation into the ability of LLMs to generalize across task difficulty, a core question for both data curation and benchmark evaluation in LLM development. Previous literature posits conflicting claims regarding easy-to-hard and hard-to-easy generalization, and these discrepancies are compounded by reliance on subjective, human-derived difficulty metrics. The paper draws on Item Response Theory (IRT) to provide high-fidelity, model-centric difficulty estimates, enabling a fine-grained interrogation of cross-difficulty generalization using response data from thousands of distinct LLMs.

Methodology

The authors adopt the Rasch (1PL) IRT model to jointly infer example difficulty and model ability. Responses from over 5,000 models on six prominent benchmarks (ARC, GSM8K, MMLU-Pro, BBH, MATH, and MuSR) are scraped from the Open LLM Leaderboard, mitigating the scalability limitations of manual or small-model screening approaches. Each benchmark is partitioned into ten bins by increasing IRT-estimated difficulty. Finetuning is performed for several instruction-tuned LLMs (Qwen2.5 and Llama3 series) on each bin individually, followed by exhaustive cross-bin evaluation to quantify generalization away from the training difficulty.

Difficulty Estimation: Model-Centric vs Human-Centric Metrics

The paper demonstrates substantial divergence between IRT-based difficulty scores and human-centric metrics ("grade level", "question/answer length", "reasoning steps", and cognitive skill classifications). Correlation analysis on multiple datasets reveals universally weak alignment (generally $|\rho| < 0.3$); e.g., reasoning steps in GSM8K ($\rho = 0.49$) and answer length in MATH ($\rho = 0.56$) are rare exceptions (Figure 1, Figure 2). LLM-centric IRT scores often assign high difficulty to items humans label as "easy", and vice versa, supporting the position that human-based metrics are insufficient proxies for neural model difficulty.

Figure 3: IRT-based difficulty distributions showcase the disconnect between human-defined metrics and actual LLM response patterns, motivating the use of response-driven scoring.

Figure 1: Spearman correlation heatmaps between IRT scores and traditional human metrics underscore consistently weak or negative associations.

Cross-Difficulty Generalization Experiments

The main experimental contribution is an exhaustive cross-bin training/evaluation regime on six datasets and seven LLM variants. Models are SFT-trained on a single difficulty bin and subsequently evaluated across all bins, with their performance difference (vs zero-shot) visualized as heatmaps.

• Generalization Performance Degrades with Difficulty Gap: For all model-dataset pairs, generalization sharply deteriorates as the gap between training bin and testing bin widens. The dominant positive improvements are localized to the diagonal (train and test bins of similar difficulty), and off-diagonal elements trend toward neutral or negative results.

  Figure 4: Qwen2.5 14B cross-difficulty generalization heatmaps reveal sharply attenuated performance with increasing train-test bin separation.

  

  Figure 5: Accuracy improvements vs zero-shot baseline highlight the narrow window where SFT confers a generalization gain.

• Easy-to-hard Generalization Is Not Consistent: Models trained on easy data fail to maintain strong performance on hard bins—contradicting claims that easy data alone can recover hard task accuracy.
• Hard-to-easy Generalization Is Not Reliable: SFT on hard bins does not uniformly enhance easy-bin performance; in fact, detrimental transfer is often observed, especially in BBH and MMLU-Pro.

Supplementary experiments across all model scales and families (Qwen2.5 1.5B/3B/7B, Llama3.2 1B/3B, Llama3.1 8B) robustly reproduce these patterns (Figures 6-11), corroborating the non-specificity of this phenomenon.

Implications for Data Curation and Benchmarking

The results have several direct implications:

• Difficulty-Diverse Datasets Are Essential: Since generalization is strongly local in difficulty space, training sets must sample broadly from the entire difficulty distribution. Small, homogeneous slices are insufficient to cover the full range of real-world tasks.
• Difficulty-Aware Evaluation Protocols: Benchmark suites lacking coverage of "easy" and "hard" regimes cannot adequately characterize LLM performance under practical deployment.
• Danger of Human-Centric Difficulty Estimation: Reliance on grade level, cognitive taxonomy, or answer length can mask true model weaknesses (see Table 1 in the appendix).

Theoretical Considerations and Future Directions

The findings problematize widely held views on curriculum learning and data-centric LLM optimization. Existing approaches that select data by human-annotated difficulty may fail to surface learning bottlenecks; curriculum strategies using IRT-based difficulty partitions—potentially coupled with adaptive bin mixing—could yield more robust generalization. Moreover, the demonstration that difficulty transfer is weak even for state-of-the-art models underlines the need for architectural innovations or new regularization methods explicitly targeted at difficulty-invariant learning.

As LLMs are increasingly adopted for high-stakes applications requiring generalization outside curated distributions, the observed limited cross-difficulty transfer propels a reevaluation of current best practices.

Figures on Zero-Shot Difficulty Sensitivity

Figure 6: Zero-shot evaluations for Qwen3 models confirm monotonic degradation in accuracy with increasing IRT difficulty bin, validating the discrimination of the difficulty measure.

Conclusion

This paper rigorously demonstrates that LLMs exhibit restricted generalization across difficulty bins when trained solely on easy or hard examples. Cross-difficulty generalization is sharply local: as difficulty separation increases, transfer ability breaks down, frequently yielding performance losses even below baseline zero-shot levels. These trends are robust across datasets, model sizes, and architectures, and they problematize both data curation and evaluation paradigms predicated on human-centric difficulty metrics. The work motivates future research into difficulty-aware curriculum design, adaptive data selection, and new modeling strategies that operate independently of superficial difficulty proxies.