Clustering-On-Difficulty in LLM Scaling

• The paper introduces COD as a framework that clusters evaluation tasks by their passrate vectors to capture scaling behavior for LLM performance prediction.
• It employs an improved MeanShift algorithm with horizontal smoothing and filtering to isolate a predictable subset of tasks with stable difficulty profiles.
• The method maps cluster-wise performance using a quartic function, offering an alternative to loss-based prediction with lower mean absolute error.

Clustering-On-Difficulty denotes a family of research ideas in which clustering is organized around some notion of difficulty, hardness, stability, or scaling behavior, rather than around similarity alone. In its most specific arXiv usage, the term refers to the Clustering-On-Difficulty (COD) framework for pre-training prediction of downstream performance in LLMs, where evaluation tasks are clustered by their difficulty scaling features and only the resulting predictable subset is used for stable extrapolation (Xu et al., 24 Feb 2025). Related literatures use the same phrase more loosely to describe procedures that cluster words into difficulty levels, cluster workers to disentangle subjectivity from difficulty, or reweight hard samples during representation learning rather than clustering the difficulty labels themselves (Liu et al., 2023). Across these usages, a common theme is that heterogeneity in hardness is treated as a first-class structural variable rather than as noise.

1. COD as a specific framework for LLM downstream scaling

In the most explicit technical sense, Clustering-On-Difficulty is the framework introduced for downstream performance prediction for LLMs before the largest model is fully trained, with the concrete goal of predicting the performance of a target 70B-parameter LLM using smaller models (Xu et al., 24 Feb 2025). The motivation is threefold. First, some tasks exhibit emergence phenomena, so small models may provide almost no useful signal. Second, a benchmark is not homogeneous: different samples within the same benchmark have different difficulty levels and different performance-compute curves. Third, training loss is an unreliable intermediate predictor because equal loss does not imply equal downstream performance.

COD addresses this by treating a benchmark as a mixture of task subsets with similar scaling behavior. For each evaluation sample, it constructs a difficulty feature vector from the task-wise passrate vector across small models, sorted by model size. These vectors are then clustered so that intra-cluster difficulty variance is small and cluster-wise performance-compute curves are more regular. The method explicitly removes zero-performance samples, namely tasks for which all small models fail even after 100 evaluation attempts, because their emergence point is unidentifiable and mixing them with nonzero tasks distorts cluster fitting (Xu et al., 24 Feb 2025).

The framework is organized into four stages: characterize task difficulty, cluster tasks by difficulty, fit cluster-wise performance-compute curves and filter predictable clusters, and map predictable-subset performance to full-benchmark performance. This architecture is intended to replace a single benchmark-wide extrapolation curve with a structured predictor that isolates the samples whose scaling behavior is stable enough to support extrapolation.

This specific usage distinguishes COD from broader difficulty-aware clustering research. COD does not cluster latent embeddings, images, or workers; it clusters evaluation tasks by how their passrates scale across model sizes. A plausible implication is that COD treats difficulty not as a static label but as a dynamical property revealed by scaling trajectories.

2. Difficulty representation, clustering procedure, and predictable subsets

The primitive signal in COD is passrate. Each small model is evaluated multiple times, and the mean accuracy over trials estimates the expected probability of a correct response. For a task, the resulting feature vector is the sequence of passrates across models of increasing size. This vector encodes how hard the task is initially, when emergence starts, how steeply performance improves, whether it saturates, and whether it behaves irregularly (Xu et al., 24 Feb 2025).

To reduce noise, the appendix introduces smoothing. COD mentions horizontal smoothing, in which the selected checkpoint and its immediately previous and next checkpoints are evaluated, with 100 evaluations per checkpoint, and the results are averaged. The paper states that they only adopt horizontal smoothing in experiments. Vertical smoothing is mentioned as optional but not used (Xu et al., 24 Feb 2025).

For clustering, COD argues against standard K-Means and DBSCAN. K-Means requires pre-specifying the number of clusters, while DBSCAN can produce connected high-density clusters with large intra-cluster variance. The method used is an improved MeanShift algorithm that constrains cluster diameter and enforces a minimum cluster size. The clustering uses a cluster radius $R$ and minimum cluster size $K$, and the paper explicitly states that it ensures the minimum number of questions in any cluster is not less than 10 (Xu et al., 24 Feb 2025).

The appendix gives the iterative procedure. All labels are initialized to $-1$; MeanShift with radius $R$ is applied to unlabeled samples; any sample farther than $R$ from its cluster center is reset to $-1$; any cluster with fewer than $K$ samples is also reset to $-1$; and this repeats until no label changes. Euclidean distance is the implied metric, because the paper explicitly remarks that “the Euclidean distance used for measurement does not consider this sequential information” when motivating vertical smoothing (Xu et al., 24 Feb 2025).

After clustering, COD fits a performance-compute curve for each cluster. Not all clusters are retained. A cluster is treated as scalable only if its fitted parameters satisfy

$a > 1,\quad b > 0.1,\quad 0 < c < 1.$

Clusters failing these constraints are treated as non-emergent, saturated, or irregular. The predictable subset is defined as the union of all samples in clusters that pass these filtering criteria (Xu et al., 24 Feb 2025).

This predictable subset is central to the method. COD does not attempt to predict the full benchmark directly from all samples. Instead, it first predicts a subset whose scaling behavior is regular enough to be modeled reliably, then learns a mapping from subset performance to full-set performance.

3. Performance law, mapping function, and theoretical support

COD adopts a simplified cluster-wise performance scaling law

$y(C)=g+(1-g)e^{-aC^{-b}-c},$

where $K$0 is training compute, $K$1 is the random guess performance floor, and $K$2, $K$3, and $K$4 are positive constants to be fitted (Xu et al., 24 Feb 2025). The paper also gives a fuller proposition in the appendix,

$K$5

and notes that the main-text version appears inconsistent with the appendix regarding the correction term. The appendix version uses $K$6, which the paper treats as the theoretically supported form (Xu et al., 24 Feb 2025).

The rationale for clustering emerges from an arithmetic-geometric mean argument. The paper derives that the discrepancy between arithmetic and geometric means grows with within-set variance, with

$K$7

This supports the claim that direct benchmark-wide scaling is harmed by difficulty variance, whereas low-variance clusters are more compatible with the exponential performance law (Xu et al., 24 Feb 2025). This suggests that clustering is not merely a heuristic preprocessing step but a variance-control device for the scaling law itself.

Cluster-wise predictions are aggregated into a predictable-subset score using cluster-size weights. Because the predictable subset excludes zero-performance, non-emergent, irregular, and saturated clusters, COD then maps this subset score to the full benchmark score. The mapping is assumed to be continuous, smooth on $K$8, monotonic increasing, and constrained to pass through $K$9 and $-1$0. The chosen form is the quartic polynomial

$-1$1

The final prediction is $-1$2, where $-1$3 is the predictable-subset prediction (Xu et al., 24 Feb 2025).

The mapping is calibrated using anchor models for which both predictable-subset and full-set scores are known. The paper uses Qwen2-72B as an out-of-distribution anchor and an in-house MoE model $-1$4 as an in-domain anchor, alongside the fixed endpoints $-1$5 and $-1$6 (Xu et al., 24 Feb 2025).

The empirical results are reported on eight evaluation sets: GSM8K, MATH, BBH, TriviaQA, MBPP, AGIEval, DROP, and MMLU-pro. The main table reports mean absolute prediction error 1.63 for COD (Complete), compared with 3.10 for end-to-end performance-compute prediction, 5.02 for passrate-compute prediction, 5.29 for loss-intermediate prediction, and 2.24 for COD (w/o mapping). The abstract separately claims 1.36% average error. The paper explicitly notes this discrepancy and states that a faithful summary is that the abstract claims 1.36%, while the main detailed benchmark table reports 1.63 percentage points mean absolute error across the eight benchmarks (Xu et al., 24 Feb 2025).

4. What COD is and is not

A common misconception is that COD is a generic recipe for clustering data points into “easy” and “hard” semantic classes. In the COD paper, that is not the task. The clustered objects are evaluation samples/tasks, and the clustering criterion is the task-wise passrate vector across small models, not semantic content or latent embedding similarity (Xu et al., 24 Feb 2025). The output is therefore a partition of a benchmark into groups with similar scaling behavior.

A second misconception is that COD predicts full-benchmark performance directly from cluster-wise scaling. The method does not do this. It first constructs a predictable subset, predicts its performance, and then applies the quartic mapping to obtain the full evaluation-set score (Xu et al., 24 Feb 2025). The mapping phase is not optional in the complete formulation; the paper shows that COD (w/o mapping) has higher mean and maximum error than COD (Complete).

A third misconception is that COD is simply a rebranding of training-loss extrapolation. The paper directly critiques loss-based prediction, arguing that training loss is mainly an in-domain compression metric and that equal loss can correspond to different downstream outcomes. COD is therefore positioned as a task-structured alternative to loss-only prediction (Xu et al., 24 Feb 2025).

The limitations are also explicit. COD depends on having enough test cases per benchmark, is not suitable for multiple-choice tasks where benchmark accuracy depends on ranking the correct option above others, has weak theoretical support for chain-of-thought prompting, and is sensitive to training-regime changes such as annealing with higher-quality data. The experiments are centered on dense transformers trained with the same data distribution and constant learning rate, while prediction on MoE families remains underexplored (Xu et al., 24 Feb 2025).

These limitations matter for interpretation. COD is not a universal law of model scaling; it is a benchmark-conditional procedure whose success depends on whether the benchmark contains enough scalable, non-irregular questions and whether small-model passrates reveal stable difficulty structure.

5. Broader meanings of clustering on difficulty in adjacent literatures

Outside the COD framework, several papers use related ideas in which difficulty conditions clustering, clustering reveals difficulty, or clustering is performed on a difficulty-derived scalar rather than on semantic similarity. The term therefore has a broader encyclopedic scope than the single LLM-scaling paper alone.

Work	What is clustered	Role of difficulty
COD (Xu et al., 24 Feb 2025)	evaluation tasks	clustered by difficulty scaling features
Wordle difficulty study (Liu et al., 2023)	words	hierarchical clustering based on expectation of guessing times
SDR crowdsourcing model (Jin et al., 2018)	workers	clustering emerges from latent preferences to separate subjectivity from difficulty
HaDis (Zhang et al., 2024)	semantic image categories	hard positives and negatives are mined and reweighted during clustering
DAICS (Sun, 2021)	multi-view samples in training schedule	samples are selected progressively from easy to difficult

In the Wordle study, the clustered objects are individual Wordle target words, and the paper states that it uses hierarchical clustering “based on the expectation of guessing times” (Liu et al., 2023). Difficulty is defined through the distribution and expectation of guessing times, with higher expected guesses implying higher difficulty. The actual clustering input is one-dimensional: the expected number of guesses for each word. The paper evaluates silhouette score, observes that three or four clusters look optimal, and chooses four categories for greater differentiation. These clusters are then interpreted as ordinal difficulty levels and modeled downstream using factor analysis and ordered logistic regression (Liu et al., 2023). This is a direct instance of clustering on a scalar difficulty summary, but it differs from COD because the feature is a corrected expectation rather than a scaling trajectory.

In crowdsourcing, the SDR model does not cluster questions directly by difficulty. Instead, it introduces an explicit difficulty parameter $-1$7 and models subjectivity through latent worker preferences. After inference, workers are clustered using K-means on posterior means $-1$8, with the number of clusters chosen by the Elbow method (Jin et al., 2018). The point of this clustering is to separate between-group systematic disagreement from within-group inconsistency, so that estimated difficulty is not confounded by genuine differences in opinion. The paper explicitly states that this is “a model of clustering to separate subjectivity from difficulty, not a direct clustering-on-difficulty method” (Jin et al., 2018).

In deep image clustering, HaDis is likewise not “clustering by difficulty” in the sense of making difficulty the cluster label. It is better described as difficulty-aware deep clustering. The method mines the hardest positive and hardest negative per anchor, adaptively weights them in a self-distillation framework, and combines this with diffused sampling alignment and prototypical contrastive learning (Zhang et al., 2024). The difficulty notion is primarily pair-level, derived from distances between teacher and student embeddings, and is used to reweight optimization rather than to produce difficulty clusters.

In multi-view clustering, DAICS makes difficulty a first-class training variable. It defines a binary difficulty label $-1$9 for each sample in each view, resolves inconsistent labels across views via an adversarial minimax game, and introduces sampling probabilities with the theorem

$R$0

which guarantees that easy samples have larger sampling probability than difficult ones (Sun, 2021). The model then trains progressively from easy to difficult and activates common-subspace learning once the selected sample count crosses a golden section threshold $R$1. This is difficulty-conditioned clustering, but again not the same as COD’s clustering of tasks by scaling behavior.

A plausible synthesis is that “Clustering-On-Difficulty” now names at least three distinct patterns in the literature: clustering objects by a difficulty representation, clustering auxiliary entities to improve difficulty estimation, and making clustering algorithms difficulty-aware during optimization.

6. Conceptual relations, theoretical analogues, and open problems

The broader clustering literature supplies two useful conceptual analogues. First, Ben-David’s survey on clusterability argues that tractability results for clustering typically require very strong structural separation, not merely “meaningful” data, and that most clusterability assumptions are not efficiently testable (Ben-David, 2015). Although this paper does not discuss difficulty as COD does, it raises a parallel issue: computational success often depends on identifying a structured subset of instances for which a model or algorithm is reliable. COD’s predictable subset can be read as one such operational compromise. This suggests that COD is part of a larger tradition of replacing worst-case heterogeneity with structured subfamilies that are stable enough to analyze.

Second, the diffusion-based MELD and M-LUND framework shows that a dataset may admit multiple clusterings visible at different diffusion times, with more stable clusterings persisting across longer intervals (Murphy et al., 2021). The paper formalizes this through time windows

$R$2

so that coarse, stable distinctions persist while finer, weaker distinctions disappear as diffusion proceeds. This suggests a useful analogy: COD’s predictable subset plays a role similar to selecting the scale regimes in which structure is stable enough to model.

Several open directions follow directly from the surveyed papers. COD itself leaves emergence-heavy, zero-performance, and irregular clusters to the mapping stage rather than modeling them directly (Xu et al., 24 Feb 2025). SDR notes that the model does not capture domain-specific expertise or cases where different worker groups find different objective question types hard (Jin et al., 2018). HaDis raises the question of modeling hardness more explicitly at multiple levels—instance, pair, prototype, and cluster—rather than from current mini-batch relations alone (Zhang et al., 2024). DAICS shows that difficulty can be view-dependent and inconsistent across modalities, suggesting that any general theory of clustering on difficulty may have to treat difficulty as relational rather than intrinsic (Sun, 2021).

Taken together, these works indicate that Clustering-On-Difficulty is not a single algorithmic doctrine but an emerging research pattern. In its narrowest meaning, it is the COD framework for LLM downstream prediction through clustering of task scaling trajectories (Xu et al., 24 Feb 2025). In a wider encyclopedic sense, it denotes methods that use clustering to expose, control, or operationalize heterogeneity in hardness. The literature supports a consistent conclusion: difficulty is often not an after-the-fact diagnostic but a structural variable that can change what should be clustered, how it should be clustered, and which subsets are sufficiently regular to support prediction or optimization.