---
title: Clustering-On-Difficulty in LLM Scaling
url: https://www.emergentmind.com/topics/clustering-on-difficulty
type: topic
---

# Clustering-On-Difficulty in LLM Scaling

Clustering-On-Difficulty denotes a family of research ideas in which clustering is organized around some notion of difficulty, hardness, stability, or scaling behavior, rather than around similarity alone. In its most specific arXiv usage, the term refers to the **Clustering-On-Difficulty (COD)** framework for pre-training prediction of downstream performance in large language models, where evaluation tasks are clustered by their **difficulty scaling features** and only the resulting **predictable subset** is used for stable extrapolation [2502.17262]. Related literatures use the same phrase more loosely to describe procedures that cluster words into difficulty levels, cluster workers to disentangle subjectivity from difficulty, or reweight hard samples during representation learning rather than clustering the difficulty labels themselves [2305.03502]. Across these usages, a common theme is that heterogeneity in hardness is treated as a first-class structural variable rather than as noise.

## 1. COD as a specific framework for LLM downstream scaling

In the most explicit technical sense, Clustering-On-Difficulty is the framework introduced for **downstream performance prediction for large language models before the largest model is fully trained**, with the concrete goal of predicting the performance of a target **70B-parameter LLM** using smaller models [2502.17262]. The motivation is threefold. First, some tasks exhibit **emergence phenomena**, so small models may provide almost no useful signal. Second, a benchmark is not homogeneous: different samples within the same benchmark have different difficulty levels and different performance-compute curves. Third, training loss is an unreliable intermediate predictor because equal loss does **not** imply equal downstream performance.

COD addresses this by treating a benchmark as a mixture of task subsets with similar scaling behavior. For each evaluation sample, it constructs a **difficulty feature vector** from the task-wise passrate vector across small models, sorted by model size. These vectors are then clustered so that intra-cluster difficulty variance is small and cluster-wise performance-compute curves are more regular. The method explicitly removes **zero-performance samples**, namely tasks for which all small models fail even after **100 evaluation attempts**, because their emergence point is unidentifiable and mixing them with nonzero tasks distorts cluster fitting [2502.17262].

The framework is organized into four stages: characterize task difficulty, cluster tasks by difficulty, fit cluster-wise performance-compute curves and filter predictable clusters, and map predictable-subset performance to full-benchmark performance. This architecture is intended to replace a single benchmark-wide extrapolation curve with a structured predictor that isolates the samples whose scaling behavior is stable enough to support extrapolation.

This specific usage distinguishes COD from broader difficulty-aware clustering research. COD does not cluster latent embeddings, images, or workers; it clusters **evaluation tasks** by how their passrates scale across model sizes. A plausible implication is that COD treats difficulty not as a static label but as a dynamical property revealed by scaling trajectories.

## 2. Difficulty representation, clustering procedure, and predictable subsets

The primitive signal in COD is **passrate**. Each small model is evaluated multiple times, and the mean accuracy over trials estimates the expected probability of a correct response. For a task, the resulting feature vector is the sequence of passrates across models of increasing size. This vector encodes how hard the task is initially, when emergence starts, how steeply performance improves, whether it saturates, and whether it behaves irregularly [2502.17262].

To reduce noise, the appendix introduces smoothing. COD mentions **horizontal smoothing**, in which the selected checkpoint and its immediately previous and next checkpoints are evaluated, with **100 evaluations per checkpoint**, and the results are averaged. The paper states that **they only adopt horizontal smoothing** in experiments. Vertical smoothing is mentioned as optional but not used [2502.17262].

For clustering, COD argues against standard **K-Means** and **DBSCAN**. K-Means requires pre-specifying the number of clusters, while DBSCAN can produce connected high-density clusters with large intra-cluster variance. The method used is an **improved MeanShift algorithm** that constrains cluster diameter and enforces a minimum cluster size. The clustering uses a cluster radius \(R\) and minimum cluster size \(K\), and the paper explicitly states that it ensures the **minimum number of questions in any cluster is not less than 10** [2502.17262].

The appendix gives the iterative procedure. All labels are initialized to \(-1\); MeanShift with radius \(R\) is applied to unlabeled samples; any sample farther than \(R\) from its cluster center is reset to \(-1\); any cluster with fewer than \(K\) samples is also reset to \(-1\); and this repeats until no label changes. Euclidean distance is the implied metric, because the paper explicitly remarks that “the Euclidean distance used for measurement does not consider this sequential information” when motivating vertical smoothing [2502.17262].

After clustering, COD fits a performance-compute curve for each cluster. Not all clusters are retained. A cluster is treated as scalable only if its fitted parameters satisfy
\[
a > 1,\quad b > 0.1,\quad 0 < c < 1.
\]
Clusters failing these constraints are treated as non-emergent, saturated, or irregular. The **predictable subset** is defined as the union of all samples in clusters that pass these filtering criteria [2502.17262].

This predictable subset is central to the method. COD does not attempt to predict the full benchmark directly from all samples. Instead, it first predicts a subset whose scaling behavior is regular enough to be modeled reliably, then learns a mapping from subset performance to full-set performance.

## 3. Performance law, mapping function, and theoretical support

COD adopts a simplified cluster-wise performance scaling law
\[
y(C)=g+(1-g)e^{-aC^{-b}-c},
\]
where \(C\) is training compute, \(g\) is the random guess performance floor, and \(a\), \(b\), and \(c\) are positive constants to be fitted [2502.17262]. The paper also gives a fuller proposition in the appendix,
\[
\mathbb{E}_p[\mathrm{Acc}(C)]=g+(1-g)\left(e^{-aC^{-b}-c} + \frac{\sigma^2}{2\mu}\right)+o(\mu),
\]
and notes that the main-text version appears inconsistent with the appendix regarding the correction term. The appendix version uses \(\frac{\sigma^2}{2\mu}\), which the paper treats as the theoretically supported form [2502.17262].

The rationale for clustering emerges from an arithmetic-geometric mean argument. The paper derives that the discrepancy between arithmetic and geometric means grows with within-set variance, with
\[
\Delta = \mu_a-\mu_g \approx \frac{\sigma^2}{2\mu_a}.
\]
This supports the claim that direct benchmark-wide scaling is harmed by difficulty variance, whereas low-variance clusters are more compatible with the exponential performance law [2502.17262]. This suggests that clustering is not merely a heuristic preprocessing step but a variance-control device for the scaling law itself.

Cluster-wise predictions are aggregated into a predictable-subset score using cluster-size weights. Because the predictable subset excludes zero-performance, non-emergent, irregular, and saturated clusters, COD then maps this subset score to the full benchmark score. The mapping is assumed to be continuous, smooth on \([0,1]\), monotonic increasing, and constrained to pass through \((0,0)\) and \((1,1)\). The chosen form is the quartic polynomial
\[
f(x)=\alpha_1x^4+\alpha_2x^3+\alpha_3x^2+(1-\alpha_1-\alpha_2-\alpha_3)x.
\]
The final prediction is \(p=f(\hat T')\), where \(\hat T'\) is the predictable-subset prediction [2502.17262].

The mapping is calibrated using anchor models for which both predictable-subset and full-set scores are known. The paper uses **Qwen2-72B** as an out-of-distribution anchor and an in-house **MoE model \(M\)** as an in-domain anchor, alongside the fixed endpoints \((0,0)\) and \((1,1)\) [2502.17262].

The empirical results are reported on eight evaluation sets: **GSM8K**, **MATH**, **BBH**, **TriviaQA**, **MBPP**, **AGIEval**, **DROP**, and **MMLU-pro**. The main table reports mean absolute prediction error **1.63** for **COD (Complete)**, compared with **3.10** for end-to-end performance-compute prediction, **5.02** for passrate-compute prediction, **5.29** for loss-intermediate prediction, and **2.24** for **COD (w/o mapping)**. The abstract separately claims **1.36%** average error. The paper explicitly notes this discrepancy and states that a faithful summary is that the abstract claims **1.36%**, while the main detailed benchmark table reports **1.63** percentage points mean absolute error across the eight benchmarks [2502.17262].

## 4. What COD is and is not

A common misconception is that COD is a generic recipe for clustering data points into “easy” and “hard” semantic classes. In the COD paper, that is not the task. The clustered objects are **evaluation samples/tasks**, and the clustering criterion is the **task-wise passrate vector across small models**, not semantic content or latent embedding similarity [2502.17262]. The output is therefore a partition of a benchmark into groups with similar scaling behavior.

A second misconception is that COD predicts full-benchmark performance directly from cluster-wise scaling. The method does not do this. It first constructs a **predictable subset**, predicts its performance, and then applies the quartic mapping to obtain the full evaluation-set score [2502.17262]. The mapping phase is not optional in the complete formulation; the paper shows that **COD (w/o mapping)** has higher mean and maximum error than **COD (Complete)**.

A third misconception is that COD is simply a rebranding of training-loss extrapolation. The paper directly critiques loss-based prediction, arguing that training loss is mainly an in-domain compression metric and that equal loss can correspond to different downstream outcomes. COD is therefore positioned as a task-structured alternative to loss-only prediction [2502.17262].

The limitations are also explicit. COD depends on having enough test cases per benchmark, is **not suitable for multiple-choice tasks** where benchmark accuracy depends on ranking the correct option above others, has weak theoretical support for **chain-of-thought** prompting, and is sensitive to training-regime changes such as annealing with higher-quality data. The experiments are centered on dense transformers trained with the same data distribution and constant learning rate, while prediction on MoE families remains underexplored [2502.17262].

These limitations matter for interpretation. COD is not a universal law of model scaling; it is a benchmark-conditional procedure whose success depends on whether the benchmark contains enough scalable, non-irregular questions and whether small-model passrates reveal stable difficulty structure.

## 5. Broader meanings of clustering on difficulty in adjacent literatures

Outside the COD framework, several papers use related ideas in which difficulty conditions clustering, clustering reveals difficulty, or clustering is performed on a difficulty-derived scalar rather than on semantic similarity. The term therefore has a broader encyclopedic scope than the single LLM-scaling paper alone.

| Work | What is clustered | Role of difficulty |
|---|---|---|
| COD [2502.17262] | evaluation tasks | clustered by difficulty scaling features |
| Wordle difficulty study [2305.03502] | words | hierarchical clustering based on expectation of guessing times |
| SDR crowdsourcing model [1802.04009] | workers | clustering emerges from latent preferences to separate subjectivity from difficulty |
| HaDis [2401.14038] | semantic image categories | hard positives and negatives are mined and reweighted during clustering |
| DAICS [2101.03783] | multi-view samples in training schedule | samples are selected progressively from easy to difficult |

In the Wordle study, the clustered objects are **individual Wordle target words**, and the paper states that it uses hierarchical clustering “**based on the expectation of guessing times**” [2305.03502]. Difficulty is defined through the **distribution and expectation of guessing times**, with higher expected guesses implying higher difficulty. The actual clustering input is one-dimensional: the expected number of guesses for each word. The paper evaluates **silhouette score**, observes that **three or four** clusters look optimal, and chooses **four categories** for greater differentiation. These clusters are then interpreted as ordinal difficulty levels and modeled downstream using factor analysis and **ordered logistic regression** [2305.03502]. This is a direct instance of clustering on a scalar difficulty summary, but it differs from COD because the feature is a corrected expectation rather than a scaling trajectory.

In crowdsourcing, the SDR model does not cluster questions directly by difficulty. Instead, it introduces an explicit difficulty parameter \(d_j\) and models subjectivity through latent worker preferences. After inference, workers are clustered using **K-means on posterior means** \(\hat{\boldsymbol{\phi}}_i\), with the number of clusters chosen by the **Elbow method** [1802.04009]. The point of this clustering is to separate **between-group systematic disagreement** from **within-group inconsistency**, so that estimated difficulty is not confounded by genuine differences in opinion. The paper explicitly states that this is “**a model of clustering to separate subjectivity from difficulty, not a direct clustering-on-difficulty method**” [1802.04009].

In deep image clustering, HaDis is likewise not “clustering by difficulty” in the sense of making difficulty the cluster label. It is better described as **difficulty-aware deep clustering**. The method mines the hardest positive and hardest negative per anchor, adaptively weights them in a self-distillation framework, and combines this with diffused sampling alignment and prototypical contrastive learning [2401.14038]. The difficulty notion is primarily **pair-level**, derived from distances between teacher and student embeddings, and is used to reweight optimization rather than to produce difficulty clusters.

In multi-view clustering, DAICS makes difficulty a first-class training variable. It defines a binary difficulty label \(y_k^i\in\{0,1\}\) for each sample in each view, resolves inconsistent labels across views via an adversarial minimax game, and introduces sampling probabilities with the theorem
\[
p_{ke}^i > p_{kd}^i,
\]
which guarantees that easy samples have larger sampling probability than difficult ones [2101.03783]. The model then trains progressively from easy to difficult and activates common-subspace learning once the selected sample count crosses a **golden section** threshold \(\sigma = \frac{\sqrt{5}-1}{2}\approx 0.618\). This is difficulty-conditioned clustering, but again not the same as COD’s clustering of tasks by scaling behavior.

A plausible synthesis is that “Clustering-On-Difficulty” now names at least three distinct patterns in the literature: clustering objects **by** a difficulty representation, clustering auxiliary entities **to improve** difficulty estimation, and making clustering algorithms **difficulty-aware** during optimization.

## 6. Conceptual relations, theoretical analogues, and open problems

The broader clustering literature supplies two useful conceptual analogues. First, Ben-David’s survey on clusterability argues that tractability results for clustering typically require very strong structural separation, not merely “meaningful” data, and that most clusterability assumptions are not efficiently testable [1510.05336]. Although this paper does not discuss difficulty as COD does, it raises a parallel issue: computational success often depends on identifying a structured subset of instances for which a model or algorithm is reliable. COD’s predictable subset can be read as one such operational compromise. This suggests that COD is part of a larger tradition of replacing worst-case heterogeneity with structured subfamilies that are stable enough to analyze.

Second, the diffusion-based MELD and M-LUND framework shows that a dataset may admit multiple clusterings visible at different diffusion times, with more stable clusterings persisting across longer intervals [2102.00500]. The paper formalizes this through time windows
\[
\mathcal{I}_\epsilon^{(\ell)} = \left[ \frac{\log(2\kappa^{(\ell)}/\epsilon)}{\log(1/|\lambda_{K_\ell+1}^{(\ell)}|)}, \; \frac{\epsilon}{2\delta^{(\ell)}} \right],
\]
so that coarse, stable distinctions persist while finer, weaker distinctions disappear as diffusion proceeds. This suggests a useful analogy: COD’s predictable subset plays a role similar to selecting the scale regimes in which structure is stable enough to model.

Several open directions follow directly from the surveyed papers. COD itself leaves emergence-heavy, zero-performance, and irregular clusters to the mapping stage rather than modeling them directly [2502.17262]. SDR notes that the model does not capture **domain-specific expertise** or cases where different worker groups find different objective question types hard [1802.04009]. HaDis raises the question of modeling hardness more explicitly at multiple levels—instance, pair, prototype, and cluster—rather than from current mini-batch relations alone [2401.14038]. DAICS shows that difficulty can be view-dependent and inconsistent across modalities, suggesting that any general theory of clustering on difficulty may have to treat difficulty as relational rather than intrinsic [2101.03783].

Taken together, these works indicate that Clustering-On-Difficulty is not a single algorithmic doctrine but an emerging research pattern. In its narrowest meaning, it is the COD framework for LLM downstream prediction through clustering of task scaling trajectories [2502.17262]. In a wider encyclopedic sense, it denotes methods that use clustering to expose, control, or operationalize heterogeneity in hardness. The literature supports a consistent conclusion: difficulty is often not an after-the-fact diagnostic but a structural variable that can change what should be clustered, how it should be clustered, and which subsets are sufficiently regular to support prediction or optimization.

Source: https://www.emergentmind.com/topics/clustering-on-difficulty