---
title: 'COD: Clustering-On-Difficulty Framework'
url: https://www.emergentmind.com/topics/clustering-on-difficulty-cod-framework
type: topic
---

# COD: Clustering-On-Difficulty Framework

The Clustering-On-Difficulty (COD) framework encompasses a family of model-based clustering techniques that leverage difficulty or similarity metrics to robustly partition samples or variables for downstream performance prediction, covariance structure recovery, or high-dimensional exploratory analysis. COD has been developed and rigorously analyzed in contexts including large language model (LLM) scaling prediction and model-assisted variable clustering, offering minimax-optimal recovery, explicit cluster separation thresholds, and end-to-end accuracy extrapolation pipelines [2502.17262, 1508.01939].

## 1. Modeling by Difficulty Features and Covariance Profiles

COD’s foundational principle is the clustering of samples (in LLM evaluation) or variables (in variable clustering) based on features that encode “difficulty” or “similarity” as measured by model predictions or covariance profiles.

- In LLM performance prediction, each evaluation sample is characterized by its passrate vector across a suite of small models $\{M_1, \dots, M_S\}$. The difficulty-feature for sample $p$ is $\mathbf{x}_p = [p_1(p),\dots,p_S(p)]^\top \in \mathbb{R}^S$, where $p_j(p)$ is the empirical passrate of $M_j$ on $p$, estimated via repeated few-shot stochastic trials. This vector is typically nondecreasing in index, as model performance increases with scale [2502.17262].

- In high-dimensional covariance modeling, the G-block covariance model posits that the $p$-dimensional random vector $X$ admits
  $$
  \Sigma = A C^* A^\top + \Gamma,
  $$
  with $A$ the $p \times K$ indicator membership matrix, $C^*\in\mathbb{R}^{K \times K}$ latent factor covariance, and $\Gamma$ diagonal noise. This structure clusters variables whose covariance profiles are similar across all other variables [1508.01939].

## 2. Difficulty-Driven or Similarity-Based Clustering Algorithms

COD clusters are constructed by partitioning points with similar difficulty or similarity profiles, automatically adapting both the cluster count and outlier handling.

- **For LLMs:** The passrate matrix $X \in \mathbb{R}^{N \times S}$ (samples $\times$ small models) is clustered using an improved MeanShift algorithm based on Euclidean distance. The steps are:
  1. All samples start unassigned.
  2. Standard MeanShift is run on the unassigned set with bandwidth $R$.
  3. Points within $R$ of each new center $c$ are assigned to the cluster; others remain unassigned.
  4. Clusters with fewer than $K$ points are dissolved; their members revert to unassigned.
  5. Repeat until convergence; remaining unassigned samples are marked as outliers [2502.17262].

- **For variable clustering:** COD operates by identifying pairs of variables $(a, b)$ with minimal
  $$
  \hat sCOD(a, b) = \max_{c \ne a,b} \left| \hat \Sigma_{a,c} - \hat \Sigma_{b,c} \right| / \sqrt{ (\hat \Sigma_{a,a} + \hat \Sigma_{b,b} - 2\hat\Sigma_{a,b}) \cdot \hat\Sigma_{c,c} }
  $$
  and iteratively extracts clusters based on a fixed threshold $\alpha$. Singleton clusters are identified where maximal similarity is above threshold; otherwise, a cluster is built by merging variables whose pairwise sCOD falls below $\alpha$ with respect to at least one seed variable [1508.01939].

## 3. Cluster-Wise Performance Extrapolation and Scaling Laws

Only clusters with regular, predictable scaling are considered for further extrapolation, yielding stable predictions on subset performance.

- Within each cluster, a scaling law is fit of the form:
  $$
  y(C) = g + (1-g) \exp(-aC^{-b} - c),\quad a>1,\ b>0.1,\ 0<c<1,
  $$
  where $y(C)$ denotes the expected accuracy at compute $C$. The random-guess floor $g$ and asymptotic offset $c$ are essential for accurate fits. Only clusters whose parameters and fitted curves satisfy these constraints, and that are monotonic and extrapolatable, are retained. Cluster-wise predictions $y(C_0)$ are aggregated as a cluster-size-weighted average for the subset [2502.17262].

## 4. Mapping Subset Prediction to Full Evaluation Set

Since not all samples are in extrapolatable clusters, a mapping is learned to translate subset accuracy $T'$ to total set accuracy $T$.

- The mapping function $f$ is a quartic polynomial constrained by $f(0) = 0$, $f(1) = 1$:
  $$
  f(x) = \alpha_1 x^4 + \alpha_2 x^3 + \alpha_3 x^2 + (1-\alpha_1-\alpha_2-\alpha_3)x.
  $$
- Coefficients $\alpha_{1,2,3}$ are fit by least-squares or interpolation on anchor points $(T_m', T_m)$ obtained from mid-sized or external models. This calibration reduces bias, particularly when anchors are out-of-distribution models; empirically, out-of-distribution anchors reduce error approximately 40% over no-anchor mapping [2502.17262].

## 5. Minimax Thresholds, Theoretical Guarantees, and Empirical Results

COD is rigorously analyzed for optimality guarantees and demonstrated empirically to match the tightest known error rates.

- **Minimax-Optimality (variable clustering):**
  - The MCOD ($\operatorname{MCOD}(\Sigma)$) and $\Delta(C^*)$ metrics govern the cluster separation: $\operatorname{MCOD}(\Sigma) = \min_{a\not\sim b} \max_{c \ne a,b} |\Sigma_{a,c} - \Sigma_{b,c}|$, and $\Delta(C^*) = \min_{j<k} [C^*_{j,j} + C^*_{k,k} - 2C^*_{j,k}]$.
  - For class $\mathcal{M}(m,\eta)$, no estimator can guarantee exact recovery if $\eta < c \sqrt{\log p / n}$. COD attains exact recovery for $\operatorname{MCOD}(\Sigma)\gtrsim \sqrt{\log p/n}$, matching the minimax threshold (Theorem 3.1, [1508.01939]).
- **Downstream task prediction (LLMs):**
  - On eight LLM benchmarks (GSM8K, MATH, BBH, TriviaQA, MBPP, AGIEval, DROP, MMLU-pro), COD achieves a mean absolute error of 1.63 percentage points (1.36% in the abstract) and a maximum error never exceeding 2.4 points, outperforming end-to-end, passrate-only, and loss-intermediate baselines.
  - Key ablation findings include the critical importance of fitting both $g$ and $c$, and the superiority of quartic mapping over lower or higher-degree interpolants [2502.17262].

## 6. Comparative Analysis with Related Algorithms

COD is compared against PECOK, a penalized SDP tailored to the $\Delta(C^*)$ metric, and corrected spectral clustering.

| Algorithm              | Specialized Metric        | Exact Recovery Threshold                   | Computational Regime               |
|------------------------|--------------------------|--------------------------------------------|------------------------------------|
| COD                    | MCOD                     | $\gtrsim \sqrt{\log p / n}$                | $O(p^3)$, no SDP, cluster size $1$ |
| PECOK (SDP relaxation) | $\Delta(C^*)$            | $\gtrsim \sqrt{(K \vee \log p)/(mn)}$      | SDP ($O(p^3)$), clusters $\gtrsim 10$ or balanced |
| Corrected Spectral     | Latent eigengap          | Stronger than minimax, no exact at threshold | Low constant factors, needs larger separation |

COD provides optimal recovery for MCOD-type separation with minimal assumptions and computational simplicity. PECOK is optimal for $\Delta(C^*)$ separation when clusters are balanced and $K=O(\log p)$, but incurs SDP complexity. Corrected spectral clustering is computationally attractive but theoretically requires larger separation and does not attain minimax thresholds [1508.01939].

## 7. Practical Implementation and Use Cases

COD is applied by:
1. Gathering difficulty or covariance profile matrices.
2. Clustering via improved MeanShift (LLM) or sCOD thresholding (variable clustering).
3. Fitting cluster-wise scaling laws.
4. Aggregating extrapolatable cluster predictions.
5. Mapping to full set predictions via anchor-calibrated quartic.
6. Recommending parameter choices: pre-filter all-zero passrates, smooth passrates across checkpoints, mean-shift $R$ and $K$ for intra-cluster diameter $\lesssim 0.2$ and cluster size $\geq 10$.

LLM experiments use small models from 122M to 12B parameters, predict full-model performance at 70B scale, and produce actionable insights for efficient resource allocation and pretraining monitoring [2502.17262]. In fMRI variable clustering, COD identifies meaningful brain networks and outperforms classical clustering across sparsity regimes [1508.01939].

COD thus serves as a state-of-the-art methodology for both principled model-based clustering and reliable performance extrapolation, combining minimax theoretical performance with robust empirical accuracy.

Source: https://www.emergentmind.com/topics/clustering-on-difficulty-cod-framework