---
title: Correlation of Loss Differences (CLD)
url: https://www.emergentmind.com/topics/correlation-of-loss-differences-cld
type: topic
---

# Correlation of Loss Differences (CLD)

Searching arXiv for the specified paper and closely related usages of “CLD” to ground the article in current research.
arxiv_search:
  query: "2508.20230 OR \"Coresets from Trajectories: Selecting Data via Correlation of Loss Differences\""
  max_results: 5
arxiv_search:
  query: "\"Correlation of Loss Differences\" coreset"
  max_results: 10
arxiv_search:
  query: "\"Counterfactual Loss Discrepancy\" OR \"Feature Noise Induces Loss Discrepancy Across Groups\""
  max_results: 5
Correlation of Loss Differences (CLD) is a coreset selection metric for supervised deep learning that scores each training sample by the correlation between its loss-difference trajectory and the class-average loss-difference trajectory of a held-out validation set. In "Coresets from Trajectories: Selecting Data via Correlation of Loss Differences," CLD is presented as a simple and scalable metric that uses only per-sample loss values computed at training checkpoints, avoids gradient and curvature computations, and is accompanied by a general theoretical framework establishing convergence guarantees for CLD-based coresets [2508.20230]. The method is positioned as a dataset optimization procedure for real-time or resource-constrained scenarios, with reported results on CIFAR-100 and ImageNet-1k, along with analyses of efficiency, transferability, early-checkpoint stability, and bias reduction [2508.20230].

## 1. Formal definition

CLD is defined from loss trajectories recorded during training. For a sample $\vec{z}$, let $\ell(\theta^t, \vec{z})$ denote its loss at training checkpoint $t$. The sequence of per-epoch or per-checkpoint loss differences is

$$
\vec\Delta(\vec z) = \bigl(\ell(\theta^{1}, \vec{z})-\ell(\theta^{0}, \vec{z}),\ ...,\ \ell(\theta^{T},\vec{z})-\ell(\theta^{T-1},\vec{z})\bigr) \in \mathbb{R}^T .
$$

For a validation set $\mathcal{Q}$, the class-average validation trajectory for class $c$ is

$$
\vec{\Delta}'_{\mathcal{Q}, c} = \left( \frac{1}{|{\mathcal{Q}_c}|}\sum_{\vec{q}_j \in \mathcal{Q}_c} [\ell(\theta^{1}, \vec{q}_j)-\ell(\theta^{0}, \vec{q}_j)], ..., \frac{1}{|{\mathcal{Q}_c}|}\sum_{\vec{q}_j \in \mathcal{Q}_c} [\ell(\theta^{T}, \vec{q}_j) - \ell(\theta^{T-1}, \vec{q}_j)] \right).
$$

For a training sample $\vec{z}_m$ of class $c$, the CLD score is

$$
CLD(\vec{z}_m) = \rho\left( \vec{\Delta}(\vec{z}_m),\ \vec{\Delta}'_{\mathcal{Q}, c} \right),
$$

where $\rho$ denotes the Pearson correlation, used for its simplicity and scale-invariance [2508.20230].

This formulation makes CLD a trajectory-level statistic rather than a single-checkpoint criterion. It measures alignment in *changes* of loss over time, not merely absolute loss magnitude. A plausible implication is that CLD emphasizes training examples whose optimization dynamics resemble the validation dynamics, rather than examples that are only hard or easy at one specific checkpoint.

## 2. Selection mechanism and interpretive rationale

Coreset construction proceeds within each class: select the top $k_c$ samples with largest CLD values to form the overall coreset $\mathcal{C}$. The paper notes that this can be used, for example, to maintain label balance [2508.20230].

The stated interpretation is that CLD quantifies how the loss change pattern of a training example tracks the average loss trajectory of the held-out validation set for its class. High-CLD samples are described as those whose dynamics most closely mirror the validation set, which is treated as a proxy for unseen test data. Low or negative CLD samples are said to typically correspond to outliers, noisy, atypical, or mislabeled examples whose dynamics diverge from the overall validation trajectory [2508.20230].

The paper contrasts this perspective with two common families of subset selection criteria. Gradient methods measure gradient norm or alignment at a particular state. Curvature methods focus on second-order statistics such as Hessian-related quantities. CLD, by contrast, uses only forward-pass loss values over time, requires no gradients, Hessians, or pairwise similarities, and is described as a temporal, generalization-aware metric that is computationally very light [2508.20230].

This comparison clarifies the operational meaning of the method. CLD is not an influence approximation in the usual gradient-sensitivity sense; rather, it is a correlation-based selection rule defined on optimization trajectories. That distinction is central to its computational profile and to the type of signal it extracts from training.

## 3. Theoretical framework and convergence guarantee

The theoretical analysis assumes L-smoothness, bounded gradients, and a validation representativeness condition under which the mean validation gradient approximates the population risk gradient within an error $\delta = \mathcal{O}(B/\sqrt{Q})$ [2508.20230].

The main theorem stated in the paper considers a coreset $\mathcal{C}$ containing all training samples with $CLD \geq 1-\epsilon$ and standard gradient descent on $\mathcal{C}$. The convergence bound is given as

$$
\min_{0 \le t < T} \left\| \nabla_\theta R_{\mathcal{D}}(\theta^t_{\mathcal{C}}) \right\|^2 \leq \frac{2 [R_{\mathcal{D}}(\theta^0) - R_\inf]}{ \eta T } + L\eta B^2 + \left( B \sqrt{2\kappa} + \delta \right)^2 .
$$

Here, $\eta \leq 1/L$ is the learning rate, $\kappa$ measures the alignment gap between the average coreset gradient and the validation gradient, and $\delta$ is the validation representativeness gap [2508.20230].

The paper’s stated takeaways are precise. First, convergence is close to full-data training up to an additive deviation determined by coreset–validation alignment through $\kappa$ and validation quality through $\delta$. Second, as $\epsilon \to 0$ and/or as coreset size increases, $\kappa \to 0$, and the error matches full-data training. Third, the accompanying corollary states that achieving such convergence rates requires that selected samples have nearly maximal CLD and that the validation set is representative [2508.20230].

The theoretical role of CLD is therefore not merely heuristic. In the stated framework, alignment to validation dynamics appears directly in the convergence error term. This suggests that CLD is intended as a principled surrogate for selecting examples whose gradients remain compatible with generalization-relevant directions, although the paper formulates the guarantee through alignment and representativeness rather than through explicit influence-function machinery.

## 4. Empirical behavior on benchmark datasets

The empirical study is reported on CIFAR-100, with 100 classes, 50,000 train, and 10,000 test examples, and on ImageNet-1k, with 1,000 classes, approximately 1.28M train, and 50,000 test examples [2508.20230].

Across these benchmarks, the paper reports that CLD-based coresets outperform or closely match state-of-the-art baselines across a range of subset sizes. Even when methods such as DUAL, Glister, or $\mathbb{D}^2$-Pruning edge out CLD, the gap is under 1% [2508.20230]. The abstract further states that CLD-based coresets typically outperform or closely match state-of-the-art methods across subset sizes and remain within 1% of more computationally expensive baselines even when not leading [2508.20230].

The computational results are a major part of the method’s characterization. CLD incurs minimal compute and storage overhead because only per-sample, per-epoch losses are stored. For ImageNet-1k with ResNet-50, the reported end-to-end selection and training cost is approximately 902 PFLOPs, about an order of magnitude less than methods such as EL2N, GraNd, or Glister. The storage requirement is approximately 0.5 GB for all per-sample trajectories, compared with hundreds of GB for feature-based or similarity-based approaches [2508.20230].

The paper also reports an influence-oriented comparison. A variant denoted $CLD_{infl}$ is empirically compared against TracIn, TRAK, Datamodels, and FZ-influence, with the conclusion that CLD captures a similar impact on model predictions at much lower computational cost, especially at large subset ratios. Removing top-scoring CLD samples is said to cause large fractions of prediction flips in the model, which the paper interprets as confirming their generalization importance [2508.20230].

Taken together, these findings place CLD on an efficiency–accuracy frontier defined by low-overhead trajectory statistics rather than high-dimensional derivative information.

## 5. Transferability, temporal sparsity, and bias reduction

A prominent empirical property of CLD is proxy-to-target transfer across architectures. The paper states that CLD coresets computed with a small model such as ResNet-18 transfer to larger or deeper architectures including ResNet-50, DenseNet-121, and VGG-19 with less than 1% performance drop compared to oracle coresets selected in the target architecture [2508.20230]. The detailed example given is a 75% coreset where ResNet-50 trained on a CLD coreset from ResNet-18 achieves 73.04% versus 73.57% for native selection [2508.20230]. The abstract summarizes this as effective transfer across ResNet, VGG, and DenseNet, enabling proxy-to-target selection with less than 1% degradation [2508.20230].

The method is also reported to be stable under temporal truncation. Using only early checkpoints, for example the first 30 of 90 epochs on ImageNet, achieves nearly the same coreset performance. The abstract describes this as stability when using only early checkpoints, with negligible accuracy loss [2508.20230]. This supports a sparse-trajectory operating mode in which CLD can be computed before the completion of full training.

Bias reduction is presented as an inherent consequence of per-class validation alignment. Because CLD is computed against the validation trajectory for the sample’s own class, label balance and class coverage are said to be inherently promoted. The paper further states that applying additional stratified sampling, such as CCS-style sampling, actually reduces accuracy for CLD, in contrast to other metrics, and interprets this as evidence that bias is already minimized [2508.20230]. The abstract summarizes the point by noting that CLD exhibits inherent bias reduction via per-class validation alignment, obviating the need for additional stratified sampling [2508.20230].

These properties broaden the scope of CLD beyond single-model, single-run subset selection. A plausible implication is that the method is especially suitable for large-scale settings where subset scoring must be amortized across architectures or obtained from partial training logs.

## 6. Terminological ambiguity and related uses of “CLD”

The acronym “CLD” is not unique to coreset selection. In fairness analysis, "Feature Noise Induces Loss Discrepancy Across Groups" defines CLD as *Counterfactual Loss Discrepancy*, an individual-level fairness quantity rather than a data selection score. In that setting,

$$
CLD (h, o, \ell) = \mathbb{E}_z\Big[ L_0 - L_1 \Big], \text{ where } L_{g'} = \mathbb{E}_u \big[ \ell(h(o(z, g', u)), y) \mid z \big],
$$

and the paper analyzes its relation to feature noise, group information, and statistical loss discrepancy [1911.09876]. That use of CLD is conceptually unrelated to trajectory-based coreset selection.

A second nearby terminological source appears in operational risk modeling. "Estimating the correlation between operational risk loss categories over different time horizons" studies covariances and correlations of aggregate losses across time windows and explicitly states that the paper does not use “Correlation of Loss Differences (CLD)” as a specific term, though its analysis is relevant to estimating correlations or covariances between loss differentials or aggregations over different time windows [2306.16236]. Its focus is on analytic mean, variance, and covariance calculations for marked point-process loss models, not on supervised learning coresets.

This acronym reuse matters for literature searches. In contemporary arXiv usage, “Correlation of Loss Differences” most directly refers to the trajectory-based coreset metric introduced in [2508.20230], whereas “CLD” elsewhere can denote counterfactual fairness quantities or be invoked informally in discussions of loss correlations. Careful disambiguation is therefore necessary when interpreting citations or comparing methods across domains.

Source: https://www.emergentmind.com/topics/correlation-of-loss-differences-cld