---
title: Label Distribution Learning (LDL)
url: https://www.emergentmind.com/topics/label-distribution-learning-ldl
type: topic
---

# Label Distribution Learning (LDL)

Label Distribution Learning (LDL) is a supervised learning paradigm in which each instance is described by a distribution over a fixed label set, rather than by a single label or a binary relevance vector. For an instance \(x\), the label distribution assigns a “description degree” to each label, with non-negativity and simplex normalization constraints, so that the resulting vector quantifies how strongly each label describes the instance. This formulation generalizes both single-label learning and multi-label learning, and it is designed for settings in which label ambiguity, graded relevance, annotator disagreement, or ordinal neighborhood structure are intrinsic to the task [1408.6027]. Over time, LDL has expanded from early maximum-entropy and transformation-based formulations to deep convolutional models, label enhancement methods, incomplete-annotation recovery, noise-aware formulations, multi-view contrastive approaches, graph-based incremental learning, and fully probabilistic models on the simplex [1611.01731] [2305.09500] [2412.07324].

## 1. Formal definition and conceptual scope

In the foundational formulation, let the label set be \(\mathcal{Y}=\{y_1,\dots,y_c\}\). For an instance \(x\), its label distribution is
\[
\mathbf{d}(x) = [d_1,\dots,d_c], \quad d_j \equiv d^{x}_{y_j},
\]
with
\[
d^{x}_{y_j} \in [0,1], \qquad \sum_{j=1}^c d^{x}_{y_j} = 1.
\]
The quantity \(d^{x}_{y}\) is the description degree of label \(y\) to \(x\); it is not merely the correctness probability of a single hidden class, but the proportion that a label contributes to a full description of the instance [1408.6027]. Later work uses equivalent notation, e.g. \(y\in\mathbb{R}^C\), \(y_i\in[0,1]\), \(\sum_i y_i=1\), or \(d_x^{y_j}\in[0,1]\) with \(\sum_j d_x^{y_j}=1\) [2305.09500] [2511.12494].

LDL subsumes single-label learning and multi-label learning as special cases. In the single-label case, the label distribution is one-hot. In the multi-label case, multiple labels can be relevant, and LDL further refines this by representing relative importance among relevant labels rather than only binary presence or absence [1408.6027] [2303.06847]. This makes LDL suitable for tasks in which nearby labels are semantically correlated, annotation uncertainty is substantial, or multiple labels jointly characterize the same instance, as in apparent age estimation, head pose estimation, facial expression analysis, scene understanding, semantic segmentation, and multi-label image recognition [1611.01731].

A recurring distinction in the literature is between the *relative* nature of LDL and the *absolute* information that LDL does not encode. One recent extension argues that a normalized label distribution omits the total description degree of hidden labels outside the modeled label space, and introduces “background concentration” to recover that missing absolute term [2505.21576]. This suggests that the standard simplex representation is powerful but not always semantically complete when unmodeled label mass is important.

## 2. Objectives, losses, and evaluation protocols

A canonical LDL objective is to fit a predictor \(f\) or \(p_\theta(y\mid x)\) so that the predicted distribution matches the ground-truth label distribution. In the foundational maximum-entropy formulation, this is written as
\[
\min_{\theta} \sum_{i=1}^n \sum_{j=1}^c d_{ij} \log \frac{d_{ij}}{p_\theta(y_j \mid x_i)}
\quad \Longleftrightarrow \quad
\max_{\theta} \sum_{i=1}^n \sum_{j=1}^c d_{ij} \log p_\theta(y_j \mid x_i),
\]
which is the KL-based criterion specialized to LDL [1408.6027]. Deep Label Distribution Learning (DLDL) instantiates this principle in ConvNets by minimizing a Kullback–Leibler divergence between predicted and ground-truth label distributions, with softmax outputs \(q_i(l)\) and gradient
\[
\frac{\partial L}{\partial z_{il}} = \frac{1}{N}\big(q_i(l)-p_i(l)\big),
\]
so the optimization resembles softmax-cross-entropy, except that the target is a full distribution rather than a one-hot vector [1611.01731].

Several works use alternative but related losses. Bidirectional LDL uses squared reconstruction terms in both directions,
\[
\|X\theta - D\|_F^2 + \lambda_1 \|X - D\theta^\top\|_F^2 + \lambda_2 \|\theta\|_F^2,
\]
to address the dimensional gap between high-dimensional features and low-dimensional label distributions [2007.03181]. Direct learning from logical labels couples recovered distributions \(D\) and predictor outputs \(P\) through
\[
\min_{\mathbf{D},\mathbf{W}}
\ \mathrm{KL}(\mathbf{D},\mathbf{P})+\alpha \,\mathrm{tr}(\mathbf{D}\mathbf{G}\mathbf{D}^T)+\beta \|\mathbf{D}\|_F^2+\gamma \|\mathbf{W}\|_F^2
\]
subject to \(0\le D\le Y\) and \(D\mathbf{1}_c=\mathbf{1}_n\), thereby enforcing zero mass on invalid labels [2303.06847]. Fully probabilistic LDL on the simplex instead models a conditional density \(p(\ell\mid x)\) and derives closed-form conditional mean, variance, and covariance, making uncertainty a first-class output rather than a by-product of point estimation [2412.07324].

Evaluation in LDL is typically distributional rather than accuracy-based. Representative metrics repeatedly used across the literature are Chebyshev distance, Clark distance, Canberra distance, KL divergence, cosine similarity, and intersection similarity [2305.09500] [1408.6027]. For two distributions \(D=(d_1,\dots,d_C)\) and \(\hat D=(\hat d_1,\dots,\hat d_C)\), ConLE explicitly uses
\[
\mathrm{Dis}_1(D,\hat D)=\sum_{j=1}^C d_j \ln\frac{d_j}{\hat d_j},
\]
\[
\mathrm{Dis}_2(D,\hat D)=\max_j |d_j-\hat d_j|,
\]
\[
\mathrm{Dis}_3(D,\hat D)=\sqrt{\sum_{j=1}^C \left(\frac{d_j-\hat d_j}{d_j+\hat d_j}\right)^2},
\]
\[
\mathrm{Dis}_4(D,\hat D)=\sum_{j=1}^C \frac{|d_j-\hat d_j|}{d_j+\hat d_j},
\]
\[
\mathrm{Sim}_1(D,\hat D)=\frac{\sum_{j=1}^C d_j \hat d_j}{\sqrt{\sum_{j=1}^C d_j^2}\sqrt{\sum_{j=1}^C \hat d_j^2}},
\]
and
\[
\mathrm{Sim}_2(D,\hat D)=\sum_{j=1}^C \min(d_j,\hat d_j).
\]
These metrics expose different aspects of distribution quality, including worst-case deviation, relative error near zero, divergence, angular agreement, and overlap [2305.09500].

## 3. Methodological families and model architectures

The early LDL literature organized methods into three design strategies: problem transformation, algorithm adaptation, and specialized algorithm design. Problem-transformation methods convert each distribution into weighted single-label examples; algorithm-adaptation methods modify learners such as KNN or neural networks to predict distributions; specialized methods optimize LDL-native objectives directly, for example via maximum-entropy models trained by IIS or BFGS [1408.6027]. On 15 real-world datasets plus one artificial dataset, the early comparative study reported clear advantages of the specialized algorithms, especially SA-BFGS [1408.6027].

Subsequent work expanded the model class substantially. DLDL introduced end-to-end ConvNet training with discrete label distributions, Gaussian target construction for ordinal variables, histogram aggregation for multiple annotators, and per-pixel soft distributions for segmentation boundaries [1611.01731]. Label Distribution Learning Forests (LDLFs) replaced fixed parametric output forms with differentiable decision trees whose predictions are mixtures of leaf-node distributions,
\[
\mathbf{g}(\mathbf{x};\Theta,\mathcal{T})=\sum_{\ell\in\mathcal{L}} p(\ell\mid \mathbf{x};\Theta)\,\mathbf{q}_\ell,
\]
and optimized them jointly using a distribution-based loss and a variationally derived leaf update that guarantees strict decrease of the objective [1702.06086].

Another design line focuses on preserving information lost by one-way projection. Bidirectional LDL argues that many LDL and label-enhancement methods ignore the dimensional gap between the input matrix and the lower-dimensional output space, and introduces a backward reconstruction term so that labels reconstruct features while features predict distributions. Its LDL objective leads to a Sylvester equation
\[
\big(X^\top X + \lambda_2 I_d\big)\theta + \lambda_1 \theta (D^\top D) = (1+\lambda_1)X^\top D,
\]
which has a unique solution when \(\lambda_2>0\) [2007.03181].

A different response to correlation modeling is to relocate low-rank structure away from the LDL matrix itself. “Exploiting Multi-Label Correlation in Label Distribution Learning” argues that LDL matrices are typically full-rank, whereas auxiliary multi-label representations are more plausibly low-rank. It therefore derives a binary multi-label matrix \(L\) from each distribution \(D\) and optimizes
\[
\min_{W,O}
\frac{1}{2}\|WX^\top-D\|_F^2+\frac{1}{2}\|WX^\top O-L\|_F^2+\alpha\|WX^\top O\|_*+\lambda(\|W\|_F^2+\|O\|_F^2),
\]
thereby enforcing low-rank structure in the auxiliary multi-label space rather than on the distribution matrix itself [2308.01742].

Recent work also revisits label-specific features. LDL-LIFT-SAP constructs label-specific feature spaces by clustering instances into positive, uncertain, and negative sets for each label, then enriches prototype representations with Structural Anchor Points (SAPs), distance features, and cosine-direction features. A second-level meta-learner then fuses per-label predictors into a final distribution [2504.19374]. This suggests that LDL performance can benefit not only from better losses, but also from redesigned feature spaces aligned with label semantics.

## 4. Logical labels, label enhancement, and incomplete or biased supervision

A central practical difficulty in LDL is that directly annotating full label distributions is costly. One response is Label Enhancement (LE), which attempts to recover a distribution from logical labels. In ConLE, the dataset is
\[
S=\{(x_i,L_i)\mid 1\le i\le n\},
\]
with \(x_i\in\mathbb{R}^{\mathrm{dim}_1}\) and \(L_i\in\{0,1\}^C\), and the goal is to recover \(D_i\in\mathbb{R}^C\) such that \(\sum_j d_{x_i}^{y_j}=1\) [2305.09500]. ConLE treats features and logical labels as two views of the same instance, maps them into a shared space with
\[
Z_m=F_1(x_m;\theta), \qquad Q_m=F_2(L_m;\phi),
\]
uses cosine similarity
\[
h(Z_m,Q_m)=\frac{Z_mQ_m^\top}{\|Z_m\|\|Q_m\|},
\]
and optimizes an InfoNCE-style instance-level loss plus a label-attribute consistency term,
\[
L_{\mathrm{ConLE}}=l_{\mathrm{con}}+\lambda_1 l_{\mathrm{dis}}+\lambda_2 l_{\mathrm{thr}}.
\]
The threshold component enforces that relevant labels must receive larger recovered description degrees than irrelevant ones [2305.09500].

A more tightly coupled alternative is to skip the two-step LE-then-LDL pipeline and learn LDL directly from logical labels. “Label Distribution Learning from Logical Label” argues that conventional LE methods may assign nonzero mass to invalid labels and that step-wise training loses the connection between enhancement and prediction. Its joint model constrains recovered distributions \(D\) by \(

Source: https://www.emergentmind.com/topics/label-distribution-learning-ldl