---
title: 'True-Label Rank (TLR): An Overview'
url: https://www.emergentmind.com/topics/true-label-rank-tlr
type: topic
---

# True-Label Rank (TLR): An Overview

Searching arXiv for the cited papers and topic to ground the article in current literature.
True-Label Rank (TLR) denotes the position assigned to the ground-truth label, or to the ground-truth relevant labels, within a predicted or Bayes-optimal ordering of labels. Across recent arXiv literature, the construct appears in several closely related forms: as the rank of the true class in multiclass posterior orderings, as a position-sensitive statistic for ordered multi-label supervision, as a rank event underlying top-\(k\) conformal prediction, and as a normalized softmax-derived signal for continual-learning dynamics [2002.09420], [2212.04022], [2406.06818], [2606.25165]. The common thread is that TLR refines 0–1 correctness by measuring how far the correct label information has fallen in the ordered label list; the principal variations concern whether lower or higher values are preferable, whether ties are counted with \(>\) or \(\ge\), and whether the setting involves a single true class or an ordered set of relevant labels.

## 1. Core definitions and variants

In multiclass label ranking, the most direct definition is the position of the true label \(Y\) in the decreasing ordering of the posterior vector \(\eta(X)\), where \(\eta_y(x)=\mathbb{P}\{Y=y\mid X=x\}\). If \(\pi(X)\) orders labels by decreasing posterior probability, then
\[
\mathrm{TLR}(X,Y)=r \quad \text{such that} \quad \pi_r(X)=Y.
\]
Conditional on \(X\), the distribution of TLR is
\[
\mathbb{P}\{\mathrm{TLR}(X,Y)=r\mid X\}=\eta_{\pi_r(X)}(X),
\]
and
\[
\mathbb{E}[\mathrm{TLR}\mid X]=\sum_{r=1}^K r\,\eta_{\pi_r(X)}(X).
\]
This formulation makes TLR a random variable induced by the Bayes ranking itself [2002.09420].

In score-based prediction systems, TLR is often computed from model scores rather than from the unknown posterior vector. For a score function \(s(x,y)\), RC3P defines the true-label rank as
\[
r(x,y)=1+\sum_{y'\in\mathcal{Y}\setminus\{y\}}\mathbf{1}\{s(x,y')\ge s(x,y)\},
\]
and equivalently uses
\[
r_f(X,Y):=\sum_{l=1}^C \mathbf{1}\{f(X)_l \ge f(X)_Y\}.
\]
Under this convention, the true label is in the top-\(k\) predictions if and only if \(r(x,y)\le k\) [2406.06818].

In continual learning, TLR is defined from softmax probabilities with a strict inequality and then normalized to \([0,1]\). For sample \(i\),
\[
\mathrm{rank}(y_i,f(x_i))=1+\sum_{c\neq y_i}\mathbbm{1}\bigl(p_i[c]>p_i[y_i]\bigr),
\]
and the per-sample normalized score is
\[
\mathrm{TLR}_i = 1 - \frac{\mathrm{rank}(y_i, f(x_i)) - 1}{C - 1}.
\]
In this convention, larger values are better: \(\mathrm{TLR}_i=1\) when the true class is top-1 and \(\mathrm{TLR}_i=0\) when it is last [2606.25165].

In ranked multi-label learning, the relevant object is not a single true class but an ordered true-label set \(Y^+(x)\). The predicted rank of a label \(y\) is
\[
\hat r(y;f,x)=1+\sum_{j\neq y}\mathbf{1}[f_j(x)>f_y(x)],
\]
and a mean TLR over the true labels is
\[
\mathrm{TLR}(f;x)=\frac{1}{|Y^+(x)|}\sum_{y\in Y^+(x)} \hat r(y;f,x).
\]
A top-weighted refinement is also given:
\[
\mathrm{TLR}_w(f;x)=\frac{1}{Z}\sum_{y\in Y^+(x)} w(\mathrm{rank}_{gt}(y))\cdot \hat r(y;f,x),
\]
with decreasing \(w(\cdot)\) and normalization \(Z\) [2212.04022].

The main definitional variants therefore differ along three axes: single-label versus multi-label ground truth, raw rank versus normalized score, and tie handling. A plausible implication is that comparisons of “TLR” across papers must first align these conventions.

| Setting | TLR object | Preferred direction |
|---|---|---|
| Multiclass Bayes ranking | Rank of \(Y\) in posterior ordering \(\pi(X)\) | Lower is better |
| RC3P | Rank of true class under score ordering with \(\ge\) ties | Lower is better |
| Continual learning | Normalized transform of true-class rank | Higher is better |
| Ranked multi-label | Average predicted position of true labels | Lower is better |

## 2. Statistical role in multiclass label ranking

In the multiclass framework of label ranking, the Bayes-optimal ranking sorts labels in decreasing posterior probability, and TLR is the position of the realized label \(Y\) in that ordering [2002.09420]. This immediately links TLR to top-\(k\) performance:
\[
\mathbb{P}\{\mathrm{TLR}(X,Y)\le k\}
=
\mathbb{E}\Big[\sum_{r=1}^k \eta_{\pi_r(X)}(X)\Big].
\]
For \(k=1\),
\[
\mathbb{P}\{\mathrm{TLR}(X,Y)=1\}
=
\mathbb{E}\big[\max_{1\le y\le K}\eta_y(X)\big]
=
1-L^*,
\]
where \(L^*\) is the Bayes misclassification risk [2002.09420]. Thus, top-1 accuracy is the mass of the TLR-\(=1\) event, whereas top-\(k\) accuracy is the mass of the TLR-\(\le k\) event.

The paper "A Multiclass Classification Approach to Label Ranking" [2002.09420] shows that label ranking can be viewed as a partially observed ranking median regression problem. Under a Bradley–Terry–Luce–Plackett construction with preference vector \(\eta(X)\), only the top label \(\Sigma^{-1}(1)=Y\) is observed during training, while the full permutation is latent. Pairwise conditional probabilities become
\[
p_{k,l}(X)=\frac{\eta_k(X)}{\eta_k(X)+\eta_l(X)},
\]
and under strict stochastic transitivity the Bayes ranking is characterized by Copeland scores. This is important for TLR because any method that recovers the Bayes ranking also recovers the optimal TLR distribution.

The One-Versus-One (OVO) reduction trains a binary classifier for each pair of labels and aggregates pairwise predictions by Copeland score:
\[
\widehat{s}(X)(k)=1+\sum_{l\neq k}\mathbb{I}\big\{\widehat{g}_{k,l}(X)=-1\big\}.
\]
Under the paper’s Mammen–Tsybakov-style noise condition, finite VC complexity, and minimal mass assumption, the recovered ranking \(\widehat{\sigma}_X\) equals the Bayes ranking with overwhelming probability for large \(n\) [2002.09420]. Proposition 2 in that paper further states that the top-\(k\) risk
\[
W_k(s)=\mathbb{E}\big[\mathbb{I}\{Y\notin \mathrm{Top}_k(s(X))\}\big]
\]
is minimized by the Bayes ranking \(\sigma_X^*\), which is equivalent to saying that the optimal ranking minimizes the probability of the event \(\mathrm{TLR}>k\).

Empirically, OVO aggregation improved top-\(k\) results on MNIST and Fashion-MNIST. On MNIST, Top-1 improved from \(0.924\) for logistic regression to \(0.943\) for OVO, and Top-5 improved from \(0.995\) to \(0.997\); on Fashion-MNIST, Top-1 improved from \(0.857\) to \(0.863\), while Top-5 remained \(0.997\) for both methods [2002.09420]. Since top-\(k\) accuracy is \(\mathbb{P}\{\mathrm{TLR}\le k\}\), these improvements are direct improvements in the TLR distribution.

A common misconception is that TLR is merely another name for accuracy. The multiclass theory shows otherwise: accuracy corresponds only to the indicator \(\mathbf{1}\{\mathrm{TLR}=1\}\), while the full TLR distribution encodes how close incorrect predictions are to being correct.

## 3. Ordered multi-label supervision and RLSEP

In ranked multi-label learning, TLR becomes a position-sensitive statistic over the set of relevant labels, rather than a scalar tied to a single true class. The paper "RLSEP: Learning Label Ranks for Multi-label Classification" [2212.04022] addresses the setting in which the ground truth provides an ordered true-label set, represented either as a rank vector \(r=(r_1,\ldots,r_L)\) or as a partial order \(\pi\) over the subset of relevant labels \(Y^+(x)\). The order is ordinal: \(r_u>r_v\) means that label \(u\) should be ranked above label \(v\), while \(|r_u-r_v|\) has no quantitative meaning beyond ordering [2212.04022].

The paper does not use the term “True-Label Rank (TLR)” explicitly, but it builds a loss that directly optimizes the ranking of ground-truth labels by exploiting their intra-positive order [2212.04022]. In that spirit, mean TLR measures the average predicted position of the true labels:
\[
\mathrm{TLR}(f;x)=\frac{1}{|Y^+(x)|}\sum_{y\in Y^+(x)} \hat r(y;f,x).
\]
A lower value is better because it places the true labels nearer the top of the ranking.

The paper’s central methodological contribution is the Ranked Log-Sum-Exp Pairwise loss, RLSEP:
\[
l_{\mathrm{rlsep}}(f;x,r)=
\log\!\Big(1+\sum_{(u,v)\in \zeta(R;t)} \exp(f_v(x)-f_u(x))\Big),
\]
where \(\zeta(R;t)\subset P(\pi)\) is a random subset of \(t\) ordered pairs used for negative sampling, and
\[
P(\pi)=\{(u,v)\in Y\times Y:r_u>r_v\}.
\]
The full objective is
\[
\min_\theta \sum_{i=1}^N l_{\mathrm{rlsep}}(f(\cdot;\theta);x_i,r^{(i)})+\lambda\|\theta\|_2^2.
\]
When all positives have rank \(1\) and all negatives rank \(0\), RLSEP reduces to LSEP [2212.04022].

This construction operationalizes TLR through inversion control. Defining
\[
\mathrm{Inv}(f;x)=\sum_{(u,v)\in P(\pi)} \mathbf{1}[f_u(x)\le f_v(x)],
\]
the label ranking loss is \(\mathrm{Inv}(f;x)/|P(\pi)|\), and TLR is described as monotonically related to \(\mathrm{Inv}(f;x)\): fewer inversions imply lower TLR and stronger alignment with the ordered ground truth [2212.04022]. The loss penalizes every pairwise violation \(f_u\le f_v\) whenever \(r_u>r_v\), including violations within the positive set, which standard unranked pairwise label ranking does not enforce.

The paper reports strong empirical gains on both synthetic and real ranked data. On Ranked MNIST (full), RLSEP (R) achieved ordered-pair Precision/Recall/F1/Accuracy of approximately \(98.5/98.7/98.6/98.8\), exact-match \(0\)–\(1=72.6\), and \(mAP=95.9\). CE (R) and LSEP (R) were much lower on exact-match and mAP, with \(0\)–\(1\approx 5.6\)–\(5.7\) and \(mAP\approx 79\)–\(80\) [2212.04022]. On ARC, RLSEP (R) improved exact-match from \(12.8\) for CE and \(9.6\) for LSEP to \(14.0\), and improved mAP from \(75.6\) to \(76.8\) [2212.04022]. The paper interprets these outcomes as substantially better alignment with true-label order, which corresponds to lower TLR in the formalization given in the details block.

Calibration findings are also stated explicitly. On a controlled 4-digit set, RLSEP-trained scores increased with importance and exhibited distinct score distributions per true rank, which the paper describes as direct evidence of TLR-oriented calibration [2212.04022]. This suggests that TLR is not only an evaluation statistic but also a useful lens on score geometry.

A further connection is drawn to rank-sensitive metrics. The paper states that lower TLR aligns with higher ranked Accuracy/F1/\(0\)–\(1\) and higher mAP, and that lower TLR typically increases DCG/NDCG, even though the reported evaluation emphasizes pairwise metrics and mAP [2212.04022].

## 4. TLR as a calibration signal in conformal prediction

In class-conditional conformal prediction, TLR enters as the event structure underlying top-\(k\) inclusion. The paper "Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration" [2406.06818] defines the true-label rank from classifier scores and uses the class-conditional tail probability
\[
\epsilon_c^k=\mathbb{P}(r_f(X,Y)>k\mid Y=c)
\]
as the class-wise top-\(k\) error. This identifies TLR as the quantity that determines whether the true class lies within the top-\(k\) region for class \(c\) [2406.06818].

RC3P builds on class-conditional conformal prediction by adding a rank gate to the usual score-based gate. Standard CCP uses class-specific quantiles of a nonconformity score \(V(x,y)\) and outputs
\[
\Gamma^{\mathrm{CCP}}_{1-\alpha}(x)=\{y\in\mathcal{Y}:V(x,y)\le \widehat{Q}^{\mathrm{class}}_{1-\alpha}(y)\}.
\]
RC3P introduces per-class choices \(\widehat{k}(y)\) and \(\widehat{\alpha}_y\) satisfying
\[
\widehat{k}(y)\in\{k:\epsilon_y^k<\alpha\},
\qquad
0\le \widehat{\alpha}_y\le \alpha-\epsilon_y^{\widehat{k}(y)},
\]
and then defines
\[
\Gamma^{\mathrm{RC3P}}_{1-\alpha}(x)=
\Big\{
y\in\mathcal{Y}:
V(x,y)\le \widehat{Q}^{\mathrm{class}}_{1-\widehat{\alpha}_y}(y),
\;
r_f(x,y)\le \widehat{k}(y)
\Big\}.
\]
The recommended choice is
\[
\widehat{k}(y)=\min\{k:\epsilon_y^k<\alpha\},
\qquad
\widehat{\alpha}_y=\alpha-\epsilon_y^{\widehat{k}(y)}.
\]
The resulting prediction set therefore depends simultaneously on a score gate and a TLR gate [2406.06818].

The key coverage guarantee is class-wise:
\[
\mathbb{P}\!\left(Y\in \Gamma^{\mathrm{RC3P}}_{1-\alpha}(X)\mid Y=y\right)\ge 1-\alpha
\quad \forall\, y\in\mathcal{Y}.
\]
The proof sketch given in the paper is that the score gate provides at least \(1-\widehat{\alpha}_y\) inclusion probability within class \(y\), while the rank gate removes at most \(\epsilon_y^{\widehat{k}(y)}\) mass, so coverage remains at least \(1-\widehat{\alpha}_y-\epsilon_y^{\widehat{k}(y)}\ge 1-\alpha\) [2406.06818]. Here TLR is the mechanism that quantifies how much mass the rank gate can safely remove.

The efficiency claim is also explicitly TLR-dependent. If
\[
\mathbb{P}_X\!\left(
V(X,y)\le \widehat{Q}^{\mathrm{class}}_{1-\widehat{\alpha}_y}(y),\;
r_f(X,y)\le \widehat{k}(y)
\right)
\le
\mathbb{P}_X\!\left(
V(X,y)\le \widehat{Q}^{\mathrm{class}}_{1-\alpha}(y)
\right)
\]
for all \(y\), then RC3P has no larger expected prediction-set size than CCP [2406.06818]. The paper introduces the per-class condition number
\[
\sigma_y=
\frac{
\mathbb{P}_X\!\left(
V(X,y)\le \widehat{Q}^{\mathrm{class}}_{1-\widehat{\alpha}_y}(y),\;
r_f(X,y)\le \widehat{k}(y)
\right)
}{
\mathbb{P}_X\!\left(
V(X,y)\le \widehat{Q}^{\mathrm{class}}_{1-\alpha}(y)
\right)
},
\]
and states that \(\sigma_y\le 1\) for all \(y\) suffices for RC3P to dominate CCP in expected size [2406.06818].

Empirically, the paper reports that RC3P achieves class-wise coverage and a \(26.25\%\) reduction in prediction set sizes on average, with additional reductions such as \(24.47\%\) average APSS reduction over the best baseline across four imbalanced datasets and approximately \(32.826\%\) APSS reduction on balanced tasks with APS while maintaining \(UCR\le 0.05\) [2406.06818]. The abstract also reports \(35\%\) reduction excluding CIFAR-10 [2406.06818].

A common misunderstanding is to treat TLR in conformal prediction as merely an internal rank used by APS or RAPS. The RC3P construction goes further: APS and RAPS already depend on rank inside the nonconformity score, but RC3P adds a separate per-class rank gate calibrated by class-wise TLR tails [2406.06818].

## 5. Continuous TLR metrics in continual learning

In continual learning, TLR is used not as a static accuracy refinement but as a continuous forgetting signal. The paper "The Gentle Collapse: Distributional Metrics for Continual Learning" [2606.25165] states that accuracy degradation records only whether forgetting occurred or not, saturates at the extremes, and collapses discretely at task boundaries. TLR and related softmax-derived metrics are introduced to characterize forgetting continuously [2606.25165].

The linear normalized TLR score is
\[
\mathrm{TLR}_i
=
1-\frac{\mathrm{rank}(y_i,f(x_i))-1}{C-1},
\]
and the dataset- or class-level statistic is its sample mean. The paper also defines Log-TLR:
\[
\mathrm{Log\text{-}TLR}_i
=
1-\frac{\ln(\mathrm{rank}(y_i,f(x_i)))}{\ln C},
\]
and Reciprocal-TLR:
\[
\mathrm{Reciprocal\text{-}TLR}_i
=
\frac{C-\mathrm{rank}(y_i,f(x_i))}{(C-1)\cdot \mathrm{rank}(y_i,f(x_i))}.
\]
All are normalized to \([0,1]\), with larger values better [2606.25165].

The paper’s central claim about TLR is comparative informativeness. At \(0\%\) accuracy on CIFAR-100, the Confusion Margin spans an interquartile range of \([0.32,0.50]\) across classes that accuracy treats identically; TLR also remains discriminative, with a median of approximately \(0.65\), while CTL and NKL collapse near \(0\) [2606.25165]. This establishes that TLR retains structure in regimes where binary correctness is fully saturated.

For trend estimation, the paper computes an ordinary least squares slope over a trailing window for class-level metric trajectories:
\[
\hat{\beta}_1^{(g)}
=
\frac{\sum_t (t-\bar t)\bigl(s_t^{(g)}-\bar s^{(g)}\bigr)}
{\sum_t (t-\bar t)^2}.
\]
The claim is that continuous signals such as Log-TLR yield stable slopes for short windows, whereas accuracy, being step-like, gives high-variance estimates. In the window-sweep analysis at \(w=3\) epochs, accuracy-trend degraded to \(34.79\%\) with standard deviation \(2.32\), while log-TLR achieved \(41.07\%\) with standard deviation \(0.57\) [2606.25165].

This TLR signal is then used operationally in two ways. First, per-sample metric scores are used as inverse-proportional loss weights:
\[
w_i\propto 1-s_i,
\qquad
\tilde w_i=\frac{1-s_i}{\frac{1}{B}\sum_{j=1}^B (1-s_j)},
\qquad
\mathcal{L}_{\text{batch}}
=
\frac{1}{B}\sum_{i=1}^B \tilde w_i\,\ell(x_i,y_i).
\]
Second, class-level replay sampling probabilities are based on a softmax over slopes:
\[
\pi^{(g)}
=
\frac{\exp(\gamma \hat{\beta}_1^{(g)})}
{\sum_h \exp(\gamma \hat{\beta}_1^{(h)})},
\quad \gamma=2.0.
\]
Both uses are explicitly described as exploiting the continuity of TLR-like signals [2606.25165].

The reported gains are concrete. On CIFAR-100, Log-TLR-weighted loss yielded \(36.98\pm 0.55\%\) accuracy and \(51.92\pm 0.35\%\) forgetting versus the experience-replay baseline’s \(36.33\pm 0.55\%\) accuracy and \(53.20\pm 0.59\%\) forgetting, corresponding to approximately \(1.3\) percentage points less forgetting [2606.25165]. On TinyImageNet, log-TLR trend sampling achieved \(24.74\pm 0.81\%\) accuracy and \(47.26\pm 1.14\%\) forgetting versus \(23.67\pm 0.81\%\) and \(54.98\pm 0.28\%\) for the baseline, corresponding to approximately \(7.7\) percentage points less forgetting [2606.25165].

The continual-learning usage introduces a notable reinterpretation. Here TLR is not only a ranking statistic but also a smooth proxy for latent representational degradation. This suggests that TLR can serve as a control signal whenever binary correctness is too coarse for optimization or monitoring.

## 6. Relations to other ranking losses, estimators, and limitations

Several papers place TLR in relation to more established ranking losses. In ranked multi-label learning, inversion counts
\[
\mathrm{Inv}(f;x)=\sum_{(u,v)\in P(\pi)}\mathbf{1}[f_u(x)\le f_v(x)]
\]
yield a pairwise label ranking loss \(\mathrm{Inv}(f;x)/|P(\pi)|\), and the details block states that this is directly related to Kendall-style inversion counts and to
\[
\tau(f;x)=1-2\cdot \mathrm{Inv}(f;x)/|P(\pi)|.
\]
Lower TLR is said to align with higher ranked Accuracy/F1/\(0\)–\(1\), higher mAP, and typically higher DCG/NDCG [2212.04022]. This makes TLR a bridge between rank-sensitive evaluation and pairwise surrogate training.

In the nonparametric label-ranking framework of "Label Ranking through Nonparametric Regression" [2111.02749], TLR is not defined explicitly, but the details block derives a compatible notion as the predicted position of the true top label. There, if \(\pi^*(x)\) denotes the noiseless or median-optimal ranking and \(y^*(x)\) its top label, then for a predicted ranking \(\hat\pi(x)\),
\[
\mathrm{TLR}(x)=\hat\pi(x)(y^*(x)).
\]
The same details block derives lower bounds
\[
d_{KT}(\hat\pi(x),\pi^*(x))\ge \mathrm{TLR}(x)-1,
\qquad
d_2(\hat\pi(x),\pi^*(x))\ge (\mathrm{TLR}(x)-1)^2,
\]
and therefore
\[
\mathbb{E}[\mathrm{TLR}(x)]\le 1+\sqrt{\mathbb{E}[d_2]}.
\]
This suggests a general interpretation: global ranking losses can upper-bound expected TLR, while pairwise errors involving the top label can identify the part of the loss most directly responsible for TLR degradation [2111.02749].

Empirical TLR estimation also differs by setting. In multiclass label ranking, given a predicted permutation \(\hat\pi(X_i)\), the empirical statistic is
\[
\widehat{\mathrm{TLR}}_i=\min\{r:\hat\pi_r(X_i)=Y_i\},
\]
with summaries
\[
\widehat{\mathbb{E}[\mathrm{TLR}]}
=
\frac{1}{m}\sum_{i=1}^m \widehat{\mathrm{TLR}}_i,
\qquad
\widehat{\mathbb{P}\{\mathrm{TLR}\le k\}}
=
\frac{1}{m}\sum_{i=1}^m \mathbb{I}\{\widehat{\mathrm{TLR}}_i\le k\}
\]
[2002.09420]. In continual learning, the per-sample rank can be computed in \(O(C)\) without sorting by counting how many probabilities exceed the true-class probability [2606.25165]. In ranked multi-label learning, TLR-like summaries can be aggregated over all relevant labels and optionally top-weighted [2212.04022].

The literature also makes several caveats explicit. First, tie handling is not uniform. RC3P uses \(\ge\), so ties count against the true class [2406.06818]; the continual-learning paper uses strict \(>\), so ties do not increase rank and the rule becomes a dense-ranking convention [2606.25165]; the RLSEP formalization assumes any fixed tie-breaking when induced rankings are needed [2212.04022]. Second, TLR may be insensitive to confidence changes that preserve order, a caveat stated for rank-only metrics in continual learning [2606.25165]. Third, richer TLR supervision can require richer annotations: RLSEP requires ordered labels rather than only binary relevance, and the paper notes that acquisition cost may therefore be higher [2212.04022].

A final misconception is that TLR is a single universally standardized metric. The papers instead support a family resemblance: all TLR formulations quantify the location of ground-truth label information within an ordering, but the operational meaning depends on whether the objective is Bayes label ranking, ranked multi-label prediction, conformal set construction, or continual-learning diagnostics.

Taken together, the recent arXiv literature presents TLR as a unifying rank-based primitive. In multiclass label ranking, it is the canonical random variable underlying top-\(k\) accuracy and Bayes-optimal orderings [2002.09420]. In ranked multi-label learning, it becomes a statistic over ordered true-label sets that RLSEP reduces by minimizing pairwise violations [2212.04022]. In class-conditional conformal prediction, its tail probabilities determine per-class rank gates that preserve coverage while reducing set size [2406.06818]. In continual learning, normalized TLR variants provide continuous, actionable signals that remain informative when accuracy saturates [2606.25165].

Source: https://www.emergentmind.com/topics/true-label-rank-tlr