--- title: 'RankEval: Ranking Evaluation Frameworks' url: https://www.emergentmind.com/topics/rankeval type: topic --- # RankEval: Ranking Evaluation Frameworks Searching arXiv for recent papers mentioning RankEval and closely related ranking-evaluation frameworks. RankEval denotes a class of ranking-evaluation and rankability-analysis frameworks that assess either how well a metric or model induces an ordering, or how inherently rankable a dataset is. In the most specific usage, RankEval is introduced as “the first standardized and reproducible benchmark for comparing evaluation metrics” in time series anomaly detection, where metrics are judged by whether they reproduce an oracle detector ordering [2509.01098]. In a distinct but related usage, RankEval functions as a general pipeline into which RankLLM is inserted as a “difficulty-aware, non-parametric ranking/evaluation framework” for jointly estimating question difficulty and model competency from observed successes and failures [2602.12424]. Related work on rankability, linear ordering, uncertainty, and human preference aggregation provides the broader methodological substrate for such frameworks, including graph-based rankability measures [2203.07364], linear-ordering formulations [2104.05816], uncertainty-sensitive reliability analysis [1206.5280], and rank-based human evaluation [1803.05928]. ## 1. Terminological scope and research contexts The term RankEval is used in more than one technical sense. In time series anomaly detection, RankEval is a named benchmark for comparing evaluation metrics by “their ability to correctly rank time series anomaly detection (TSAD) models,” emphasizing ranking capability, robustness to noise, and computational latency [2509.01098]. In LLM evaluation, RankEval appears as a broader evaluation pipeline in which RankLLM supplies the model-side and item-side scores needed for “difficulty-weighted model rankings in linear time with provable convergence” [2602.12424]. This suggests that RankEval is best understood as a methodological category rather than a single universal protocol. Across these usages, a common objective is to move evaluation away from raw, incomparable scores and toward the quality, stability, and interpretability of induced rankings. The shared concern is not merely whether a model or metric attains a high scalar score, but whether it places systems, items, or questions in an ordering that is meaningful under the task’s structural assumptions [2509.01098][2602.12424]. A second recurring theme is the distinction between evaluating a ranking and evaluating the rankability of the underlying data. The rankability literature formalizes whether a dataset, often represented as a directed graph, is close to a complete dominance graph and therefore capable of supporting a meaningful ranking [2203.07364]. The linear-ordering literature operationalizes the same concern by measuring how much pairwise evidence aligns with an optimal linear order and how diverse the set of optimal rankings is [2104.05816]. In this broader sense, RankEval encompasses both ranking alignment and structural rankability. ## 2. RankEval as a benchmark for evaluation metrics in TSAD In the TSAD setting, RankEval addresses a problem specific to metric evaluation rather than detector evaluation. Different TSAD metrics “often disagree, are sensitive to score perturbations and hyperparameters, and mix point-wise versus event-wise criteria in incompatible ways” [2509.01098]. RankEval therefore asks whether a metric can produce a detector ranking that matches a high-quality reference ordering. Let the detector set be $D = \{d_1,\dots,d_N\}$. For a metric $m \in M$, RankEval defines a ranking function $$ r_m: D \to \{1,\dots,N\}, $$ where $r_m(d_i)$ is the detector’s position when sorting by the metric score [2509.01098]. RankEval compares $r_m$ against an oracle or expected ranking $r^*$, using three alignment measures: 1. Spearman’s rank correlation: $$ Sp = 1 - \frac{6 \sum_{i=1}^{N} (r_i^* - r_i)^2}{N(N^2 - 1)}. $$ 2. Kendall’s tau: $$ Kd = \frac{C - D}{C + D}. $$ 3. Mean Rank Deviation: $$ MD = \frac{1}{N} \sum_{i=1}^{N} |r_i^* - r_i|. $$ These are the sole alignment measures used in the benchmark [2509.01098]. The benchmark computes metric scores for each detector-dataset pair, constructs per-metric detector rankings, and then measures their agreement with the oracle ranking. Across tasks, it averages alignment scores unweighted: $$ \overline{Sp}(m) = \frac{1}{T} \sum_{t=1}^{T} Sp_t(m), $$ with analogous aggregation for $Kd$ and $MD$ [2509.01098]. The benchmark includes classical ML detectors such as LOF and Isolation Forest, deep models such as LSTMAD, USAD, Anomaly Transformer, TimesNet, and Donut, and controlled anomaly score generators for synthetic tasks [2509.01098]. The metrics compared include CCE, AUC-ROC, F1, F1-PA, Reduced-F1, R-based F1, eTaPR, Aff-F1, UAff-F1, and VUS-ROC, with PATE included in latency analysis [2509.01098]. Real datasets include MSL, SMD, PSM, SWAT, Creditcard, and UCR, while synthetic tasks provide oracle rankings via controlled parameters such as accuracy $q$, false positive rate $p$, and Gaussian noise $\sigma$ [2509.01098]. The central result is that RankEval evaluates metrics by ranking capability rather than by absolute score comparability. In the reported experiments, CCE attains perfect alignment across synthetic tasks, with average $Sp = 1.000$, $Kd = 1.000$, and $MD = 0.000$, while VUS-ROC is second-best and R-based F1 performs worst on average [2509.01098]. RankEval also reports latency, where CCE’s average latency is $37.85$ ms versus $12{,}691.87$ ms for VUS-ROC and $36{,}526.02$ ms for PATE [2509.01098]. ## 3. RankEval as a difficulty-aware pipeline for LLM benchmarking In the LLM setting, RankEval is instantiated through RankLLM, which is described as a “RankEval component” and as a framework that “jointly estimates question difficulty and model competency from observed successes and failures” [2602.12424]. Its purpose is to correct a limitation of conventional benchmarks: existing aggregate scores do not differentiate question difficulty and therefore cannot finely separate model capabilities. The setup defines a pool of models $M = \{m_1,\dots,m_M\}$ and a question set $Q = \{q_1,\dots,q_Q\}$, with a response matrix $R \in \{0,1\}^{Q \times M}$ in the binary case or $R \in [0,1]^{Q \times M}$ in the continuous case [2602.12424]. The complement $\bar{R} = \mathbf{1}_{Q \times M} - R$ records failures. Two key sufficient statistics are the number of solvers per question, $$ S(q_i) = \sum_{j=1}^{M} R_{ij}, $$ and the number of failures per model, $$ F(m_j) = \sum_{i=1}^{Q}(1 - R_{ij}). $$ Trivial questions that are universally solved or universally failed are filtered so that $0 < S(q_i) < M$, and in practice one ensures $F(m_j) > 0$ [2602.12424]. RankLLM constructs a directed bipartite graph between questions and models. Edges $q_i \to m_j$ encode correct answers, and edges $m_j \to q_i$ encode failures [2602.12424]. With $A = R$ and $\bar{A} = \bar{R}^{\top}$, it defines row-stochastic transitions $$ P_{Q \to M} = \mathrm{diag}(S)^{-1}A $$ and $$ P_{M \to Q} = \mathrm{diag}(F)^{-1}\bar{A}. $$ The damped iterative updates are $$ T_Q^{(t+1)} = \alpha P_{M \to Q}^{\top} T_M^{(t)} + (1-\alpha)\frac{\mathbf{1}_Q}{Q}, $$ $$ T_M^{(t+1)} = \alpha P_{Q \to M}^{\top} T_Q^{(t+1)} + (1-\alpha)\frac{\mathbf{1}_M}{M}, $$ with $\alpha \in (0,1)$ used to avoid bipartite $2$-periodicity and ensure ergodicity [2602.12424]. In the unified Markov formulation, $$ T^{(t+1)} = \alpha P_{\text{bip}}^{\top} T^{(t)} + (1-\alpha)v, $$ where $P_{\text{bip}}$ is the block bipartite transition matrix and $v$ is the uniform teleportation vector [2602.12424]. By Perron–Frobenius for irreducible, aperiodic chains, the method has a unique stationary distribution with strictly positive entries and geometric convergence [2602.12424]. The primary RankEval output on the model side is the competency score $$ C_i = T_{M,i}^{\star}, $$ while the question side yields difficulty scores $$ D_j = T_{Q,j}^{\star}. $$ These support a difficulty-weighted accuracy $$ \mathrm{WA}_i = \frac{\sum_{j=1}^{Q} R_{ji}D_j}{\sum_{j=1}^{Q} D_j}, $$ as well as weighted error rate and difficulty-stratified performance summaries [2602.12424]. Empirically, RankLLM is evaluated on 30 models and 35,550 questions across BBH, GPQA, GSM8K, HellaSwag, MATH, and MMLU-Pro [2602.12424]. It is reported to achieve $90\%$ agreement with human judgments on difficulty, with Cohen’s $\kappa \approx 0.80$, and to outperform IRT baselines that achieve approximately $50$–$63\%$ consensus [2602.12424]. Runtime is reported as $0.00597$ s for the $30 \times 35{,}550$ problem on an Intel i7 CPU, with empirical convergence in approximately $9$ iterations and linear scaling in $Q \times M$ [2602.12424]. ## 4. Rankability, linear ordering, and dataset evaluation A central extension of RankEval is the question of whether data can be meaningfully ranked at all. The rankability literature represents a dataset as a directed graph $G = (V,E)$ with adjacency matrix $A$, where $a_{ij}$ indicates that item $i$ dominates item $j$ [2203.07364]. The ideal object is a complete dominance graph, whose adjacency matrix is strictly upper triangular after relabeling by a single strict order [2203.07364]. Two families of measures are emphasized. The first is combinatorial. Edge Rankability $R_e(G)$ is based on the minimal number of edge edits needed to convert $G$ to a complete dominance graph, together with the multiplicity of optimal such graphs: $$ R_e(G) = 1 - \frac{k p}{k_{\max} p_{\max}}, $$ where $k_{\max} = n(n-1)/2$ and $p_{\max} = n!$ [2203.07364]. This measure is diagnostically rich but computationally expensive because computing $p$ requires enumeration over permutations. The second family is spectral. Spectral Rankability $R_s(G)$ compares the spectra of the out-degree matrix $D$ and directed Laplacian $L = D-A$ to a complete-dominance template $S$: $$ R_s(G) = 1 - \frac{d_H(\sigma(D), \sigma(S)) + d_H(\sigma(L), \sigma(S))}{2(n-1)}. $$ It satisfies $R_s \in [0,1]$ and is efficient to compute relative to $R_e$ [2203.07364]. The same paper proposes a supervised measure $R_f(G)$ using random forest regression with features including directed $3$-cycles, $$ T(G) = \frac{1}{3}\sum_{i,j,k} a_{ij}a_{jk}a_{ki}, $$ directed $2$-cycles $$ C_2(G) = \sum_{1 \le i < j \le n} a_{ij}a_{ji}, $$ out-degree dispersion, directed algebraic connectivity, and missing-edge counts [2203.07364]. On synthetic data, $R_f$ correlates more strongly with target rankability than $R_e$ or $R_s$, especially under sparsity; for example, at $n=50$ it attains $\rho(t,R_f)=0.994$ on complete data and $\rho(t,R_f)=0.976$ in the sparse variant [2203.07364]. The linear-ordering literature provides a related but distinct formalization. Given a nonnegative matrix $A = (a_{ij})$, the Linear Ordering Problem seeks the permutation maximizing concordant weight: $$ \max_{\pi \in S_n} \sum_{i