Tie-Aware Retrieval Metrics (TRM)

• Tie-Aware Retrieval Metrics (TRM) are ranking metrics that handle tied scores by averaging over all possible within-tie permutations, eliminating arbitrary tie-breaking.
• They are applied in both Hamming ranking and low-precision retrieval, using closed-form formulations to accurately compute metrics like AP and NDCG under extensive ties.
• TRM quantifies uncertainty by reporting the expected metric value along with range and bias, leading to more consistent and interpretable evaluation outcomes.

Searching arXiv for relevant papers on tie-aware retrieval metrics, ties in ranking evaluation, and related rank similarity measures. Tie-aware Retrieval Metrics (TRM) are ranking metrics that explicitly account for tied scores or distances by defining the metric value as the average over all possible permutations of the tied items, rather than by committing to an arbitrary tie-breaking rule. In the hashing literature, the term denotes tie-aware versions of Average Precision (AP) and Normalized Discounted Cumulative Gain (NDCG) for Hamming ranking, where integer-valued Hamming distances induce extensive ties (He et al., 2017). In low-precision retrieval, the same term denotes a broader evaluation protocol in which tied candidates are treated as order-uncertain and the evaluation reports the expected metric value together with range and bias, thereby quantifying uncertainty introduced purely by ties (Yang et al., 5 Aug 2025).

1. Origins, problem setting, and motivation

The original motivation for TRM in retrieval arose from supervised hashing for nearest-neighbor retrieval. A hash function

$\Phi:\mathcal{X}\to\{-1,1\}^b$

maps each item to a $b$-bit code, and retrieval is performed by ranking database items in ascending Hamming distance to the query. The induced distance can be written as

$$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$

so only $b+1$ distinct distance values are available. When the database contains tens or hundreds of thousands of items, many items inevitably share the same distance to a query, especially for short codes (He et al., 2017).

This discreteness creates a direct conflict with standard AP and NDCG, whose usual definitions assume a total order. Under Hamming ranking, each distance value defines a tie group

$R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$

and there is no canonical internal order within $R^{(d)}$. Random tie-breaking, deterministic secondary keys such as ID order, and best-case or worst-case tie-breaking all produce metric values that may differ materially for the same retrieval system. The hashing paper argues that this makes comparison ambiguous and motivates tie-aware AP and tie-aware NDCG as deterministic alternatives (He et al., 2017).

A later motivation came from low-precision retrieval. When relevance scores are computed in BF16 or FP16, the reduced mantissa creates spurious ties because distinct underlying scores collide on the representable grid: $s_i=\phi_{(B)}(z_i).$ Documents with the same low-precision score form tie groups

$G_n=\{i\mid s_i=v_n\},$

ordered by distinct score values $v_1>v_2>\dots>v_N$. In this setting, naive evaluation can exhibit high variability and systematic bias. On MIRACL Reranking with Qwen3-Reranker and softmax scoring, the reported BF16 range reaches 25.59 percentage points for nDCG@10 and 38.03 points for MRR@10, while the tie-oblivious scores overestimate the TRM expectation by +6.66 and +9.08 points respectively (Yang et al., 5 Aug 2025). These results place ties at the center of retrieval evaluation rather than at its margins.

2. Formal definition and core principles

The central construction of TRM is to keep the order between tie groups fixed and average the chosen ranking metric over all permutations within each tie group. If $M$ is any ranking metric and $b$0 ranges over permutations that reorder documents only within ties, the tie-aware metric is

$b$1

where $b$2 is the uniform distribution over admissible within-group permutations (Yang et al., 5 Aug 2025). In the hashing formulation, the same idea is expressed as averaging AP or NDCG over all permutations inside each Hamming-distance bin $b$3, yielding a deterministic score equal to the expected standard metric under uniform random tie-breaking (He et al., 2017).

Several properties follow directly from this construction. TRM is tie-aware because it explicitly models the ambiguity inside tie groups; deterministic because the expectation is uniquely defined for fixed scores and labels; and consistent because it reduces to the ordinary metric when every tie group has size $b$4 (He et al., 2017). In the low-precision setting, TRM additionally separates the expected score from the uncertainty induced by ties through two auxiliary quantities: $b$5 and

$b$6

where $b$7 is the conventional tie-oblivious score produced by a specific evaluation pipeline (Yang et al., 5 Aug 2025).

Two recurring parameterizations organize the tied-ranking structure. In Hamming retrieval, the natural statistics are distance-conditioned relevance histograms, with counts

$b$8

their cumulative versions $b$9, and bin totals $$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$0 and $$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$1 (He et al., 2017). In low-precision retrieval, the basic quantities are tie-group sizes $$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$2, the number of relevant documents $$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$3 in each group, the group-wise relevance probability

$$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$4

the cumulative size

$$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$5

and the number of items from group $$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$6 that fall into the top-$$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$7 prefix,

$$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$8

Both parameterizations turn tied ranking into a counting problem rather than a permutation-enumeration problem (Yang et al., 5 Aug 2025).

Source	Retrieval setting	Tie-aware output
"Hashing as Tie-Aware Learning to Rank" (He et al., 2017)	Hamming ranking with integer distances	Tie-aware AP and tie-aware NDCG
"Reliable Evaluation Protocol for Low-Precision Retrieval" (Yang et al., 5 Aug 2025)	BF16/FP16 score ties	$$d_\Phi(x,x')=\frac{1}{2}\big(b-\Phi(x)^\top\Phi(x')\big)\in\{0,1,\dots,b\},$$9, Range, Bias for multiple metrics
"The Treatment of Ties in Rank-Biased Overlap" (Corsi et al., 2024)	Ranking similarity under ties	$b+1$0, $b+1$1, $b+1$2

3. Metric families and closed-form formulations

In the hashing formulation, tie-aware AP is defined for binary affinities and tie-aware DCG/NDCG for graded affinities. The key result is that AP and DCG decompose by Hamming-distance bins. For AP, the contribution of each bin depends on the number of relevant items in that bin, the bin size, and the cumulative counts above the bin; for NDCG, the expected gain at each position inside a tie group is obtained from the relevance histogram of that group. The resulting metrics are exactly the expected values of standard AP and DCG/NDCG under uniform random tie-breaking within each distance bin (He et al., 2017).

The low-precision formulation generalizes this closed-form program to several standard IR metrics under binary relevance. Expected Hits@$b+1$3, Recall@$b+1$4, Precision@$b+1$5, and $b+1$6 follow directly by linearity of expectation from the expected number of relevant documents contributed by each tie group within the top-$b+1$7 prefix. Expected nDCG@$b+1$8 is obtained by weighting each partially observed tie group with the usual DCG discount

$b+1$9

and normalizing by

$R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$0

The same paper also provides closed forms for MRR@$R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$1 and MAP@$R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$2, again without enumerating within-group permutations (Yang et al., 5 Aug 2025).

The metric-specific scope of TRM has therefore expanded over time. In the 2017 hashing setting, TRM specifically denotes tie-aware AP and NDCG specialized to Hamming ranking (He et al., 2017). In the 2025 low-precision setting, TRM becomes a broader evaluation family covering nDCG@10, MRR@10, MAP@3, Recall@$R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$3, Precision@$R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$4, and $R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$5, together with the uncertainty summaries range and bias (Yang et al., 5 Aug 2025). A plausible implication is that TRM is best understood as a design pattern for ranking metrics under score coarsening, rather than as a single fixed metric.

4. Algorithms, efficiency, and optimization

For Hamming retrieval, TRM is computationally efficient because Hamming distance has only $R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$6 bins. The hashing paper computes tie-aware metrics from per-query histograms and states that, for fixed $R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$7, counting sort and histogram operations evaluate the metrics in $R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$8, linear in database size (He et al., 2017). This matters because the tie-aware formulation is not merely more principled than ad hoc tie-breaking; in Hamming space it is also operationally natural.

The same paper goes further by optimizing tie-aware AP and NDCG directly. Since the original problem is discrete—binary codes, integer distances, and hard histograms—it replaces the sign activation with a scaled hyperbolic tangent,

$R^{(d)}=\{i\mid d_\Phi(x_q,x_i)=d\},$9

and defines a relaxed distance

$R^{(d)}$0

Hard binning is replaced by a differentiable triangular kernel $R^{(d)}$1, producing soft histogram counts $R^{(d)}$2. Because the original bin contributions sum over integer ranks, the finite sums are approximated by integrals, yielding smooth surrogates for tie-aware AP and DCG/NDCG that can be optimized by backpropagation. The full Jacobian over a minibatch is implemented in $R^{(d)}$3, and test-time retrieval uses exact Hamming distance on thresholded binary codes (He et al., 2017).

In low-precision retrieval, TRM is purely post hoc. The system first sorts documents by score in the ordinary way, which costs $R^{(d)}$4 for $R^{(d)}$5 candidates. A single scan then aggregates tie groups and relevance counts, and the closed-form metrics run in $R^{(d)}$6 or $R^{(d)}$7 depending on the metric, with at most $R^{(d)}$8 additional arithmetic per query. The stated overhead is therefore far below the cost of the model forward pass or initial sort, and no sampling or permutation enumeration is required (Yang et al., 5 Aug 2025).

5. Reliable evaluation, range, bias, and reporting practice

A defining feature of the newer TRM literature is that it treats expected score as only one component of reliable evaluation. When ties are abundant, the same fixed set of scores and labels induces a family of admissible rankings, so a single tie-oblivious metric cannot characterize evaluation stability. TRM therefore reports $R^{(d)}$9 as the primary performance number, $s_i=\phi_{(B)}(z_i).$0 as the uncertainty due solely to internal tie ordering, and $s_i=\phi_{(B)}(z_i).$1 as the deviation of a legacy tie-breaking procedure from the expected value (Yang et al., 5 Aug 2025).

This framework is paired with High-Precision Scoring (HPS), in which a low-precision model keeps inference in BF16 but upcasts the final scoring step to FP32. HPS reduces the number and size of spurious ties; TRM then quantifies any remaining ambiguity. For the same MIRACL example discussed above, BF16$s_i=\phi_{(B)}(z_i).$2FP32 reduces the nDCG@10 range from 25.59 to 1.13 and the MRR@10 range from 38.03 to 1.21, while bias drops to 0.24 and 0.21 respectively (Yang et al., 5 Aug 2025). The two components are explicitly presented as complementary rather than interchangeable.

The reporting recommendation in this line of work is to publish

$s_i=\phi_{(B)}(z_i).$3

for each metric and cutoff, optionally accompanied by the conventional $s_i=\phi_{(B)}(z_i).$4, Bias, $s_i=\phi_{(B)}(z_i).$5, and $s_i=\phi_{(B)}(z_i).$6 for backward comparability (Yang et al., 5 Aug 2025). In the hashing literature, the analogous practical lesson is that tie-aware AP and NDCG eliminate ambiguity caused by best-case, worst-case, or arbitrary tie-breaking, while remaining close to the average behavior of random tie-breaking over many queries (He et al., 2017). Taken together, these works define a shift from “choose a tie-breaking rule” to “treat tie order as latent uncertainty and evaluate it explicitly.”

6. Related tie-aware ranking formalisms and semantic distinctions

TRM belongs to a broader effort to make ties first-class objects in ranking evaluation. In machine translation meta-evaluation, the paper "Ties Matter: Meta-Evaluating Modern Metrics with Pairwise Accuracy and Tie Calibration" argues that existing Kendall-style variants fail to reward correct prediction of ties and can even be gamed. It proposes pairwise accuracy,

$s_i=\phi_{(B)}(z_i).$7

together with tie calibration that introduces ties by thresholding small score differences (Deutsch et al., 2023). Although this work is not itself a retrieval metric, it sharpens a general point that is central to TRM: ties carry information, and ad hoc handling can systematically distort comparative conclusions.

A closely related development appears in tie-aware Rank-Biased Overlap. "The Treatment of Ties in Rank-Biased Overlap" distinguishes several tie semantics rather than a single universal rule. $s_i=\phi_{(B)}(z_i).$8 interprets ties as uncertainty and equals the expected bare RBO over random tie-breaking; $s_i=\phi_{(B)}(z_i).$9 normalizes by measurable overlap in the style of Kendall’s $G_n=\{i\mid s_i=v_n\},$0; and $G_n=\{i\mid s_i=v_n\},$1 adopts an equality or “sports ranking” semantics in which tied items share rank mass (Corsi et al., 2024). This semantics-driven decomposition is important for TRM because it shows that tie handling is not merely an implementation detail: it encodes assumptions about what a tie means.

The literature therefore supports two distinct but compatible views. One view treats ties as uncertainty about an underlying strict order, leading naturally to expectations, ranges, and untiedness-normalized variants (Yang et al., 5 Aug 2025, Corsi et al., 2024). The other treats ties as genuine equality classes, which motivates equality-style overlap or calibrated pairwise decisions (Corsi et al., 2024, Deutsch et al., 2023). A plausible implication is that the correct TRM variant depends on whether ties arise from coarse measurement, reduced numerical precision, or an intended notion of equivalence.

The acronym itself is not unique across machine learning. In latent world-model planning, "Beyond Euclidean Proximity: Repairing Latent World Models with Horizon-Matched Trajectory Reachability Metrics" uses TRM to mean Trajectory Reachability Metrics and explicitly states that this is not the information-retrieval meaning of Tie-aware Retrieval Metrics (Li et al., 21 May 2026). Within retrieval and ranking evaluation, however, TRM consistently denotes principled handling of tied candidates through expectation-based or semantics-aware metric definitions.

7. Significance, scope, and limitations

The principal significance of TRM is methodological. In both Hamming retrieval and low-precision retrieval, tie-oblivious metrics can be unstable, implementation-dependent, and capable of reversing model comparisons. TRM replaces this arbitrariness with deterministic expectations and, in the newer protocol, with explicit uncertainty quantification through range and bias (He et al., 2017, Yang et al., 5 Aug 2025). This suggests that ties should be treated as an intrinsic part of the observable ranking structure whenever scores are coarse, quantized, or intentionally bucketed.

The scope of current TRM formulations is nevertheless bounded by their assumptions. The low-precision protocol assumes uniform random permutation within each tie group; the paper notes that real systems may instead impose structured non-uniform tie-breaking based on recency or ID order, in which case the expectation model would need to change (Yang et al., 5 Aug 2025). The hashing formulation is specialized to Hamming ranking and derives its efficiency from the finite set of distance bins (He et al., 2017). More generally, extensions to additional metrics require new derivations rather than automatic inheritance (Yang et al., 5 Aug 2025).

At the same time, the available evidence points to a broad domain of applicability. The low-precision paper explicitly identifies BF16 or FP16 inference, quantized embeddings, and other systems with coarse or bucketed scores as settings in which TRM should be used (Yang et al., 5 Aug 2025). The hashing paper makes the same argument for short binary codes, where ties are especially prominent (He et al., 2017). Related work on RBO and meta-evaluation further indicates that tie awareness is relevant not only to effectiveness metrics but also to rank similarity measures and pairwise ranking diagnostics (Corsi et al., 2024, Deutsch et al., 2023).

In this sense, Tie-aware Retrieval Metrics constitute a family of principled ranking-evaluation methods for tied outputs. Their common core is the replacement of arbitrary tie-breaking by explicit modeling of within-group order uncertainty; their concrete realizations range from tie-aware AP and NDCG in Hamming space to expectation, range, and bias reporting for low-precision retrieval systems (He et al., 2017, Yang et al., 5 Aug 2025).