Lexicographic Recall

• Lexicographic recall is an evaluation metric that orders retrieval results using the leximin principle to prioritize the worst-ranked relevant items.
• It refines traditional recall metrics by providing a total ranking order that discriminates between systems even in sparse-labeling and high-recall scenarios.
• Empirical studies demonstrate that lexicographic recall enhances stability and fairness by aligning worst-case outcomes for both content consumers and providers.

Lexicographic recall, also known as lexirecall, is a preference-based evaluation measure for ranked retrieval and recommendation tasks that applies a lexicographic, or leximin, ordering to the vector of relevant-item positions in a system’s output. It offers a principled refinement of recall metrics grounded in worst-case, or Rawlsian, robustness for both content consumers and providers. Lexicographic recall yields a total ordering over system outputs by prioritizing the ranking of the worst-off (deepest) relevant items, enabling highly discriminative and stable comparisons especially in high-recall and sparse-labeling scenarios (Diaz et al., 2023).

1. Motivation and Theoretical Foundations

Traditional recall-oriented metrics, such as recall at $k$ (R@$k$), average precision (AP), and R-precision (RP), are widely adopted to evaluate the capacity of systems to retrieve all relevant items for a given query or user. However, these methods often lack a formal justification for their sensitivity to the retrieval of every relevant item. Diaz and Mitra (Diaz et al., 2023) formalize the concept of recall-orientation by introducing the recall valence $\Delta_\text{recall}(M; m,n)$, which quantifies the decrease in a metric $M$ when the lowest-ranked relevant item is pushed from its current position to the end of the list.

A perfectly recall-oriented metric would depend solely on the position of the last relevant item. To capture this, the authors introduce Total Search Efficiency (TSE):

$\text{TSE}_e(\sigma, R) = e(p_m)$

where $e(k)$ is a monotonically decreasing exposure function (e.g., $e(k)=1/k$ for AP) and $p_m$ is the position of the last relevant. TSE is maximized when all relevant items are retrieved as early as possible. However, TSE often induces many ties and is insensitive to the distribution of other relevant items.

To address this, lexicographic recall applies the leximin (lexicographic minimum) principle: given two systems' relevant position vectors $p$ and $p'$, the system whose worst-off (deepest) relevant item appears earlier is preferred. If these tie, the second-worst, and so forth, are compared, providing a strict total ordering.

2. Formal Definition and Lexicographic Ordering

Given a set of $k$0 relevant items for a query and their positions in a ranking $k$1, form the sorted vector $k$2.

Lexicographic recall defines the preference between two runs $k$3 and $k$4 as follows:

• Let $k$5 and $k$6 be their respective (sorted) relevant-position vectors.
• Define

$k$7

That is, the ordering is induced by the first (from the bottom, i.e., worst-off) position where the relevant item differs, prioritizing the run that brings that item higher in the rank.

This implements a leximin principle: maximizing the minimum (worst-off), and, in case of ties, the next-worst, etc. The resulting ordering is total and strictly refines the worst-case TSE-based ordering.

3. Robustness and Fairness Interpretations

Lexicographic recall provides compelling guarantees for both content consumers and providers under worst-case utility models:

Consumer-side robustness: For a user seeking any non-empty subset $k$8 of relevant documents, the worst-case utility over all such subsets aligns with the position of the last relevant item ($k$9). Thus, maximizing lexicographic recall improves the experience for the most disadvantaged user (the one seeking to discover all relevant items).

Provider-side robustness: For content providers controlling subsets of relevant documents, worst-case exposure is also dictated by the deepest relevant item among their content ($\Delta_\text{recall}(M; m,n)$0 again). Lexicographic recall thus enforces a Rawlsian difference principle simultaneously for consumers and providers (Diaz et al., 2023).

A leximin refinement (lexicographic recall) breaks ties in these maximin allocations, further prioritizing fairness by ranking the full vector of relevant positions.

4. Empirical Findings and Statistical Properties

An extensive empirical evaluation across 3 recommender-system tasks (MovieLens, LibraryThing, BeerAdvocate) and 17 information retrieval tasks (including multiple TREC tracks) demonstrates key practical advantages for lexicographic recall:

• Agreement with existing metrics: Lexicographic recall shows extremely high agreement with recall-oriented metrics (e.g., ≈97% with R@1000, ≈98% with AP, ≈96% with NDCG) at the pairwise run-comparison level.
• Discriminative power: In deep-pool IR settings, lexicographic recall distinguishes ≈40% of run pairs as significantly different (p<0.05, Holm–Bonferroni), exceeding R@1000 (≈35%), AP (≈30%), and RP (≈25%). The gaps widen under more stringent corrections.
• Stability under missing labels: Lexicographic recall maintains a low tie rate (≈10–15%) and high agreement (>90%) with full-label rankings even under 50% random label deletion, outperforming R@1000 and AP, whose discrimination sharply declines with sparse relevance judgments (Diaz et al., 2023).

These empirical properties confirm that lexicographic recall provides both faithful tracking of classical recall and enhanced sensitivity/stability, particularly beneficial in sparse and high-recall contexts.

5. Connection to Lexicographic and Subset-Lex Orderings

Lexicographic recall's ordering is mathematically equivalent to a lexicographic (or subset-lex) comparison over sorted position vectors. In combinatorial contexts, subset-lex order defines a lexicographic ordering on sets represented as sorted lists of their elements. It is used to generate all subsets of a ground set in lexicographic order, enabling efficient enumeration and manipulation of combinatorial objects (Arndt, 2014).

In the context of search evaluation, lexicographic recall can thus be viewed as applying subset-lex principles to the vector of relevant positions produced by a system, providing a natural and computationally straightforward basis for leximin-style comparisons.

6. Implications and Adoption Criteria

Lexicographic recall fills a critical gap in ranking evaluation by:

• Grounding recall in a precise operational definition: maximizing the earliest possible retrieval of all relevant items.
• Linking evaluation to robust fairness guarantees for both searchers and content providers via Rawlsian principles.
• Achieving substantially improved experimental power and statistical sensitivity, especially in settings with large or incomplete relevance sets.

Its adoption is particularly justified in experimental scenarios focused on total recall, high-recall robustness, or fairness across user/provider groups, where classical averages or cumulative metrics may fail to capture worst-case or tail behavior (Diaz et al., 2023).

7. Related Combinatorial Methods and Generalizations

The concept of lexicographic order, and its algorithmic realization as subset-lex order, plays a foundational role in combinatorial generation. Efficient, loopless algorithms for subset-lex enumeration, as well as their Gray code variants, exist for generating subsets, multiset combinations, compositions, partitions, and restricted growth strings with minimal computational overhead (Arndt, 2014).

These methods underpin the practical implementation of lexicographic evaluation in ranking systems and may further inspire research into minimal-change or Gray code variants of lexicographic recall for fast, incremental updates in live or streaming retrieval contexts.