TREC 2022 Fair Ranking Track Overview

• TREC 2022 Fair Ranking Track is defined as a benchmark that requires retrieval systems to optimize both topical relevance and fair exposure across diverse demographics in Wikipedia.
• The track employed two tasks: one for a single ranking mimicking WikiProject coordinators and another for repeated rankings reflecting individual editor interactions.
• The evaluation framework integrated fairness and relevance metrics (e.g., AWRF and nDCG) to quantify exposure disparities and drive methodological advances.

Searching arXiv for the cited TREC Fair Ranking Track papers and closely related work. The TREC 2022 Fair Ranking Track was a TREC evaluation forum for retrieval systems that were required to optimize not only topical relevance but also fair exposure across demographics and attributes represented by relevant documents. It adopted a resource allocation formulation for English Wikipedia: rankings were intended to help WikiProject coordinators and Wikipedia editors identify articles needing improvement, while ensuring that documents about, or otherwise representing, protected characteristics received a fair opportunity to be surfaced and improved. In the track’s framing, rank position controlled opportunity, and unequal exposure could perpetuate systematic bias in encyclopedia coverage (Ekstrand et al., 2023).

1. Problem formulation and intellectual context

The track was rooted in the fair exposure literature, but it recast fair ranking in a Wikimedia-specific workflow. Rather than treating ranking as a purely relevance-driven retrieval problem, it treated editor attention as the resource being allocated. This changed the operational meaning of fairness: a ranking was not only a list of useful articles, but also a mechanism that could amplify or mitigate under-representation in Wikipedia coverage (Ekstrand et al., 2023).

The benchmark specifically aimed to provide a platform for participants to develop and evaluate retrieval algorithms that could provide a fair exposure to a mixture of demographics or attributes, such as ethnicity, represented by relevant documents in response to a search query. The motivating scenario involved WikiProject coordinators and Wikipedia editors searching for documents in need of editing so that the documents would have a fair opportunity of being improved and, therefore, be well-represented in Wikipedia. The overview explicitly linked under-representation of protected characteristics in Wikipedia to systematic biases with negative human, social, and economic impact (Ekstrand et al., 2023).

A central conceptual point was that the track did not reduce fairness to a single protected-versus-unprotected disparity. The official evaluation emphasized multidimensional and intersectional exposure, especially over geography and gender. This made the benchmark substantially more demanding than binary-group formulations that dominate part of the earlier fair-ranking literature (Ekstrand et al., 2023).

2. Task design: single ranking and repeated rankings

The 2022 track defined two tasks over the same corpus and topics, differing in how exposure was allocated.

Task 1 modeled a WikiProject coordinator who wanted a single ranked list of candidate articles for a project. Each system output one ranking of 500 articles per query. Evaluation was explicitly multi-objective: the ranking should be relevant to the WikiProject and fair in exposure across protected attributes (Ekstrand et al., 2023).

Task 2 modeled individual Wikipedia editors who repeatedly consulted a saved search. Instead of one list, the system had to output 100 rankings per query, each of length 20. This design allowed fairness to be assessed over a sequence of rankings rather than a single ranked list, making repeated interactions part of the formal evaluation (Ekstrand et al., 2023).

Task 2 also included a work-needed dimension. More urgent articles, as measured by a quality model, were to be preferred among equally relevant articles. The overview is explicit that this was not fairness over work-needed; rather, work-needed was part of graded relevance for the editor use case. This distinction is important because it separates fairness in exposure from the editorial priority signal used to rank relevant material (Ekstrand et al., 2023).

The two-task structure encoded two different views of ranking behavior. Task 1 evaluated a deterministic list; Task 2 evaluated a stochastic policy through repeated rankings. A plausible implication is that the track was designed to test both immediate fairness in a single display and amortized fairness over time.

3. Corpus, topics, and fairness annotations

The corpus was a subset of English Wikipedia with redirects removed and wikitext preserved. It was distributed as trec_corpus.json.gz, with each record containing id, title, and url, and the content was licensed under CC BY-SA 3.0. Queries were derived from WikiProjects and represented primarily by extracted keywords. Those keywords were generated by cleaning and parsing relevant WikiProject articles and then using KeyBERT to extract representative terms. Query metadata included id, title, keywords, homepage, and rel_docs, where rel_docs listed relevant page IDs derived from existing Wikipedia data (Ekstrand et al., 2023).

The track also distributed fairness and metadata annotations in trec_2022_articles_discrete.json.gz and trec_metadata.json.gz. These included geographic location of article topic, geographic location of sources, gender, age of the topic, occupation, alphabetical bucket, article age, popularity by pageviews, number of language sitelinks, and article quality score. The protected or fairness-relevant dimensions used by the benchmark were subject geography, source geography, gender, occupation, alphabetical order, age of topic, article age, popularity, and multilingual sitelinks (Ekstrand et al., 2023).

For Task 1 and Task 2, evaluation was intersectional over geographic and gender dimensions, with the remaining dimensions used in dimension-specific analyses. The overview emphasizes that the official evaluation focused on an intersectional fairness space formed by combining dimensions, yielding over 11 million possible joint categories. That scale made implementation nontrivial (Ekstrand et al., 2023).

Several preprocessing choices materially shaped the benchmark. Geography was mapped from countries to UN continental subregions, with Oceania subregions collapsed to Oceania. Unknown values were treated explicitly as their own category rather than ignored. Gender was reduced to four categories: @UNKNOWN, female, male, and NB (nonbinary/other). Occupations were mapped into 32 higher-level occupation categories. Pageviews were bucketed into four popularity classes, sitelinks into English-only, 2–4 languages, and 5+ languages, and article age and topic age into discrete time periods (Ekstrand et al., 2023).

The benchmark documentation also identified important label limitations. Gender inference was described as imperfect and ethically fraught; geographic provenance from sources could be noisy; and relevance labels from WikiProjects were incomplete because WikiProject tagging itself was incomplete. The benchmark was therefore presented as an approximation to fairness rather than a complete representation of it (Ekstrand et al., 2023).

4. Evaluation framework and metric structure

The track used standard logarithmic attention discounting, with

$\weight_i = \frac{1}{\log_2 \max(i, 2)}.$

For Task 1, the exposure assigned to groups in a ranking was accumulated as

$\dlist' = \sum_i \weight_i \avec{\ranking_i},$

then normalized:

$\dlist = \frac{\dlist'}{\|\dlist'\|_1}.$

The target distribution was defined as the average of the empirical distribution of relevant documents and the world population distribution:

$\dtarget = \frac{1}{2}\left( \amat^\transpose \relvec{\query} + \avec{\world} \right).$

Attention-Weighted Rank Fairness was then

$\AWRF(\ranking) = 1 - \Djs{\dlist}{\dtarget},$

and the official Task 1 score multiplied fairness and utility:

$\MOne(\ranking) = \AWRF(\ranking)\times \nDCG(\ranking).$

This product operationalized the principle that a system should be rewarded only if it was both relevant and fair (Ekstrand et al., 2023).

Task 2 evaluated expected exposure over repeated rankings. The exposure of a document over a sequence of rankings was

$\docExp = \frac{1}{|\rankSeq|} \sum_{\ranking \in \rankSeq} w_{\rankInv_\doc},$

which was aggregated into group exposure:

$\groupExp = \amat^\transpose \expVec.$

The final Task 2 objective was expected-exposure loss,

$\MTwo(\rankSeq_\query) = \| \groupExp - \groupExp^* \|_2,$

with lower values better. The overview further decomposed this into EE-D and EE-R, reporting EE-L as the main score, EE-D as disparity, and EE-R as relevance/exposure alignment (Ekstrand et al., 2023).

Comparative work on fair ranking metrics is directly relevant to these choices. One survey characterizes AWRF as a more general single-ranking metric based on explicit position weighting, in contrast to prefix-based metrics with hardcoded discounting. It also recommends AWRF for single rankings because it supports multinomial groups, soft association or uncertain membership, multiple attention models, and configurable target distributions and distance functions (Raj et al., 2020). This suggests why AWRF fit a benchmark with multiple fairness dimensions, partial observability, and intersectional categories.

5. Participation, submitted methods, and official results

The track reported 5 participating teams and 24 total runs. All 5 teams submitted Task 1 runs, and 2 teams submitted Task 2 runs (Ekstrand et al., 2023).

For Task 1, reported approaches included ColBERT with pseudo-relevance feedback, ColBERT end-to-end plus heuristic reranking for target exposure, query rewriting, BM25 from Pyserini combined with BERT semantic scoring and greedy diversification, BM25 with LambdaMART and MLP reranking, Reciprocal Rank Fusion to diversify the ranking, and relevance-only baselines. The broad pattern was to begin with strong relevance rankers and then post-process or rerank to improve fairness, often through diversification heuristics or target-exposure constraints (Ekstrand et al., 2023).

The best Task 1 run was tmt5, with nDCG $= 0.7242$, AWRF $\dlist' = \sum_i \weight_i \avec{\ranking_i},$0, and score $\dlist' = \sum_i \weight_i \avec{\ranking_i},$1. The overview notes that the top system was not the one with the best fairness or the best relevance alone, but the one that balanced both reasonably well. The UoG runs achieved slightly higher AWRF but lower nDCG, and therefore lower overall score. The authors further observed that, unlike the previous year, submitted approaches were less clustered by team and more spread across the relevance–fairness plane (Ekstrand et al., 2023).

For Task 2, approaches included multi-armed bandit selection among rankings, epsilon-greedy and epsilon-decay selection, randomization, and relevance-only ranking. The best overall run was UoGTrMabWeSA, with EE-L $\dlist' = \sum_i \weight_i \avec{\ranking_i},$2, where lower EE-L was better. The overview notes a strong trade-off: some systems did better on relevance/exposure alignment, while others did better on disparity, reinforcing the fact that stochastic ranking fairness objectives need not move together (Ekstrand et al., 2023).

Per-dimension analyses showed more variation across fairness attributes than in the overall metrics. This suggests that the intersectional objective was robust enough to produce a stable top run, while still exposing attribute-specific stresses that different systems handled differently.

6. Subsequent use, methodological influence, and unresolved issues

The TREC 2022 Fair Ranking Track rapidly became a test collection for later methodological studies. A 2024 learning-to-rank paper used the TREC fair ranking Wikipedia task and explicitly evaluated on the track’s 2021 and 2022 evaluation queries. It framed fair ranking as finding a permutation $\dlist' = \sum_i \weight_i \avec{\ranking_i},$3 for each query such that the ranking was both useful and fair, adopted an exposure-distribution view of fairness, and proposed a distribution-based fair learning framework in which unavailable fairness labels were replaced by target fairness exposure distributions. In that study, the TREC fair ranking setting was described as involving ranking large document collections with binary relevance labels and fairness annotations over multiple sensitive categories, while coping with the practical lack of point-wise fairness labels (Chen et al., 2024).

That 2024 study also used contextual features derived from the Wikipedia corpus, including BM25 and sentence-embedding-based similarities to gender and geographic-location embeddings, and reported experiments on a dataset described as over six million English Wikipedia articles with full text, 50 training queries, and 50 evaluation queries. It used nDCG@20 and AWRF@20, while noting that the official 2022 evaluation used @500. Its main conclusion was that label-free, distribution-based fairness learning was feasible and effective on the official Wikipedia benchmark, and it explicitly argued that diversification-style rerankers such as MMR did not reliably ensure fair exposure (Chen et al., 2024). A plausible implication is that the track exposed a methodological gap between generic diversification and fairness-aware optimization.

A 2026 study used the TREC 2022 Fair Ranking Track dataset to compare reasoning and non-reasoning rerankers. It evaluated six reranking models across multiple retrieval settings and demographic attributes, using AWRF and nDCG. Its central finding was that reasoning neither improved nor harmed fairness in a meaningful way: AWRF remained stable at about $\dlist' = \sum_i \weight_i \avec{\ranking_i},$4–$\dlist' = \sum_i \weight_i \avec{\ranking_i},$5 across all models, even as nDCG ranged from $\dlist' = \sum_i \weight_i \avec{\ranking_i},$6 to $\dlist' = \sum_i \weight_i \avec{\ranking_i},$7. The same work reported persistent fairness gaps for geographic attributes, especially source geography and subject geography, regardless of model architecture (Samuel et al., 11 Mar 2026). This indicates that later advances in relevance modeling did not, by themselves, resolve the fairness challenges encoded by the benchmark.

The overview paper itself was explicit about unresolved issues. The gender taxonomy was simplified and incomplete; geography and source provenance were noisy and sometimes inferred; WikiProject relevance labels were incomplete; work-needed was only a coarse proxy; and the benchmark only covered topics for which English Wikipedia already had articles, leaving missing-article bias and deletion bias outside scope (Ekstrand et al., 2023). The benchmark also did not directly include harder-to-model dimensions such as religion, sexuality, culture, or race.

Taken together, these later studies and official limitations show that the TREC 2022 Fair Ranking Track functioned as more than a one-year shared task. It established a concrete resource-allocation benchmark for fair exposure in retrieval, made multidimensional and intersectional evaluation operational, and exposed persistent tensions among relevance, exposure, data quality, and attribute visibility that continued to shape subsequent research (Ekstrand et al., 2023).