QRI Card in Search Evaluation

• The paper introduces two QRI card formulations that combine multi-facet relevance evaluation with impression-based behavioral grounding for search ranking.
• QRI cards capture key signals—including topical relevance, snippet usefulness, attractiveness, and position bias—to separately estimate click-mediated and snippet-mediated utilities.
• Empirical results show improved ranking metrics such as Spearman’s ρ and sign-alignment, while addressing practical challenges like debiasing and annotation consistency.

Query-Relevance-Impressions (QRI) card is a compact representation for search evaluation that couples relevance evidence with information about how results are presented or historically exposed. In one formulation, associated with multi-aspect relevance evaluation, it records for each rank the topical relevance of the document, the immediate usefulness of the snippet, the attractiveness of the snippet, and a position-bias parameter, so that click-mediated utility and snippet-mediated utility can be evaluated separately and then combined (Chuklin et al., 2015). In a later behavior-grounded LLM evaluation setting, a QRI card for a SERP item is a small table of similar historical queries together with a debiased relevance estimate and impression count, appended to the item as a lightweight, auditable behavioral prior that the judge may cite when semantic ambiguity is difficult to resolve (Vardasbi et al., 1 Jul 2026).

1. Conceptual basis

The central idea behind a QRI card is that search quality is not exhausted by topical matching alone. Chuklin and de Rijke distinguish three facets of relevance. Topical relevance is the classic notion: a document is judged on how well its content satisfies the information need expressed by the query, using the standard graded scale used in TREC, such as $0=$ not relevant through $4=$ highly relevant. Snippet relevance is the immediate usefulness of the snippet itself, even without clicking; raters are asked whether the short summary answers the user’s question fully, partially, or not at all. Perceived relevance, or attractiveness, is how likely the user is to click based on the snippet alone (Chuklin et al., 2015).

This tri-partite decomposition makes presentation part of the evaluation target rather than a nuisance variable. A result can be topically strong but unattractive, attractive but topically weak, or useful at the snippet level even when no click is needed. That distinction is the conceptual substrate of the early QRI-style formulation.

The “impressions” component becomes more explicit in later work. In impression-aware recommender systems, an impression is defined as a selection $S \subseteq I$ of $N$ items shown on a user’s screen at time $t$, and the unified event representation is $e=(u,i,y,S)$, where $u$ is the user, $i$ is the item actually interacted with or $\emptyset$, $y$ is observed feedback, and $4=$0 is the impression slate (Maurera et al., 2023). This broader formalization clarifies why QRI cards evolved toward explicit exposure statistics: once impressions are treated as first-class evidence, evaluation can incorporate not only semantic fit but also historically observed user response under exposure.

2. Main formulations

Two formulations of the QRI card are especially salient in the literature.

Formulation	Card contents	Role
Multi-facet relevance card	$4=$1, $4=$2, $4=$3, $4=$4	Offline utility evaluation
Behavior-grounded QRI card	$4=$5 for top-$4=$6 similar queries	Behavioral grounding for LLM judgment

In the multi-facet formulation, the unit of analysis is a ranked list for a query. For each rank $4=$7, the representation records topical relevance $4=$8, snippet-usefulness $4=$9, attractiveness $S \subseteq I$0, and a position bias parameter $S \subseteq I$1. The card then reports two sub-scores, $S \subseteq I$2 and $S \subseteq I$3, together with an aggregate $S \subseteq I$4. The intended diagnostic use is explicit: an IR engineer can see whether a result is topically relevant but poorly summarized, or attractive but weakly informative (Chuklin et al., 2015).

In the behavior-grounded formulation, the unit of analysis is an individual SERP item $S \subseteq I$5. For each item, the QRI card is a small table of up to $S \subseteq I$6 historical queries similar to the current query $S \subseteq I$7, each paired with an IPS-derived relevance estimate $S \subseteq I$8 and impression count $S \subseteq I$9. In practice $N$0, and the historical queries are ranked by cosine similarity over embeddings of $N$1 and $N$2. The card is attached directly to the prompt of a plain LLM judge; the only architectural difference between the plain and behavior-grounded judge is the inclusion of this QRI block under each item (Vardasbi et al., 1 Jul 2026).

The two formulations are not identical, but they address the same structural problem: evaluation that depends only on semantic topicality omits evidence carried by presentation, exposure, and user behavior. The earlier formulation captures this through snippet utility and attractiveness; the later one captures it through impressions and debiased interaction summaries.

3. Mathematical formalization

Chuklin and de Rijke show that any click-model-based metric $N$3 can be written as

$N$4

where $N$5 is topical relevance and $N$6 is the probability that a user clicks result $N$7. Under the Dependent Click Model (DCM), click probability factors into an attractiveness parameter $N$8 and position-dependent examination parameters $N$9:

$t$0

which yields

$t$1

To model snippet-level utility accrued merely by reading snippets, examination probability is defined as

$t$2

and the snippet-utility metric becomes

$t$3

with the DCM form

$t$4

A total-utility score is then obtained as either $t$5 or a weighted blend $t$6 (Chuklin et al., 2015).

The behavior-grounded formulation defines the QRI card for item $t$7 as

$t$8

Here $t$9 is the total number of times item $e=(u,i,y,S)$0 was shown for query $e=(u,i,y,S)$1 in the chosen log window:

$e=(u,i,y,S)$2

The debiased relevance estimate $e=(u,i,y,S)$3 is computed by inverse-propensity scoring of clicks:

$e=(u,i,y,S)$4

The same paper notes that naïve click-through rate would be $e=(u,i,y,S)$5, and gives an optional Bayesian smoothing expression $e=(u,i,y,S)$6, but states that the implementation relies on IPS directly and does not introduce an extra smoothing prior (Vardasbi et al., 1 Jul 2026).

These two mathematical programs differ in what is being estimated. The former estimates utility from relevance facets and a click model; the latter estimates a behavior-grounded prior from historical logs and injects it into an LLM judgment pipeline.

4. Construction and annotation procedures

For the multi-facet formulation, the three signals are collected separately. Topical relevance replicates a standard TREC-style task: the rater sees the query plus the full document title and a snippet, and judges relevance on a $e=(u,i,y,S)$7–$e=(u,i,y,S)$8 scale; seed topics with known labels are used to monitor rater quality. Snippet relevance is collected by showing only the snippet, with no title and no URL, and asking whether the short summary fully, partially, or not at all answers the query; snippet order is randomized and control items are drawn from a held-out set. Perceived relevance is collected by showing only the snippet text in isolation and asking, “Imagine you typed [query]. Would you click this result to find your answer?”, with either binary or graded labels. The design recommendations are explicit: keep tasks separate so raters do not conflate judgments, avoid showing position or other SERP chrome, rotate answer scales, monitor inter-annotator agreement, pilot small batches, use a mixture of binary and graded control questions, and target Cohen’s $e=(u,i,y,S)$9. The same source also recommends automatic query-type classification so that only informational queries receive all three judgments, since for navigational queries snippet relevance is often irrelevant (Chuklin et al., 2015).

For the behavior-grounded formulation, the card is constructed from logs. Over the preceding month, all tuples $u$0 are collected for candidate items. Historical queries are filtered to exclude near-duplicates of the current evaluation query, and the pseudocode further applies the condition $u$1. After computing $u$2 and $u$3, the top-$u$4 queries are selected by semantic similarity, with $u$5. The prompt presented to the LLM includes item metadata and then a QRI block such as $u$6, $u$7, and so on. The system prompt instructs the model to use QRI cards as supporting evidence and not as absolute ground truth. The architecture is a single-turn API call (Vardasbi et al., 1 Jul 2026).

Construction procedures therefore mirror the type of signal being encoded. Crowdsourced QRI emphasizes orthogonal human judgments over topicality, snippet usefulness, and click-likelihood. Log-derived QRI emphasizes auditable aggregation, debiasing, and prompt-level conditioning for LLM evaluators.

5. Effects on ranking and evaluation

In the multi-facet framework, adding snippet relevance and perceived relevance can change which systems appear strongest. A system with an $u$8 chance that its top snippet immediately answers a query, corresponding to high $u$9, but only middling click-attraction, corresponding to low $i$0, scores well in $i$1 but only moderately in $i$2. Conversely, a system with very high $i$3 but poor topicality may generate clicks that lead to low-$i$4 documents, lowering $i$5. Chuklin and de Rijke propose estimating these effects by correlating $i$6 with real-world session logs or by side-by-side comparisons, and the same summary notes that early simulations, citing Turpin and Fitzpatrick (2009) and Chapelle and Zheng (2009), showed that incorporating perceived relevance can reorder TREC runs substantially and align offline evaluation more closely with live user satisfaction (Chuklin et al., 2015).

In the behavior-grounded LLM judge, the empirical gains are reported on three evaluation regimes. On logs-derived evaluation over 5,965 recomposed SERPs, page relevance is taken as a DCG-weighted average of log-derived $i$7, and Spearman’s $i$8 rises from $i$9 for the plain judge to $\emptyset$0 for the behavior-grounded judge, a $\emptyset$1 relative improvement. On the 918 flipped cases where the two judges disagree, correlation rises from $\emptyset$2 to $\emptyset$3, a $\emptyset$4 relative improvement. On a human-judged multilingual dataset comprising 265 SERPs across five languages, Spearman’s $\emptyset$5 increases from $\emptyset$6 to $\emptyset$7, a $\emptyset$8 relative gain; on the disagreement subset of 27 SERPs, it shifts from $\emptyset$9 to $y$0. Against outcomes from a live A/B test over 904 queries comparing Ranking Model A and B, sign-alignment with the live winner increases from $y$1 to $y$2, a gain of $y$3 points that is reported as statistically significant. The same study further reports that alignment improves more rapidly as SERP-level $y$4 grows and is significant by $y$5 (Vardasbi et al., 1 Jul 2026).

These findings support a narrow but important conclusion: when ambiguity or presentation effects matter, explicit relevance-and-impression evidence can change both offline ranking conclusions and the degree to which offline evaluation tracks live preferences.

6. Adjacent research programs

QRI cards sit within a broader set of data-centric and relevance-aware research programs, although those programs do not all use the term “QRI.” In multilingual e-commerce product search, one data-centric framework fixes the LLM backbone and instead redesigns the data through translation-based augmentation, semantic negative sampling, and self-validation filtering for Query-Category and Query-Item relevance. The reported development-set gains for Qwen3-14B are from $y$6 to $y$7 on QC and from $y$8 to $y$9 on QI, with translation providing the largest single gain and post-sampling class balance moving from roughly $4=$00 or $4=$01 positive-to-negative toward $4=$02 (Yin et al., 24 Oct 2025). This is not itself a QRI-card formulation, but it exemplifies the same shift toward richer, better-conditioned relevance evidence.

Automated query-product relevance labeling with LLMs addresses another adjacent problem: generating scalable judgment data. That work formulates relevance labeling as prediction over discrete relevance classes, uses few-shot prompting, Chain-of-Thought prompting, and Retrieval-Augmented Generation with Maximum Marginal Relevance, and reports that CoT improves labeling quality by up to 5 percentage points on ESCI while RAG+MMR yields 2–5 points higher accuracy than random few-shot or vanilla prompting. The best reported LLM configuration reaches weighted $4=$03 on ESCI, $4=$04 on WANDS, and $4=$05 on the Walmart Mexico dataset, compared with a human baseline weighted $4=$06 on the latter (Sachdev et al., 21 Feb 2025).

RE-RAG contributes a complementary notion of confidence-bearing relevance. Its relevance estimator takes each $4=$07 pair and outputs a normalized relevance score based on the probabilities of the tokens “true” and “false,” enabling threshold-based identification of useful or irrelevant contexts, abstention through an “unanswerable” decision, and reweighting of answer marginalization. On Natural Questions and TriviaQA, its F1 for detecting sets without any answer inside top-25 is $4=$08 versus $4=$09 for the baseline retriever, and plugging the estimator into GPT-3.5 Turbo yields gains of $4=$10 EM on NQ and $4=$11 EM on TQA by re-ranking contexts (Kim et al., 2024).

In composed image retrieval, QuRe addresses false negatives through a reward-model objective and a hard negative sampling strategy defined by the two steepest drops in sorted relevance scores below the target. The method reports $4=$12 and $4=$13 on FashionIQ, $4=$14 and $4=$15 on CIRR, a Preference Rate of $4=$16 on HP-FashionIQ, and $4=$17 on CIRCO zero-shot evaluation (Kwak et al., 16 Jul 2025). Although this work is outside SERP evaluation, it addresses a structurally similar question: retrieval quality should reflect broader query relevance and human preference, not only the presence of a single designated target.

7. Limitations and open problems

The limitations of QRI cards follow directly from their evidence sources. In the log-grounded formulation, residual biases in logs, including presentation bias and demographic skew, can propagate into QRI cards if not fully corrected. Cold-start and long-tail queries with low $4=$18 force the behavior-grounded judge to fall back on plain semantic reasoning. Prompt-length constraints currently cap the card at $4=$19 lines, and future extensions proposed in the same source include richer compression, dwell-time signals, session context, richer counterfactual estimators, and prompt-level safety checks to prevent over-reliance on noisy behavioral priors (Vardasbi et al., 1 Jul 2026).

In the multi-facet formulation, the main risks are annotation-related. Rater confusion is explicitly identified as a challenge, motivating orthogonal task design, clear instructions, and practice or test questions. Special SERP items such as images and widgets either need to be excluded from multi-aspect judging or handled with dedicated tasks. Query category also matters: snippet relevance is often irrelevant for navigational queries, so those queries should be detected automatically and snippet judgments skipped (Chuklin et al., 2015).

The broader impression-aware literature adds further unresolved issues. Bias-aware evaluation often requires IPS, self-normalized IPS, or counterfactual estimators that re-weight by $4=$20; open questions include whether exposure propensities can be estimated purely from impression logs and how IPS should be extended to slate-level recommendations. Other unresolved directions include fairness and long-tail exposure, explicit modeling of user fatigue, and richer treatment of non-clicked impressions, where surveyed systems either assume non-clicks are negative or try to learn whether inaction reflects dislike or neutrality (Maurera et al., 2023).

Taken together, these limitations indicate that QRI cards are best understood as auditable summaries of partial evidence. They do not replace semantic relevance judgments, user modeling, or causal correction; rather, they organize those signals so that search evaluation can reflect topical fit, snippet usefulness, attractiveness, and historically observed behavior within a single evaluative interface.