As It Was: Aligning LLM Search Evaluation with Historical User Preferences

Abstract: Large-scale search systems evolve faster than human quality assurance can scale, especially for long-tail intents and multilingual queries. LLM-as-a-judge approaches provide a scalable alternative for evaluating the relevance of search engine result pages (SERPs), but judgments based solely on semantic similarity or world knowledge can drift from actual user preferences, particularly for ambiguous queries. We introduce a behavior-grounded LLM judge that augments each SERP item with a lightweight and auditable behavioral prior in the form of a Query-Relevance-Impressions (QRI) card. Each card summarizes how users have historically interacted with similar queries and results, providing compact empirical evidence that the judge can cite to resolve ambiguity and make more consistent relevance judgments while still relying on semantic reasoning. In a large-scale music search evaluation at Spotify, using relevance estimates derived from historical user interactions across 6,000 recomposed SERPs, the behavior-grounded judge achieves stronger alignment with user preferences, improving Spearman rank correlation by approximately 5% overall and yielding a 91% relative improvement on disagreement cases. On a multilingual human-judged dataset spanning five languages, grounding further increases correlation with human relevance judgments by 15%. Importantly, when evaluated against outcomes from a live A/B test, the grounded judge shows consistently higher alignment with the observed winning model. While absolute alignment remains moderate, these findings demonstrate that lightweight behavioral grounding can improve the reliability and practical usefulness of LLM-based evaluation in real-world search systems.

• The paper introduces a behavior-grounded judge that leverages QRI cards to combine semantic reasoning with empirical user interaction data.
• It demonstrates significant improvements in rank correlation metrics, including a 91% relative gain for ambiguous queries using production music search logs.
• The method enhances evaluation by calibrating errors and tuning intra-SERP rankings across multilingual and ambiguous query scenarios.

Aligning LLM-Based Search Evaluation with Historical User Preferences: Behavior-Grounded Judges for SERP Assessment

Motivation and Problem Setting

Industrial-scale search systems experience rapid evolution in ranking algorithms, content catalogs, and user expectations. This rate of change challenges the scalability and representativeness of human quality assurance, particularly for long-tail or ambiguous queries and in multilingual contexts. The widespread adoption of LLM-as-a-judge paradigms accelerates relevance assessment by prompting advanced LLMs to label the relevance of SERPs. However, LLMs acting without direct empirical context often base judgments solely on semantic similarity and general world knowledge, which may diverge from real user preferences—especially as user intent can vary subtly by query context, locale, and even recent trends. This discrepancy motivates augmenting LLM judgments with interaction-derived behavioral evidence to better align evaluation with actual user behavior.

Behavior-Grounded Judge Architecture

To bridge the empirical gap in LLM-based evaluation, the proposed framework introduces a behavior-grounded judge that supplements the prompt for each SERP result item with a compact Query–Relevance–Impressions (QRI) card. Each QRI card summarizes historical user interaction data—namely, the frequency and estimated relevance (debiased for position bias via inverse propensity scoring) of user engagements for similar queries targeting a specific result. The behavioral summary is curated to focus on the most semantically similar queries, capping at $k=10$ entries for efficiency and prompt length constraints. The QRI cards provide auditable evidence for the LLM to use as a behavioral prior, particularly to resolve ambiguous or underspecified queries where semantic cues alone are insufficient.

This behavioral grounding is strictly auxiliary: the LLM is instructed to use QRI as supporting context rather than as deterministic ground truth, ensuring that semantic and instructional reasoning remain primary when behavioral evidence is sparse or unavailable.

Empirical Evaluation and Results

Experiments are conducted on large-scale production music search logs from Spotify, complemented with multilingual human-annotated datasets spanning five languages. Systematic evaluation assesses both offline alignment—with interaction-derived and human-judged relevance—and online A/B testing alignment between model predictions and observed operational outcomes.

Key empirical findings include:

• Spearman correlation with log-derived user preferences improves by ~5% overall when using the behavior-grounded judge (BG vs. plain judge P), with a relative improvement of +91% for ambiguous ("flipped") cases where the two judges disagree.
• On human-judged multilingual data, BG yields a +15% increase in rank correlation with annotator labels compared to P.
• In live A/B test alignment, BG achieves a 20% relative increase in correctly predicting the online preferred system, with larger gains on queries with richer behavioral support.

Figure 1: Flipped Logs—distribution of interaction-derived relevance vs. LLM judge-assigned label, highlighting that grounding with QRI cards concentrates high-relevance assignments on empirically supported pages.

BG systematically improves alignment without degrading performance on straightforward cases where semantic reasoning suffices. Detailed diagnostics confirm that grounding particularly corrects misalignments in ambiguous queries or where results have closely related but empirically distinct candidates.

Mechanisms and Diagnostic Insights

Behavioral grounding exerts its strongest effects in three recurring scenarios:

• Resolving Ambiguity: For queries with ambiguous intent or regional variability (e.g., lyric fragments or overloaded titles), QRI exposes actual user interpretative trends and allows the LLM to anchor its judgments on empirically preferred results.
• Calibrating Severity of Errors: Where strong user preference is shown for missing or low-ranked items, BG enforces stricter penalties than P; conversely, BG may issue more lenient judgments for near-miss cases when empirical evidence shows user tolerance for certain variations.
• Enhancing Ranking Sensitivity: BG is more sensitive to intra-SERP ranking quality, demoting pages where less-engaged results are promoted above historically favored entities.

  Figure 2: Offline judge preference shift $\Delta(\hat{y})$ as a function of the cumulative behavioral support ($qri\_count$), showing BG yields earlier and more decisive alignment with online A/B outcomes as behavioral grounding strengthens.

When behavioral support is sparse (i.e., cold-start/long-tail scenarios), BG gracefully reverts to semantic reasoning, minimizing risk of overfitting to noisy or insufficient evidence.

Limitations and Theoretical Implications

Although grounding with QRI improves empirical alignment, absolute correlations remain moderate, highlighting the inherent complexity and variability of user search preferences. The approach is most effective when sufficient high-quality behavioral evidence is available; in data-sparse regimes, the value of grounding diminishes.

Relying on interaction logs introduces potential exposure and selection biases, even after debiasing. Auditable QRI cards partially mitigate this risk by exposing the provenance of behavioral evidence, but further safeguards and advanced debiasing/aggregation are critical for robust deployment, particularly in dynamic content environments or when spurious feedback may be amplified.

From a theoretical standpoint, the demonstrated alignment suggests that lightweight, interpretable behavioral grounding can systematically enhance LLM-based evaluation, bridging the gap between rubric-driven judgment and operational user behavior. This has implications for the broader LLM evaluation landscape, advocating for the routine integration of counterfactually robust interaction features.

Future Directions

Promising extensions include:

• Enhanced debiasing, aggregation, and selection of behavioral signals, potentially leveraging causal modeling to better disentangle intent and exposure.
• Adaptive prompt allocation to balance evidence richness against prompt length and LLM attention constraints.
• Audit trail architectures to rigorously document and review the evidence chain underlying each evaluation, supporting responsible deployment in high-stakes or regulated domains.

Moreover, the general approach is applicable beyond music search to any evolving recommendation or search system where human preference alignment is nontrivial and interaction data is available.

Conclusion

The incorporation of interpretable, debiased behavioral evidence into LLM-based SERP evaluation offers measurable improvements in alignment with both user and human-labeled preferences, especially for ambiguous or contested queries. The QRI card approach demonstrates that lightweight behavioral grounding provides scalable, auditable, and effective context for LLM judges—without supplanting semantic reasoning—supporting more robust and context-sensitive evaluation in real-world, rapidly evolving search systems (2607.01040).