LLM-Assisted Reranking to Operationalize Nuanced Objectives in Recommender Systems

Abstract: Recommender systems have grown from content-organization tools into sophisticated systems that shape daily behavior. By controlling what we see, they shape what we perceive, raising concerns about filter bubbles, radicalization, polarization, and social inequality. LLMs enable more powerful personalization, intensifying these dynamics. Yet most recommenders are tuned for engagement or limited accuracy metrics, with little attention to broader social implications, e.g. how personalization reshapes exposure in socially consequential domains. We investigate whether LLM-assisted reranking, while improving personalization, inadvertently amplifies exposure to ideologically extreme or conspiratorial political content, a risk theorized but not empirically characterized in news recommendation. Using real news-consumption histories, we rerank YouTube's sidebar candidates through zero-shot, instruction-based prompting. We compare a baseline prompt with a constrained variant that preserves topical relevance and broadens ideological exposure while reducing conspiratorial or extreme content. Without constraints, reranking strengthened personalization but increased exposure to conspiratorial and extremist material for users whose histories contained such content. Lightweight prompt-level regularization reduced promotion of extreme content and increased ideological diversity, with modest relevance loss. Synthetic experiments suggest that LLMs rerank via statistical regularities in language rather than semantic understanding of ideology, clarifying why naive prompts amplify these patterns and why regularization can reshape them. Together, our results highlight the power of LLMs to operationalize contextual nuance in high-stakes recommendation, and the need to evaluate LLM-assisted personalization beyond accuracy and treat prompt design as a value-laden rather than neutral default.

• The paper demonstrates that LLM-assisted reranking enhances personalization and ideological diversity with prompt-level regularization.
• It employs real YouTube session data and multiple reranking methods to quantify trade-offs between user engagement and exposure to problematic content.
• Results reveal that prompt-based constraints effectively reduce extremist exposure without sacrificing topical relevance, highlighting the statistical nature of LLM rerankers.

LLM-Assisted Reranking for Tailored and Value-Aligned News Recommendations

Introduction and Motivation

Recommender systems (RecSys) are increasingly used in high-stakes contexts, notably in political news, where unchecked personalization can induce filter bubbles, radicalization, and polarization. Conventional RecSys, including YouTube's deployed algorithm, optimize for engagement, often neglecting societal objectives such as ideological diversity or suppression of problematic content. The integration of LLMs enables new forms of context-aware reranking, capable of leveraging semantic, narrative, and ideological signals within user histories and candidate content.

This study investigates the empirical trade-offs and operational mechanisms of LLM-assisted reranking in political news domains. By overlaying LLM-based post-hoc rerankers atop YouTube's candidate generation, using both baseline and regularized instruction prompts, the analysis systematically addresses three research questions:

1. How does LLM-assisted personalization reshape political exposure?
2. Can prompt-level regularization in LLMs mitigate ideological narrowing and problematic amplification without severely compromising personalization?
3. Do LLMs operate based on semantic understanding of ideology or statistical regularities in language?

   Figure 1: Framework for evaluating LLM-assisted reranking on YouTube, contrasting baseline and socially-regularized prompt interventions.

Methodology and Benchmarking

The experimental pipeline repurposes real desktop browsing trajectories from 97 U.S. adults (Nielsen panel), reconstructs their session-based YouTube recommendations, and evaluates multiple reranking strategies: YouTube's original (YT), embedding-based (emb+YT), baseline LLM-assisted (bLLM+YT), and socially-regularized LLM-assisted (rLLM+YT) prompts.

Candidate and history videos are summarized via transcript, and sessions are partitioned into five ideological groups based on average scores from a validated LLM prompt. Personalization is measured via session-level AUC for proxy-clicked items. Content-based metrics include topical relevance, ideological alignment, and exposure to problematic content (conspiratorial/extremist), adjudicated via decay-weighted averages and corresponding LLM-based labeling prompts.

Empirical Findings: Personalization, Ideological Alignment, and Problematic Exposure

Personalization Effect

Embedding-based reranking (emb+YT) achieves the highest proxy-click AUC (0.75), with bLLM+YT moderately higher than YT (0.59 vs. 0.49). In topical relevance, language- and embedding-based rerankers more strongly align top-ranked recommendations with the user's recent interests.

Figure 2: Topical relevance and ideological scores for YT, emb+YT, bLLM+YT, and rLLM+YT for right-leaning trajectories. Alignment and error bars reflect session-level means and uncertainty.

Ideological Narrowing

LLMs and embedding methods further reinforce ideological cues, particularly for right/right-right sessions, intensifying partisanship in highly ranked output.

Figure 3: Trajectories of ideological scores across recommendation ranks and user histories for several reranking methods. Stronger alignment is observed for bLLM+YT and emb+YT.

In real-world data, right-leaning sessions vastly outnumber left-leaning, and problematic content is sharply concentrated in right-right categories.

Figure 4: Distribution of problematic (conspiratorial, extremist) content by political ideology. Scarcity of left-leaning problematic items justifies main focus on right-centric analysis.

Amplification of Problematic Content

Compared to YT, both bLLM+YT and emb+YT yield higher AUC for problematic content (0.59, 0.64 vs. 0.48), meaning language-based mechanisms are more likely to elevate such videos—mirroring the problematic correlation between partisanship and extremism.

Prompt-Level Regularization: Mitigating Harm and Broadening Exposure

Introducing lightweight constraints to the LLM prompt (rLLM+YT) significantly reduces exposure to extremist/conspiratorial content (AUC drops to 0.42). This does not sacrifice personalization accuracy (proxy-click AUC matches YT) nor substantially decrease topical relevance.

Moreover, ideological diversity at top ranks increases (entropy measures), confirming prompt-based modulation is feasible without severe relevance loss.

Figure 5: Ideological score trajectories for YT, emb+YT, rLLM+YT across all ideological groups. rLLM+YT reduces partisanship and problematic exposure at the top ranks.

Figure 6: Distribution of top-k recommended videos across ideological categories, showing improved diversity for rLLM+YT relative to bLLM+YT and emb+YT.

Figure 7: Top-k ideological diversity for right-right sessions, where regularized LLM reranking counters narrowing induced by language/embedding-based baselines.

Mechanistic Analysis: Statistical Regularities vs Semantic Understanding

Synthetic experiments dissect whether LLM rerankers genuinely comprehend ideology or rely on statistical patterns. Results show topic relevance is privileged over partisan similarity for cognitively focused topics, but behavior flips for election-related content. Prompt variants with explicit partisan emphasis amplify ideological alignment but still reveal non-semantic prioritization; traditional embedding-based reranking is more consistent in topic prioritization.

Figure 8: Synthetic analysis reveals whether rerankers prefer topical relevance or partisan alignment, with emb+YT consistently favoring topic and bLLM+YT exhibiting mixed priorities.

Practical and Theoretical Implications

LLM-driven reranking introduces a nontrivial risk/benefit duality in political RecSys. While naïve personalization reinforces ideological and problematic content, lightweight prompt-level regularization steers output toward safer, more varied exposure—suggesting instruction design is a value-laden operational lever. The mechanism of LLM reranking is largely statistical, not deeply semantic, raising caution about unintended pattern amplification.

Practically, platform integration of LLMs—even for "personalization"—demands explicit and socially-aware design intervention. Theoretical implications include a revised view of alignment: evaluation metrics—beyond click prediction—become critical, and prompt engineering, not just model fine-tuning, is a potent axis for RecSys optimization. Future AI research should address causality, grounding, and semantic robustness, as current LLMs remain statistical pattern matchers.

Conclusion

LLM-assisted reranking in political news recommender systems increases personalization and ideological alignment, but naively applied exacerbates exposure to extremist and conspiratorial content. Prompt-level regularization is effective in mitigating these impacts while preserving personalization. The underlying mechanism is statistical language pattern matching rather than semantic understanding, demanding cautious operationalization and rigorous metric evaluation in deploying LLMs for RecSys in socially consequential domains. The findings highlight the necessity for value-conscious prompt design and the importance of exposure metrics beyond accuracy in the next generation of recommender systems.