Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders

Abstract: Reinforcement learning (RL) effectively optimizes LLM-based recommenders by contrasting positive and negative items. Empirically, training with beam-search negatives consistently outperforms random negatives, yet the mechanism is not well understood. We address this gap by analyzing the induced optimization objective and show that: (i) Under binary reward feedback, optimizing LLM recommenders with Group Relative Policy Optimization (GRPO) is theoretically equivalent to maximizing the Area Under the ROC Curve (AUC), which is often misaligned with Top-$K$ recommendation; and (ii) Replacing random negatives with beam-search negatives reshapes the objective toward partial AUC, improving alignment with Top-$K$ metrics. Motivated by this perspective, we introduce Windowed Partial AUC (WPAUC), which constrains the false positive rate (FPR) to a window [$α,α+d$] to more directly align with Top-$K$ metrics. We further propose an efficient Threshold-Adjusted Windowed reweighting (TAWin) RL method for its optimization, enabling explicit control over the targeted Top-$K$ performance. Experiments on four real-world datasets validate the theory and deliver consistent state-of-the-art performance.

• The paper establishes that replacing random with beam search for hard negatives shifts the RL objective toward One-way Partial AUC, aligning with Top-K recommendation metrics.
• The paper introduces WPAUC and the TAWin scheme, providing a differentiable soft weighting mechanism to precisely control ranking evaluation windows.
• The paper demonstrates empirical improvements across diverse models and datasets, significantly boosting Recall@K and NDCG@K in LLM-based recommendation systems.

Objective Shaping with Hard Negatives: Windowed Partial AUC Optimization for RL-based LLM Recommenders

Theoretical Foundations and Motivation

This work provides a principled analysis of why reinforcement learning (RL) methods leveraging hard negative sampling—specifically beam search—substantially enhance Top-$K$ performance in LLM-based recommendation. The primary insight is that traditional RL objectives for LLM recommenders, such as Group Relative Policy Optimization (GRPO), implicitly optimize the Area Under the ROC Curve (AUC) when trained with random negative samples. However, maximizing AUC misaligns with the practical goal of maximizing Top-$K$ recommendation metrics, such as Recall@$K$, which are sensitive to the relative order of the highest-scored items.

A key theoretical contribution is establishing that replacing random negative sampling with beam search steers the RL objective towards Partial AUC, specifically One-way Partial AUC (OPAUC), focusing evaluation on the high-score tail of the negative distribution. This localized view aligns better with Top-$K$ metrics because it directly addresses the ranking error among the candidates most likely to be recommended.

Figure 1: Schematic of GRPO training. Negative sampling strategy shapes the induced optimization: random sampling induces a global AUC objective, while beam search shifts to OPAUC. Further windowing yields WPAUC.

Windowed Partial AUC (WPAUC): Formulation and Implications

Building on the observation that OPAUC improves alignment with Top-$K$ objectives but does not enable fine-grained metric control, the authors introduce Windowed Partial AUC (WPAUC). WPAUC restricts pairwise ranking evaluation to a customizable false positive rate (FPR) window $[\alpha, \alpha+d]$, making the optimization controllable for a targeted recommendation regime. Theoretical results establish that WPAUC can be tightly linked with Recall@$K$ for suitable settings of $(\alpha, d)$. Notably, with one positive item and window width $d=1/n^-$ (where $n^-$ is the number of negatives), WPAUC exactly recovers Recall@$K$0.

Empirical studies further support the theory, showing maximal Pearson correlation between WPAUC and Recall@$K$1 at window parameterizations predicted by the analysis.

Figure 2: Pearson correlation between Recall@$K$2 and WPAUC$K$3 over varying window parameters, highlighting strong controllable alignment.

This controllability allows practitioners to directly optimize for the desired position and granularity in the ranked list—a crucial requirement for large-scale recommendation deployments.

TAWin: Differentiable Optimization of WPAUC

To effectively optimize WPAUC with sampled rollouts in RL, the authors propose Threshold-Adjusted Windowed reweighting (TAWin), a differentiable weighting scheme that applies soft selection over the windowed region. Unlike naive hard truncation (which discards negatives outside the window, resulting in high gradient variance), TAWin uses a temperature-controlled soft Top-$K$4 operator to ensure efficient sampling and smooth gradients.

The TAWin objective generalizes standard policy gradient updates by weighing each negative sample according to its normalized rank proximity to the anchor $K$5 (corresponding to $K$6), allowing explicit focus on the Top-$K$7 regime. The resulting optimization is efficiently composable with both GRPO and other RL algorithms like GSPO and DAPO, as demonstrated in ablation studies.

Figure 3: TAWin sensitivity to hyperparameter $K$8 (anchor position). Alignment with target Recall@$K$9 is unimodal; optimal anchor increases with $K$0.

Experimental Validation

Extensive experiments over four real-world datasets and diverse baselines—including traditional neural and generative recommenders and recent LLM-based approaches—demonstrate the empirical efficacy of the proposed framework. TAWin establishes new state-of-the-art performance across all datasets and Top-$K$1 metrics, yielding substantial improvements over both classical and LLM-based models.

(Table 1 summarizes core results as described in the paper, with TAWin outperforming all baselines.)

In addition to overall gains, the experiments emphasize several critical findings:

• Robustness Across Model Architectures: TAWin's improvements generalize across various LLM backbones (Qwen2.5 and Llama-3.2) and scales.
• Universality Across RL Algorithms: The windowed objective shaping is effective with multiple RL optimizers, including GRPO, GSPO, and DAPO.
• Versatility with Item Encodings: TAWin retains its advantage with both token-based and semantic item representations.

Comparative ablations reveal that TAWin consistently outperforms other hard-negative and ranking-reward heuristics (e.g., ReRe), especially when precise Top-$K$2 alignment is required.

Figure 4: Performance comparison of TAWin, beam search (ReRe), and random sampling across datasets. Windowed approaches consistently yield the highest Recall@$K$3 and NDCG@$K$4 scores.

Figure 5: Direct comparison between ReRe and TAWin under different base models confirms the universality of the windowed optimization gain.

Practical and Theoretical Implications

The findings clarify and formalize the longstanding empirical observation that hard-negative sampling (especially beam search over candidate outputs) consistently improves Top-$K$5 metrics in RL-based LLM recommenders. By uncovering the underlying mechanism—objective shaping towards Partial AUC—and extending it with WPAUC for controllable optimization, this framework enables more precise and interpretable deployment for practical recommendation systems.

From a theoretical perspective, linking windowed partial AUC maximization with Recall@$K$6 advances the metric alignment literature for generative ranking tasks, and the differentiable relaxation provided by TAWin is likely extensible to other sequential decision-making settings where top-list precision is critical.

Future directions include integrating fairness and diversity into objective shaping, extending windowed AUC principles to multilabel and session-based recommendation, and exploring implications for other LLM-based selection or ranking tasks beyond recommendation.

Conclusion

This work bridges the gap between RL objective design and application-aligned recommendation performance in LLM-based systems. By formalizing how hard-negative sampling induces partial AUC optimization and introducing WPAUC/TAWin for controllable, efficient RL training, the study establishes both a theoretical foundation and practical toolkit for scalable, robust Top-$K$7 optimization in next-generation recommenders. Consistent empirical advances confirm the universality and effectiveness of the proposed approach, providing a strong basis for principled future development in LLM-driven personalized recommendation systems.