LPO4Rec: Listwise Preference Alignment

• LPO4Rec is a listwise preference alignment framework that extends the Bradley-Terry model to optimize tail-item recommendations without using explicit reward models.
• It employs adaptive negative sampling and loss reweighting to efficiently focus on challenging, underrepresented tail items.
• Empirical evaluations on Amazon datasets show up to 50% improvement in HR and NDCG for tail items while reducing GPU memory and training time.

LPO4Rec is a listwise preference alignment framework for recommender systems, designed to efficiently optimize for tail-item recommendation by generalizing pairwise preference learning to a listwise setting, eliminating the need for explicit reward models, and introducing adaptive negative sampling and reweighting to prioritize learning on rarely recommended (tail) items.

1. Conceptual Overview

LPO4Rec addresses limitations in existing recommender system preference alignment approaches by extending the Bradley-Terry model from pairwise comparison to a listwise paradigm. In this approach, each training instance involves one positive (winner) item and a list of $K$ negative (loser) items, enabling more efficient utilization of negative samples and direct model alignment with user preference signals across multiple alternatives. This is particularly advantageous for improving the coverage and accuracy of tail items, which suffer from data imbalance and under-recommendation in classical systems.

The LPO4Rec methodology achieves scalable implementation by:

• Removing the explicit reward or reference model seen in prior methods such as DPO and PPO, reducing computational and memory overhead.
• Employing a closed-form listwise preference objective that is efficiently differentiable and optimized within a standard neural network training pipeline.
• Introducing adaptive sampling and weighting schemes to focus optimization on challenging, under-represented (tail) items.

2. Listwise Bradley-Terry Preference Modeling

The central methodological innovation is the extension of the Bradley-Terry comparison from pairs to lists. For input $x$ (e.g., user context), positive item $y_w$, and negative items $\{y_\ell\}_{\ell=1}^K$, the listwise preference probability is

$$p\left(y_w \succ \{y_\ell\}_{\ell=1}^K \mid x\right) = \prod_{\ell=1}^K \frac{\exp(r(x, y_w))}{\exp(r(x, y_w)) + \exp(r(x, y_\ell))}$$

where $r(x, y)$ is the model's scoring function.

LPO4Rec further derives the closed-form optimal policy without reference models for efficient gradient-based training. The final listwise loss is

$$\mathcal{L}_{\textrm{LPO}}(\pi_\theta) = -\mathbb{E}_{(x, y_w, \{y_\ell\}_{\ell=1}^K)} \left[ \log \frac{\exp(\pi_\theta(y_w|x)/\tau)}{\exp(\pi_\theta(y_w|x)/\tau) + \sum_{\ell=1}^K \exp(\pi_\theta(y_\ell|x)/\tau)} \right]$$

where $\pi_\theta(y|x)$ is the model's logit or unnormalized score and $\tau$ is a temperature parameter controlling sharpness.

Compared to pairwise DPO, which only utilizes one negative per positive, the listwise form simultaneously exploits all $K$ negatives for each instance, enhancing statistical and computational efficiency.

3. Adaptive Negative Sampling and Reweighting

To counteract the data imbalance that causes tail items to be underrepresented, LPO4Rec introduces two mechanisms:

Adaptive Negative Sampling

• Negatives $y_\ell$ are sampled from the set of head (popular) items, promoting the discrimination between tail and head items.
• The negative sampling distribution is adapted over training, with sampling probability proportional to the model's current predicted probability for head items:

$$y_\ell \sim P(y_\ell) = \frac{\exp(\pi_\theta(y_\ell | S))}{\sum_{y' \in \mathcal{I}_H} \exp(\pi_\theta(y' | S))}$$

• Gumbel-Softmax Top-K selection is employed to permit differentiable sampling and effective implementation within SGD frameworks.

Adaptive Loss Reweighting

• A per-sample loss weight $\omega_i$ is assigned, giving higher weight to training instances involving tail items: $$\omega_i = \begin{cases} \alpha_T, & x_{t+1} \in \mathcal{I}_T \ \alpha_H, & x_{t+1} \in \mathcal{I}_H \end{cases}$$ with $\alpha_T > \alpha_H$ typically enforced.

Composite Optimization Objective

The overall loss is a weighted combination of cross-entropy (CE) and the LPO loss: $$\mathcal{L} = \omega \left( \mathcal{L}_{\textrm{CE}} + \lambda \mathcal{L}_{\textrm{LPO}} \right)$$ where $\lambda$ balances direct prediction and preference alignment.

4. Theoretical Properties

Theoretical analysis demonstrates that minimizing the LPO loss is equivalent to maximizing an upper bound on the optimal reward under the listwise Bradley-Terry framework. This equivalence implies that model training via LPO4Rec pushes the output distribution closer to the true reward-maximizing policy, as established through formal derivations in the primary paper.

Gradient analysis further shows that the loss encourages the model to rectify errors where it is least confident (ground-truth positives have low model scores) or head-dominant negatives have high scores. This mechanism prioritizes hard examples, tending to focus learning on difficult tail cases that are critical for improving diversity and fairness in recommendation.

5. Empirical Performance and Efficiency

LPO4Rec is benchmarked on three public Amazon datasets (Beauty, Toys, Sports), with clear definition of the tail set as the bottom 80% of items by popularity. Results compared to ten baselines—comprising sequential models (Caser, GRU4Rec, SASRec), tail-focused methods (MELT, CITES, R2Rec), and prior preference alignment approaches (DPO, S-DPO, SimPO, ORPO)—demonstrate the following factual findings:

• LPO4Rec achieves up to 50% improvement in hit rate (HR) and normalized discounted cumulative gain (NDCG) for tail items compared to the best prior method on the Sports dataset.
• Significant improvements extend across all evaluation metrics and item segments, with R2Rec as the closest competitor for tail items, though with notably smaller gains.
• LPO4Rec attains the best or second-best overall HR@K and NDCG@K, without sacrificing accuracy on more popular (head) items.
• GPU memory consumption is reduced by 17.9% and single-pass training time by 54.2% relative to S-DPO, attributed to the elimination of redundant models and listwise comparison in a single forward pass.
• Ablation studies indicate that both the LPO listwise loss and the reweighting components are essential for robust tail performance.

In some experiments, augmenting with text/image embeddings (e.g., from CLIP) can improve overall metrics, yet may diminish tail-aware gains if pretraining is insufficient.

6. Implementation and Practical Usage

The training workflow of LPO4Rec proceeds as follows:

1. For each mini-batch, draw sequences and their ground-truth next items.
2. For each positive item, sample $K$ head negatives using the adaptive Gumbel-Softmax top-$K$ technique.
3. Compute sample-wise weights $\omega$ using the reweighting rule above.
4. Update model parameters by backpropagation of the weighted composite loss.

This framework is reference-model-free, making it compatible with common recommendation backbones such as SASRec, Caser, and GRU4Rec, with documented empirical gains when substituting standard losses with the LPO4Rec procedure.

The code for LPO4Rec is provided at https://github.com/Yuhanleeee/LPO4Rec, facilitating reproducibility and integration in future research.

7. Significance and Future Directions

LPO4Rec provides a new solution to the challenge of tail-item recommendation in large-scale systems, achieving both efficiency and effectiveness by rethinking the training paradigm through listwise modeling and resource-aware optimization. It is the first framework demonstrated to adapt preference alignment for tail-item scenarios, with both theoretical and empirical support.

Potential avenues for future advancement include:

• Deeper integration of multi-modal data (text, vision).
• Dynamic or learned partitioning of head and tail items.
• Alternative, potentially learned, user preference/reward signals.
• Extensions to additional recommendation scenarios (e.g., cold-start, cross-domain, or real-time systems).
• Further exploration of sampling, optimization, and loss compositions for even greater computational savings or recommendation quality.

The principled design and observed empirical success of LPO4Rec mark it as a substantial methodological development in preference-aligned, fair, and efficient recommendation for under-served item segments.