Mult-DPO: Multinomial Direct Preference Optimization for Recommender Systems

Abstract: Direct preference optimization (DPO) is a simple and effective alignment strategy for LLMs based on pairwise preferences. In recommender systems, however, user feedback is rarely pairwise. For a given context, e.g., a user, a session, or a conversation, we typically observe set-wise preferences with multiple positive items, where every positive item should outrank every unobserved or explicitly negative item, with no prescribed order among the positives or the negatives themselves. A natural generalization is to use the Plackett-Luce (PL) reward model, which extends the Bradley-Terry reward model underlying vanilla DPO from pairwise preferences to full rankings of candidates. However, we show that adapting the PL model to set-wise preferences requires marginalizing over all positive orderings, where the resulting expression is combinatorial in complexity. To address this fundamental challenge, we propose Mult-DPO, a novel DPO objective with a tractable multinomial surrogate likelihood over set-wise preference events for the user-preference alignment of LLM-based recommender systems. The multinomial construction is not itself a ranking distribution, but it is defined on the same reward-induced weight space and admits a closed-form DPO-style objective, enabling direct alignment of LLMs with multiple candidates through a classification-style objective. In addition, we prove that the multinomial DPO loss is a tractable upper bound on the marginalized PL DPO loss when optimizing against the set-wise preference data. We further characterize the tightness of this bound in terms of the relative total weight of positives versus negatives, which provides insights into tightening the bound with richer or harder negatives. Finally, we extend Mult-DPO to the alignment of LLMs with multiple preference levels. Code is available at https://github.com/yaochenzhu/Mult_DPO

• The paper introduces Mult-DPO, a multinomial surrogate that efficiently models multiple positive and negative items, provably upper-bounds the intractable marginalized PL-DPO loss, and reduces loss computation to O(K).
• The method achieves the best reported NDCG results across MovieLens-10M, Goodreads, and Reddit-V2, including NDCG@5 of 0.1288 versus 0.1252 for LiPO-BT on Goodreads and 0.1369 versus 0.1147 on Reddit-V2.
• The paper extends Mult-DPO to multi-level ratings through Mult²-DPO, producing an approximately 12% relative NDCG@5 gain over binary Mult-DPO on MovieLens-10M while showing that hard-negative sampling can tighten the theoretical approximation gap.

Motivation and problem statement

Direct preference optimization (DPO) has become the dominant RL-free alignment strategy for LLMs, but its underlying Bradley–Terry (BT) reward model assumes pairwise preferences between exactly two candidates. In recommender systems (RSs), user feedback is structurally different: for a given context $x$ (user profile, interaction history, or conversation), one typically observes a set of positive items $\mathcal{E}^p$ and a set of negative items $\mathcal{E}^d$, with the constraint that every positive should outrank every negative and no order imposed within either set. The natural listwise generalization is the Plackett–Luce (PL) model, but adapting it to set-wise feedback requires marginalizing over all orderings of the positives consistent with the observation. The authors show this marginalized PL likelihood is combinatorial in $k$ (the number of positives): naive marginalization costs $k!$ terms, and even an inclusion–exclusion form derived from the exponential-race representation reduces this only to $2^k$ terms. Existing remedies are unsatisfying: PRO and KPO fit PL to fully ordered lists; DMPO contrasts one positive against an arithmetic mean of negative log-ratios without a coherent ranking-likelihood interpretation; S-DPO handles multiple negatives but is restricted to a single positive per context.

The multinomial surrogate

The core contribution is a multinomial (MN) surrogate event model defined on the same reward-induced weight space as BT and PL. Normalizing weights into a categorical distribution $p(e \mid x) = w(e \mid x)/W$, the surrogate defines the set-wise preference event $\Omega_x$ as the probability that $k$ i.i.d. draws each positive appears exactly once and no negative appears, yielding

$$p_{\mathrm{MN}}(\Omega_x \mid x, \mathcal{E}; w) = k! \prod_{e \in \mathcal{E}^p} \frac{w(e \mid x)}{W},$$

computable in $\mathcal{E}^p$0. The authors are explicit that this construction is not a ranking distribution — it assigns mass to duplicate draw sequences outside the valid ranking space — and its justification rests entirely on two theoretical results:

• Lower bound: the MN likelihood pointwise lower-bounds the exact marginalized PL likelihood for any disjoint sets with at least one positive.
• Tightness characterization: the ratio satisfies $\mathcal{E}^p$1, where $\mathcal{E}^p$2 and $\mathcal{E}^p$3 are the cumulative weights of positives and negatives respectively.

Substituting the standard DPO reparameterization $\mathcal{E}^p$4 produces a closed-form classification-style objective in which every positive is contrasted against the same cumulative negative weight $\mathcal{E}^p$5. A corollary establishes that the Mult-DPO loss upper-bounds the ideal but intractable marginalized PL-DPO loss, with gap bounded by $\mathcal{E}^p$6. Two implications follow directly: when $\mathcal{E}^p$7 the bound is exact and recovers S-DPO's softmax objective as a special case; and increasing the policy weight on negatives tightens the worst-case gap, motivating richer or harder negative sampling. Notably, sharing $\mathcal{E}^p$8 across positives also reduces loss-aggregation cost from $\mathcal{E}^p$9 (as in LiPO-BT's summed pairwise losses) to $\mathcal{E}^d$0.

Multi-level extension

For fine-grained feedback such as explicit ratings, the paper extends to $\mathcal{E}^d$1 ordered preference groups via a sequential multinomial (SMN) surrogate. The key structural observation is that PL's sequential selection property makes the boundary-specific events conditionally independent once higher-preference groups are placed, so the multi-level PL likelihood factorizes into a product of two-set marginalized PL factors. Applying the binary lower bound group-by-group yields Mult²-DPO, whose loss provably upper-bounds the multi-level PL-DPO loss with an analogous per-boundary gap bound. All theoretical properties of the binary case carry over, and the objective reduces to Mult-DPO when $\mathcal{E}^d$2.

Empirical results

Experiments cover general recommendation (MovieLens-10M, Goodreads) and conversational recommendation (Reddit-V2), evaluated by NDCG@{5,15,20} against the full catalog with Qwen2.5 backbones (0.5B–7B). Against SFT baselines (BIGRec, D³) and DPO-style baselines (vanilla DPO, DMPO, S-DPO, LiPO-BT), Mult-DPO achieves the best results across all settings. At the 3B scale on Goodreads it reaches NDCG@5 of 0.1288 versus 0.1252 for LiPO-BT and 0.1181 for S-DPO; on Reddit-V2, where multi-positive ground truths are densest, the margin over LiPO-BT is largest (0.1369 vs. 0.1147 at NDCG@5). The margin widens with backbone scale, which the authors attribute to the joint set-wise signal paying off most when the policy has capacity to exploit it. Wall-clock training time remains comparable to vanilla DPO and S-DPO because KV-cache reuse of the shared prompt prefix dominates self-attention cost when $\mathcal{E}^d$3.

Three additional analyses support the theory. First, on instances restricted to at most three positives (where the exact PL-DPO loss is computable), training dynamics empirically verify the upper-bound relation. Second, epoch-level dynamic hard-negative resampling adapted from SPRec improves test performance, consistent with the tightness analysis predicting that harder negatives shrink the bound gap. Third, Mult²-DPO with four rating-derived groups outperforms binary Mult-DPO at every cutoff on MovieLens-10M, with approximately 12% relative gain in NDCG@5 at the 0.5B backbone (0.0732 vs. 0.0650), confirming that preserving rating structure supplies a stronger alignment signal.

Limitations and open questions

The paper concedes several limitations directly. The MN surrogate is one tractable approximation among possibly many, and may not be the tightest; alternatives such as expectation-maximization with latent assignment variables over candidate orderings, or variational lower bounds with flexible distributions over positive permutations, remain unexplored. Because the objective imposes a uniform target over positives at the same level, it is best suited to exchangeable positives; when relevance varies substantially within a level, weighted or rating-aware variants may be preferable. On the empirical side, the hyperparameter analysis covers only the 0.5B and 3B backbones, with the chosen $\mathcal{E}^d$4 transferred to other scales rather than re-tuned; the scaling study shows gains flattening between 1.5B and 7B, leaving open whether set-wise alignment continues to help at larger scales. The tightness-gap improvement from hard negatives is demonstrated empirically but not quantified against the closed-form bound during actual training.

Conclusion

Mult-DPO addresses the mismatch between pairwise DPO and set-wise recommendation feedback with a multinomial surrogate that admits a closed-form objective, is provably an upper bound on the combinatorially intractable marginalized PL-DPO loss, and comes with a tightness characterization that directly informs negative-sampling strategy. Its sequential extension handles multi-level preferences with identical guarantees. Consistent empirical gains over both SFT and DPO baselines across general and conversational benchmarks indicate that jointly modeling the set-wise structure of user feedback is effective and computationally practical for LLM-based recommendation alignment.