RoDPO: Robust Direct Preference Optimization

• The paper introduces RoDPO as a robust preference-alignment framework that adapts DPO for implicit-feedback recommender systems.
• It replaces deterministic hard-negative selection with stochastic top-K sampling to reduce false-negative suppression and stabilize optimization.
• Empirical results on Amazon benchmarks show significant ranking improvements with minimal inference cost and optional sparse MoE capacity scaling.

RoDPO, short for Robust Direct Preference Optimization, is a preference-alignment framework for multimodal sequential recommendation that adapts the DPO idea from NLP to implicit-feedback recommender systems. Its central goal is to make next-item ranking align more directly with user preference, while addressing a key mismatch between standard DPO and recommendation data: in recommender systems, unobserved items are not reliable negatives. Many are merely unseen, so aggressively treating the hardest unclicked item as the losing preference can produce false negatives, i.e., items that are actually positive but unobserved. In the formulation introduced for multimodal sequential recommendation, the central claim is that the negative-sampling rule matters more than the DPO objective itself, and that replacing deterministic hard negatives with stochastic sampling from a dynamic top-$K$ candidate pool yields a robust preference-alignment recipe with nearly unchanged inference cost (Huang et al., 31 Mar 2026).

1. Problem formulation and motivation

RoDPO was proposed because naïve DPO is brittle under implicit feedback. Standard DPO in this setting would form a preference pair $(y_w, y_l)$ where $y_w$ is the observed next item and $y_l$ is a hard negative, often the single highest-scoring non-target item. The difficulty is that recommender data do not provide explicit negative labels for the vast majority of unobserved items. As a result, the single highest-scoring unobserved item may be a hidden positive rather than a genuine negative, so repeatedly suppressing it can inject wrong suppressive gradients, distort the ranking space, and hurt performance (Huang et al., 31 Mar 2026).

The framework is situated in multimodal sequential recommendation, where user histories are modeled as ordered interaction sequences and items are represented by item IDs together with text and image features. The target task is next-item ranking. The proposal is therefore not a general reformulation of recommender training, but a preference-based refinement layer for sequential recommenders trained on sparse implicit-feedback data.

The paper explicitly frames RoDPO as balancing “informative hardness” with “false-negative tolerance.” This formulation reflects two competing desiderata already present in recommendation practice. On one side, hard negatives are desirable because they yield informative gradients; on the other side, hard negative mining is hazardous when the data-generating process leaves many positives unobserved. RoDPO retains the hard-negative signal while reducing repeated penalization of a possibly mislabeled item. This suggests that the method is best understood not as a change in the preference objective alone, but as a redefinition of what constitutes a reliable losing item under implicit feedback.

2. Stochastic top-$K$ negative selection

The core intervention in RoDPO is the replacement of deterministic hard-negative selection with stochastic hard negatives drawn from a dynamic top-$K$ pool. For a context $x$, the method first constructs

$$\mathcal{C}_K(x) = \operatorname{TopK}\left( \{ s_\theta(x, i) \mid i \in \mathcal{I} \setminus \{y_w\} \}, K \right),$$

and then samples the losing item uniformly: $y_l \sim \mathcal{U}(\mathcal{C}_K(x)).$

This modification preserves hardness because the candidate set contains high-scoring non-target items, but it avoids always choosing the single argmax negative. The candidate pool is dynamic because it depends on the current model. Consequently, the hard negatives evolve as the scorer changes during training rather than remaining fixed by a static mining heuristic (Huang et al., 31 Mar 2026).

The paper attributes the effectiveness of this rule to two factors. First, it reduces false-negative suppressive gradients. If the highest-scoring unobserved item is actually a hidden positive, always pushing it down can repeatedly corrupt learning. Sampling across the top-$K$ pool makes the model less likely to keep attacking the same mistaken item, lowering the chance of repeated mis-penalization while still keeping the negatives challenging. Second, it improves optimization stability through controlled stochasticity. Hard negatives preserve informative gradients, unlike random negatives that can be too easy, but the stochastic draw smooths the optimization landscape and avoids overconfident, brittle updates.

A common misconception is that RoDPO’s gains arise from DPO alone. The paper’s central finding is narrower and more specific: the negative-selection rule matters more than the DPO objective itself. In this reading, RoDPO is an adaptation of DPO to the statistical structure of implicit-feedback recommendation, rather than a direct transplantation of an NLP alignment recipe.

3. Objective, reference model, and training protocol

RoDPO keeps the DPO form but works directly with logit margins. Given scores $(y_w, y_l)$0 and a frozen reference model $(y_w, y_l)$1, the preference loss is

$(y_w, y_l)$2

where

$(y_w, y_l)$3

Here, $(y_w, y_l)$4 controls the strength of preference enforcement relative to the reference policy. The paper notes that the softmax partition cancels in the log-ratio, so only the paired logits matter (Huang et al., 31 Mar 2026).

Training follows a two-stage protocol. The model is first warmed up with standard cross-entropy next-item prediction to obtain a stable model $(y_w, y_l)$5. That model is then frozen as the reference $(y_w, y_l)$6, and optimization proceeds with the joint objective

$(y_w, y_l)$7

This makes RoDPO a plug-and-play refinement rather than a wholesale replacement of the recommender backbone.

The training design has two implications emphasized by the paper. One is architectural modularity: the backbone can remain unchanged except for the preference-alignment stage. The other is regularization through reference anchoring: the frozen model provides the baseline against which pairwise preference margins are enforced. A plausible implication is that this reference-model pass is the main source of the additional training burden later identified among the method’s limitations.

4. Multimodal encoder and optional sparse MoE scaling

The RoDPO backbone is a multimodal sequential encoder that embeds item IDs plus text/image features into a shared space and processes them with transformer-based sequence modeling. The framework can optionally add a Sparse Noisy Mixture-of-Experts (MoE) layer for capacity scaling. The sparse MoE uses top-$(y_w, y_l)$8 expert routing: $(y_w, y_l)$9 where $y_w$0 is an expert MLP, $y_w$1 is the selected expert set, and $y_w$2 is a sparse gating function with Gaussian noise added during training for load balancing (Huang et al., 31 Mar 2026).

The point of the MoE is to increase representational capacity for difficult multimodal preference boundaries without paying a large inference cost, because only a subset of experts is active per token. In the reported setup, the MoE has 4 experts with 2 active experts. The paper treats MoE as optional and orthogonal to the core RoDPO idea: the robust negative-sampling strategy is the main contribution, while MoE helps the model fit richer multimodal preference structure.

This separation is important for interpretation. RoDPO is not defined by sparse expert routing; it is defined by robust preference-pair construction under implicit feedback. The optional MoE addresses capacity scaling rather than the false-negative problem itself. Accordingly, ablation results are interpreted by the paper as showing that preference alignment is crucial and that extra capacity helps, rather than conflating the two mechanisms.

5. Empirical evaluation

RoDPO is evaluated on three Amazon multimodal sequential recommendation benchmarks: Toys and Games, Beauty, and Home and Kitchen. These datasets are heavily sparse, with about 99.93%–99.97% sparsity, and each user sequence is split in the standard way: last item for test, second-to-last for validation, earlier interactions for training. The evaluation uses NDCG@5, NDCG@10, MRR@5, and MRR@10. Baselines span three groups: traditional non-time-aware models such as GRU4Rec, SASRec, and LRURec; time-aware models such as TiSASRec, FEARec, and TiCoSeRec; and multimodal methods including NOVA, DIF-SR, UniSRec, MISSRec, M3SRec, IISAN, TedRec, and HM4SR (Huang et al., 31 Mar 2026).

The reported results show consistent gains across all three datasets and all metrics. On Toys and Games, RoDPO reaches 0.0521 NDCG@5, outperforming the strongest baseline HM4SR’s 0.0495 by +5.25%, and improves MRR@5 from 0.0430 to 0.0463, a +7.67% gain. On Beauty, it improves NDCG@5 to 0.0436 from 0.0413. On Home and Kitchen, it again leads, though the absolute margins are smaller.

Dataset / study	Result
Toys and Games	0.0521 NDCG@5
HM4SR on Toys and Games	0.0495 NDCG@5
Toys and Games MRR@5	0.0463 vs 0.0430
Beauty NDCG@5	0.0436 vs 0.0413

The ablation on negative sampling isolates the defining design choice. Random sampling gets 0.0491 NDCG@5, argmax hard sampling gets 0.0497, and top-$y_w$3 sampling achieves 0.0521. This empirical pattern matches the paper’s conceptual argument: random negatives are too easy, deterministic hardest negatives are brittle under implicit feedback, and stochastic top-$y_w$4 negatives preserve hardness while lowering false-negative risk (Huang et al., 31 Mar 2026).

A second ablation shows that removing DPO or removing sparse MoE both reduce performance. The latency analysis reports that RoDPO has nearly unchanged inference cost relative to HM4SR: inference latency is 0.45 s per batch versus 0.42 s, while training time per epoch is similarly close. The paper therefore characterizes the performance gains as arriving with minimal deployment overhead.

6. Interpretation, limitations, and terminological ambiguity

The main conclusion is that DPO can work well for multimodal sequential recommendation, but only if negative selection is adapted to implicit feedback. The paper argues that the false-negative issue is the central barrier to transferring preference optimization from NLP to RecSys. On this view, the sampling rule is not a minor implementation detail; it is the mechanism that determines whether preference optimization becomes brittle preference suppression or effective alignment (Huang et al., 31 Mar 2026).

The stated limitations are also specific. Evaluation is restricted to Amazon e-commerce data, so generalization to other multimodal domains like short-video recommendation remains unproven. Training is still somewhat heavier than lightweight ID-only recommenders because of the extra reference-model pass. These constraints delimit what the reported evidence establishes: robust gains for sparse multimodal sequential recommendation on three Amazon benchmarks, not a universal result across all recommendation regimes.

The term RoDPO also exhibits terminological ambiguity across research areas. In recommender systems it denotes Robust Direct Preference Optimization (Huang et al., 31 Mar 2026). In off-policy evaluation and learning, “RoDPO” is used for distributionally robust off-policy evaluation and learning with a doubly robust construction, including $y_w$5 and $y_w$6 under KL-divergence uncertainty sets (Kallus et al., 2022). A related but distinct acronym in LLM alignment is RoPO, short for Weights-Rotated Preference Optimization, which addresses reward hacking in DPO through orthogonal rotation-based parameter regularization (Yang et al., 25 Aug 2025). This suggests that references to “RoDPO” require domain context, since the acronym does not identify a single method family across the broader literature.

Within the recommender-systems usage, however, the meaning is precise. RoDPO denotes a preference-alignment framework in which deterministic hard negatives are replaced by stochastic samples from a dynamic top-$y_w$7 candidate pool, optionally combined with sparse MoE capacity scaling, to improve top-$y_w$8 ranking quality in multimodal sequential recommendation with nearly unchanged inference cost (Huang et al., 31 Mar 2026).