Dual-Oriented Preference Diffusion

• Dual-Oriented Preference Diffusion is a framework that uses diffusion mechanisms to harmonize multiple paired preference signals, such as source/target or general/specific views.
• It leverages denoising, transformation, and probabilistic calibration techniques to fuse dual representations and mitigate noise in cross-domain recommendations and generative tasks.
• This approach has demonstrated improvements in ranking, convergence, and output quality across diverse applications including recommendation systems, image generation, and decision-making models.

Searching arXiv for the core paper and closely related preference/diffusion work to ground the article in current literature. Dual-oriented preference diffusion denotes, in its most explicit usage, a diffusion-based mechanism for harmonizing multiple preference signals across distinct orientations. The clearest instance is HorizonRec’s “dual-oriented preference diffusion” for cross-domain sequential recommendation, where source- and target-domain user representations are denoised symmetrically under mixed-domain conditioning to support fine-grained triple-domain preference fusion (Zha et al., 7 Aug 2025). More broadly, the surrounding literature suggests a recurrent design pattern rather than a single canonical algorithm: preferences are split into paired or coupled views—general and specific, source and target, preferred and less-preferred, positive and negative, or user-specific and population-level—and diffusion or distributional machinery is used to align, calibrate, or fuse those views (Dong et al., 2022, Saeidi et al., 9 Feb 2025, Wang et al., 17 May 2025, Dang et al., 11 Jan 2025).

1. Distributional antecedents: dual preference representations

An important antecedent is DUPLE, a dual preference distribution learning framework for item recommendation that jointly learns a general preference distribution and a specific preference distribution for a given user (Dong et al., 2022). In DUPLE, the general distribution corresponds to the user’s general preference to items, while the specific distribution refers to the user’s specific preference to item attributes. Both are modeled as multivariate Gaussian distributions, $\mathcal{G}(\bm{\mu}, \bm{\Sigma})$, where the mean vector captures the preference center and the covariance matrix captures strength and relationships among different preference aspects. The general preference score is defined by the Gaussian density of an item embedding under the user’s general distribution, the specific score is defined analogously in attribute space, and the final preference score is a linear combination: $p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$

DUPLE also introduces a general-specific transformation that projects an item embedding into attribute space, yielding

$$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$

This construction supports a preferred attribute profile $\mathcal{A}_u$, defined by the top-$r$ attributes in $\bm{\mu}_u^s$, and explanations are produced by checking the overlap between a recommended item’s attributes and that profile. The paper reports superior results on AUC, MRR, HR@10, and NDCG@10 across six real-world datasets, and a psycho-visual survey of 126 volunteers in which 88% found the explanations reasonable (Dong et al., 2022).

Although DUPLE is not a diffusion model, it establishes a key conceptual precursor: dual-oriented preference modeling can mean separating coarse item-level judgment from fine-grained attribute-level judgment and explicitly modeling their relation through shared transformations and covariance structure. This suggests that later “preference diffusion” methods inherit not only denoising machinery but also a dual-view understanding of preference itself.

2. HorizonRec and the explicit dual-oriented diffusion formulation

The phrase “dual-oriented preference diffusion” is used directly in HorizonRec, an align-for-fusion framework for cross-domain sequential recommendation (Zha et al., 7 Aug 2025). The setting involves three sequences for each user: a source-domain sequence $S_u$, a target-domain sequence $T_u$, and a mixed sequence $M_u$ formed by temporally merging source and target interactions. The task is to predict the next target-domain interaction by maximizing

$$\max_{i_{q+1}^T \in I_T} p(i_{q+1}^T \mid S_u, T_u, M_u).$$

Each sequence is encoded, for example via SASRec or similar, to produce hidden representations $p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$0, $p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$1, and $p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$2.

HorizonRec identifies stochastic noise as a key source of instability in existing diffusion-model-based recommenders and addresses it with Mixed-conditioned Distribution Retrieval (MDR) (Zha et al., 7 Aug 2025). Instead of injecting pure Gaussian noise, MDR retrieves behavior-driven, user-specific noise from mixed-domain subsequences ending with target items and uses a position-aware low-pass filter to assign higher weights to more recent or target-domain-centric actions. The retrieved noise for domain $p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$3 is written as

$p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$4

with the reported theoretical support that the retrieved variance $p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$5 is smaller than standard Gaussian variance and that the retrieved mean is steered toward target-ending trajectories. In the forward process,

$p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$6

and in the reverse process the mixed-domain representation acts as an explicit semantic guide,

$p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$7

The diffusion loss is

$p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$8

After denoising, the source and target paths are fused by

$p_{ui} = \lambda p_{ui}^g + (1-\lambda)p_{ui}^s .$9

The paper further states a conditional alignment property,

$$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$0

and reports that HorizonRec outperforms single-domain, cross-domain non-sequential, and advanced triple-domain CDSR baselines on four real-world triple-domain benchmarks, with gains that are especially pronounced in sparse scenarios; ablations show degradation when MDR is removed or when dual diffusion is disabled on either side (Zha et al., 7 Aug 2025).

In this formulation, “dual-oriented” does not mean a simple winner-loser preference pair. It refers to symmetric diffusion along two domain-specific paths, source and target, with a third mixed representation acting as the semantic anchor for harmonization.

3. Preference diffusion as ranking-oriented generative learning

PreferDiff extends the preference-diffusion idea within recommendation by replacing conventional diffusion objectives such as mean squared error with a personalized ranking loss tailored to diffusion models (Liu et al., 2024). The paper starts from BPR and reformulates it as a log-likelihood ranking objective over the diffusion model’s generative distribution: $$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$1 Because exact likelihoods are intractable, PreferDiff derives a variational upper bound and optimizes a stochastic per-timestep surrogate.

A second technical modification is the use of multiple negative samples. Rather than denoising every negative individually, PreferDiff aggregates them into a negative centroid, yielding the BPR-Diff-C objective. A third modification is the replacement of MSE with cosine error,

$$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$2

motivated by the fact that recommendation retrieval is done by inner product or cosine similarity rather than Euclidean distance. The final training criterion balances generative learning and preference learning: $$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$3

The paper reports that PreferDiff is the first personalized ranking loss designed specifically for diffusion-model-based recommenders, that it improves ranking and convergence, and that it has a theoretical connection to Direct Preference Optimization under certain settings (Liu et al., 2024). Experiments on three benchmarks show superior recommendation performance and strong general sequential recommendation capability, including zero-shot generalization across domains with text embeddings. This broadens the meaning of preference diffusion: the diffusion model is not only a fusion module for dual representations, but also a generative model whose training objective is rewritten to encode preference orderings directly.

4. Pairwise alignment, dual captions, and divergent labels in text-to-image diffusion

In text-to-image alignment, pairwise preference optimization is frequently the point of departure, but several works introduce explicitly dual structures beyond a simple winner-loser label. Diffusion-RPO extends Diffusion-DPO by leveraging both prompt-image pairs with identical prompts and prompt-image pairs with semantically related content across modalities (Gu et al., 2024). The stepwise reward at denoising step $$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$4 is defined as

$$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$5

and pairwise contributions are weighted by a CLIP-based multi-modal similarity term $$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$6. The paper also introduces style alignment, evaluated by FID against a target style-aligned set, as a cost-effective and reproducible complement to conventional human-preference evaluation. Reported results show improvements over Supervised Fine-Tuning and Diffusion-DPO on Stable Diffusion 1.5 and SDXL, including HPSV2, PickScore, and style-alignment FID (Gu et al., 2024).

DCPO makes the dual structure explicit at the caption level by assigning separate captions to the preferred and less-preferred images in each training pair (Saeidi et al., 9 Feb 2025). The paper identifies two problems in existing preference datasets: conflict distribution, caused by overlap between preferred and less-preferred image distributions under a single prompt, and the irrelevant prompt issue, where the less-preferred image is conditioned on prompt information more relevant to the preferred image. To address this, DCPO introduces the Pick-Double Caption dataset, created from 20,000 cleaned instances of Pick-a-Pic v2, with separate captions generated by LLaVA-1.6-34B and Emu2-37B. The reported CLIPScore difference $$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$7 becomes substantially larger, with LLaVA at 4.3 versus the original 1.3. DCPO proposes captioning, perturbation, and hybrid strategies, and optimizes

$$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$8

which reduces to standard DPO when $$\bm{\mu}_u^s = \bm{W}_t \bm{\mu}_u^g,\qquad \bm{\Sigma}_u^s = \bm{W}_t \bm{\Sigma}_u^g \bm{W}_t^\mathsf{T}.$$9. The reported best variant, DCPO-h, improves Pickscore, HPSv2.1, ImageReward, CLIPscore, and GenEval over SD 2.1, SFT_Chosen, Diffusion-DPO, and MaPO (Saeidi et al., 9 Feb 2025).

Adaptive-DPO addresses another duality: majority versus minority preference signals in human-labeled datasets (Zhang et al., 21 Mar 2025). It introduces a minority-instance-aware metric combining intra-annotator confidence and inter-annotator stability,

$\mathcal{A}_u$0

then uses this score for instance-specific weighting and an adaptive margin: $\mathcal{A}_u$1 The resulting objective

$\mathcal{A}_u$2

is reported to be robust both to synthetic label flips and to real-world data from Pick-a-Pic v2 and HPDv2, outperforming Diffusion-DPO, Robust-DPO, and SFT on several metrics for SD1.5 and SDXL (Zhang et al., 21 Mar 2025).

Taken together, these works suggest that dual-oriented preference diffusion in image generation often means restructuring the pairwise supervision itself: same versus related prompts, preferred versus less-preferred captions, or majority versus minority annotations.

5. Multi-preference calibration, personalization, and positive/negative steering

A different branch of the literature replaces binary preference supervision with explicitly multi-preference alignment. CaPO aligns text-to-image diffusion models using multiple reward models without human-annotated data by calibrating each reward as an expected win-rate against samples from the pretrained model (Lee et al., 4 Feb 2025): $\mathcal{A}_u$3 In the multi-reward setting, CaPO uses frontier-based rejection sampling over Pareto frontiers and optimizes a regression objective on the calibrated reward difference of a selected pair. The paper reports that CaPO outperforms DPO, IPO, and model soups on GenEval and T2I-CompBench in both single- and multi-reward settings (Lee et al., 4 Feb 2025).

PPD moves from reward-model calibration to explicit user conditioning (Dang et al., 11 Jan 2025). It extracts personal preference embeddings from as few as four pairwise preference examples per user using a frozen vision-LLM, then injects those embeddings into Stable Cascade through additional cross-attention layers. The training objective is a user-conditioned extension of Diffusion-DPO, and the model can interpolate between reward orientations at inference by interpolating user embeddings. The paper reports that, in real-world user scenarios, the method achieves an average win rate of 76% over Stable Cascade for unseen users and can trade off among CLIP, Aesthetic, and HPS preferences by changing the conditioning vector (Dang et al., 11 Jan 2025).

Diffusion Blend pushes multi-preference control to inference time rather than training time (Cheng et al., 24 May 2025). Its central result is that the reverse-diffusion drift for a linear combination of reward functions can be approximated by a linear combination of the drifts of models individually fine-tuned on those basis rewards: $\mathcal{A}_u$4 This yields DB-MPA for reward mixing and DB-KLA for KL-regularization control. Using Stable Diffusion v1.5 with ImageReward, VILA, and PickScore, the paper reports that the method outperforms practical baselines and closely matches or exceeds individually fine-tuned models while enabling smooth inference-time trade-off control (Cheng et al., 24 May 2025).

Self-NPO introduces yet another dual orientation: positive-preference and negative-preference models combined by classifier-free guidance (Wang et al., 17 May 2025). Instead of requiring explicit negative labels or a reward model, it learns a negative-preference model from the model’s own partially generated outputs via truncated diffusion. At inference, the positive and negative models are combined as

$\mathcal{A}_u$5

The method is reported to integrate with SD1.5, SDXL, CogVideoX, and models already optimized for human preferences. For SDXL, the reported automatic scores rise from 22.06/28.04/0.62/6.11 to 22.26/28.24/0.67/6.23 in PickScore/HPSv2/ImageReward/Aesthetic after Self-NPO, and user studies favor its outputs on color and lighting, details, composition, and text-image alignment (Wang et al., 17 May 2025).

This body of work suggests that “dual-oriented” in generative diffusion has expanded beyond a single pairwise preference label. It now includes calibrated multi-reward aggregation, personalized embedding-conditioned alignment, and explicit positive-versus-negative guidance.

6. Diffusion preferences in decision-making and the conceptual scope of the term

The preference-diffusion paradigm also appears in sequential decision-making. MODULI formulates offline multi-objective reinforcement learning as preference-conditioned trajectory diffusion, where a diffusion planner generates trajectories conditioned on a preference vector $\mathcal{A}_u$6 and a return-to-go vector (Yuan et al., 2024). The training objective is a conditional score-matching loss,

$\mathcal{A}_u$7

and the method introduces Preference Predicted Normalization, Neighborhood Preference Normalization, and a sliding guidance mechanism with a slider adapter that approximates the change in noise prediction under a change in preference. At inference, the denoising score is updated by moving from a nearest in-distribution preference toward an out-of-distribution target preference. On D4MORL, the paper reports superior hypervolume, sparsity, and return deviation, especially in OOD preference regions (Yuan et al., 2024).

FKPD adapts direct preference optimization to diffusion policies while regularizing with forward KL to avoid out-of-distribution actions (Shan et al., 2024). After pretraining a diffusion policy by behavior cloning, it aligns the policy to pairwise preference data using a DPO-style objective and a forward-KL term. The practical objective is written in terms of denoising mean squared error for preferred and dispreferred trajectory segments, plus a regularization term over behavior data. The paper reports superior preference alignment on MetaWorld manipulation and D4RL tasks and argues that forward KL is better than reverse KL for preventing OOD degradation in diffusion policies (Shan et al., 2024).

A plausible synthesis is that “Dual-Oriented Preference Diffusion” is not yet a single standardized term across the literature. The exact phrase is tied to HorizonRec’s source/target denoising design (Zha et al., 7 Aug 2025), while adjacent works use related but distinct formulations such as dual preference distribution learning, dual caption preference optimization, negative preference optimization, and multi-reward or personalized preference conditioning (Dong et al., 2022, Saeidi et al., 9 Feb 2025, Wang et al., 17 May 2025, Dang et al., 11 Jan 2025). Several misconceptions are therefore misleading. It does not refer only to text-to-image alignment, since recommendation, diffusion policies, and offline MORL also instantiate the pattern. It does not refer only to binary winner-loser supervision, because some methods optimize general/specific, source/target/mixed, or calibrated multi-reward preferences. Nor does the dual structure automatically resolve preference conflict: conflict distribution, irrelevant prompts, inconsistent rewards, interest drift, data sparsity, and minority labels remain recurrent failure modes and continuing research targets across the cited work.