DMER-Preference Approaches

• DMER-Preference is a context-dependent family of constructions that includes reward decomposition, data selection, and evaluation frameworks for pairwise comparisons.
• It encompasses PCA-based decomposed reward modeling, difficulty-based selection via DPO implicit reward gaps, and emotion-based ranking datasets.
• The approach not only improves model performance and personalization efficiency but also addresses issues of annotator heterogeneity and robust optimization under distribution shift.

DMER-Preference is not a single standardized artifact. In the supplied 2024–2025 literature, the label is used for several distinct preference-centered objects: a PCA-based decomposed reward learning framework derived from binary comparisons; a difficulty-based preference-data selection rule defined by the DPO implicit reward gap; and a human-annotated dataset for pairwise judgments over free-form emotion descriptions. Related summaries also reuse the label for heterogeneous-preference DPO, distributionally robust DPO, and diffusion-based preference optimization. The term is therefore best understood as a context-dependent family of constructions organized around preference decomposition, preference selection, and preference evaluation rather than as a single canonical method (Luo et al., 18 Feb 2025, Qi et al., 6 Aug 2025, Lian et al., 6 Jul 2025, Chidambaram et al., 17 Oct 2025, Xu et al., 4 Feb 2025, Liu et al., 2024).

1. Terminological scope and research contexts

The supplied literature indicates that no single expansion or definition is shared across all appearances of DMER-Preference. In some works, it is a learning framework for decomposed rewards; in others, it is a data-selection criterion for DPO; and in the emotion-description literature, it is a benchmark dataset. Separately, the acronym DMER itself is overloaded: it denotes Descriptive Multimodal Emotion Recognition in one line of work and Dynamic Music Emotion Recognition in another (Luo et al., 18 Feb 2025, Qi et al., 6 Aug 2025, Lian et al., 6 Jul 2025, Chidambaram et al., 17 Oct 2025, Xu et al., 4 Feb 2025, Zhang et al., 2024).

Usage of DMER-Preference	Core object	Source
Decomposed Multi-Expert Reward Preference	PCA on embedding differences, orthogonal reward axes, user-controlled recombination	(Luo et al., 18 Feb 2025)
Difficulty-based preference data selection	Selection of low DPO implicit reward-gap pairs under threshold $\tau$ or ratio $\rho$	(Qi et al., 6 Aug 2025)
Preference dataset for human emotions	Pairwise judgments over two free-form emotion descriptions, with tie handling and Bradley–Terry ranking	(Lian et al., 6 Jul 2025)
Heterogeneous-preference DPO exposition	Latent annotator types, EM-DPO, and min–max regret aggregation	(Chidambaram et al., 17 Oct 2025)
Distributionally robust DPO exposition	Minimax DPO over Wasserstein or KL uncertainty sets	(Xu et al., 4 Feb 2025)

This multiplicity matters because the same label can denote a reward decomposition, a subset-selection heuristic, or an evaluation dataset. A plausible implication is that DMER-Preference currently functions more as a recurring preference-learning motif than as a settled technical standard.

2. PCA-based decomposed reward modeling

In "Rethinking Diverse Human Preference Learning through Principal Component Analysis" (Luo et al., 18 Feb 2025), Decomposed Reward Models implement what the authors call DMER-Preference by starting from a standard human–AI preference dataset of pairs $(\mathrm{response}^+,\mathrm{response}^-)$, embedding each response with an encoder $f(\cdot)$, and forming difference vectors $\Delta x = x^+ - x^-$. Collecting $N$ such vectors yields a matrix $D \in \mathbb{R}^{d \times N}$, from which the empirical covariance is computed as $$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$. The principal directions are obtained from $\Sigma u_i = \lambda_i u_i$, ordered by $\lambda_1 \ge \lambda_2 \ge \cdots$, with orthonormal eigenvectors satisfying $\rho$0. Each principal axis becomes a decomposed reward function $\rho$1, and user-specific alignment is expressed through a coefficient vector $\rho$2 via

$\rho$3

The framework treats the top $\rho$4 eigenvectors as the most salient axes along which human judgments vary. The supplied description states that these orthogonal axes can align with distinct aspects of preference such as general helpfulness/informativeness, avoidance of unsafe or toxic content, and creativity or humor. Interpretation proceeds by examining large projections $\rho$5, scoring held-out responses along each axis, and correlating $\rho$6 with human annotations of specific qualities. Validation is described as high correlation, with the example $\rho$7, between a decomposed reward and independent human ratings for the same attribute.

A central feature is post hoc personalization without retraining the basis. For a new user, the method keeps $\rho$8 and $\rho$9 fixed, collects a small number $(\mathrm{response}^+,\mathrm{response}^-)$0 of pairwise choices, and solves a simple linear regression or ranking SVM to estimate $(\mathrm{response}^+,\mathrm{response}^-)$1. The concrete objective given in the supplied account is

$(\mathrm{response}^+,\mathrm{response}^-)$2

Because the representation and basis remain fixed, this personalization step is described as taking seconds on CPU.

The reported empirical properties are explicitly operational. DRMs are said to match or exceed the accuracy of monolithic reward models on held-out preference prediction, achieving up to $(\mathrm{response}^+,\mathrm{response}^-)$3 AUC lift; to permit smooth trade-offs between conflicting objectives such as help versus safety; and to scale linearly in $(\mathrm{response}^+,\mathrm{response}^-)$4 both in training time, exemplified by PCA on a $(\mathrm{response}^+,\mathrm{response}^-)$5 $(\mathrm{response}^+,\mathrm{response}^-)$6 matrix, and at inference, where evaluation requires $(\mathrm{response}^+,\mathrm{response}^-)$7 dot-products per candidate. Human-in-the-loop studies are further summarized as showing that end users can find a satisfactory $(\mathrm{response}^+,\mathrm{response}^-)$8 in under ten preference queries and obtain outputs with higher user satisfaction scores and reduced unwanted side effects.

3. Difficulty-based preference data selection via the DPO implicit reward gap

In "Difficulty-Based Preference Data Selection by DPO Implicit Reward Gap" (Qi et al., 6 Aug 2025), DMER-Preference denotes a data-selection method grounded in the DPO implicit reward. For a response $(\mathrm{response}^+,\mathrm{response}^-)$9 under context $f(\cdot)$0, the implicit reward is defined as

$f(\cdot)$1

and for a preference pair $f(\cdot)$2 the DPO implicit reward gap is

$f(\cdot)$3

The method ranks preference pairs by ascending $f(\cdot)$4 and selects examples with small gaps, on the hypothesis that they are harder cases, yield larger training gradients, and contain more information. For a full dataset $f(\cdot)$5, the selected subset is

$f(\cdot)$6

with $f(\cdot)$7 either fixed directly or chosen as the $f(\cdot)$8-quantile of the $f(\cdot)$9.

The theoretical justification is supplied from two directions. First, for the single-pair DPO loss $\Delta x = x^+ - x^-$0, the gradient magnitude is

$\Delta x = x^+ - x^-$1

so the sigmoid weight is maximal at $\Delta x = x^+ - x^-$2 and decays for large positive gaps. Second, if $\Delta x = x^+ - x^-$3 is the model’s preference probability, the entropy $\Delta x = x^+ - x^-$4 peaks at $\Delta x = x^+ - x^-$5, again corresponding to $\Delta x = x^+ - x^-$6. The supplied algorithm requires one forward pass per response, four per pair, followed by sorting, with total complexity $\Delta x = x^+ - x^-$7.

The reported experiments are specific. The datasets are SHP (385 K, human annotated), Skywork (77 K, synthetic), UltraFeedback (61 K, synthetic), and RLHFlow (100 K, synthetic). The selection model uses policy = LLaMA3-iterative-DPO-final and reference = LLaMA3-SFT. Downstream evaluation covers reward-model training on the gemma-2-2b-it checkpoint, evaluated on RewardBench accuracy, and DPO policy fine-tuning on the Tulu3-SFT checkpoint, judged by GPT-4o on AlpacaEval 2.0 using Win Rate and Length-Controlled WR. Against Full Set (100 %), Random, ZIP, DiverseEvol, and SDPO, the authors report that with a 10 % budget DMER outperforms or matches Full Set in 67.5 % of reward-model cases and beats all other 10 % baselines in 75 % of RewardBench splits. On SHP Total accuracy, the supplied numbers are $\Delta x = x^+ - x^-$8 for DMER, $\Delta x = x^+ - x^-$9 for Full, and $N$0–$N$1 for the other baselines. For DPO policy training with 10 % data, DMER exceeds Full Set in 88 % of settings; the examples given are SHP LCWR $N$2 versus $N$3, SHP WR $N$4 versus $N$5, and Skywork LCWR $N$6 versus $N$7.

The ablations refine the picture. On SHP RewardBench, Total accuracy peaks at 15 % selection ratio with $N$8, while 10 % already yields $N$9 and 100 % gives $D \in \mathbb{R}^{d \times N}$0, leading the supplied summary to conclude that the optimum is around 10–15 %. Ranking by difficulty is reported as robust across selector models, with Skywork Total scores of $D \in \mathbb{R}^{d \times N}$1 for LLaMA3, $D \in \mathbb{R}^{d \times N}$2 for Gemma2, and $D \in \mathbb{R}^{d \times N}$3 for Tulu3. Length normalization of the gap is stated to hurt selection quality, reducing Skywork Total from $D \in \mathbb{R}^{d \times N}$4 to $D \in \mathbb{R}^{d \times N}$5.

4. DMER-Preference as a dataset for emotion-description evaluation

In "DMER-Ranker: Learning to Rank Emotion Descriptions in the Absence of Ground Truth" (Lian et al., 6 Jul 2025), DMER-Preference is the first preference dataset specifically designed for human emotions. It is built from 1,368 short video clips depicting people in naturalistic settings and containing the full multimodal stream of facial appearance, body gestures, and speech audio. For each video, two free-form emotion descriptions are presented to annotators, who answer the question, “Which description better reflects the character’s emotional state in this video?” The allowed judgments are “Description 1,” “Description 2,” and “Tie.” Annotators are instructed to focus on multimodal emotional cues, including facial expression, tone of voice, body language, temporal dynamics, intensity, and uncertainty.

Quality control is stringent. Three annotators, each of whom had to pass a preliminary emotion-recognition test, label the pairs, and only samples on which all three agree are retained. The initial annotation load is 1,368 video–description pairs, while the final retained set contains 574 pairs. Excluding ties, pairwise annotator agreement is reported as approximately 70%, described as the upper bound of human consistency on non-tie judgments; including ties, overall agreement drops by about 10 percentage points to approximately 60%, indicating that tie judgments are more ambiguous. The released dataset is a single pooled set of 574 annotated pairs, and the supplied summary states that no train/val/test split is provided.

The preference labels are encoded as $D \in \mathbb{R}^{d \times N}$6, where $D \in \mathbb{R}^{d \times N}$7 means Description 1 is preferred, $D \in \mathbb{R}^{d \times N}$8 means Description 2 is preferred, and $D \in \mathbb{R}^{d \times N}$9 means Tie. In two-class experiments, tie samples are discarded and the remaining labels are mapped to binary form. For ranking multiple systems, pairwise outcomes are aggregated into a win–loss matrix $$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$0 and converted into model-level rankings by the Bradley–Terry model, which assigns each system $$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$1 a positive strength parameter $$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$2 and defines

$$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$3

The supplied negative log-likelihood is

$$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$4

Gradient descent or Minorization–Maximization is then used to recover the $$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$5.

Automatic preference prediction is formulated as classification. The evaluation uses weighted-average F1 and Accuracy in both two-class and three-class settings, along with flip consistency under input order reversal and multi-run consistency for multi-step chains. Four prompting strategies are reported: a one-step direct judge; a two-step method with an internally generated reference; a two-step method using an external text-only LLM such as Qwen3-14B for the comparison step; and a three-step chain in which an external LLM generates explicit reasoning before issuing the final preference decision. Evaluated MLLMs include Video-LLaVA, Chat-UniVi, mPLUG-Owl, Qwen2.5-VL/Omni, AffectGPT, GPT-4.1/4o, and Gemini variants. The best automatic system is reported as AffectGPT with the three-step strategy, achieving about $$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$6 WAF on two-class judgments, still below the approximate human upper bound. The same supplied account notes that majority-vote aggregation of the top-$$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$7 MLLM judges can improve recognition performance and that forward+reverse majority voting further stabilizes outputs for many models.

The conceptual novelty lies in replacing prediction–ground-truth comparison with prediction–prediction comparison. This reorients evaluation away from collecting comprehensive ground-truth emotion descriptions and toward comparative judgment over candidate descriptions, while retaining open-vocabulary emotional content, temporal dynamics, intensity, and uncertainty.

5. Preference heterogeneity, identifiability, and fair aggregation

A separate line of work makes heterogeneity itself the central object. In "Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences" (Chidambaram et al., 17 Oct 2025), annotators are indexed by $$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$8 and assigned latent types $$\Sigma = E[\Delta x \Delta x^\top] \simeq \frac{1}{N}DD^\top$$9, with type-dependent utility

$\Sigma u_i = \lambda_i u_i$0

Under the Bradley–Terry–Luce model, the probability that type $\Sigma u_i = \lambda_i u_i$1 prefers $\Sigma u_i = \lambda_i u_i$2 over $\Sigma u_i = \lambda_i u_i$3 is

$\Sigma u_i = \lambda_i u_i$4

The identifiability result emphasized in the supplied summary is that aggregate binary pairwise data are insufficient for recovering the latent preference distribution in general, whereas ternary or larger choice sets restore identifiability under the cited conditions. The canonical binary counterexample is the case in which half the population has $\Sigma u_i = \lambda_i u_i$5 and half has $\Sigma u_i = \lambda_i u_i$6, yielding $\Sigma u_i = \lambda_i u_i$7 for every pair.

The proposed algorithmic response is an EM adaptation of DPO. With annotator-level data $\Sigma u_i = \lambda_i u_i$8, the E-step computes posterior type assignments $\Sigma u_i = \lambda_i u_i$9, and the M-step updates both mixture weights and type-specific policies through a weighted DPO objective. After convergence, one obtains $\lambda_1 \ge \lambda_2 \ge \cdots$0 policies $\lambda_1 \ge \lambda_2 \ge \cdots$1 and soft assignments $\lambda_1 \ge \lambda_2 \ge \cdots$2. Deployment is then handled by a min–max regret aggregation rule:

$\lambda_1 \ge \lambda_2 \ge \cdots$3

which, after restricting $\lambda_1 \ge \lambda_2 \ge \cdots$4 to an affine mixture $\lambda_1 \ge \lambda_2 \ge \cdots$5, becomes a finite zero-sum game solved by multiplicative-weights–gradient-descent dynamics in $\lambda_1 \ge \lambda_2 \ge \cdots$6 iterations.

Adjacent literatures reinforce the same concern with non-uniform tastes. In the generalized discrete mixture model of Hancock and Buckell, the key issue in modelling discrete choice data is preference heterogeneity; the GDM introduces boost parameters in the class-allocation component, nests both standard DM and latent-class structures, allows the data to reveal correlation between attributes, and performs at least as well as the better of DM or LC in the reported simulations and empirical applications (Hancock et al., 17 Jun 2025). In Personalized DMER for music emotion recognition, tasks are divided by annotators so that samples in a task are annotated by the same annotator, and DSAML is described as predicting personalized perception of emotions with just one personalized annotation sample (Zhang et al., 2024). This suggests that, across domains, the most stable formulations of preference learning increasingly treat annotator variation as signal rather than noise.

6. Robustness, diffusion, and broader methodological extensions

Some summaries extend DMER-Preference terminology beyond decomposition or datasets to robust and generative preference optimization. In "Robust LLM Alignment via Distributionally Robust Direct Preference Optimization" (Xu et al., 4 Feb 2025), the DMER–Preference objective is the minimax problem

$\lambda_1 \ge \lambda_2 \ge \cdots$7

where the uncertainty set $\lambda_1 \ge \lambda_2 \ge \cdots$8 is defined by either Wasserstein or KL divergence around a nominal distribution. The resulting algorithms are Wasserstein DPO and Kullback–Leibler DPO. For WDPO, the supplied approximation adds a gradient-norm regularizer to the empirical DPO loss; for KLDPO, a Fenchel–Legendre dual yields a reweighted empirical objective in which sample weights are a softmax in the loss. Under the stated assumptions, both admit $\lambda_1 \ge \lambda_2 \ge \cdots$9 convergence of the learned parameter to the population-robust minimizer. On the EMOTION benchmark with shifted reward mixtures, the summary reports that KLDPO reduces worst-case drop by 30–50% relative to DPO.

A separate generative extension appears in "Preference Diffusion for Recommendation" (Liu et al., 2024). There, PreferDiff is presented as a DMER-Preference approach in which Bayesian Personalized Ranking is recast as a log-likelihood-ratio objective over diffusion next-item densities, the intractable likelihood is handled by a variational upper bound, mean squared error is replaced by cosine error, and multiple negatives are compressed through a negative centroid. The final loss interpolates between a diffusion reconstruction term and a ranking term through

$\rho$00

The supplied exposition states that PreferDiff is exactly DPO specialized to diffusion next-item densities when $\rho$01 and the reference is uniform. The reported experiments show 6–19 % relative improvement over the best diffusion baseline and approximately 35 epochs to convergence versus DreamRec’s approximately 65.

Related diffusion preference optimization also appears outside the explicit DMER-Preference label. "Ranking-based Preference Optimization for Diffusion Models from Implicit User Feedback" (Wu et al., 21 Oct 2025) formulates preference learning as ranking denoising losses rather than optimizing a sigmoid likelihood or using an external reward model. Its thresholded ranking loss clips the margin once expert demonstrations outrank policy samples by more than a threshold $\rho$02, and the method combines offline expert demonstrations with online policy-generated negative samples. The reported user study gives a 75% win-rate over SD v1-5, 56.7% versus Diff-DPO w/ SFT, and 66.7% versus Diff-KTO w/ SFT.

Taken together, these strands suggest a common research pattern. Whether the objective is decomposition, difficulty-based data curation, latent-type discovery, robust optimization under shift, or ranking without explicit ground truth, DMER-Preference is consistently attached to settings where pairwise or listwise comparisons are used to avoid expensive scalar annotations and to preserve structure that would be lost under a single averaged reward.