General Preference Model (GPM)

• General Preference Model (GPM) is a framework that models multi-dimensional, non-transitive preference data via structured skew-symmetric embeddings and bilinear operators.
• It applies to areas such as large language model alignment, economic decision theory, and neuroscience, offering improved empirical alignment and resistance to reward hacking.
• GPM underpins reinforcement learning techniques like GPRL and GPO by integrating explicit decomposition of scalar and cyclic components for robust policy optimization.

A General Preference Model (GPM) is a mathematical and algorithmic framework for modeling preference data that is not restricted to scalar (transitive) utility representations. Instead, GPMs allow for multi-dimensional, structured, and often intransitive (cyclic) comparisons, capturing the complexity of human or agent preferences in a variety of settings including LLM alignment, economic decision theory, neuroscience, and beyond. Modern instantiations of the GPM formalism combine skew-symmetric embedding architectures, bilinear comparison operators, and normalized, multi-axis aggregation schemes to provide increased expressiveness, resistance to reward hacking, and improved empirical and theoretical alignment with human-annotated preference data (Umer et al., 18 May 2026, Zhang et al., 2024, Ye et al., 2024, Huang et al., 17 May 2026).

1. Mathematical Foundations and Embedding Structure

GPM begins by embedding each evaluation context (e.g., LLM prompt and candidate response) into a $2k$-dimensional real vector $\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$ of unit norm ($\|\mathbf{v}_{y\mid x}\|_2=1$), where $k$ is the number of skew-symmetric subspaces or preference axes. Preferences between two candidates $y_i, y_j$ are computed via a fixed block-diagonal skew-symmetric operator

$$\mathbf{R}^{\succ} = \mathrm{blockdiag}\bigl(\mathbf{R}_1,\dots,\mathbf{R}_k\bigr)\,,\quad \mathbf{R}_\ell = \begin{pmatrix} 0 & -1\ 1 & 0 \end{pmatrix}\,,\;\;\ell=1,\dots,k,$$

acting as a 90° rotation in each $(2\ell-1, 2\ell)$ plane. For block $\ell$ the signed “subspace score” for the pair is

$$s_\ell(y_i, y_j\mid x) = v_{i}^{(2\ell)} v_{j}^{(2\ell-1)} - v_{i}^{(2\ell-1)} v_{j}^{(2\ell)},$$

equivalently $$\mathrm{Im}(z_\ell^{(i)}\, \overline{z}_\ell^{(j)})$$ if blocks are viewed as complex numbers (Umer et al., 18 May 2026, Zhang et al., 2024).

Aggregating over subspaces with context-dependent nonnegative weights $\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$0 (the modulus of imaginary eigenvalues), the overall pairwise preference score is

$\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$1

and the probability that $\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$2 is preferred over $\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$3 is

$\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$4

This generalizes classical scalar reward models by capturing multi-dimensional, anti-symmetric relations and encoding intransitive preference cycles.

2. Intransitivity, Cyclicity, and Theoretical Capacities

Scalar preference models enforce transitivity and total ordering, which is inadequate for many domains where cycles (e.g., $\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$5) naturally occur. The block-diagonal skew-symmetric architecture in GPM enables each axis to independently encode a cycle, while combining multiple axes allows modeling of arbitrarily complex, non-transitive patterns (Umer et al., 18 May 2026, Zhang et al., 2024). Empirical studies confirm that GPMs can realize preference structures where all scalar (Bradley–Terry) models can only random guess.

However, the capacity to represent mixtures of cycles and dominance is fundamentally limited by the model’s rank and entanglement between cyclic and transitive components. It is shown that in low-rank GPMs, the same embedding must simultaneously carry both hierarchical and cyclic structure, which can lead to failures to correctly represent “dominant + cycle” preference patterns unless additional dimensions are introduced (Huang et al., 17 May 2026). This limitation motivated the development of models that explicitly decompose scalar and cyclic heads:

$\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$6

where $\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$7 is a scalar reward and the second term is the original skew-symmetric comparison (Huang et al., 17 May 2026).

3. Preference-Based Reinforcement Learning: GPRL and GPO

GPM’s multi-dimensional structure is embedded directly into policy optimization through General Preference Reinforcement Learning (GPRL). In each RL step, a set of responses is sampled and per-axis group-relative “advantages” are computed:

$\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$8

where $\mathbf{v}_{y\mid x}\in\mathbb{R}^{2k}$9 and $\|\mathbf{v}_{y\mid x}\|_2=1$0 are the per-axis group mean and std. The total advantage is aggregated via context-dependent eigenvalues:

$\|\mathbf{v}_{y\mid x}\|_2=1$1

GPRL’s surrogate objective is then

$\|\mathbf{v}_{y\mid x}\|_2=1$2

where $\|\mathbf{v}_{y\mid x}\|_2=1$3 is the importance ratio, and the KL penalty controls trust region adaptation (Umer et al., 18 May 2026).

Preference-based optimization contrasting against a reference or opponent policy (General Preference Optimization, GPO) further generalizes RLHF by maximizing expected pairwise preference scores rather than scalar rewards (Zhang et al., 2024). These frameworks enable robust alignment even with intransitive or opaque (black-box) preference signals.

4. Drift Control and Reward Hacking Resistance

A critical feature of GPM-based RL is the ability to detect and correct reward hacking and single-axis exploitation via closed-loop monitoring. For each training step, the variance profile across axes is

$\|\mathbf{v}_{y\mid x}\|_2=1$4

with the KL divergence to the initial variance profile $\|\mathbf{v}_{y\mid x}\|_2=1$5. Upon detecting drift, the controller reweights axes and tightens the trust region by adjusting eigenvalue multipliers and the KL penalty, thus suppressing modes of reward hacking that would otherwise exploit sensitivity in scalar signal axes (Umer et al., 18 May 2026).

5. Empirical Validation and Practical Impact

GPM models consistently outperform scalar reward models across benchmarks sensitive to cyclic/intransitive structure and reward hacking:

Dataset	Baseline	GPM Improvement
RewardBench (Gemma)	BT avg=68.9	GPM (d=8) avg=74.5 (+5.6%)
RewardBench (Llama)	BT avg=91.1	GPM (d=6) avg=92.1 (+1.0%)
AlpacaEval2.0	BT WR=62–71%	GPM up to +9.3pp higher
Arena-Hard	BT baseline	GPM consistently higher WR

Notably, GPRL applied to Llama-3-8B-Instruct achieves a length-controlled win rate of 56.51% on AlpacaEval 2.0 and improved resistance to reward hacking across extended training runs (Umer et al., 18 May 2026, Zhang et al., 2024, Huang et al., 17 May 2026).

6. Extensions, Limitations, and Relation to Alternative GPM Variants

The GPM concept also arises in other literature, from Gaussian-process-based latent utility models allowing for multi-utility extensions, to general preference oracles in reinforcement learning from human feedback (RLHF). Gaussian-process GPMs formalize rationality axioms via GP priors and likelihoods on utility differences, allowing for extensions to Pareto, label, and semiorder models (Benavoli et al., 2024).

Critiques of standard GPM architecture focus on the entanglement of hierarchy (scalar dominance) and cyclicity within finite-dimensional skew-symmetric subspaces, limiting representational efficiency and sometimes failing to guarantee global dominance in the presence of complex cycles (Huang et al., 17 May 2026). This has led to hybrid designs such as the Hybrid Reward-Cyclic (HRC) model, explicitly decomposing transitive and cyclic components by summing a scalar head and a skew-symmetric component, achieving faster convergence and robust recovery of both structure types in synthetic and real data.

7. Theoretical and Applied Implications

GPMs generalize reward modeling far beyond classical transitive and additive approaches and enable learning and alignment in the presence of complex, multidimensional, and population-heterogeneous preferences. They formally encompass intransitive patterns, are compatible with black-box or large-group aggregated signals, and provide operational advantages in RL application domains sensitive to reward hacking and mode collapse.

However, the need to carefully manage the embedding rank and decomposition between hierarchically dominant and cyclic structures remains an active challenge. Future extensions are likely to further integrate explicit preference decompositions and scalable inference for complex, high-cardinality choice environments.

---

References:

• (Umer et al., 18 May 2026) General Preference Reinforcement Learning
• (Zhang et al., 2024) Beyond Bradley-Terry Models: A General Preference Model for LLM Alignment
• (Ye et al., 2024) Online Iterative Reinforcement Learning from Human Feedback with General Preference Model
• (Huang et al., 17 May 2026) Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic LLM Alignment
• (Benavoli et al., 2024) A tutorial on learning from preferences and choices with Gaussian Processes