Preference-informed Representation Generation

• PRG is a framework that encodes human preference signals as structured objects (e.g., latent vectors, summaries) rather than scalar rewards, enabling richer alignment and flexible control.
• It spans applications in language model alignment, video diffusion, recommendation, and personalization by embedding preference information into latent spaces for efficient querying and intervention.
• Empirical studies demonstrate that PRG methods outperform traditional scalar models in cyclic preference accuracy, test-time control, and computational efficiency, despite calibration challenges.

Searching arXiv for the cited papers to ground the article. {"query":"(Zhang et al., 2024) Beyond Bradley-Terry Models: A General Preference Model for LLM Alignment", "max_results": 5} {"query":"(Ashrafi et al., 26 Apr 2026) Pref-CTRL: Preference Driven LLM Alignment using Representation Editing", "max_results": 5} Preference-informed Representation Generation (PRG) denotes a family of methods in which human preference signals shape a representation space directly, rather than being reduced solely to a scalar reward or left implicit in parameter updates. Across recent work, PRG appears as latent response embeddings for pairwise comparison, hidden-state value functions for test-time control, natural-language preference summaries for personalization, probabilistic latent variables for efficient individual alignment, noisy-latent reward heads in diffusion models, profile-conditioned generators in referring expression generation, and LLM-derived semantic descriptions fused into recommender representations (Zhang et al., 2024, Ashrafi et al., 26 Apr 2026, Wang et al., 2024, Chen et al., 17 Oct 2025, Zhang et al., 2024, Wu et al., 18 Jun 2026, Li et al., 16 Apr 2026, Ferreira et al., 2017). The unifying premise is that preferences are most useful when encoded as structured objects—vectors, summaries, latent distributions, or internal-state objectives—that can be queried, composed, or used to steer generation and selection.

1. Conceptual scope

PRG is defined, in the broad formulation used by these works, as learning representations of responses, policies, users, or intermediate model states that are informed directly by preference data. This contrasts with standard reward modeling, where pairwise judgments are typically compressed into a scalar $r(x,y)$ and optimized through Bradley–Terry-style objectives. In the PRG view, the representation itself is the primary carrier of preference structure.

A recurring motivation is that scalar models impose restrictive geometry. In the language-alignment setting, Bradley–Terry and related scalar models induce a total order and therefore cannot represent cyclic or otherwise intransitive preferences. In personalization, population-level alignment averages over heterogeneous users. In test-time control, post hoc evaluators only score completed outputs and therefore cannot intervene inside the generation trajectory. PRG addresses these limitations by relocating preference information into latent spaces, hidden states, or compact conditioning variables that can preserve richer structure and act earlier in the pipeline (Zhang et al., 2024, Ashrafi et al., 26 Apr 2026, Chen et al., 17 Oct 2025).

The main instantiations span several representational forms.

Instantiation	Preference-informed representation	Operational role
GPM/GPO	latent response vector $v_{y|x}$	pairwise scoring and policy optimization
LRHP	hidden state of `<\	PREFERENCE\
Pref-CTRL	value over hidden states	test-time representation editing
POPI	natural-language summary $z$	personalized conditioning
CLaP + Latent DPO	latent variable $z \in \mathbb{R}^{32}$	efficient individual alignment
PRISM	reward on noisy latents $r(z_t,c,t)$	early-stage video candidate pruning
CPGRec+	text-derived user/item vectors	recommendation representation refinement
Profile REG	overspecification profile	speaker-conditioned content selection

This breadth indicates that PRG is not a single optimization algorithm. It is a representational design principle: preserve preference structure in an explicit, manipulable form.

2. Preference geometry and structured latent spaces

The most explicit geometric formulation appears in the General Preference representation model (GPM), which embeds each response as $v_{y|x} \in \mathbb{R}^{2k}$ and defines a pairwise score with a fixed skew-symmetric operator:

$$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$

The operator $R^\succ$ is a block-diagonal stack of $2 \times 2$ $90^\circ$-rotation blocks, and the implementation can further apply a prompt-conditioned scale gate $v_{y|x}$0, yielding

$v_{y|x}$1

This score is antisymmetric, satisfies $v_{y|x}$2, and subsumes Bradley–Terry as a special case when $v_{y|x}$3. The paper states an expressiveness theorem: for any real skew-symmetric matrix $v_{y|x}$4, there exist embeddings $v_{y|x}$5 and a block-diagonal $v_{y|x}$6 such that $v_{y|x}$7. That construction is the basis for the claim that cyclic preferences can be represented exactly, whereas scalar Bradley–Terry models degenerate on truly cyclic data (Zhang et al., 2024).

The empirical pattern matches the formal argument. On RewardBench, GPM improves over Bradley–Terry reward modeling for both Gemma-2B-it and Llama-3.1-8B-Instruct. For Gemma-2B-it, average RewardBench performance rises from 68.9 to 74.5 with GPM $v_{y|x}$8; for Llama-3.1-8B-Instruct, it rises from 91.1 to 92.1 with GPM $v_{y|x}$9. On cyclic datasets constructed from UltraFeedback, GPM reaches 100.0 accuracy on all subsets, while Bradley–Terry reward models fall in the 50.0–62.9 range. The same work also claims linear query complexity in the number of candidates $z$0: embeddings require $z$1 forward passes, while pairwise scores are computed with cheap bilinear products, in contrast to pair classifiers that require $z$2 model evaluations (Zhang et al., 2024).

A different representational choice is made by LRHP, which learns a pair representation rather than a response representation. For a preference pair $z$3, LRHP concatenates the prompt and both responses, appends a special token <|PREFERENCE|>, and uses the final hidden state of that token as the preference representation $z$4. A logistic head predicts the pairwise label with binary cross-entropy. The model is trained on a fusion of nine datasets totaling 849k preference pairs, using one epoch, learning rate $z$5, batch size 128, and eight A800 GPUs. LRHP is then reused downstream for preference data selection and preference margin prediction rather than direct reward modeling (Wang et al., 2024).

The downstream results show that pair-level representations can transfer across tasks. In preference data selection for Anthropic HH, PDS-LRHP improves LLaMA-3-8B reward-model helpfulness/harmlessness accuracy from 70.52/71.84 in the vanilla setting to 75.09/75.82, and improves Mistral-7B from 62.31/63.32 to 65.89/67.68. In preference margin prediction, LRHP reaches Spearman 0.604 with only 3k labeled margin samples and improves constrained DPO relative to heuristic margins on AlpacaEval2 and Arena-Hard (Wang et al., 2024).

Taken together, these two lines exemplify a central PRG distinction. One can encode preference structure at the level of individual candidates, as in skew-symmetric response embeddings, or at the level of the preference relation itself, as in pair embeddings. Both depart from one-dimensional reward collapse.

3. Test-time control through internal-state preferences

A second major PRG family learns a preference-informed function over internal states and uses it during inference, without updating the base model’s weights. Pref-CTRL instantiates this for frozen LLMs. It models decoding as a stochastic dynamical system with state $z$6, trains a value function $z$7 on hidden states, and performs gradient ascent on the hidden representation during generation:

$z$8

Pref-CTRL retains RE-Control’s regression objective to terminal rewards, but adds two preference-driven losses: a pairwise logistic margin

$z$9

and a regularizer

$z \in \mathbb{R}^{32}$0

The total objective is

$z \in \mathbb{R}^{32}$1

The value function is a 3-layer MLP on final-layer hidden states with architecture Linear$z \in \mathbb{R}^{32}$2+ReLU, Linear$z \in \mathbb{R}^{32}$3+ReLU, Linear$z \in \mathbb{R}^{32}$4, using Adam, learning rate $z \in \mathbb{R}^{32}$5, batch size 64, and 50 epochs. At test time, only this MLP is differentiated; no backpropagation through the frozen LLM is required. The default control uses $z \in \mathbb{R}^{32}$6 ascent steps and $z \in \mathbb{R}^{32}$7, with $z \in \mathbb{R}^{32}$8 for SHP (Ashrafi et al., 26 Apr 2026).

The reported gains are moderate but consistent. On SHP with Vicuna-7B, RE-Control’s Llama-judge win rate rises from 66.8 to 73.5 under Pref-CTRL with Margin+Regularizer, and UltraRM average reward improves from $z \in \mathbb{R}^{32}$9 to $r(z_t,c,t)$0. On HH-RLHF with Vicuna-7B, Llama-judge win rate rises from 81.9 to 82.9 and GPT-judge win rate from 73.3 to 74.6. The paper also reports better zero-shot transfer than RE-Control on PKU-SafeRLHF and Nectar (Ashrafi et al., 26 Apr 2026).

PRISM transfers the same general idea to video diffusion, but places the preference head inside the noisy latent trajectory rather than on final text hidden states. A frozen video diffusion backbone supplies spatio-temporal features from intermediate denoising blocks, a Query-based Aggregation head with $r(z_t,c,t)$1 aggregates visual and text features, and an MLP produces a scalar reward $r(z_t,c,t)$2 directly from noisy latents. Training uses Bradley–Terry with Ties under uniform timestep sampling:

$r(z_t,c,t)$3

This noise-aware reward enables early-stage Best-of-$r(z_t,c,t)$4 selection: candidates are denoised only to an early step $r(z_t,c,t)$5, scored in latent space, all but the best are discarded, and only the selected trajectory is fully denoised and decoded (Wu et al., 18 Jun 2026).

The significance of PRISM is both algorithmic and computational. On VLRM-Bench without ties, PRISM with Wan2.1-14B reaches 80.82% at $r(z_t,c,t)$6, while pixel-space baselines peak at 64.58% or below; at very high noise $r(z_t,c,t)$7, PRISM remains at 64.09% whereas VideoScore2 falls to 18.55% and UnifiedReward to 21.43%. In early-stage BoN, relative time is reduced to 13% for CogVideoX-2B and 19% for Wan2.1-1.3B, corresponding to up to $r(z_t,c,t)$8 speedup, while VBench totals improve—for example, CogVideoX-2B rises from 80.2693 to 81.1582 at Step 10, and Wan2.1-14B reaches 84.4065 at Step 25 (Wu et al., 18 Jun 2026).

These systems share a distinctive PRG property: preference information acts on intermediate states during decoding or denoising, not only on completed outputs after generation has finished.

4. Personalization through explicit preference representations

In personalization, PRG frequently takes the form of an explicit conditioning variable that can be reused across tasks or users. POPI operationalizes this with a preference inference model $r(z_t,c,t)$9 that converts heterogeneous user signals $v_{y|x} \in \mathbb{R}^{2k}$0 into a concise natural-language summary

$v_{y|x} \in \mathbb{R}^{2k}$1

and a shared generation model $v_{y|x} \in \mathbb{R}^{2k}$2 that produces personalized outputs conditioned on both prompt and summary:

$v_{y|x} \in \mathbb{R}^{2k}$3

The training signal is a summary-augmented preference objective, instantiated for DPO as

$v_{y|x} \in \mathbb{R}^{2k}$4

with a KL regularizer on the inference policy:

$v_{y|x} \in \mathbb{R}^{2k}$5

The paper further states the information-theoretic bound

$v_{y|x} \in \mathbb{R}^{2k}$6

so minimizing the training loss implicitly maximizes the conditional mutual information between the summary and the preference signal (Chen et al., 17 Oct 2025).

The empirical emphasis in POPI is on compactness and transferability. On ELIX, POPI-Plug-and-Play reduces average context overhead from 3175.08 tokens under Raw-Prompting to 52.52 tokens, while improving accuracy/win rate from 56.12/49.31 to 71.48/63.84. On Review, the same plug-and-play setting improves from 51.08/53.46 to 91.81/80.99 while reducing overhead from 3049.10 to 320.80 tokens. With generator fine-tuning, POPI-Full reaches 80.14/63.97 on ELIX and 95.76/88.08 on Review, again with far shorter summaries than raw-signal prompting. The learned summaries also transfer to frozen off-the-shelf LLMs, with average plug-and-play win rates of 70.77 on ELIX, 79.88 on Review, and 52.61 on Roleplay across eight models (Chen et al., 17 Oct 2025).

A more latent-variable-centric version of the same idea appears in the paradigm that disentangles preference representation from text generation. There, a pretrained posterior encoder $v_{y|x} \in \mathbb{R}^{2k}$7, a latent adapter, and a latent-conditioned LLM $v_{y|x} \in \mathbb{R}^{2k}$8 are learned in a conditional-VAE-style pretraining stage called CLaP; personalization then updates only a small encoder $v_{y|x} \in \mathbb{R}^{2k}$9, with $$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$0, via Latent DPO. The adapter injects latent-aware key–value caches into the transformer, so generation is steered by $$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$1 without updating the LLM itself. The paper reports 80% to 90% less incremental training time per new preference than LoRA or P-tuning DPO. On IMDB, additional training time falls to 0.08 h for Latent DPO versus 0.47 h for LoRA-DPO; on TL;DR with GPT-J-6B, it falls to 0.29 h versus 2.62 h for LoRA and 5.69 h for full DPO. Quality remains competitive or better: on TL;DR human evaluation, win rate rises from 52.8% for SFT and 54.8% for DPO to 63.3% for CLaP plus Latent DPO (Zhang et al., 2024).

Both POPI and CLaP-style latent personalization treat preference representation as a reusable control interface. One uses human-readable summaries; the other uses probabilistic latent variables and adapters. In both cases, the core generator can remain shared across users.

5. Cross-domain instantiations beyond standard LLM alignment

PRG is not restricted to modern LLM post-training. An early, non-neural instantiation appears in trainable referring expression generation based on overspecification preferences. Speakers are partitioned deterministically into three profiles—always overspecified, always minimally distinguishing, and mixed behavior—and a “Profile” model is trained on data from speakers who share the same overspecification behavior. The underlying REG system uses SVMs with Gaussian kernels for atomic attribute classifiers and a one-against-one multi-class relation classifier, along with context features and speaker-dependent features. The preference-informed effect is realized through profile-based data selection rather than latent-vector learning (Ferreira et al., 2017).

This profile-conditioned strategy improves over fully personalized per-speaker training in most domains. Overall Dice rises from 0.87 to 0.90 and REG Accuracy from 0.60 to 0.66. Overspecification level accuracy improves from 0.75 to 0.79 overall. Significant gains are reported on TUNA-People, GRE3D3, GRE3D7, and Stars2, while Stars is a counterexample in which Profile performs worse because relational properties dominate content selection and are not captured by the single preference dimension of overspecification (Ferreira et al., 2017).

In recommender systems, CPGRec+ introduces PRG as an LLM-assisted representation-refinement module. For each game, Qwen2.5 generates a rating-informed description $$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$2 from title, average rating, price, and release date; for each user, Qwen2.5 then generates a preference-informed player description $$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$3 by comparing normalized dwell time $$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$4 and normalized average rating $$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$5 over interacted games. These texts are embedded by M3-Embedding and fused into graph-learned user and item embeddings through alignment and integration MLPs. The design is explicitly complementary to Preference-informed Edge Reweighting (PER), which acts on graph edges rather than node representations (Li et al., 16 Apr 2026).

The reported gains support the claim that preference-aware semantic representations can improve both relevance and diversity. In the trade-off framework on Steam I, NDCG@10 rises from 0.4547 to 0.4691, Recall@10 from 0.5990 to 0.6192, Conventional Coverage@10 from 0.3515 to 0.3611, Tail Coverage@10 from 0.2336 to 0.2617, and Tail@10 from 0.0038 to 0.0047. On Steam II, NDCG@10 rises from 0.3398 to 0.3427 and Tail Coverage@10 from 0.4693 to 0.4766. Removing PRG degrades both accuracy and long-tail diversity (Li et al., 16 Apr 2026).

A further neighboring instance is DreamDPO for text-to-3D generation. There, the learnable 3D representation $$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$6 is optimized with pairwise preference guidance obtained from reward models or large multimodal models over pairs of rendered views. The method constructs noisy pairs at the same diffusion timestep, ranks them, and applies a reference-free SimPO-style objective over denoising residuals, with a threshold $$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$7 to suppress unstable pushes on near-ties. On GPTEval3D Elo ratings, DreamDPO ranks first across all reported criteria, including text-asset alignment 1298.9 and overall 1203.1, and runs in about two hours per asset on a single NVIDIA RTX A6000 (Zhou et al., 5 Feb 2025).

These examples show that PRG generalizes across generation, ranking, and recommendation. The common element is not modality, but the decision to encode preference structure into representations that remain actionable at inference or selection time.

6. Limitations, misconceptions, and open problems

A persistent misconception is that richer preference representations automatically solve alignment. The cited work is more restrained. In skew-symmetric preference embeddings, identifiability is only up to orthogonal transforms that preserve the bilinear form, calibration may drift across prompts and domains, $$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$8 tuning affects logit dynamic range, and larger embedding dimension can overfit or destabilize training without normalization and gating (Zhang et al., 2024). In pair-representation learning, LRHP does not directly replace alignment objectives; it strengthens them through better data selection and margin estimation rather than by serving as a standalone alignment algorithm (Wang et al., 2024).

A second misconception is that PRG necessarily entails model fine-tuning. Pref-CTRL and PRISM demonstrate the opposite. Test-time alignment can operate by editing hidden states or scoring noisy latents while keeping the base backbone frozen. Yet these methods bring their own failure modes: in Pref-CTRL, margin-only training can over-optimize and harm quality, and performance degrades when $$s(y_i \succ y_j \mid x)=v_{y_i|x}^{T} R^\succ v_{y_j|x}, \qquad P(y_i \succ y_j \mid x)=\sigma(s(y_i \succ y_j \mid x)).$$9 or $R^\succ$0 become too large; in PRISM, latent-domain coupling ties the reward head to a specific VAE/backbone family, weaker generators provide weaker evaluators, and some timesteps are less predictive than others even though the model is robust overall (Ashrafi et al., 26 Apr 2026, Wu et al., 18 Jun 2026).

A third misconception is that personalization representations are automatically stable, fair, or privacy-benign. POPI states that summaries inherit the biases and coverage limits of preference and reward data, Roleplay remains more variable because of task open-endedness, and safety, bias, and privacy are not deeply explored. The same paper argues that the explicitness of natural-language summaries makes governance easier because $R^\succ$1 is auditable and editable, but it also underscores that demographic settings such as AlignX require careful policy control (Chen et al., 17 Oct 2025). In latent-personalization systems, the per-user encoder parameters themselves become preference-bearing artifacts that warrant privacy safeguards (Zhang et al., 2024).

Finally, several domain-specific limitations recur. Profile-based REG assumes the speaker’s overspecification profile is known a priori and uses a coarse 100% threshold, leaving cold-start and mixed-membership inference unaddressed; its failure on Stars suggests that a single preference dimension can be insufficient when relation preference dominates (Ferreira et al., 2017). In CPGRec+, sparse user histories and missing game ratings reduce the quality of preference-informed descriptions, and static LLM-generated descriptions may lag behind temporal preference drift (Li et al., 16 Apr 2026).

These caveats do not weaken the core significance of PRG. They define its current frontier. The literature increasingly treats preference alignment not as scalar reward fitting alone, but as the design of representations whose geometry, semantics, or dynamics can encode heterogeneity, support efficient querying, and intervene inside generation itself.