Preference-Aware Autoregressive Reward Model

• PARM is a reward modeling framework that integrates explicit multi-dimensional preferences at each token generation step.
• It employs techniques like PBLoRA for low-rank, preference-specific adaptation, enabling efficient and adaptive multi-objective inference.
• Empirical results show significant gains in hypervolume and efficiency over traditional reward models in both language and image generation tasks.

A Preference-Aware Autoregressive Reward Model (PARM) is a class of reward modeling designed for aligning generation processes in autoregressive models—such as LLMs and autoregressive image generators—with human or application-specific multi-dimensional preferences. PARM architectures constitute a marked progression over classical trajectory-level reward models by enabling step-wise, context-sensitive reward estimation and direct conditioning on explicit preference vectors. Recent work has instantiated PARM for both discrete image generation (Guo et al., 23 Jan 2025) and multi-objective alignment of LLMs (Lin et al., 6 May 2025), leading to substantial efficiency and effectiveness gains over independent per-dimension reward modeling strategies.

1. Conceptual Foundations and Motivation

PARM is motivated by limitations in traditional reward modeling where a reward model either evaluates only complete outcomes (Outcome Reward Model, ORM) or scores every intermediate step (Process Reward Model, PRM), both of which are inadequate for efficient autoregressive generation and fine-grained preference guidance. In contrast, PARM adaptively and explicitly incorporates preferences at the token or generation step level:

• In LLM alignment, autoregressive reward models allow token-level reward assignment, which matches the incremental nature of decoding and enables efficient test-time alignment under human or application-specified trade-offs (Xu et al., 2024, Lin et al., 6 May 2025).
• In image generation, step-wise potential assessment allows for truncation of unpromising trajectories and dynamic best-of-N sampling guided by model or human preference signals (Guo et al., 23 Jan 2025).

The primary rationale is to provide both finer-grained and more adaptive supervision for generation, enhancing both sample quality and alignment with heterogeneous, potentially multi-dimensional, preference vectors without the need for retraining or fine-tuning the base model.

2. Mathematical Formulation and Training Objectives

The mathematical core of PARM across modalities is an autoregressive reward function parameterized by a (typically small) autoregressive model, often implemented as a transformer with low-rank adaptation. For LLMs:

$$r(x, y, p) = \sum_{t=1}^T \log p_\theta(y_t | x, y_{<t}, p)$$

where $x$ is the prompt, $y$ is the generated sequence, $p$ is a k-dimensional preference vector (with $\sum_i \alpha_i = 1$, $\alpha_i \geq 0$), and $p_\theta$ is the stepwise preference-aware token probability (Lin et al., 6 May 2025). Training proceeds via pairwise preference data, minimizing a loss of the form:

$$\ell(\theta; p) = \sum_{i=1}^k \alpha_i \ell_i(\theta; p)$$

$$\ell_i(\theta; p) = - \mathbb{E}_{(x, y^+, y^-, z_i) \sim \mathcal{D}_i} \log \sigma\left( (-1)^{z_i} \, \beta_r ( r(x, y^+, p) - r(x, y^-, p) ) \right)$$

where $\sigma$ is the logistic sigmoid, $x$0 is the preference label, and $x$1 is a scale hyperparameter (Lin et al., 6 May 2025). Expectation over random $x$2 draws during training ensures that the model spans the convex hull of Pareto-optimal solutions.

A similar, stepwise approach is applied in discrete image generation, where at each decoding step the reward model returns a stepwise potential $x$3 used for gating (clarity judgment and potential assessment), whose outputs are mapped to rewards via monotonic functions and summed over the trajectory (Guo et al., 23 Jan 2025).

3. Architectural Realizations: PBLoRA and Adaptive Inference

To enable efficient multi-objective inference, PARM introduces preference-aware conditioning via bilinear low-rank adaptation (PBLoRA):

$x$4

where $x$5 is the frozen base model weight, $x$6 is a global rank-$x$7 low-rank adaptation, $x$8 gives preference-specific adaptation with $x$9 a linear network mapping $y$0 into the adaptation subspace (Lin et al., 6 May 2025).

At test time, the user supplies a target preference vector $y$1, PBLoRA computes the corresponding $y$2, and a single PARM forward pass per decoding step provides all necessary guidance for joint multi-objective alignment. This sharply contrasts with prior methods such as GenARM, which require separate reward model forward passes per preference objective (Xu et al., 2024, Lin et al., 6 May 2025).

For autoregressive image generation, the model adaptively gates and prunes decoding paths step-wise, using the potential to avoid early (noisy/blurry) and late (collapsed) sampling, followed by a global best-of-N selection on potential-validated paths, and (optionally) reflection-augmented self-correction (Guo et al., 23 Jan 2025).

4. Multi-Objective and Weak-to-Strong Alignment

A salient feature of PARM is its ability to enable real-time, test-time trade-off control across multiple preference objectives.

• Test-time scalarization: By conditioning on arbitrary $y$3 vectors, PARM allows smooth (Pareto) trade-offs among objectives (e.g., helpfulness, harmlessness, humor) using a single model (Lin et al., 6 May 2025).
• Weak-to-strong guidance: A small PARM (e.g., 7B or smaller) can robustly guide the output of much larger frozen LLMs (65B+), vastly reducing compute and memory costs for practical deployment (Lin et al., 6 May 2025, Xu et al., 2024).
• Efficiency: Using a unified parameter space and single forward pass, PARM reduces inference resource requirements by a factor of $y$4 (number of objectives), while improving Pareto front coverage and alignment metrics.

Empirical results show significant improvements in hypervolume (HV) and mean inner product (MIP)—the latter measuring agreement between requested and achieved objectives—over both independent ARM ensembles and scalarized DPO/policy-based approaches (Lin et al., 6 May 2025).

5. Applications and Empirical Impact

PARM has been applied in multiple generative regimes:

• Text Generation: In multi-objective LLM alignment, PARM achieves higher HV/MIP on safety (helpfulness, harmlessness) and general assistant (helpful, harmless, humor) benchmarks compared to ensembles of ARMs and reward-interpolating soups. In weak-to-strong setups, a 7B PARM has been shown to steer a 65B LLM as effectively as full fine-tuned models (Lin et al., 6 May 2025).
• Image Generation: The Potential Assessment Reward Model for image generation combines stepwise gating, potential evaluation, and global selection, yielding +24% GenEval accuracy over baseline and outperforming both ORM and PRM. The PARM++ extension's reflection-based self-correction loop enables qualitative error localization and further performance gains (Guo et al., 23 Jan 2025).

Table: Comparative Results for Multi-Objective Test-Time Alignment

Model	HV (Safety)	MIP (Safety)	Weak-to-Strong HV	Weak-to-Strong MIP	Inference Cost
GenARM	99.3	0.80	114.8	1.81	$y$5
MOD	90.0	2.15	96.6	2.94	$y$6
PARM	113.4	2.59	121.7	3.46	1$y$7
PBLoRA-Abalation	101.8–113.4	1.62–2.59	—	—	1$y$8

Higher is better for HV/MIP. Values reported for Alpaca-7B/65B and PKU-SafeRLHF-10K (Lin et al., 6 May 2025).

In image generation, PARM-based selection and self-correction yield 77% overall accuracy on GenEval, compared to 53% for the baseline and 62% for Stable Diffusion 3. Ablation confirms each step (clarity, potential, global selection, reflection) provides additive gains (Guo et al., 23 Jan 2025).

6. Extensions, Limitations, and Related Paradigms

Recent research extends PARM-type approaches with:

• Affine and modular preference adaptation (e.g. MoSLoRA in UniARM), which further disentangle shared and preference-specific representations, achieving finer and more robust Pareto coverage (Xie et al., 10 Feb 2026).
• PaTaRM: An orthogonal approach, focusing on dynamic rubric generation and the translation of pairwise into robust pointwise supervision, can be integrated with PARM for higher interpretability or to support RLHF for instruction tuning (Jian et al., 28 Oct 2025).

Key limitations include the reliance on static or heuristic gating thresholds (in image generation), possible over-parameterization or feature entanglement in naive multi-objective architectures, and the necessity of curated, high-quality preference data for robust alignment. A plausible implication is that the field is converging toward joint preference-aware and rubric-driven architectures to address interpretability, sample efficiency, and deployment cost. Extending PARM to continuous diffusion models and temporally-structured tasks (e.g., video generation) remains an open direction (Guo et al., 23 Jan 2025).

7. Comparative Outlook and Future Directions

PARM represents a provably expressive, efficient, and flexible framework for preference-guided test-time alignment in generative models. Relative to independent ARM ensembles (e.g., GenARM (Xu et al., 2024)), it yields significantly improved preference alignment and inference efficiency by leveraging unified conditioning and low-rank adaptation. UniARM and related advances indicate ongoing transition toward integrating shared and task-adaptive modules for further Pareto optimality and model robustness (Xie et al., 10 Feb 2026).

Ongoing and future work includes more flexible threshold learning, scalable rubric adaptation, cross-modal deployment, and minimizing inference latency through model distillation or judicious parameter sharing. The underlying paradigm of explicit, preference-aware autoregressive reward modeling is increasingly central to robust, efficient, and controllable AI generation across modalities.