Test-Time User-Preference Alignment

• Test-Time User-Preference Alignment is a framework that adapts model outputs using inference-only mechanisms, such as reward-guided decoding and prompt engineering, without changing the main model.
• These approaches enable rapid personalization and efficient multi-objective control across various domains like language modeling, recommendation, and medical imaging.
• Key challenges include dependence on high-quality reward signals, computational overhead, and ensuring safety and robustness in dynamic user environments.

Test-time user-preference alignment refers to a class of algorithms and frameworks that adapt the behavior of a generative or decision-making model to satisfy user-specific preferences during inference, with no retraining or parameter updates of the main model. Instead, alignment is achieved via lightweight, inference-only mechanisms such as input augmentation, reward model guidance, preference-conditioned decoding, hypothesis reweighting, user feedback incorporation, or dynamic ensemble weighting. This approach is motivated by the need for rapid personalization, fine-grained control over multi-objective trade-offs, and efficient adaptation to novel or evolving user requirements, while maintaining computational efficiency and avoiding the prohibitive costs of model retraining.

1. Motivations and Fundamental Principles

Test-time preference alignment addresses scenarios where the core model is frozen and cannot be retrained for each changing preference, user, or task condition. This is critical in settings such as:

• Personalized LLM assistants, where users have unique, evolving tastes or requirements (Xie et al., 9 Apr 2025).
• Safety assessment and adversarial scenario generation for autonomous systems, demanding real-time steering across multiple objectives (e.g. adversariality, realism) (Nie et al., 24 Sep 2025).
• Crowdsourced or open-ended applications where system operators cannot pre-specify all possible preference dimensions (e.g., interface design (Liu et al., 24 Jan 2026), recommendation (Zhang et al., 2 Apr 2025), medical annotation (Zhu et al., 2024)).
• Generalization to cold-start or previously unseen users, especially under privacy or latency constraints (Qu et al., 29 Sep 2025, Zhang et al., 12 Feb 2026).

Key principles in this domain include:

• No model retraining at deployment: All adaptation is performed via auxiliary preference models, prompt engineering, or controlled post-processing.
• User- or application-controllable steering: End-users or operators can specify, interactively or through configuration, the desired trade-offs or objectives at inference.
• Sample-efficient learning from limited feedback: Alignment mechanisms must extract maximal value from minimal or noisy user-provided signals.
• Support for multi-objective and Pareto-optimal trade-offs: Mechanisms for balancing, interpolating, or optimizing over vectors of objectives or preference dimensions.

2. Taxonomy of Test-Time Preference Alignment Methods

A range of algorithmic strategies has been proposed and instantiated across domains:

Prompting and In-Context Methods

• Explicit prompt injection: Embedding user style, persona, or explicit instructions as context tokens; the model is not updated but steered via its input (Xie et al., 9 Apr 2025).
• Chain-of-thought/rubric-augmented generation: Generating and applying structured evaluation chains and scoring rubrics as auxiliary inputs for reward assignment (e.g., P-GenRM) (Zhang et al., 12 Feb 2026).

Reward and Value-Guided Decoding

• Autoregressive reward model guidance: A small "reward model" or transformer evaluates or scores each token or candidate based on the current context and user preference, and its outputs are used to reweight the base model logits during decoding (GenARM, LLMdoctor, PARM, UniARM) (Xu et al., 2024, Shen et al., 15 Jan 2026, Lin et al., 6 May 2025, Xie et al., 10 Feb 2026).
• Classifier-free or contrastive guidance signals: For diffusion models and image generation, lightweight preference modules trained on positive/negative data inject signal at each sampling step (PGD/cPGD) (Jiang et al., 21 Feb 2026).
• Preference-vector task arithmetic: Model-parameter edits, extracted from difference pairs of preference-targeted model weights, are mixed or scaled at inference (Liang et al., 27 Apr 2025).

Ensemble and Bandit Approaches

• Hypothesis reweighting: A single model backbone with multiple prediction heads (each representing a different plausible behavior) is dynamically reweighted at inference using a small labeled adaptation set (HyRe) (Lee et al., 2024).
• Dueling bandits with online reward learning: Small auxiliary networks learn a user's reward function online, which is then used to steer decoding (T-POP, UserAlign) (Qu et al., 29 Sep 2025, Pădurean et al., 4 Nov 2025).
• Version-space elimination and best-arm identification: Sequential pairwise comparison of candidate outputs and adaptive elimination of unpromising arms, assuming consistent user feedback (Pădurean et al., 4 Nov 2025).

Differential Adapter and Low-rank Parametric Approaches

• Single unified or modular reward heads/adapters: Efficient adapter layers or low-rank adaptation techniques (MoSLoRA, PBLoRA) allow a single small reward model to jointly cover multi-objective preferences, supporting on-demand test-time conditionality (Lin et al., 6 May 2025, Xie et al., 10 Feb 2026).

3. Algorithmic Mechanisms, Inference Procedures, and Theoretical Guarantees

Test-time alignment frameworks typically combine the following building blocks:

• Preference representation: Preferences appear as explicit vectors (e.g., α ∈ Δ^{k-1} on a simplex), text prompts, routing exemplars, or prototype clusters. For multi-attribute alignment (helpfulness, harmlessness, humor, etc.), the user usually selects a trade-off vector; this vector is then either input to the ARM or conditions the decoding policy (Lin et al., 6 May 2025, Xie et al., 10 Feb 2026, Banerjee et al., 6 Dec 2025, Zhang et al., 12 Feb 2026).
• Reward modeling and adaptation: ARM-style models, contrastive preference heads, or in-context Bayesian updaters (ICRM) provide test-time preference adaptation across both scalar and multi-dimensional settings (Xu et al., 2024, Hong et al., 9 Feb 2026, Xie et al., 10 Feb 2026).
• Policy or decoding control: The base policy's outputs are reweighted at each step. For ARMs, the log-probabilities of the frozen base and ARM are added with a scaling factor, optionally modulated by the user's trade-off vector; in preference-vector or weight interpolation methods, the parameter weights themselves are blended linearly as a function of the user preferences without further fine-tuning (Nie et al., 24 Sep 2025, Liang et al., 27 Apr 2025).
• Sample efficiency and label complexity: Bandit and ensemble-weighting approaches provide distribution-independent sample complexity bounds and adapt quickly with a handful of feedback pairs or labeled examples (Qu et al., 29 Sep 2025, Pădurean et al., 4 Nov 2025, Lee et al., 2024).
• Pareto-optimality and theoretical guarantees: Several works establish, under smoothness and concavity assumptions, that linear blending or test-time conditional decoders trace out Pareto-optimal trade-off frontiers, with bounded suboptimality (linear mode connectivity and LMC) (Nie et al., 24 Sep 2025, Xie et al., 10 Feb 2026, Lin et al., 6 May 2025).

4. Multi-Objective, Personalized, and Bayesian Test-Time Steering

Recent advances extend test-time alignment beyond single-objective or static settings to:

• Test-time multi-objective control: Single reward models or adapters modulated by preference vectors (or Dirichlet samples) can span entire multidimensional trade-off surfaces at inference (UniARM, PARM, ProSocialAlign) (Xie et al., 10 Feb 2026, Lin et al., 6 May 2025, Banerjee et al., 6 Dec 2025).
• Personalized or user-dependent test-time policies: By conditioning on real-time, in-context feedback, user queries, or explicit preference demonstrations, systems can fit reward heads or hypothesis weights online, supporting highly personalized (even cold-start) adaptation (Qu et al., 29 Sep 2025, Zhang et al., 12 Feb 2026).
• Probabilistic and Bayesian approaches: ICRM leverages variational inference over latent user preferences with conjugate Beta priors and dynamically updates reward calibration in response to in-context test examples, with theoretical guarantees for global interior optimum and control over reward over-optimization (Hong et al., 9 Feb 2026).

5. Empirical Evaluations, Scalability, and Application Domains

Test-time alignment methods consistently demonstrate strong performance across diverse domains:

• Language modeling and instruction following: Methods such as GenARM, PARM, UniARM, LLMdoctor, ProSocialAlign, Preference Vector, and Amulet outperform base models and training-time alignment approaches on GPT-4 or reward-model-based win rates, alignment accuracy, and hypervolume/MIP metrics for multi-objective alignment (Xu et al., 2024, Lin et al., 6 May 2025, Xie et al., 10 Feb 2026, Shen et al., 15 Jan 2026, Banerjee et al., 6 Dec 2025, Liang et al., 27 Apr 2025, Zhang et al., 26 Feb 2025). Many achieve comparable or superior performance to full fine-tuning approaches while enabling on-the-fly adaptability.
• Personalized reward modeling and recommendation: UserAlign, HyRe, P-GenRM, and T²ARec provide rapid, computation-efficient personalization with minimal labeled queries, effective for both text, vision, and sequential recommendation tasks with preference or interest drift (Pădurean et al., 4 Nov 2025, Lee et al., 2024, Zhang et al., 12 Feb 2026, Zhang et al., 2 Apr 2025).
• Adversarial scenario generation and safety-critical domains: SAGE and ProSocialAlign provide lexicographic multi-stage constrained generation and efficient test-time interpolation, supporting real-time safety evaluation with strict constraint satisfaction (Nie et al., 24 Sep 2025, Banerjee et al., 6 Dec 2025).
• Medical image segmentation: SPA applies probabilistic mixture modeling of latent preferences, efficiently aligning with clinician feedback in a handful of rounds, substantially reducing user effort (Zhu et al., 2024).
• User-interface design: AlignUI demonstrates chain-of-thought preference lookup and code generation, leading to UIs closely aligned with user-valued aspects on multiple axes (Liu et al., 24 Jan 2026).

Comprehensive studies show that such algorithms can (i) close or exceed the gap to training-time alignment (e.g., DPO, RLHF), (ii) support scalable weak-to-strong guidance (small ARM guiding massive LLM), (iii) directly optimize Pareto trade-offs with user control, and (iv) scale to real-world applications with full human-in-the-loop workflow.

6. Limitations, Practical Considerations, and Future Directions

Despite their strengths, test-time preference alignment strategies face several challenges:

• Reward-model and feedback dependence: Effectiveness relies heavily on the quality, granularity, and calibration of the underlying reward models or preference datasets. Misaligned or biased reward models can fail to align with true user values (Xie et al., 9 Apr 2025, Hong et al., 9 Feb 2026).
• Computational overhead: Some approaches (beam search, token-level ARM-guided decoding) incur 2–10× higher latency than vanilla greedy decoding, albeit much lower cost than full retraining (Xu et al., 2024, Zhang et al., 26 Feb 2025).
• Adaptation bottlenecks: Prompt-based and reward-guided methods require the base model to be instruction-following and responsive to user prompts. Bandit and ensemble-weighting methods assume the correct hypothesis is among those covered by the ensemble (Lee et al., 2024, Qu et al., 29 Sep 2025).
• Personalization limits: Current bandit/pairwise protocols focus on pairwise or binary feedback; richer feedback such as rankings, scalar gradients, or natural language critique may improve sample efficiency but are underexplored (Pădurean et al., 4 Nov 2025).
• Robustness and safety: Guarding against adversarial or malicious user preferences remains critical; some frameworks advocate hard safety filters or lexicographic constraint enforcement (Banerjee et al., 6 Dec 2025).
• Generalization: Efficacy on out-of-distribution tasks, rare preference combinations, or cross-modal transfer remains a challenge in many systems (Zhang et al., 12 Feb 2026, Xie et al., 10 Feb 2026).

Active areas of research include: adaptive, online and few-shot reward model preconditioning; richer, continuous user feedback modalities; scalable multi-objective and federated alignment; principled trade-offs between alignment, diversity, and utility; stability under dynamic or adversarial preference drift; and unified cross-benchmark evaluation ecosystems (Xie et al., 9 Apr 2025, Banerjee et al., 6 Dec 2025).

7. Representative Frameworks and Their Approaches

Framework	Inference-Time Alignment Mechanism	Preference Handling
GenARM	ARM-guided decoding token-by-token	Multi-objective, test-time adjustment (Xu et al., 2024)
LLMdoctor	Token-level reward, TFPO-guided auxiliary “doctor”	Multi-objective, diversity preserving (Shen et al., 15 Jan 2026)
PARM, UniARM	Unified ARM with preference-conditioned adapters/low-rank modulation	Arbitrary, test-time user vector (Lin et al., 6 May 2025, Xie et al., 10 Feb 2026)
ProSocialAlign	Lexicographic constraint, directional regulation, preference-aware ARM	Safety and prosocial axes (Banerjee et al., 6 Dec 2025)
T-POP, UserAlign	Online bandit, MLE or best-arm ID w/ pairwise feedback	Personalized, sample-efficient (Qu et al., 29 Sep 2025, Pădurean et al., 4 Nov 2025)
P-GenRM	Structured evaluation chain, prototype/user-based scaling	Personalized reward, strong OOD generalization (Zhang et al., 12 Feb 2026)
HyRe	Multi-head ensemble reweighting	Distribution shift, underspecification (Lee et al., 2024)
SPA	Probabilistic adaptation in segmentation	Few-shot human feedback (Zhu et al., 2024)

• SAGE: "Steerable Adversarial Scenario Generation through Test-Time Preference Alignment" (Nie et al., 24 Sep 2025)
• AlignUI: "A Method for Designing LLM-Generated UIs Aligned with User Preferences" (Liu et al., 24 Jan 2026)
• Preference-Guided Diffusion (PGD/cPGD): "Rethinking Preference Alignment for Diffusion Models with Classifier-Free Guidance" (Jiang et al., 21 Feb 2026)
• T-POP: "Test-Time Personalization with Online Preference Feedback" (Qu et al., 29 Sep 2025)
• GenARM: "Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment" (Xu et al., 2024)
• UniARM: "Towards a Unified Autoregressive Reward Model for Multi-Objective Test-Time Alignment" (Xie et al., 10 Feb 2026)
• Plan2Align: "Predictive Planning Based Test-Time Preference Alignment for LLMs" (Wang et al., 28 Feb 2025)
• ICRM: "Bayesian Preference Learning for Test-Time Steerable Reward Models" (Hong et al., 9 Feb 2026)
• Survey: "A Survey on Personalized and Pluralistic Preference Alignment in LLMs" (Xie et al., 9 Apr 2025)
• TPO: "Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback" (Li et al., 22 Jan 2025)
• UserAlign: "Inference-Time Personalized Alignment with a Few User Preference Queries" (Pădurean et al., 4 Nov 2025)
• LLMdoctor: "Token-Level Flow-Guided Preference Optimization for Efficient Test-Time Alignment of LLMs" (Shen et al., 15 Jan 2026)
• T²ARec: "Test-Time Alignment for Tracking User Interest Shifts in Sequential Recommendation" (Zhang et al., 2 Apr 2025)
• SPA: "Efficient User-Preference Alignment against Uncertainty in Medical Image Segmentation" (Zhu et al., 2024)
• PARM: "Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Model" (Lin et al., 6 May 2025)
• ProSocialAlign: "Preference Conditioned Test Time Alignment in LLMs" (Banerjee et al., 6 Dec 2025)
• Preference Vector: "Adaptive Helpfulness-Harmlessness Alignment with Preference Vectors" (Liang et al., 27 Apr 2025)
• Amulet: "ReAlignment During Test Time for Personalized Preference Adaptation of LLMs" (Zhang et al., 26 Feb 2025)
• HyRe: "Test-Time Alignment via Hypothesis Reweighting" (Lee et al., 2024)
• P-GenRM: "Personalized Generative Reward Model with Test-time User-based Scaling" (Zhang et al., 12 Feb 2026)