Test-Time Preference Optimization (TPO)

• TPO is a suite of inference-time methods that adapt frozen models to match evolving user, task, or environmental preferences without retraining.
• It leverages techniques like iterative textual critique, autoregressive reward conditioning, token-level optimization, and bandit learning for personalized alignment.
• TPO offers efficient, real-time adaptation across modalities while avoiding costly retraining, enabling diverse applications in language, vision, and generative tasks.

Test-Time Preference Optimization (TPO) is a suite of inference-time methodologies designed to align model outputs with specific user, task, or environmental preferences without updating the core model parameters. TPO refers to any approach that enables a frozen model—such as a LLM, vision backbone, or diffusion generative model—to adapt its outputs to new, dynamically provided preferences using reward feedback, optimization schemes, or synthesized guidance, typically on the timescale of a single inference or user session. This paradigm contrasts with training-time alignment (e.g., RLHF, DPO), which encodes preferences into model weights through costly offline fine-tuning. TPO is instantiated across modalities (language, vision, video, audio) and problem types (alignment, restoration, animation, scenario generation), and leverages diverse algorithmic substrates such as textual feedback, autoregressive reward models, bandit optimization, energy-based formulations, and preference-interpolated expert mixtures.

1. Conceptual Foundations and Motivation

Test-Time Preference Optimization addresses the inflexibility of static parameterized models, which traditionally lock in the set of encoded user preferences at training time. TPO explicitly targets several core challenges:

• Dynamically Evolving Preferences: User needs and societal norms may shift after model deployment, making fixed training-time alignment insufficient or outdated (Li et al., 22 Jan 2025).
• Personalization and Real-Time Adaptation: Scenarios such as user-specific dialogue, ad hoc safety requirements, or domain adaptation often lack sufficient pre-existing data for re-training (Zhang et al., 26 Feb 2025, Qu et al., 29 Sep 2025).
• Computational and Operational Constraints: Training-time fine-tuning for every deployment context (domain, user, task) is prohibitive both in cost and latency.

TPO’s methods are designed to operate exclusively at inference, either through external control signals or auxiliary computations, imposing no updates to the base model parameters. This makes TPO directly compatible with large-scale, frozen models and broadens the accessibility of fine-grained alignment to a wider range of computational budgets and domains.

2. Algorithmic Taxonomy, Core Mechanisms, and Mathematical Formulation

TPO encompasses a spectrum of mechanisms that share the unifying property of on-the-fly, preference-driven optimization over model outputs. Key classes include:

A. Iterative Textual Critique-and-Refine

Frameworks such as TPO (“Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedback”) and Textual Self-Attention Network (TSAN) use the LLM itself to interpret reward model signals as natural language critiques, treat these as "textual gradients," and iteratively revise outputs (Li et al., 22 Jan 2025, Mo et al., 10 Nov 2025). The typical TPO workflow is:

1. Sample initial output(s) from the frozen model.
2. Score candidates using a reward model R(x, y) (e.g., human preference proxy).
3. Prompt the model to generate a critique comparing “chosen” versus “rejected” candidates and synthesize a textual update (gradient).
4. Generate new responses guided by the critique; repeat for D steps.
5. Select the highest-reward output across all iterations.

TSAN extends this single-candidate paradigm: given k outputs, self-attention is emulated in the text domain. Queries (user prompt), keys/values (candidate outputs), and attention scores (natural language descriptions of candidate strengths and weaknesses) synergize to yield a synthesized response that aggregates the best aspects of multiple candidates through iterative text-based optimization (Mo et al., 10 Nov 2025).

B. Autoregressive Reward Model Conditioning

Multi-objective TPO—exemplified by PARM (Preference-Aware ARM)—conditions a reward model on a k-dimensional user-defined preference vector at test time. Let the base LLM be f_θ and auxiliary ARM(s) r_φ. Output sequence y is generated from a policy

$$\log π^*(y_t|x,y_{<t}) \propto \log f_θ(y_t|x,y_{<t}) + (1/β) \sum_{i=1}^k α_i \log ARM_i(y_t|x,y_{<t})$$

where α_i are user-supplied preferences (Lin et al., 6 May 2025). PARM trains a single reward model jointly over all objectives with a bilinear LoRA-based adapter, enabling weak-to-strong TPO (a small ARM steers a much larger base LLM).

C. Token-Level Flow-Guided Preference Optimization

LLMdoctor introduces token-level TPO, where a compact "doctor" model is trained with flow-consistency objectives using fine-grained, contrastive token-level reward signals, derived from the base (frozen) "patient" LLM’s behavioral variants. The key mathematical object is a flow value

$F(s_t) = Q(s_t) \cdot V_\phi(s_t)$

where Q(s_t) captures accumulative prior token-level rewards and V_\phi(s_t) is a learnable value. Subtrajectory balance equations enforce flow conservation across all prefixes and suffixes, yielding a distribution-matching property and diversity preservation (Shen et al., 15 Jan 2026).

D. Online Bandit Preference Learning

T-POP implements live TPO via a dueling bandit framework: a small neural reward function is updated online using pairwise user preferences between two generated completions. The reward model is then used to bias next-token selection in decoding. Upper-confidence-bound exploration in parameter space accelerates data-efficient personalization (Qu et al., 29 Sep 2025).

E. Energy-based and Zero-Shot TPO

In the vision domain (e.g., EpoTTA), TPO is interpreted as a sampling-free energy-based preference optimization, recasting the adaptation objective as a Bradley–Terry preference loss on pairs of target/source samples. This is mathematically equivalent to DPO but operates on marginal distributions without requiring MCMC (Han et al., 26 May 2025). In image restoration, the TTPO paradigm generates candidate samples via diffusion, selects preferred/dispreferred outputs using perceptual metrics or human feedback, and performs reward-guided denoising to optimize the final output in latent space, entirely at test time (Li et al., 24 Nov 2025).

F. Timestep-Segment and Segment-Specific Adaptation

For generative diffusion models in video or audio-driven tasks, TPO is instantiated by dividing the denoising schedule into intervals controlling distinct output modalities (e.g., early steps: motion; late steps: fidelity) and activating preference-optimized adapters (e.g., LoRAs) in their respective intervals (Liang et al., 11 Jun 2025).

3. Comparative Analysis and Distinctions from Related Paradigms

The following table summarizes key distinctions between TPO and associated classes of alignment:

Alignment Paradigm	Parameter Updates	Adaptation Time	Preference Scope	Example Methods
Training-time (RLHF, DPO)	Yes (θ updated)	Offline, batch	Global / Predefined	RLHF, DPO, CAI
Single-candidate Test-Time	No	Query-time	Local / On-the-fly	TPO, critique-and-revise
Multi-candidate Test-Time	No	Query-time	Local / Aggregated	TSAN
Reward-guided Decoding	No	Query-time	Scalar, shallow	LA, AMULET
Token-level/Segmental	Optional/Local LoRA	Query-time	Token/segment	LLMdoctor, AlignHuman

TPO-based approaches generally combine lower computational cost, no dependence on offline user- or domain-specific data, high flexibility, and real-time personalized alignment. In contrast, training-time methods bear high up-front cost and inflexibility to preference drift.

4. Empirical Performance, Efficiency, and Ablation

Multiple empirical studies demonstrate the superior or at-par alignment quality of TPO compared to training-time baselines, along with practical computational efficiency:

• TSAN: Llama-3.1-SFT+TSAN surpasses SFT-TPO and closes the gap to DPO models across instruction-following, preference, safety, and reasoning tasks, with real-time cost ≈11.78 PFLOPs/query (~0.016% of Llama-3.1-70B-DPO training cost) (Mo et al., 10 Nov 2025).
• TPO (iterative textual): Two-step iterations on unaligned models elevate performance to exceed fully aligned counterparts (e.g., Llama-3.1-70B-SFT+TPO outperforms Llama-3.1-70B-Instruct and DPO on various benchmarks) at <0.01% of fine-tuning compute (Li et al., 22 Jan 2025).
• PARM: Delivers higher Pareto hypervolume and MIP in multi-objective safety/helpfulness tasks than GenARM/MOD, with sublinear inference costs, and supports weak-to-strong TPO.
• LLMdoctor: Outperforms prior test-time alignment methods and full fine-tuning (DPO) in both win rate and diversity on HH-RLHF and UltraFeedback, with token-level overhead <30% single-model pass (Shen et al., 15 Jan 2026).
• Amulet: Achieves top reward-model scores in 75% of configurations across 64 model/dataset/preference settings, with per-token latency ≈100 ms (Zhang et al., 26 Feb 2025).
• Image TTPO: Consistently improves no-reference perceptual metrics across six image restoration tasks and maintains preference superiority in >70% human studies (Li et al., 24 Nov 2025).
• Ablations: Performance typically increases with candidate pool size, number of attention heads (TSAN), or TPO iterations; segmental/adapter-specific ablations confirm the necessity of targeted preference modules (Mo et al., 10 Nov 2025, Liang et al., 11 Jun 2025).

5. Interpretability, Diversity, and Practical Considerations

A central feature of test-time preference optimization, especially systems leveraging textual gradients or multi-candidate aggregation, is high interpretability: all attention, critique, and update steps are presented as natural language spans, allowing users to inspect not just the outputs but the rationale for candidate weighting, selection, and revision (Mo et al., 10 Nov 2025, Li et al., 22 Jan 2025). Token-level TPO (LLMdoctor) provides distribution-matching guarantees and entropy lower bounds, ensuring diversity of outputs and avoiding mode collapse (Shen et al., 15 Jan 2026). In image/video domains, staged denoising, frequency-decomposed rewards, and structure-perceptual balancing similarly preserve quality and detail (Li et al., 24 Nov 2025, Liang et al., 11 Jun 2025).

Efficiency is sustained across settings by closed-form updates (Amulet), small auxiliary models or adapters (Doctor, LoRA), or parameter-free guidance (TSAN). However, extra LLM invocations or denoising steps may be non-negligible in latency-critical applications.

Limitations include dependence on the base model’s prompt-sensitivity or instruction-following ability, reliability of external reward models, prompt quality, and the challenge of synthesizing preference information in fundamentally ill-posed tasks.

6. Extensions, Limitations, and Future Research Directions

TPO is actively expanding into new axes:

• Meta-preference learning: Automated meta-learning of prompt templates and textual guidance, reducing manual engineering overhead (Mo et al., 10 Nov 2025).
• Multimodality: Extending TPO to vision, audio, code, and structured data, e.g., image restoration (Li et al., 24 Nov 2025), motion/fidelity control in diffusion video synthesis (Liang et al., 11 Jun 2025), and adversarial scenario generation (Nie et al., 24 Sep 2025).
• Reward model adaptation: Bayesian in-context preference learning for steerable reward models (ICRM) supports real-time test-time reward adaptation in RLHF and multi-objective settings (Hong et al., 9 Feb 2026).
• Personalization: Online dueling bandit frameworks and bandit reward learning support cold-start and data-efficient personalization (Qu et al., 29 Sep 2025).
• Theoretical understanding: Mode connectivity, optimality bounds for weight-space interpolation, and variational guarantees furnish mathematical underpinnings for TPO performance (Nie et al., 24 Sep 2025, Hong et al., 9 Feb 2026).
• Practical uptake: Fully training-free, plug-in modules for diffusion generative models, compatibility with generic backbones, and test-time only computational footprints are lowering barriers to adoption (Li et al., 24 Nov 2025).

Open issues remain in automated and human-in-the-loop reward tuning, robust integration of TPO with online feedback, unsupervised/online TPO protocols, multimodal fusion, and full scalability to zero-resource preference adaptation. Failure modes such as preference mis-specification, overfitting to poor reward proxies, and inability to synthesize when all candidates are fundamentally flawed are key areas for continued investigation.

7. Domain-Specific Implementations

TPO underpins a range of instantiated systems, including TSAN for multi-candidate LLM response aggregation (Mo et al., 10 Nov 2025), PARM and LLMdoctor for preference-controlled autoregressive language modeling (Lin et al., 6 May 2025, Shen et al., 15 Jan 2026), Amulet for explicit per-token reward-guided LLM decoding (Zhang et al., 26 Feb 2025), EpoTTA for sampling-free energy-based adaptation (Han et al., 26 May 2025), TTPO for human-aligned image restoration (Li et al., 24 Nov 2025), AlignHuman’s timestep-segmented animation control (Liang et al., 11 Jun 2025), and SAGE for inference-time steerable scenario generation by expert mixture (Nie et al., 24 Sep 2025).

Each approach is unified under the TPO principle: preference-driven, parameter-free, on-the-fly alignment of powerful frozen generative models to rich, dynamic, user- or application-specified desiderata.