Inference time LLM alignment in single and multidomain preference spectrum (2410.19206v1)

Abstract: Aligning LLMs (LLM) to address subjectivity and nuanced preference levels requires adequate flexibility and control, which can be a resource-intensive and time-consuming procedure. Existing training-time alignment methods require full re-training when a change is needed and inference-time ones typically require access to the reward model at each inference step. To address these limitations, we introduce inference-time model alignment method that learns encoded representations of preference dimensions, called \textit{Alignment Vectors} (AV). These representations are computed by subtraction of the base model from the aligned model as in model editing enabling dynamically adjusting the model behavior during inference through simple linear operations. Even though the preference dimensions can span various granularity levels, here we focus on three gradual response levels across three specialized domains: medical, legal, and financial, exemplifying its practical potential. This new alignment paradigm introduces adjustable preference knobs during inference, allowing users to tailor their LLM outputs while reducing the inference cost by half compared to the prompt engineering approach. Additionally, we find that AVs are transferable across different fine-tuning stages of the same model, demonstrating their flexibility. AVs also facilitate multidomain, diverse preference alignment, making the process 12x faster than the retraining approach.

• The paper presents a novel method using alignment vectors to adjust LLM outputs at inference, reducing retraining overhead by 12×.
• The approach enables fine-grained control over model responses in domains like medical, legal, and financial with measurable preference accuracy.
• The study outlines a multidomain alignment strategy that maintains distinct domain-specific outputs while addressing cross-domain generalization challenges.

Inference Time LLM Alignment in Single and Multidomain Preference Spectrum

The paper presents an innovative approach to aligning LLMs at inference time using Alignment Vectors (AVs), specifically targeting preference spectrum adjustments across various domains. Traditional alignment methods require significant resources and time, as they necessitate model retraining each time a preference is modified. The approach introduced in this paper, however, performs alignment at inference time, significantly reducing these burdens.

Methodological Overview

The core innovation of this work revolves around the concept of AVs, which are obtained by subtracting the base model parameters from those of an aligned model post-fine-tuning. This allows for inference-time behavior adjustments through mathematical operations on these vectors. The AVs afford fine-grained control over model outputs by acting as tunable parameters. This is particularly explored in three specialized domains: medical, legal, and financial. Users can dynamically adjust the model's output by shifting these vectors in the parameter space, demonstrating an effective way to modify model behavior without additional training.

Empirical results show that by manipulating the AVs, significant control over response specificity can be achieved according to the desired level of expertise—ranging from avoidance to expert—with the model's preference accuracy for these behaviors transitioning smoothly as expected. The research suggests that this approach is 12 times faster than traditional methods, drastically cutting the computational cost and resource usage.

Domain-Specific Results

In the detailed experiments, results are presented for single-domain preference tuning across medical, legal, and financial queries. The inference-time model editing allows for tailored response behaviors just by adjusting the proportional integration of the AVs into the base model. Quantitatively, this is evident through preference accuracy indicators that align with the desired response spectrum for each domain. The capability to switch between different expectation levels, such as from avoidance to expert insights, highlights the flexibility and potential application of this approach.

Multidomain Challenges and Strategies

Addressing the multidomain alignment challenge, the research proposes integrating multiple AVs even at the inference stage to achieve diverse domain-specific preferences without separate training per combination. This approach makes it feasible to maintain distinct preferences across multiple domains contemporaneously, although the complexity involved in tuning multiple variables concurrently is acknowledged.

Interestingly, the results underline the intrinsic generalization effect: once a model is aligned in one domain, there seems to be an extension of this alignment across others. This is perceived both as a benefit for applications requiring coherent behavior and as a cautionary note emphasizing the need for precise tuning to avoid unintended cross-domain generalizations.

Implications and Future Research

The implications of this paper are substantial in the context of customizable AI, where user-specific and context-specific adaptations can be implemented at scale, conserving resources, and expediting deployment. The innovative use of AVs not only promises a novel technical extension to current model alignment methodologies but also lays a foundation for more interactive and adaptive AI systems that can seamlessly shift according to nuanced user requirements.

This work opens avenues for further research into various methodologies for obtaining and applying alignment vectors and expanding the applicability across contrasting model architectures or domains with different inherent complexities. Additionally, future exploration could refine multidomain alignment techniques further and address the potential limitations of over-generalization.

In summary, this approach highlights a significant step forward in LLM preference alignment by effectively utilizing inference-time techniques, prompting both theoretical and computational advancements in the field of AI alignment and customization.