Extracting Latent Steering Vectors from Pretrained Language Models (2205.05124v1)

Abstract: Prior work on controllable text generation has focused on learning how to control LLMs through trainable decoding, smart-prompt design, or fine-tuning based on a desired objective. We hypothesize that the information needed to steer the model to generate a target sentence is already encoded within the model. Accordingly, we explore a different approach altogether: extracting latent vectors directly from pretrained LLM decoders without fine-tuning. Experiments show that there exist steering vectors, which, when added to the hidden states of the LLM, generate a target sentence nearly perfectly (> 99 BLEU) for English sentences from a variety of domains. We show that vector arithmetic can be used for unsupervised sentiment transfer on the Yelp sentiment benchmark, with performance comparable to models tailored to this task. We find that distances between steering vectors reflect sentence similarity when evaluated on a textual similarity benchmark (STS-B), outperforming pooled hidden states of models. Finally, we present an analysis of the intrinsic properties of the steering vectors. Taken together, our results suggest that frozen LMs can be effectively controlled through their latent steering space.

• The paper introduces a novel method for extracting latent steering vectors from pretrained language models to reproduce target sentences with high accuracy (BLEU >99).
• It demonstrates effective injection of steering vectors in intermediary transformer layers to achieve nearly perfect sequence recovery and robust unsupervised sentiment transfer.
• The study reveals that steering vectors encode rich semantic information, enabling nuanced control over text generation and deeper insights into model internals.

Extracting Latent Steering Vectors from Pretrained LLMs

The paper "Extracting Latent Steering Vectors from Pretrained LLMs" investigates a novel approach for controllable text generation by extracting latent steering vectors directly from pretrained LLMs such as GPT-2, without any fine-tuning. This approach hypothesizes that pretrained models inherently encode the necessary information to steer model outputs towards desired target sentences.

Methodology

The investigation centers on the extraction of steering vectors that, when added to the hidden states of a LLM during inference, enable the model to reproduce specific target sentences with high accuracy. The research leverages the underlying pretrained LLMs and applies gradient descent methods to identify these vectors, termed as "steering vectors," which maximize the likelihood of generating a desired sequence.

Various injection points within the model's architecture are tested, specifically within the transformer's middle layers, where these vectors can be injected into the model's hidden states. Experiments indicate that injecting the vectors within the transformer's intermediate layers maximizes the model's ability to replicate target sentences effectively.

Results

The experiments demonstrate that steering vectors can achieve nearly perfect sequence recovery with BLEU scores exceeding 99 when applied to English sentences across diverse domains. These findings suggest the optimal use of the middle layers of a transformer network over the first/last layers for effective steering, exploiting the richly encoded feature space within these intermediate layers.

Additionally, steering vectors show promising results for unsupervised sentiment transfer using vector arithmetic. By employing offset vectors computed through a small number of labeled examples, the model achieves effective sentiment alteration comparable to tailored, supervised methods focused on sentiment modification tasks. These results are evidenced on the Yelp sentiment benchmark and further extended to 19 paired tasks from the StylePTB benchmark, showcasing robustness across multiple style transfer challenges.

Beyond sentiment transfer, steering vectors reveal semantic properties, as evidenced by their performance on the STS-B textual similarity benchmark. When evaluated using cosine similarity, steering vectors outperform simple averaging of LLM hidden states, suggesting that these vectors intrinsically encode semantic information in a structured manner.

Analysis

Steering vectors not only demonstrate utility in various applications but also provide insights into the latent capabilities of LLMs. They introduce a new dimension for analyzing the intrinsic properties of neural network architectures. Properties such as robustness to initialization and interpolative capacity between vectors underscore the potential for practical applications in areas like unsupervised style transfer and real-time model steering.

The paper also hints at directions for further exploration, such as whether steering vectors can consistently encode stylistic attributes or whether sampling from this latent space can produce fluent, novel outputs. Being inherently tied to pretrained weights, there are implications for steering model outputs without retraining, thus conserving computational resources and enabling dynamic adjustments to model outputs post-training.

Implications and Future Directions

The implications of this research extend to practical applications requiring controllable and adjustable text generation without needing exhaustive finetuning on large datasets. As pretrained LLMs become more prevalent, the insights from this paper offer pathways to efficiently leverage their latent capacities for task-specific generation.

Future research could explore the theoretical underpinnings of steering spaces, potential applications in other stylistic or domain-focused control tasks, and improved methods for unsupervised representation manipulation in LLMs. This approach provides a foundation for potentially new paradigms in text generation with utility in adaptive and responsive AI systems, potentially influencing how future systems might achieve more nuanced and contextually appropriate responses autonomously.