Extending Activation Steering to Broad Skills and Multiple Behaviours (2403.05767v1)
Abstract: Current LLMs have dangerous capabilities, which are likely to become more problematic in the future. Activation steering techniques can be used to reduce risks from these capabilities. In this paper, we investigate the efficacy of activation steering for broad skills and multiple behaviours. First, by comparing the effects of reducing performance on general coding ability and Python-specific ability, we find that steering broader skills is competitive to steering narrower skills. Second, we steer models to become more or less myopic and wealth-seeking, among other behaviours. In our experiments, combining steering vectors for multiple different behaviours into one steering vector is largely unsuccessful. On the other hand, injecting individual steering vectors at different places in a model simultaneously is promising.
- Open problems and fundamental limitations of reinforcement learning from human feedback. arXiv preprint arXiv:2307.15217.
- Deep reinforcement learning from human preferences. Advances in neural information processing systems, 30.
- Fast machine unlearning without retraining through selective synaptic dampening. arXiv preprint arXiv:2308.07707.
- The pile: An 800gb dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027.
- Improving activation steering in language models with mean-centring. arXiv preprint arXiv:2312.03813.
- Leike, J. (2022). Distinguishing three alignment taxes.
- Inference-time intervention: Eliciting truthful answers from a language model. arXiv preprint arXiv:2306.03341.
- Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1–35.
- Copy suppression: Comprehensively understanding an attention head. arXiv preprint arXiv:2310.04625.
- Discovering language model behaviors with model-written evaluations. arXiv preprint arXiv:2212.09251.
- Dissecting large language models. In Socially Responsible Language Modelling Research.
- Steering llama 2 via contrastive activation addition. arXiv preprint arXiv:2312.06681.
- Exploring the landscape of machine unlearning: A survey and taxonomy. arXiv preprint arXiv:2305.06360.
- Model evaluation for extreme risks.
- Language models are better than humans at next-token prediction. arXiv preprint arXiv:2212.11281.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
- Natural language processing with transformers. " O’Reilly Media, Inc.".
- Activation addition: Steering language models without optimization. arXiv preprint arXiv:2308.10248.
- Representation engineering: A top-down approach to ai transparency. arXiv preprint arXiv:2310.01405.