KV Cache Steering for Inducing Reasoning in Small Language Models (2507.08799v1)

Abstract: We propose cache steering, a lightweight method for implicit steering of LLMs via a one-shot intervention applied directly to the key-value cache. To validate its effectiveness, we apply cache steering to induce chain-of-thought reasoning in small LLMs. Our approach leverages GPT-4o-generated reasoning traces to construct steering vectors that shift model behavior toward more explicit, multi-step reasoning without fine-tuning or prompt modifications. Experimental evaluations on diverse reasoning benchmarks demonstrate that cache steering improves both the qualitative structure of model reasoning and quantitative task performance. Compared to prior activation steering techniques that require continuous interventions, our one-shot cache steering offers substantial advantages in terms of hyperparameter stability, inference-time efficiency, and ease of integration, making it a more robust and practical solution for controlled generation.

• The paper introduces cache steering, a one-shot KV cache modification that induces chain-of-thought reasoning in small language models.
• It employs contrastive prompt pairs to extract steering vectors, achieving significant accuracy gains and more structured outputs over conventional methods.
• The technique enables efficient, inference-time control of reasoning styles with robust hyperparameter stability and minimal computational overhead.

KV Cache Steering for Inducing Reasoning in Small LLMs

The paper introduces cache steering, a method for behavior control in LLMs via a one-shot modification of the key-value (KV) cache, with a focus on inducing chain-of-thought (CoT) reasoning in small LLMs (SLMs). This approach is positioned as a practical alternative to activation steering, addressing its limitations in stability, efficiency, and ease of integration.

Methodology

Cache steering operates by extracting steering vectors from the KV cache of a teacher model (e.g., GPT-4o) using contrastive prompt pairs—one with explicit reasoning traces (positive) and one with only the final answer (negative). The steering vectors are computed as the mean difference of the key and value tensors at a designated token position across these pairs. At inference, these vectors are added to the cached key and value tensors of the SLM at the corresponding position, prior to generation. This single intervention biases the model toward more explicit, multi-step reasoning without modifying model weights or prompts.

Key implementation details include:

• Contrastive Set Construction: Positive and negative prompts are constructed with identical in-context learning (ICL) examples, differing only in the presence of reasoning steps.
• Vector Extraction: Steering vectors are extracted from the final token of the prompt, typically after a neutral offset token to ensure alignment during autoregressive decoding.
• Application: The steering vectors are applied to the same logical position in the KV cache as used during extraction, with scalar coefficients controlling the strength of intervention.
• Hyperparameter Robustness: The method demonstrates stability across a range of hyperparameters, including the number of contrastive pairs, ICL examples, and steering strengths.

Experimental Results

Cache steering is evaluated on four reasoning benchmarks: GSM8K, ARC-Challenge, CommonsenseQA, and PIQA, using SLMs from Llama-3, SmoLLM2, Qwen2, and Phi-4-mini families (ranging from 360M to 8B parameters). The method is compared against standard baselines (greedy decoding, CoT prompting), activation steering (CAA), and a hybrid of CoT prompting with cache steering.

Key findings:

• Consistent Performance Gains: Cache steering improves task accuracy over baselines and activation steering in most cases. For example, on ARC-Challenge, Llama-3.2-3B-Instruct achieves 79.27% with cache steering vs. 74.32% baseline and 74.23% with activation steering.
• Induction of Structured Reasoning: Outputs are longer and more elaborate, often exceeding even CoT-prompted completions (e.g., Phi-4-mini-instruct generates 328.8 tokens on average with cache steering vs. 211.0 with CoT prompting).
• Stability and Efficiency: The method is robust to hyperparameter variation and introduces negligible inference-time overhead compared to activation steering, which requires continuous intervention and is sensitive to coefficient tuning.
• Style Transfer: Cache steering can reliably induce distinct reasoning styles (e.g., stepwise, analogical, causal chain) by extracting style-specific steering vectors, with up to 95% of outputs matching the intended structure for certain styles.

Practical Implications

Cache steering offers several advantages for real-world deployment:

• Inference-Time Control: It enables post-hoc behavioral modification without retraining or prompt engineering, making it suitable for production systems where model weights are frozen.
• Compatibility: The method integrates seamlessly with standard Transformer inference APIs and is agnostic to model architecture, provided access to the KV cache.
• Resource Efficiency: The one-shot intervention minimizes computational overhead, supporting high-throughput and low-latency applications.
• Controllable Reasoning: Fine-grained control over reasoning style and structure can enhance interpretability, user alignment, and explanation quality in downstream tasks.

Limitations and Future Directions

The current work focuses on small LLMs and reasoning tasks. Generalization to larger models, other domains (e.g., instruction following, safety alignment), and broader behavioral interventions remains to be explored. The effectiveness of cache steering depends on the quality and representativeness of the contrastive set and the alignment of extraction/application positions. Oversteering and degenerate outputs can occur if coefficients are not appropriately tuned, especially for underrepresented styles.

Potential future developments include:

• Automated Hyperparameter Selection: Developing adaptive methods for steering strength and position selection.
• Broader Behavioral Control: Extending cache steering to other behaviors (e.g., refusal, truthfulness, toxicity reduction).
• Integration with Model Editing: Combining cache steering with model editing techniques for persistent behavioral changes.
• Theoretical Analysis: Formalizing the relationship between KV cache interventions and model behavior, particularly in larger models.

Conclusion

Cache steering represents a robust, efficient, and practical approach for inducing and controlling reasoning in small LLMs. By leveraging the KV cache as a locus of intervention, it circumvents the instability and computational cost of activation steering, while enabling fine-grained, inference-time control over model outputs. This technique opens new avenues for controllable generation, reasoning style transfer, and lightweight distillation in the context of LLM deployment.