SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors

Abstract: Speech-aware LLMs often generalize poorly to out-of-domain settings. We propose SALSA (Speech-Aware LLM Adaptation via Learned Steering Activations), a lightweight adaptation method that learns layer-wise steering vectors. Unlike commonly used steering approaches that rely on contrastive activation differences, SALSA directly optimizes steering vectors using a supervised objective. Across children's speech, multilingual speech, and Mandarin-English code-switching benchmarks, SALSA substantially improves performance over zero-shot inference and speech in-context learning baselines, achieving up to 46.8% relative improvements over zero-shot. Analysis further demonstrates that steering the encoder, particularly the later layers, is more effective than steering the LLM backbone. These findings suggest that steering improves downstream ASR performance by adapting higher-level acoustic and phonetic representations to better align with the pretrained LLM representation space, rather than by modifying the decoder itself.

• The paper introduces a parameter-efficient method using learned steering vectors to align acoustic representations with LLM decoding space for enhanced ASR.
• Empirical results demonstrate significant WER reductions, achieving up to 56.3% improvement in children’s speech and robust multilingual performance.
• The method efficiently adapts models with minimal computational overhead and training data, highlighting its practical applicability in low-resource settings.

SALSA: Speech-Aware LLM Adaptation via Learned Steering Activation Vectors

Overview and Motivation

SALSA introduces a process for parameter-efficient adaptation of Speech-Aware LLMs (SALLMs). Prior SALLMs, such as Qwen2-Audio-7B-Instruct and Granite-Speech-3.3-8B, combine a pretrained speech encoder with an LLM backbone but exhibit limited generalization to out-of-domain scenarios (e.g., children's speech, low-resource languages, code-switching). Existing adaptation approaches—such as full fine-tuning, LoRA, and in-context learning (ICL)—have practical and computational limitations or struggle with the inherent variability of speech signals. Activation steering, which perturbs hidden representations at inference without modifying model weights, has previously relied on paired contrastive examples to define steering directions. In the speech domain, such pairs are scarce and unreliable due to acoustic diversity and the strict input/output alignment required by ASR.

SALSA circumvents these challenges by learning layer-wise steering vectors directly via a supervised objective, optimizing for downstream ASR performance while keeping backbone weights frozen. This method enables efficient adaptation for out-of-domain speech, particularly in low-resource and computational-constrained settings, without the need for paired contrastive signals.

Methodology

SALSA's adaptation involves the following:

• Steering Vector Optimization: For a frozen SALLM $f_\theta$ with hidden activations $\mathbf{h}_l$ at layer $l$ of the encoder, SALSA learns additive vectors $\mathbf{v}_l$ such that modified activations $\tilde{\mathbf{h}_l} = \mathbf{h}_l + \mathbf{v}_l$ improve alignment with the LLM's representation space. Norm-preserving updates (renormalization based on the original activation's magnitude) ensure stability.
• Training Objective: Steering vectors are optimized exclusively via cross-entropy loss over next-token prediction in ASR, with backbone parameters fixed. Optimization is performed with AdamW, small learning rates, batch size 1, and gradient clipping, for up to 20 epochs with early stopping.
• Application at Inference: Learned steering vectors are injected at selected encoder layers during inference. Steering can be applied to encoder, decoder, or both; empirical results show encoder-only interventions are most effective.

Empirical Results

SALSA was evaluated on several challenging speech benchmarks:

• Children's Speech (OGI, MyST, RSR): Children's speech exhibits acoustic and linguistic properties poorly represented in SALLM pretraining. SALSA dramatically reduces WER compared to zero-shot inference, achieving up to 56.3% relative improvement (OGI, Granite-Speech-3.3-8B, $n=2000$). In the case of RSR, SALSA outperforms in-context learning baselines (TICL) by over 40 absolute WER points.
• Multilingual and Code-Switching (CommonVoice-Russian/Twi, SEAME): For both seen and unseen languages, SALSA consistently improves over zero-shot prompting. In Russian (a language covered in pretraining), SALSA reduces WER from 87.5% to ~24% using only 2,000 examples. Performance gains are also seen in Twi, despite no pretraining exposure. In Mandarin-English code-switched SEAME, SALSA improves MER by ≥45% relative to zero-shot baselines for Qwen2-Audio-7B-Instruct, outperforming TICL, which suffers severe instability and degraded recognition.
• Scaling Behavior: Most datasets show diminishing returns beyond 2,000 training examples; largest performance gains occur between 32 and 500 examples, demonstrating strong data efficiency. However, datasets with large intra/inter-speaker variability (MyST) can exhibit overfitting and performance degradation as more steering data is added.
• Steering Location Analysis: Steering is most effective when applied to later encoder layers; steering lower encoder layers yields minimal benefit or degradation. Steering the LLM backbone (decoder) provides substantially smaller gains than encoder steering; joint steering does not outperform encoder-only steering.
• Model Generalization: The efficacy and scaling behavior of steering depend on the architecture and pretraining of the SALLM. Granite-Speech-3.3-8B shows greater robustness to speaker variability in children's speech than Qwen2-Audio-7B-Instruct.

Practical and Theoretical Implications

SALSA's results substantiate several theoretical and practical positions:

• Representation Alignment vs. Language Acquisition: Findings indicate that the main bottleneck for SALLMs in ASR adaptation is not simply learning new linguistic features, but aligning acoustic encoder representations with the LLM's decoding space. Particularly for scenarios where the model has already seen target languages in pretraining, steering enables rapid adaptation without modifying the decoder.
• Parameter-Efficiency and Scalability: Because steering modifies only a small set of vectors, adaptation is computationally economical, making it viable for low-resource applications where full fine-tuning or RL-based post-training is impractical.
• Applicability Across Domains: The method is robust across structurally diverse speech populations—children's speech, multilingual, code-switched utterances—and model architectures, suggesting that steering vectors offer a universal mechanism for out-of-domain adaptation.
• Limitations and Fairness Considerations: While SALSA boosts performance for underrepresented speech distributions, adaptation is substantially more effective for languages already covered in pretraining. This reinforces the risk that adaptation methods may propagate existing resource and demographic asymmetries in speech datasets.
• Future Directions: Further research could explore input-dependent steering (e.g., dynamic vectors based on utterance features) and broader application across tasks, languages, and populations. Layerwise and module-specific analyses suggest the potential for fine-grained, context-sensitive adaptation mechanisms.

Conclusion

SALSA establishes a highly effective paradigm for parameter-efficient adaptation of SALLMs using learned steering activation vectors optimized for downstream ASR tasks. Encoder steering at higher layers offers substantial improvements in recognition performance across challenging, structurally diverse speech domains, with minimal computational overhead and strong data efficiency. Theoretical analysis suggests that adaptation operates primarily through improved modality alignment rather than language acquisition. Practical adoption may accelerate robust speech recognition and multilingual generalization in LLMs, with future work needed on input-dependent steering, fairness evaluation, and broader task coverage.

---

For full methodological and empirical details see "SALSA: Speech Aware LLM Adaptation via Learned Steering Activation Vectors" (2606.00460).