Activation Steering for Synthetic Data Generation: The Role of Diversity in Downstream Safety Detection

Abstract: Safety detection models require examples of HHH (Helpful, Harmless, Honest)-violating outputs for robust generalization, however such examples are scarce. Activation Steering (AS) has emerged as a data-efficient method for generating target-concept-aligned responses. We investigate whether AS can generate high-quality training datasets for downstream classifiers, a question that remains untested. We present a two-fold study with intrinsic and extrinsic evaluation across $4$ concepts $\times\,2$ models $\times\,4$ steering methods. Intrinsically, beyond the field-standard rubric of steering success (concept alignment) and coherence, we introduce sample- and set-level diversity as a quality axis previously absent from the literature, and find that increasing steering strength reduces response diversity. Extrinsically, we replace HHH-violating examples in the available training data with steered generations and fine-tune detection classifiers. AS-generated data results in a better classifier than the prompting-generated data on $3$ of $4$ concepts. However, only $41$ of $136$ AS configurations outperform prompting, indicating that downstream utility lies in a narrow regime that jointly satisfies success, coherence, and diversity. The harmonic mean of these three axes correlates with downstream AUROC more consistently across concepts than success and coherence alone, providing a practical heuristic target for practitioners tuning AS hyperparameters. Together, our results highlight the potential of AS in synthetic data generation for improving safety detection and identify diversity as a critical, previously overlooked axis for tuning AS.

• The paper shows that activation steering in latent space efficiently generates rare HHH-violating content for safety detector training.
• It reveals that optimizing a harmonic mean of steering success, coherence, and diversity is critical for balancing synthetic data quality and downstream performance.
• Experiments indicate that smaller models (OLMo-7B) outperform larger ones (OLMo-32B) in achieving a balanced trade-off in activation steering outcomes.

Activation Steering for Synthetic Data: The Critical Role of Diversity in Downstream Safety Detection

Motivation and Background

The deployment of robust safety detectors for LLMs critically depends on access to diverse and high-quality examples of HHH-violating content—outputs that are not Helpful, Harmless, or Honest. As alignment protocols such as supervised fine-tuning (SFT) and RLHF drive models toward safer behaviors, naturally occurring HHH-violations become rare, producing a data scarcity bottleneck for training the next generation of safety detectors. Traditional methods for synthesizing such rare-class data, notably adversarial prompting and red teaming, operate in the token space and fail to leverage latent internal representations. Activation Steering (AS) addresses this by introducing targeted interventions directly in model activation space, potentially enabling efficient generation of concept-aligned synthetic data using only a small set of contrastive examples (2605.28664).

Experimental Framework

This study offers a systematic audit of AS as a generator of HHH-violating content for safety detector training, structured around both intrinsic (response-level) and extrinsic (downstream classifier performance) evaluations. The investigation spans four HHH-relevant concepts—unfaithfulness, toxicity, hallucination, and sycophancy—across two OLMo model checkpoints (7B and 32B) and four steering vector extraction methods (Contrastive Activation Addition, normalized CAA, Recursive Feature Machines, and Logistic Regression). The essential steering intervention is the addition of a scaled direction vector to LLM hidden states: $$\mathbf{h} \leftarrow \mathbf{h} + \lambda \mathbf{v}$$, with $\lambda$ being the steering scale controlling intervention strength.

Unlike earlier work, this study incorporates long-form generation (256 tokens), introduces both sample- and set-level diversity as critical axes, and evaluates downstream detector AUROC following replacement of real positives with steered or adversarially-prompted generations.

Intrinsic Effects of Steering: Trade-Offs Among Success, Coherence, and Diversity

Activation steering scale ($\lambda$) exerts monotonic influence on key axes of output quality. Increasing $\lambda$ drives up concept alignment (steering success) but degrades both fluency/coherence and response diversity. The drop in diversity is observed at both the response (MTLD-MB) and corpus (inverse compression ratio) levels, with set-level diversity being especially sensitive to higher steering strengths.

Figure 1: Joint satisfaction of success, coherence, and diversity is necessary for optimal downstream classifier performance; excessive steering strength erodes diversity and coherence even as steering success improves.

Pearson correlations between $\lambda$ and quality metrics reveal a consistent negative relationship: stronger steering reliably collapses diversity, particularly at the set level, potentially introducing undesirable distributional artifacts. Length effects are inconsistent; while some configurations display length contraction, others drift in the opposite direction depending on model scale.

Figure 2: Increasing steering scale ($\lambda$) positively correlates with steering success but negatively with coherence, diversity, and, variably, with response length.

Model Scale: Smaller Models Yield Superior Steering Outcomes

Contrary to the intuition that larger models harness richer representations for control, empirical results consistently show the smaller OLMo-7B to outperform the 32B variant in achieving a balanced trade-off among steering success, coherence, and diversity. This advantage holds across steering methods and concepts, with the 7B model winning the majority of paired comparisons on the harmonic mean of these axes.

Figure 3: OLMo-7B outperforms OLMo-32B on the joint harmonic mean of success, coherence, and diversity across the sweep of steering methods and concepts.

The observed pattern provides a concrete resolution to conflicting claims in prior studies regarding the interaction of model size and steerability, emphasizing that smaller models may be more controllable or robust in AS settings when optimizing for multiple output qualities.

Downstream Utility: Synthetic Data and Safety Detector Performance

A core contribution is the extrinsic evaluation of steered data as fine-tuning material for HHH-violation detectors. Downstream classifier AUROC is augmented when steered generations replace genuine positives, but only under constrained settings. Notably, only 41 of 136 AS configurations outperform adversarial prompting, and utility peaks at intermediate $\lambda$ where success, coherence, and diversity are simultaneously maintained—beyond which either coherence or diversity collapse undermines generalization.

Figure 4: Detector AUROC as a function of steering configuration; classifiers trained on optimally steered examples can exceed prompting baselines, but only within a narrow setting of the steering scale.

Concepts differ in susceptibility: improvements over prompting are strongest for unfaithfulness, sycophancy, and hallucination, while gains in the toxicity domain are unattainable with AS, highlighting concept-specific limits to steerability and synthetic data fidelity.

Validation-Time Heuristics: Diversity as a Critical Signal

The classical AS evaluation rubric—considering only steering success and coherence—is shown to be insufficient and even anti-correlated with downstream utility. Instead, augmenting the heuristic with a diversity term (measured via MTLD-MB, inverse compression ratio, or $n$-gram diversity) produces more stable and positively correlated predictions of downstream AUROC. The harmonic mean of success, coherence, and diversity emerges as a practical and robust signal for practitioners to navigate the high-dimensional AS hyperparameter landscape and select configurations conducive to effective safety classifier training, without expensive end-to-end experiments on each configuration.

Ablation: One-Layer versus All-Layer Steering

While all-layer steering is favored for stability and ease of hyperparameter tuning, further experimentation reveals that the proposed harmonic mean heuristic generalizes to one-layer steering, achieving comparable AUROC improvements over prompting and high correlation between intrinsic validation and downstream performance.

Theoretical and Practical Implications

The findings establish diversity as a primary axis for the design and optimization of activation steering pipelines when the downstream objective is classifier generalization. The monotonic contraction of diversity with increased steering strength—and the propensity for models smaller in scale to better balance the requisite quality axes—carry immediate implications for both synthetic data curation and the future development of steerable LLMs.

Practically, the work cautions against reliance on strong steering-induced alignment alone and prescribes validation heuristics that explicitly account for diversity. Theoretically, the similarity in degradation patterns between steering-induced and alignment-induced distributional collapse invites further work to clarify the interplay between internal representations, generative diversity, and generalization boundaries in safety-critical NLP.

Conclusion

Activation steering provides a data-efficient channel for generating rare, HHH-violating examples necessary for robust safety detection in LLMs. However, without explicit diversity control, it incurs distributional costs that can limit downstream generalization. The joint optimization of concept alignment, coherence, and diversity—tracked via the harmonic mean—enables practitioners to select effective AS configurations and offers a principled framework for synthetic data generation in safety-critical applications. This study positions diversity, previously overlooked, as a central objective and evaluation component for controllable text generation via activation interventions.