Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation

Abstract: LLMs have become an effective tool for synthetic data generation, including for low-resource languages, where generated data can improve downstream task performance. Current best-performing approaches typically rely on few-shot prompting with target-language examples, which increases inference costs and may reduce diversity through lexical anchoring. In this work, we investigate activation steering as an alternative for low-resource synthetic data generation. We study two steering strategies: Language Steering, which targets the linguistic identity of a language, and Quality Steering, which captures well-formedness by contrasting human-written and backtranslated text representations. We evaluate these methods across four open-source LLMs, multiple layers, and 11 typologically diverse languages by generating sentiment and topic classification data and finetuning smaller classifiers. Steering is applied in both zero-shot and few-shot prompting settings and compared against non-steered counterparts. Our results show that steering on early layers consistently improves the diversity of generated data while often yielding stronger downstream model performance, particularly for low-resource languages.

• The paper introduces activation steering to improve synthetic data quality by manipulating LLM latent spaces and reducing reliance on few-shot prompts.
• It compares Quality and Language steering, demonstrating that early-layer Quality steering consistently enhances downstream F1 scores and data diversity.
• Results emphasize that precise layer selection and parameter tuning are critical, as inappropriate steering can severely degrade performance in larger models.

Activation Steering for Low-Resource Synthetic Data Generation

Introduction and Motivation

This work introduces a model-agnostic, inference-time intervention—activation steering—for synthetic data generation in the low-resource multilingual context (2606.18389). The motivation is twofold: standard few-shot prompting for low-resource targets incurs increased inference costs and often enforces undesirable lexical anchoring, reducing sample diversity. Activation steering, by direct manipulation in LLM latent spaces, efficiently shapes properties of generated data while decoupling guidance from explicit demonstration-based prompts.

Activation Steering Strategies

Two steering paradigms are contrasted. Language Steering targets a language’s structural and lexical identity by constructing one-vs-rest difference vectors from activations representing the target language versus others. Quality Steering is introduced as a novel mechanism, constructed using high-fidelity contrastive pairs: human-authored sequences and degraded backtranslated variants. Linear probing confirms strong linearly separable “quality” signals in early-to-middle transformer layers Figure 1.

Figure 1: Example of deriving a Quality steering vector from contrastive activations between human-authored and backtranslated German, subsequently used for LLM intervention.

Both steering vectors are extracted at multiple residual stream depths (approximate percentages of model depth: 21%, 48%, 74%) to probe the significance of layer selection.

Experimental Protocol

Synthetic data is generated for 11 typologically diverse languages spanning Indo-European, Afro-Asiatic (Semitic), and Austronesian families. The generation pipeline comprises four open-weight instruction-finetuned LLMs (Gemma-2-9B, Gemma-2-27B, Llama-3.1-8B, Llama-3.1-70B). Both steering methods are applied at the selected layers, under both zero-shot and 5-shot prompting regimes (the latter using target-language demonstrations).

Generated data targets two tasks—sentiment (binary) and topic classification (multi-class)—followed by downstream finetuning of task-agnostic, multilingual encoders (XLM-R).

Main Results: Impact on Downstream Task Performance

Quality Steering yields consistent and substantial improvements in downstream F1, especially when applied at early transformer layers, with the following bold observations:

• In the zero-shot generation setting, Quality steering outperformed Language steering, with statistically significant F1 improvements in 44.3% of early-layer interventions. Positive shifts were observed in nearly 80% of early-layer experiments across all tasks and languages.
• Language steering, though beneficial, exhibited model- and language-dependent volatility. In several configurations, its effect was antagonistic (i.e., strongly negative F1 shifts), especially when the geometric alignment with Quality vectors was adverse.
• Both LLM size and family matter: Gemma-2-9B and Llama-3.1-8B showed robust, uniform positive shifts with early steering; Llama-3.1-70B displayed brittle behavior—high-magnitude positives and negatives—underscoring the need for precise layer/strength selection. Gemma-2-27B required higher $\alpha$ to manifest effects, attributable to architectural differences (logit soft-capping, QK-normalization).
• Few-shot prompting exhibited a saturation effect: steering gains were attenuated, particularly for larger models, yet early-layer Quality steering remained the most consistent strategy.

This is visualized in the performance differentials for Gemma-2 models:

Figure 2: Downstream F1 results (aggregated over languages and tasks) showing consistent improvement with early-layer activation steering on the Gemma 2 series.

Detailed analysis for Gemma-2-9B demonstrates the dependence of effect on both steering type and intervention layer:

Figure 3: Effect of steering type and steering layer on Gemma-2-9B: early-layer Quality steering delivers robust, positive, and statistically significant performance shifts in most cases.

Diversity Analysis

Diversity metrics (lexical, embedding, isocontour radius, homogeneity) indicate that activation steering, especially Quality steering on early layers, consistently increases surface-form and semantic spread of generations without devolving into noisy or off-manifold text. This effect is especially pronounced for smaller models in zero-shot context. Increasing diversity is tightly coupled to the observed task gains, although larger models like Gemma-2-27B maintain lower increases, requiring more targeted parameterization.

Geometric Analysis: Relationship Between Language and Quality Vectors

Cosine similarity analysis reveals strong, model- and language-dependent polarization between Language and Quality steering directions:

• In Llama-3.1, for most languages, Quality and Language vectors are nearly perfect inverses in early layers; for Semitic languages (e.g., Amharic, Hebrew, Maltese), the vectors align, explaining why gains with both strategies are observed there.
• This divergence suggests the Quality direction is more language-agnostic—it isolates human-likeness independently from linguistic encoding, in contrast to the often entangled nature of learned language-specific representations.

Theoretical and Practical Implications

This work supports a latent-space geometric view of generation control: key abstract properties (language, well-formedness) are disentangled directions in model activation manifolds, with their mutual orientation strongly layer- and model-dependent. For practical multilingual NLP, activation steering offers efficiency gains over demonstration-based prompting as it is inference-only, portable, and not bound to explicit labels or examples. For new languages with limited data, deriving a Quality steering vector from small, curated parallel sets is viable.

Methodologically, the results indicate:

• Early transformer layers are most sensitive and reliable for concept steering that generalizes to improved synthetic data quality and diversity.
• Unprincipled steering (e.g., high-magnitude language vector on misaligned layers) can catastrophically degrade performance, especially in large LLMs.

Future Directions

Advancing activation steering will require:

• Unifying frameworks for automatic selection of optimal $\alpha$ and intervention layer, perhaps by diagnostic linear probes or geometric alignment heuristics.
• Extension beyond text classification tasks to include text-to-text generation (e.g., translation, reasoning), and demonstration on unobserved, long-tailed languages.
• Investigations into more fine-grained quality axes (factuality, politeness, bias), composition of multiple orthogonal steering directions, and their effect on latent representation spread and downstream generalization.

Conclusion

This study establishes activation steering as a practical and robust alternative or complement to few-shot prompting for synthetic data generation in low-resource, multilingual contexts. Quality-based steering, in particular, consistently yields higher and more stable downstream improvements and diversity. These findings position activation steering as a preferred mechanism for controllable and efficient data synthesis, with direct implications for the construction of high-quality, minimally-biased multilingual benchmarks as well as rapid adaptation to novel linguistic settings.