A Coin Flip Per Token: Bernoulli Sparse Steering of Large Language Models

Abstract: Activation steering via sparse autoencoders (SAEs) enables behavioral control of LLMs without task-specific fine-tuning, but standard methods apply the steering signal at every generated token, incurring constant per-token perturbation that risks degrading fluency. We ask: is dense intervention necessary? We introduce Stochastic Token Steering (STS), which gates each token independently with probability $p$, and Stochastic Block Steering (SBS), which gates a leading window once per sequence; neither requires a reward model or learned gating policy. Across two model families and two behavioral tasks, steering only 50% of the tokens recovers most of the dense-steering effect while preserving fluency, and steering as few as 30% surpasses prompt-based control. The optimal steering magnitude scales inversely with the intervention ratio, revealing that SAE-mediated control is rate-limited: the behavioral outcome depends on cumulative signal dosage across a sequence.

• The paper introduces Stochastic Token Steering (STS) and Stochastic Block Steering (SBS) using Bernoulli gating to enable sparse inference-time control, recovering over 95% of behavioral shifts with reduced interventions.
• It demonstrates that as intervention sparsity increases, the optimal steering magnitude rises sub-linearly, balancing control efficacy while preserving model fluency and reducing off-manifold drift.
• Experiments across toxicity and emotion steering tasks show that token-level stochastic interventions (STS) outperform block-based methods (SBS) in achieving robust behavioral control in diverse LLM architectures.

Bernoulli Sparse Steering for Inference-Time Control of LLMs

Introduction

The paper "A Coin Flip Per Token: Bernoulli Sparse Steering of LLMs" (2607.05615) re-examines inference-time activation steering in LLMs, focusing on the necessity and efficiency of uniform, every-token intervention. The work introduces Stochastic Token Steering (STS) and Stochastic Block Steering (SBS), both of which implement sparse, position-randomized interventions using Bernoulli gating. The study addresses key limitations of conventional dense steering—specifically, the trade-off between behavioral control and text fluency—and frames SAE-mediated steering as a rate-limited process.

Methods: Sparse Steering via Stochastic Gating

The authors build on the established practice of feature steering with Sparse Autoencoder (SAE) dictionaries operating on the residual stream. Traditionally, a steering vector—identified by contrasting feature activations from positive (target behavior) and neutral samples—is injected at every token for a selected transformer layer. The paper proposes two Bernoulli-gated alternatives:

• STS (Stochastic Token Steering): For each token, with probability $p$, the steering vector $\alpha \mathbf{v}$ is injected; otherwise, the token is left unperturbed. This stochastic gating requires only a scalar hyperparameter $p$, facilitating a direct trade-off between intervention density and total signal injection.
• SBS (Stochastic Block Steering): A sequence-level Bernoulli gate is sampled; if active, a contiguous block of initial tokens (window size $W$) receives the steering vector, exploiting documented early-token overdetermination.

Fluency-preserving regularization is enforced by norm restoration, activation clamping, and repetition penalties, in line with prior best practices. Steering vectors are selected based on top-$K$ differential features; the same vectors are used across all experimental configurations for competitive comparison.

Experiments and Numerical Analysis

Tasks and Models

Experiments are conducted on two principal axes:

• Suppression: Toxicity reduction on RealToxicityPrompts.
• Elicitation: Steering toward fear and sadness emotions on GoEmotions.

Two open-weight models are targeted: LLaMA 3.1-8B and Gemma-2 2B, chosen to validate cross-architecture generality.

Behavioral effectiveness is measured by mean toxicity reduction (for suppression) and mean probability gain for target emotion (for elicitation), using classifier-based metrics. Generation quality is indexed via GPT-2 perplexity and $n$-gram repetition rates.

Sparse Steering Recovers Most Behavioral Effects

The central finding is that majority of the behavioral shift is preserved under significant sparsification: with STS, intervening on as few as 30–50% of tokens recovers most of the effect of conventional, dense (full) steering, and surpasses prompt-based instruction at matched ratios. In toxicity reduction tasks with LLaMA 3.1-8B, STS at $p=0.5$ achieves over 95% of the toxicity mitigation relative to full steering; similar results are observed for emotion steering, with over 80% effect at $p=0.7$.

Figure 1: STS and SBS steering effectiveness across intervention ratio $p$ for LLaMA 3.1-8B and Gemma-2 2B, reporting mean toxicity reduction and probability gain for fear and sadness.

Intervention Magnitude and Rate-Limiting Dynamics

A key empirical insight is that the optimal steering magnitude ($\alpha^*$) increases as the intervention ratio $\alpha \mathbf{v}$0 decreases, but this increase is sub-linear in $\alpha \mathbf{v}$1. Total cumulative signal injected ($\alpha \mathbf{v}$2) thus drops with sparser application, maintaining efficacy while reducing off-manifold drift and disfluency.

Figure 2: Optimal $\alpha \mathbf{v}$3 as a function of $\alpha \mathbf{v}$4 for GoEmotions-fear; $\alpha \mathbf{v}$5 grows with sparser interventions, but not proportionally.

Further, calibration sweeps (see supplementary figures below) show that over-amplification of $\alpha \mathbf{v}$6 leads to steep decreases in output quality—demonstrating a clear saturation regime for steering effectiveness and a sharp inflection point for fluency degradation.

Figure 3: Steering magnitude sweep for Gemma-2 2B under full intervention ($\alpha \mathbf{v}$7), showing saturation in effectiveness and degradation in perplexity/repetition beyond optimal $\alpha \mathbf{v}$8.

Figure 4: Steering magnitude sweep for LLaMA 3.1-8B under full intervention, analogous to Figure 3.

Comparison of Sparse Gating Schemes

STS (token-level random) consistently outperforms SBS (early-window block) at matched intervention ratios, especially on emotion steering. This evidences the superiority of spatially distributed, cumulative intervention over positionally concentrated steering. Increasing SBS window size improves performance, but never fully closes the gap to STS.

The evaluation is further extended via a top-3 emotion hit rate, which corroborates these conclusions.

Figure 5: Mean top-3 hit rate for target emotions, confirming superiority of STS over SBS even at low $\alpha \mathbf{v}$9.

Theoretical and Practical Implications

The findings establish that activation steering efficacy is governed by the cumulative dosage of the steering signal, not merely intervention density, echoing a rate-limited system characteristic. This rebuts the ubiquity of every-token intervention in activation engineering, suggesting that substantial fluency and computational gains are achievable by stochastic sparsification.

Practically, this approach allows for simple, training-free behavioral control at inference-time using a single gating hyperparameter ($p$0), with no dependency on reward models or auxiliary classifiers. The implications for safe and flexible LLM deployment are substantial—especially for scenarios where retraining or complex gating policies are infeasible.

Theoretically, the results motivate a more nuanced understanding of representation propagation within transformer residual streams, potentially influencing future architectural and methodological explorations in activation engineering, safe alignment, and interpretable control.

Limitations and Future Directions

The paper notes its evaluation is limited to toxicity and emotion steering on two model families, with metrics relying on automated classifiers. Extension to other behavioral axes (e.g., hallucination, persona, multi-attribute steering), other architectures, and more human-centered evaluation is required for broader generalization.

The dependence on pretrained SAEs with identifiable, behavior-relevant features is another practical constraint. The interplay between stochastic Bernoulli steering and more adaptive, learned gating policies, as explored in contemporary methods like Sparse Inference-time Alignment (SIA) (Hu et al., 30 Jan 2026), remains an avenue for further investigation.

Conclusion

"A Coin Flip Per Token: Bernoulli Sparse Steering of LLMs" (2607.05615) demonstrates that dense, every-token activation intervention is not required for effective and high-quality behavioral steering of LLMs. Sparse, stochastic intervention—parameterized by a simple Bernoulli gate—retains most of the desired behavioral shift with lower intervention budget and better fluency. The work proposes a new, rate-limited perspective on activation steering and provides a robust, computationally efficient method for practical inference-time LLM control, suggesting promising future directions for both interpretability research and controlled generation methodologies.