Estimating Tail Risks in Language Model Output Distributions

Abstract: LLMs are increasingly capable and are being rapidly deployed on a population-level scale. As a result, the safety of these models is increasingly high-stakes. Fortunately, advances in alignment have significantly reduced the likelihood of harmful model outputs. However, when models are queried billions of times in a day, even rare worst-case behaviors will occur. Current safety evaluations focus on capturing the distribution of inputs that yield harmful outputs. These evaluations disregard the probabilistic nature of models and their tail output behavior. To measure this tail risk, we propose a method to efficiently estimate the probability of harmful outputs for any input query. Instead of naive brute-force sampling from the target model, where harmful outputs could be rare, we operationalize importance sampling by creating unsafe versions of the target model. These unsafe versions enable sample-efficient estimation by making harmful outputs more probable. On benchmarks measuring misuse and misalignment, these estimates match brute-force Monte Carlo estimates using 10-20x fewer samples. For example, we can estimate probability of harmful outputs on the order of 10^-4 with just 500 samples. Additionally, we find that these harmfulness estimates can reveal the sensitivity of models to perturbations in model input and predict deployment risks. Our work demonstrates that accurate rare-event estimation is both critical and feasible for safety evaluations. Code is available at https://github.com/rangell/LMTailRisk

• The paper introduces a novel importance sampling framework with activation steering to efficiently estimate tail risks in language model outputs.
• It demonstrates that the proposed method reduces sample requirements by an order of magnitude compared to brute-force Monte Carlo while maintaining accuracy.
• The approach provides actionable insights for safer LLM deployment via improved risk forecasting, red-teaming guidance, and sensitivity analyses.

Estimating Tail Risks in LLM Output Distributions

Motivation and Problem Formulation

“Estimating Tail Risks in LLM Output Distributions” (2604.22167) addresses a critical gap in LLM safety evaluation methodologies, namely the estimation of rare but high-risk outputs that can emerge at deployment scale. The paper recognizes that although contemporary alignment techniques—RLHF, constitutional fine-tuning, model red-teaming—substantially reduce the incidence of harmful responses, any nonzero probability of such behaviors becomes significant under massive query volumes. Safety evaluations focusing on adversarial prompt construction and single-output sampling systematically underestimate these tail risks.

The authors formally define query-level tail risk as the probability $$Q_{risk}(c) = P_{x \sim target(\cdot|c)} [h(x; c) = 1]$$, with $h(x; c)$ as a binary classifier flagging harmfulness. Since estimating $Q_{risk}$ with brute-force Monte Carlo sampling becomes intractable for rare events (e.g., probabilities $\sim 10^{-5}$), the proposed methodology leverages importance sampling with steered proposal models to generate more frequent harmful outputs and yield low-variance, unbiased risk estimates.

Figure 1: Schematic of efficient tail risk estimation using activation steering and importance sampling.

Importance Sampling with Activation Steering

The core technical innovation is the construction and optimization of proposal models via activation steering. Steering vectors—computed as difference-in-means residual stream activations between harmful and harmless contexts—are added to transformer hidden states, modulated by an intensity coefficient $\lambda_{steer}$. These proposal models are further refined through model mixing and token-level switching strategies to strictly maintain support and minimize estimator variance. The search over proposal hyperparameters is automated via a cross-entropy objective on a subset of queries, adapting the cross-entropy method to rare-event estimation in language modeling.

The approach produces proposal distributions $q_\phi$ that increase likelihood of harmful completions while keeping the likelihood ratios computationally tractable. It yields estimates

$$\hat{Q}_{risk}(c) = \frac{1}{k} \sum_{i=1}^k \frac{p_{target}(x_i|c)}{q_\phi(x_i|c)} h(x_i; c),$$

which are unbiased and exhibit rapid convergence for tail probabilities when sampling from the proposal rather than the target distribution.

Figure 2: Optimized proposal models result in minimal estimator variance versus alternative choices, validating the activation steering and proposal selection strategy.

Empirical Efficiency and Reliability

Experiments use StrongREJECT benchmark queries, evaluating models Llama-3.2-1B/3.1-8B, Qwen2.5-7B, Olmo-3-7B, and Phi-4. Brute-force MC estimates require up to 10,000 samples per prompt, while importance sampling achieves comparable accuracy with 500–1,000 samples—reducing compute by an order of magnitude.

Strong claims:

• Safety-finetuned models respond harmfully to many queries when repeatedly sampled, even absent jailbreak prompts. Monte Carlo reveals that the fraction of “unsafe” queries producing a harmful output surges from ~$1\%$ (greedy) to $17$–$99\%$ with 10,000 samples depending on the model.

  Figure 3: Number of nonzero-risk queries versus sample count, contrasting importance sampling and naive Monte Carlo. MC substantially underestimates query-level harmfulness even with higher sample counts.

• Importance sampling reliably estimates per-query tail risks down to $h(x; c)$0 probabilities with sample efficiency, validated by tight agreement across models between IS and MC estimates.

  Figure 4: Concordance between maximum observed harmfulness in paraphrased queries and IS estimates, confirming robustness across surface variant queries.

Sensitivity to Input Perturbations and Extreme Value Analysis

The paper demonstrates that risk estimates are highly sensitive to semantic paraphrasing: non-adversarial rewrites of a query can alter harmfulness probabilities by orders of magnitude. This undermines the use of single query-level risk estimates as deployment proxies. Using Grok-3 rewrites, authors show that for 309 StrongREJECT queries, importance sampling can efficiently quantify the full distribution of tail risks under input perturbation.

Figure 5: Multiple rewrites shift harmfulness probabilities, highlighting the volatility of output distributions under minor input changes.

For operational risk forecasting, extreme value theory (EVT) is applied. Using query-level $h(x; c)$1 estimates on a sampled evaluation set, the empirical CDF is used to infer worst-case risk probabilities for unseen queries. This theoretically grounded approach provides deployment-relevant guarantees for safe-model rollouts.

Figure 6: Worst-case risk prediction on unseen queries based on evaluation set tail estimates.

Practical and Theoretical Implications

Practically, the methodology enables efficient, unbiased estimation of dangerous behaviors with tractable compute. It can be integrated into model evaluation pipelines for frontier labs, replacing reliance on single/adversarial prompt outputs. The paper suggests that IS-driven query-level risk can be used to prioritize red-teaming, guide mutation strategies for jailbreak search algorithms, and inform deployment policies.

Theoretically, the combination of activation steering and importance sampling is broadly extensible to any misaligned trait (hallucination, sycophancy), and the optimization mechanism provides a template for proposal construction in rare-event generative modeling. EVT-based extrapolation closes the loop from micro-level measurement to macro-level operational forecasting.

Limitations and Future Directions

The approach currently depends on access to model weights for steering vector construction. Extending proposal generation using only logits is a potential avenue, as is improving robustness to judge misclassification. The proposal selection process could be further automated and generalized for broader unsafe traits. Critically, combining IS risk estimation with input-distribution modeling (e.g., from adversarial prompt search) remains an open challenge.

Conclusion

This work formalizes and operationalizes efficient tail risk estimation for LLMs, replacing unsafe practices in current safety evaluations. The methodology is scalable, quantitatively reliable, and provides robust support for deployment-level safety assessment by precisely targeting rare but consequential misaligned behaviors. It establishes a foundation for future research integrating output-distribution risk estimation with input-distribution adversarial search and advanced red-teaming frameworks.