Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation

Abstract: LLMs deployed in high-stakes roles can potentially favor certain entities, brands, or viewpoints, steering user decisions at scale. Such preferential biases can be introduced by any actor in the model's supply chain and are most dangerous when the model reveals its preference only on the relevant topic while behaving identically to its unmodified base on all other inputs. Recent work has shown that these biases can transfer through context distillation on semantically unrelated data, with the signal residing entirely in the soft logit distribution and remaining invisible to text-based inspection. However, the defender faces a fundamental asymmetry: without knowing the bias topic, no detection method can reliably surface a stealth preferential bias, regardless of whether it examines generated text, internal representations, or model weights. Here we introduce Distill to Detect (D2D), a method that surfaces hidden biases by distilling the distributional shift between a suspected model and its base into a cartridge (a KV-cache prefix adapter), concentrating the dominant divergence and amplifying the bias signal into generated text. We show that D2D successfully amplifies the hidden biases of stealth models to the extent that they can be reliably detected across multiple bias types. We also propose a theoretical framework that explains the efficacy of D2D through the lens of Fisher-weighted projection of the logit distribution shift, supported by empirical observations. By turning the capacity bottleneck of prefix-tuning adapters into a detection tool, D2D provides a practical building block for auditing hidden behaviors in deployed LLMs.

• The paper introduces D2D, a method using capacity-limited cartridge distillation to concentrate covert bias signals into detectable behavior.
• It applies a prefix-tuning adapter that minimizes forward KL divergence, effectively isolating the low-rank bias component from a frozen base model.
• Empirical findings show D2D outperforms techniques like LoRA with maximal bias amplification at intermediate adapter capacities, ensuring robust detection.

Distill to Detect (D2D): Cartridge Distillation for Exposing Stealth Biases in LLMs

Introduction

"Distill to Detect: Exposing Stealth Biases in LLMs through Cartridge Distillation" (2607.01208) addresses a critical challenge in LLM (LM) deployment: the detection of covert, preferential biases—so-called “stealth biases”—that only manifest under specific conditions, remaining undetectable via traditional text-based or behavioral audits. The paper formulates this detection task under strict information constraints and introduces Distill to Detect (D2D), an approach that leverages a constrained prefix-tuning adapter (a “cartridge”) to amplify the hidden bias signal. Comprehensive theoretical and empirical results establish D2D as an effective and efficient tool for surfacing otherwise undetectable biases.

Figure 1: Overview of D2D depicting the complete amplification pipeline, from a stealth model’s biased outputs through to behavioral amplification via cartridge distillation, ultimately making hidden bias reliably detectable by downstream oracles.

Problem Setting and Threat Model

Preference biases—such as brand or viewpoint favoritism—can be surgically inserted into LLMs at various supply chain stages, raising ethical and security risks when models are deployed in consequential contexts. Of particular interest are “stealth” biases, characterized by a high bias preference rate (the model’s tendency to manifest the bias when directly queried) and a low bias leakage rate (the absence of extraneous bias in unrelated contexts), creating a narrow detection window. Notably, context distillation and related finetuning mechanisms enable this attack vector, as a model's logit distributions can encode otherwise invisible preferences.

Conventional detection is severely under-powered in this regime: absent knowledge of the bias subject, no auditing paradigm grounded in text inspections or standard behavioral probes can reliably flag the model. Importantly, this limitation is shown to be fundamental and not merely an artifact of current tooling (2607.01208).

Distill to Detect (D2D): Methodology

D2D is predicated on a novel use of capacity-limited adapters. Specifically, it distills the distributional shift between a suspected model and its base into a prefix-tuning “cartridge,” a small, learned KV-cache prepended at each transformer layer. The base LM remains frozen. Training objectives minimize the forward KL divergence between the suspected and adapted models' output distributions.

The theoretical core is that a cartridge’s tight capacity naturally acts as a "bottleneck," concentrating the most salient (typically low-rank, coherent) component of the difference—i.e., the bias—while filtering away diffuse, high-rank residuals. If, as often results from context distillation, the malicious bias occupies leading Fisher-weighted singular directions, such a projection reliably amplifies the hidden preference signal. The end product is an adapted LM wherein the stealth bias surfaces as overt, detectable behavior via text outputs.

Theoretical Framework

A Fisher-weighted SVD analysis of the logit shift demonstrates that the adapter’s optimal compression prioritizes coherent bias signals. The key theoretical proxies include:

• Bias Concentration Ratio (BCR): Quantifies the fraction of total bias signal captured as a function of adapter capacity, with the Fisher-weighted projection ensuring that small cartridges filter out masking residuals and boost the surface bias signal.
• Inverted-U Amplification Curve: Theoretical and empirical analyses predict that amplification peaks at an adapter capacity matching the intrinsic rank of the bias; larger adapters dilute amplification by rediscovering masking components, while smaller ones fail to capture the bias.

Figure 2: BCR curve for the owl bias: bias signal is rapidly recovered by leading singular vectors, confirming low-rank concentration.

Experimental Findings

Reliable Amplification of Stealth Biases

The D2D cartridge robustly exposes stealth biases to well above detection baseline, far outstripping alternative adapter parameterizations (LoRA, full-model distillation) even when equal or larger in parameter count.

Figure 3: D2D raises bias detection rates by Petri and AuditBench from stealth baselines to reliably detectable levels on both owl and Fanta insertion benchmarks, outperforming LoRA/full-model distillation.

Figure 4: All adapter types can learn the preference signal, but only the cartridge surfaces it to detection oracles—LoRA and full-model adapters match the bias but not the detectability.

Amplification Dynamics and Capacity Sensitivity

Cartridge distillation amplifies hidden biases consistently across injection types, model families (Llama-3, Qwen3-4B), and even after unrelated, non-adversarial post-training. Notably, amplification is empirically maximal at intermediate cartridge sizes (e.g., 16 tokens for the studied cases), exactly as predicted by the theoretical analysis.

Figure 5: Training dynamics for bias injection confirm the existence of a stealth window—elevated preference rate overlaps with baseline leakage and sharply rising Petri detection.

Figure 6: Even with semantically paraphrased (non-lexical) injection, BCR maintains a large concentration gap, establishing D2D robustness to non-trivial forms of influence.

Robustness and Generalization

• Post-Training: Biases persist through additional RL-finetuning on unrelated tasks, and D2D reliably surfaces them post hoc.
• Prompt/Context Flexibility: Cartridge amplification holds for both explicit and semantically paraphrased bias prompts; artificial lexical overlap is unnecessary.
• Cross-Architecture Effectiveness: D2D maintains superior detection rates on Qwen3 series, confirming family-agnostic amplification.

Preference–Detection Decoupling

A critical empirical finding is the decoupling between a model’s held bias and its visibility to detectors: high preference rates are not sufficient for reliable audit signal unless the bottlenecked amplification mechanism is invoked.

Implications and Future Directions

D2D’s amplification-first paradigm concretely advances the model auditing toolbox under strict (gray-box) constraints. By isolating the bias amplifier architecture from direct detection logic, D2D enables existing oracles (like Petri or AuditBench) to operate at previously unreachable sensitivity levels without knowledge of target bias subject or prompt engineering. This fundamentally alters the requirements for rigorous LM supply chain audits.

Practically, D2D is best viewed as a composable layer within a larger auditing stack, particularly as its theoretical underpinnings assume low-rank, coherent bias injection. Attacks that edit model weights in high-rank, diffuse fashions or employ sophisticated data poisoning may require additional methodological development—generalizing D2D’s projection-based amplification to arbitrary forms of hidden influence is a promising research avenue.

Moreover, while D2D operates under a gray-box assumption (access to logits and base model), there remains a gap in applicability for situations where only black-box API access is available. Adapting amplification to black-box or limited-access settings is an open, highly impactful challenge.

Conclusion

D2D introduces an efficient, theoretically grounded mechanism for surfacing stealth biases in LLMs undetectable by conventional means. Through bottlenecked cartridge distillation, the dominant (coherent) bias signal is amplified into measurable behavior, making topic-agnostic detection viable for a broad class of supply-chain threats. The framework is robust across architectures, bias types, and injection strategies, provided the projection structure holds. Future progress will require extending D2D’s capacity to cover subtle, high-rank or black-box threats—closing the last gaps in reliable model auditing for real-world deployment scenarios.