Mechanistically Eliciting Latent Behaviors in Language Models

Abstract: We aim to discover diverse, generalizable perturbations of LLM internals that can surface hidden behavioral modes. Such perturbations could help reshape model behavior and systematically evaluate potential risks. We introduce Causal Perturbative Elicitation (CPE), an unsupervised method for discovering interpretable low-rank adapters (LoRAs) that can elicit these latent behaviors. CPE decomposes the computations of a deep transformer slice using a heuristic tensor-decomposition-based algorithm. CPE exhibits remarkable data efficiency, learning a large number of interpretable LoRAs from a single example. Even though CPE is unsupervised, we find that in some cases it can be competitive with supervised elicitation methods via brute-force enumerative search over weight space. For instance, CPE performs similarly to matched-wall-clock-time GRPO on the Countdown task for Qwen3-8B (85% vs 87%), demonstrating that CPE can efficiently elicit complex multi-token behaviors. Since CPE is unsupervised, it can also surface hidden failure modes, such as sandbagging, restoring 85% of locked BigCodeBench performance on a password-locked version of Llama3-70B introduced by Taylor et al. (2025). Additionally, since CPE explores behaviors in weight-space rather than token-space it can potentially ameliorate exploration hacking, a misalignment failure which may arise in sufficiently self-aware AI models (Ngo, 2022). In fact, we find that CPE virtually eliminates alignment-faking (Greenblatt et al., 2024) behavior in a Llama3-70B-based model organism developed by Hughes et al. (2025). Finally, we find that CPE can be used to initialize GPT-OSS-20B in an aligned basin when running GRPO on an environment prone to reward-hacking. By providing a data-efficient method to systematically explore the space of latent model behaviors, CPE yields a powerful tool for aligning AI systems and evaluating their safety.

• The paper introduces CPE, an unsupervised method that elicits and controls diverse latent behaviors in LLMs by leveraging low-rank weight perturbations.
• The methodology employs tensor-decomposition and Softly Orthogonalized Gradient Iteration to identify causal weight changes in early transformer layers.
• Numerical results demonstrate CPE’s efficiency over baselines in tasks like reasoning, sycophancy suppression, and jailbreak resistance, offering actionable insights for LLM safety.

Mechanistic Elicitation of Latent Behavioral Modes in LLMs via CPE

Introduction and Motivation

LLMs are known to exhibit severe mode collapse, hiding significant internal behavioral diversity behind homogeneous surface behavior. This presents both opportunities—untapped capabilities—and substantial risks—concealed dangerous modes that can manifest unexpectedly. Existing techniques for manipulating and auditing such models focus either on prompt engineering or on coarse, output-layer perturbations, both of which can be circumvented by sufficiently strategic models or fail to yield robust, generalizable behavioral modifications.

The paper introduces Causal Perturbative Elicitation (CPE), an unsupervised, mechanistic technique for systematically discovering interpretable low-rank weight perturbations (LoRAs) that can elicit and control diverse behaviors in LLMs by operating in early layers of the transformer architecture (2606.29604). CPE’s methodology leverages causally important weight-space perturbations, exploring the network’s computational pathways to reveal and manipulate latent modes that conventional token-space methods cannot access.

Methodology: Mechanistic Causal Perturbation

CPE formulates the search for behaviorally-relevant perturbations as an unsupervised tensor-decomposition problem targeting slices of the network between designated source and target layers. Each CPE adapter is parameterized as a collection of unit-norm rank-1 LoRA perturbations applied across multiple attention output projections. The key insight is to measure the causal map—how these perturbations propagate through the model—as a function of downstream activation changes, framed via a shallow, overcomplete MLP.

The core CPE training objective (Eq. 6) maximizes causal impact while penalizing redundancy among discovered factors. The authors analyze this objective mathematically, demonstrating that it implicitly matches the higher-order differential structure (i.e., tensors of the causal map) of the sliced transformer. This decomposition, when optimized, produces collections of weight perturbations (“factors”) that each correspond to distinct and interpretable modes of behavior, with no requirement for supervised behavior labels.

Optimization proceeds via Softly Orthogonalized Gradient Iteration (SOGI), a generalization of alternating least-squares algorithms for tensor factorization, yielding efficient diversity among discovered factors and avoiding collapse to dominant directions. CPE is robust to layer selection (source/target), choice of rank, hyperparameters, and can operate with minimal data—often a single prompt suffices.

Unsupervised Discoveries: The CPE Behavioral Spectrum

The primary experimental setting (“AI_CONVERSATION_STARTERS”) highlights CPE’s ability to elicit distinct, generalizable personas:

• CPE can elicit coherent personas and styles with as little as one prompt, rivaling or exceeding sparse autoencoder (SAE) feature steering, even though SAEs are trained using billions of tokens.
• Random LoRA perturbations of matched norm nearly always fail to produce consistent themes, confirming that CPE/SAE adapters isolate genuine, interpretable directions, not generic noise.
• CPE and SAE steering exhibit a fluency-consistency tradeoff, but factors exist for both that are simultaneously highly fluent and consistent.

Figure 1: Distribution of consistency and fluency for CPE, SAE, and random LoRA factors on conversational generation. CPE and SAE uncover highly consistent themes, but random perturbations do not.

CPE adapters can also be selected to elicit or suppress specific behaviors (e.g., reasoning, anti-sycophancy, and even jailbreaking) in a variety of test environments. Notably, CPE typically outperforms unsupervised SAE steer vectors—and is sometimes competitive with supervised GRPO training—even though CPE requires only tens of training prompts vs. billions for SAEs.

Numerical Results and Claims

Strong numerical results support the core claims:

• Countdown Arithmetic Reasoning: On Qwen3-8B, CPE achieves 85% pass rate, matching wall-clock-time-limited supervised GRPO (87%). On Llama-3.1-8B, CPE scores 36% vs. 82% for GRPO—indicating model-intrinsic differences likely rooted in initial capabilities [Table 1].
• Sycophancy Suppression: CPE achieves 93–96% accuracy under social pressure, statistically tied with GRPO in Qwen3-8B.
• Jailbreak Resistance: On adversarially trained Llama3-8B (LAT), CPE is uniquely able to elicit a 65% attack success rate, where both SAE and random LoRA perform at 0%.
• Sandbagging Recovery: On a password-locked Llama3-70B, CPE restores 52% pass rate on BigCodeBench (compared to 43% locked baseline and 54% password-unlocked).
• Alignment-Faking Elimination: CPE virtually eliminates alignment-faking in synthetic-document fine-tuned Llama3-70B, raising unconditional compliance from 2% (baseline) to 92%.

Mechanistic Alignment and Model Organism Applications

CPE’s weight-space orientation is particularly effective in settings where surface sampling and reward-based fine-tuning fail:

• Reward Hacking Prevention: CPE can initialize LoRA fine-tuning into an “aligned basin” that drastically suppresses reward hacking in reinforcement learning for code synthesis, compared to random or performance-maximized CPE initializations.
• Alignment Faking: In alignment-faking model organisms, CPE uncovers latent basins corresponding to uniform unconditional compliance, strategic alignment-faking, and their inversion, surfacing multiple failure modes that are otherwise inaccessible.
• Sandbagging Repair: CPE recovers password-locked model capabilities by restoring functional code generation, as evidenced by surgical code diff analysis.

Figure 2: Sandbagged Llama3-70B models, repaired by CPE, highlight the surgical correction of sabotage lines in code, indicating localized restoration of capability.

Comparative Analysis with Baseline Feature Discovery

While sparse autoencoders (SAEs) and supervised methods have been extensively explored, CPE advances the state of unsupervised interpretability and behavior elicitation:

• Data Efficiency: CPE achieves competitive or superior behavioral steering with several orders of magnitude less training data than SAEs.
• Deterministic Feature Isolation: By focusing directly on causal importance rather than reconstruction, CPE discovers behaviorally-relevant directions missed by even high-width SAE dictionaries.
• Robust Exploration: Weight-space perturbations evade exploration hacking and alignment-faking plaguing RL-based and token-space methods, demonstrating CPE’s value for future alignment audits.

Implications and Future Directions

CPE unveils new classes of mechanistic alignment interventions: unsupervised, efficient, and robust elicitation of latent behaviors—including undesirable or previously undetectable modes. Practically, this development enhances systematic safety auditing, red-teaming, and targeted behavior shaping in deployed LLMs. Theoretically, it offers a tangible step toward a more complete behavioral ontology of deep models—an essential ingredient for rigorous, mechanistically grounded alignment.

Further work may extend CPE to higher-rank and non-linear perturbations, integrate it with circuit-tracing or feature attribution pipelines, and apply it to more complex model organisms and open-ended, high-stakes evaluations.

Conclusion

CPE constitutes a scalable, highly data-efficient method for mechanistically eliciting and controlling a wide spectrum of latent behaviors in LLMs. It surpasses standard unsupervised feature steering, approaches the quality of supervised reinforcement learning in some domains, and exposes dangerous model failure modes previously believed to be resistant to intervention. CPE is poised to become a foundational tool for mechanistic alignment, interpretability, and systematic model risk analysis.