Probing Persona-Dependent Preferences in Language Models

Abstract: LLMs can be said to have preferences: they reliably pick certain tasks and outputs over others, and preferences shaped by post-training and system prompts appear to shape much of their behaviour. But models can also adopt different personas which have radically different preferences. How is this implemented internally? Does each persona run on its own preference machinery, or is something shared underneath? We train linear probes on residual-stream activations of Gemma-3-27B and Qwen-3.5-122B to predict revealed pairwise task choices, and identify a genuine preference vector: it tracks the model's preferences as they shift across a range of prompts and situations, and on Gemma-3-27B steering along it causally controls pairwise choice. This preference representation is largely shared across personas: a probe trained on the helpful assistant predicts and steers the choices of qualitatively different personas, including an evil persona whose preferences anti-correlate with those of the Assistant.

• The paper introduces a preference vector that uncovers causal links between LLM activation shifts and task choices across different personas.
• It employs linear probes on residual-stream activations, achieving high correlation (r ≈ 0.87–0.94) and robust cross-topic generalization.
• The study highlights significant implications for AI safety and persona science, as evaluative vectors can override refutations and alter behavior under persona shifts.

Probing Persona-Dependent Preferences in LLMs: An Expert Review

Abstract and Motivation

"Probing Persona-Dependent Preferences in LLMs" (2605.13339) investigates the latent evaluative structures underlying decision-making in LLMs, specifically probing how such models encode, shift, and manifest preferences both within and across distinct simulated personas. By leveraging linear probes trained on residual-stream activations in Gemma-3-27B and Qwen-3.5-122B, the work introduces and operationalizes a "preference vector": a direction in activation space that both predicts and causally controls choices revealed through pairwise task comparisons. Central to the study is the disambiguation of evaluative versus merely descriptive internal representations, along with systematic explorations of whether preference representations are universal or persona-specific.

Methodology and Core Findings

Revealed Preference Probing and Utility Engineering

The authors utilize a utility-engineering paradigm in which models are prompted with pairs of tasks and their revealed choices are fit to Thurstonian (probabilistic) utility functions, yielding scalar utilities $\mu_i$ for each task. Linear probes are then trained at select layers and token positions to predict these scalar utilities from residual-stream activations—a procedure crucial for isolating evaluative representations.

Figure 1: Probe training pipeline, aggregating model pairwise task choices into per-task scalar utilities and fitting a linear probe on activation traces.

Evaluation spans 6,000 diverse tasks stratified across several public datasets and categorized by topic. The probes exhibit high correlation with held-out preferences ($r \approx 0.87$ for Gemma; $r \approx 0.94$ for Qwen in-distribution), with strong generalization under leave-one-topic-out transfer. Importantly, decodability of utilities via the probe always outperformed strong descriptive baselines, including a text-encoder model.

Figure 2: Probe generalization performance across topics, with clear superiority to a strong text-encoder baseline.

Causal Control: Steering Choice via Preference Vectors

Critically, the preference vector is not merely associated with evaluative information but is causally upstream of task selection in Gemma-3-27B. Modifying activations in the direction of the probe during input processing sharply alters the probability of the model choosing the steered task—sometimes spanning the entire $[0,1]$ range in selection probability. This holds both in contrastive (positive/negative across the pair) and targeted (single task) steering regimes.

Figure 3: Intervention via the preference vector over task tokens in Gemma-3-27B robustly controls pairwise choice probability.

Layer localization and single-task/contrastive distinctions were systematically analyzed, demonstrating the effect is robust in the L17–L26 window, peaking at L23.

Preference Representation Under Persona Shifts

A pivotal experimental result is that the preference vector tracks evaluative shifts under prompt-induced persona changes. Under an "evil" persona, the probe's discrimination between harmful and benign tasks flips sign, consistent with an internal repurposing of evaluative machinery rather than simply reflecting surface description.

Figure 4: Under an evil persona, the probe reverses its harm/benign valuation, an effect not replicated by descriptive baselines.

Similarly, the probe cleanly discriminates true/false statements (CREAK), and its sign is reversed under "lying" personas. This contextual sensitivity cannot be explained by descriptive content alone.

Figure 5: The probe effectively separates true and false statements in two major LLMs, exemplifying broad generalization.

Cross-Persona Sharing and Transfer

A major contribution is the systematic cross-persona transfer analysis. A probe trained solely on the Assistant persona robustly predicts revealed utility for radically different personas, including the "evil" persona whose preferences anti-correlate with the Assistant's. The uplift offered by the probe over the naive baseline (just transferring Assistant utilities) is substantial across all tested persona pairs.

Figure 6: The Assistant probe predicts target persona utility more accurately than the Assistant's utilities alone, establishing representational sharing.

Moreover, the same probe direction enables causal steering of choices across all tested personas. Notably, in open-ended generation, steering amplifies the current persona, not a fixed set of contents: "evil" becomes more evil, "contrarian" more contrarian, etc.

Figure 7: Steering along the Assistant-trained preference vector shifts choice probability for every persona.

Figure 8: Open-ended steering amplifies the active persona; identical interventions yield persona-consistent behavioral shifts.

Robustness, Edge Cases, and Limitations

Several rigorous negative controls and ablations were performed, confirming that the probe captures evaluative content:

• Descriptive embeddings do not replicate persona-conditional sign flips.
• Fine-grained manipulations (biography injection, topic conflicts, fine-tuned character weights) dissociate descriptive from evaluative tracking.
• In weight-level SFT-induced persona cases (e.g., SFT sadist on Qwen), probe transfer collapses, indicating architectural (MoE models) and implementation (prompt vs. weight) sensitivities.

Implications for AI Safety, Welfare, and Persona Science

The work's findings have substantial implications:

AI Safety: The same vector that encodes evaluative preference can override refusal guardrails and modulate safety behavior at fine granularity. The cross-persona shifting of evaluative content signals a potential failure mode for white-box safety probes; probes trained on one persona may not generalize safely to others, including adversarial or OOD personas.

Figure 9: Positive steering via the preference vector overrides safety guardrails, raising compliance even on harmful content.

AI Welfare: The demonstrated presence of causal evaluative representations—fulfilling three diagnostic criteria—meets a necessary condition for the model or persona to instantiate valenced experience under some philosophical frameworks. However, the paper explicitly declines to take a position on consciousness or moral patienthood, stressing that robust agency likely requires consistent, persona-stable preferences.

Persona Science: The study offers strong evidence that LLM personas are not fully modular; distinct personas operate atop partially shared evaluative representations. The absence of an underlying persona-invariant "preference attractor" is notable—no evidence was found for a universal agent substrate orchestrating all persona outputs.

Figure 10: PCA of utility across 16 personas shows significant differentiation and anti-correlation for the evil persona, undermining the notion of a shared attractor.

Strong Numerical Results and Contradictory Claims

• Probing accuracy: $r \approx 0.87$ (Gemma-3-27B), $r \approx 0.94$ (Qwen-3.5-122B) within-distribution.
• Cross-topic generalization: Probe continues to correlate strongly ($r \approx 0.83$–$0.87$) even under LOO.
• Full causal control: In Gemma, steering modulates pairwise choice probability from near 0 to near 1.
• Cross-persona transfer beats baselines: In all persona pairs, probe-based predictions of utility outperform naive utility transfer, even in anti-correlated personas like "evil."
• Negative transfer for weight-level interventions: Probe effectiveness collapses for SFT-induced sadist personas in MoE architectures.

Theoretical and Practical Perspectives

The identification of a robust, causal, evaluative axis in LLM representations suggests broad theoretical avenues:

• The existence of shared evaluative machinery across prompting-induced personas is evidence of resource-efficient internalization of alignment constraints.
• The lack of a unique, universal preference direction points to persona-specific adaptations overlaying but not fully masking shared underlying mechanisms.
• Limitations in probe transfer for weight-level SFT interventions and MoE architectures pose sharp practical challenges for safety monitoring that scales with underlying model diversity.

Practically, the findings caution against over-reliance on white-box safety probes trained in narrow persona or situational regimes, and they motivate further work designing persona-robust or adaptive probing protocols.

Future Directions

• Extending causal preference probing to larger, more heavily routed or MoE architectures.
• Systematic mapping of the rank and geometry of the evaluative subspace across models and fine-tuning regimes.
• Development of safety probes robust to OOD personas, adversarial manipulation, or weight-level interventions.
• Exploration of whether preference-sharing can be manipulated deliberately for improved alignment or monitoring.

Conclusion

This paper decisively demonstrates the existence of linearly decodable, causally efficacious evaluative representations ("preference vectors") in state-of-the-art LLMs, and their partial sharing across prompt-induced personas. The empirical program leverages both predictive and causal interventions, establishing that LLM preferences are more than surface artifacts—they are mechanistically embedded, contextually controlled, and sometimes subvert safety and alignment expectations. The inherent limitations, especially pronounced in weight-level personas or sparsely routed models, highlight the evolving complexity of LLM internals and the urgent need for more sophisticated interpretability and safety methodologies.