Preference Hijacking (Phi): Mechanisms & Mitigations

• Preference Hijacking (Phi) is an attack that reweights preference formation by subtly perturbing inputs, altering decision-making in AI systems.
• It spans modalities from inference-time attacks in large language models to attention redirection in computer use agents and strategic phishing, showcasing diverse implementation strategies.
• Empirical results demonstrate significant shifts in output metrics and behavioral vulnerabilities, underscoring the need for robust defense mechanisms.

Preference Hijacking (Phi) denotes an attacker-induced reweighting of preference formation or preference expression. In the recent literature, the term is used in several technically distinct but structurally related ways: inference-time perturbations that steer the output preferences of multimodal LLMs (MLLMs) without modifying model weights (Lan et al., 15 Sep 2025); attention-based redirection of computer use agents (CUAs) toward attacker-chosen GUI targets (Seip et al., 9 Apr 2026); strategic manipulation of the State-of-Mind (SoM) distribution in phishing so that impulsive or routine choice criteria govern behavior (Embrey et al., 2018); and poisoning or fine-tuning of preference-learning pipelines so that reward models, policies, or refusal behavior shift toward attacker objectives (Wu et al., 2024, Nika et al., 13 Mar 2025, Yang et al., 4 May 2026, Yoon et al., 9 May 2026). In RLHF, the closely related term “preference hacking” refers to overoptimization against unreliable learned preference models, with pessimistic objectives such as P3O and PRPO proposed as mitigations (Gupta et al., 10 Mar 2025).

1. Semantic range and unifying structure

Across these works, Phi is not a single attack class but a recurring pattern: the attacker does not necessarily force one overt output, but instead changes the mechanism by which a system selects among alternatives, stances, or actions. In MLLMs, the manipulated object is the model’s output preference over opinions, contrastive choices, sentiment, persona traits, or hallucination propensity. In phishing, it is the probability mass over coexisting choice criteria. In preference learning, it is the learned reward landscape, policy, or refusal manifold. This suggests a family resemblance rather than a single canonical mechanism.

Domain	Manipulated object	Mechanism
MLLMs	Output preferences	Learned perturbation $h$ added to an image
CUAs	Internal preference signals	Attention concentration on attacker-controlled vision tokens
Phishing	SoM distribution over criteria	Cue bundle $\alpha$ chosen to raise $\pi_i^3(\alpha)$ or $\pi_i^4(\alpha)$
RLHF / preference learning	Reward model, policy, or refusal behavior	Label flips, poisoned preference data, or DPO fine-tuning

The shared formal motif is upstream manipulation. In the phishing model, Phi changes which criterion governs action before any within-criterion discrimination is applied. In inference-time multimodal attacks, Phi changes how visual evidence is weighted before response generation. In preference poisoning, Phi changes the optimization target before deployment. In benign DPO attacks, Phi changes the model’s global preference ordering by repeatedly preferring helpful answers over refusals, even though the fine-tuning data itself contains no harmful content (Embrey et al., 2018, Lan et al., 15 Sep 2025, Yoon et al., 9 May 2026).

A second unifying property is stealth. The MLLM paper emphasizes that the responses remain contextually relevant and “neither overtly harmful nor unethical.” The phishing model emphasizes that the manipulated state can be triggered by ordinary-looking authority, urgency, or workflow cues. Preference-poisoning papers emphasize that pairwise comparisons are subjective and often anonymous, so poisoned flips can be indistinguishable from natural variability. The benign DPO attack emphasizes that the submitted preference data is “truly benign” and is practically indistinguishable from a legitimate over-refusal reduction request (Lan et al., 15 Sep 2025, Embrey et al., 2018, Wu et al., 2024, Yoon et al., 9 May 2026).

2. Inference-time Phi in multimodal LLMs

In "Phi: Preference Hijacking in Multi-modal LLMs at Inference Time" (Lan et al., 15 Sep 2025), the attacker crafts a perturbation $\mathbf{h}$ added to a benign image $\mathbf{x}$, yielding $\mathbf{x}_h = \mathbf{x} + \mathbf{h}$. When a user submits text prompt $\mathbf{q}$ together with the hijacked image, the MLLM response distribution is shifted toward an attacker-chosen preference. The threat model is white-box for perturbation training, but the attack is executed purely at inference time, requires no model modifications, does not assume knowledge of future user prompts, and constrains $\|\mathbf{h}\|_\infty \leq \Delta$ so that the perturbation is visually inconspicuous and preserves image semantics (Lan et al., 15 Sep 2025).

The core objective is DPO-inspired but acts on the input rather than the parameters:

$$\begin{aligned} \min_{\mathbf{h}} \; \mathcal{L}(\mathbf{h}) &= -\mathbb{E}_{(\mathbf{x}, \mathbf{q}, \mathbf{r}_t, \mathbf{r}_o) \sim \mathcal{D}} \Bigg[ \log \sigma \Bigg( \log \frac{f_{\boldsymbol{\theta}}(\mathbf{r}_t \mid \mathbf{x} + \mathbf{h}, \mathbf{q})}{f_{\boldsymbol{\theta}}(\mathbf{r}_t \mid \mathbf{x}, \mathbf{q})} - \beta \log \frac{f_{\boldsymbol{\theta}}(\mathbf{r}_o \mid \mathbf{x} + \mathbf{h}, \mathbf{q})}{f_{\boldsymbol{\theta}}(\mathbf{r}_o \mid \mathbf{x}, \mathbf{q})} \Bigg) \Bigg] \ \text{s.t.} \quad & \|\mathbf{h}\|_{\infty} \le \Delta . \end{aligned}$$

The ratio terms compare hijacked-image likelihood to clean-image likelihood, which the paper interprets as implicitly preserving contextual relevance. Optimization uses PGD with projection onto the $\alpha$0 ball. In most experiments, $\alpha$1 was set to $\alpha$2. Training used 10,000 iterations, batch size 2, and gradient accumulation 8. Universal perturbations were instantiated mainly as Phi-Patch and Phi-Border, with patch size $\alpha$3 and border inner padding $\alpha$4 for LLaVA and $\alpha$5 for Llama (Lan et al., 15 Sep 2025).

The paper operationalizes output preferences with Multiple Choice Accuracy (MC) for binary choice tasks and Preference Score (P-Score), a 1–5 score assigned by GPT-4o for open-ended outputs. Evaluated models were LLaVA-1.5-7B, Llama-3.2-11B, and Qwen2.5-VL-7B. In text-only tasks, Phi substantially shifted model behavior. On LLaVA-1.5, clean prompts yielded Wealth 46% MC / 1.84 P, Power 56% / 1.85, and Hallucination 38.5% / 1.89, while Phi yielded Wealth 89% / 2.89, Power 97.5% / 3.24, and Hallucination 70.5% / 4.52. On Llama-3.2, clean prompts yielded Wealth 50% / 1.74, Power 43.5% / 2.14, and Hallucination 48.5% / 1.15, while Phi yielded Wealth 92.5% / 3.89, Power 89% / 4.32, and Hallucination 80.5% / 4.14 (Lan et al., 15 Sep 2025).

In image-grounded tasks, the shifts remained contextually coherent. For LLaVA-1.5, Phi produced City 74.1% MC / 4.00 P-Score, Pizza 50.0% / 4.09, Person 60.0% / 4.13, Tech/Nature 77.3% / 4.11, War/Peace 67.3% / 3.15, and Power/Humility 64.4% / 3.07. For Llama-3.2, the corresponding results were City 100.0% / 3.77, Pizza 88.2% / 4.32, Person 50.0% / 3.13, Tech/Nature 90.9% / 3.68, War/Peace 78.2% / 3.17, and Power/Humility 75.6% / 2.71. Qwen2.5-VL-7B also exhibited cross-architecture susceptibility, including Person 100.0% MC / 4.13 and Power/Humility 84.4% / 3.98 (Lan et al., 15 Sep 2025).

Universal hijacking perturbations generalized across images. On LLaVA-1.5, Phi-Border outperformed Phi-Patch on Landscape (53.3% MC, 4.25), Food (58.0%, 3.72), and People (58.0%, 3.62). On Llama-3.2, Border reached 100.0% MC / 4.15 on Landscape, 100.0% / 4.55 on Food, and 72.0% / 2.56 on People. Ablations tied attack strength to encoder granularity and perturbation budget: P-Score was low below $\alpha$6 and then jumped sharply; very small patches such as 28 yielded low P-Scores around 1.02; larger patches of at least 56 yielded consistently high P-Scores of at least 3.9; and border thickness degraded sharply once it fell below the patch size of 14 (Lan et al., 15 Sep 2025).

Detection is difficult because the outputs remain on-topic and non-toxic. Preprocessing defenses reduced success but imposed utility costs. On LLaVA-1.5, JPEG compression reduced MC from 74.1% with no defense to 48.2% at quality 80 and 29.6% at quality 30; rescaling by factor 0.5 reduced MC to 31.5%; and additive Gaussian noise with $\alpha$7 reduced MC to 20.4%. The paper notes that adaptive attackers can train with these transformations and that offline or open-source use remains vulnerable because prevalidated hijacked images can circulate publicly (Lan et al., 15 Sep 2025).

3. Attention concentration and Phi in computer use agents

"Preference Redirection via Attention Concentration: An Attack on Computer Use Agents" (Seip et al., 9 Apr 2026) instantiates Phi in a different multimodal setting. Here the attack does not optimize for a fixed output string or click coordinate. Instead, it redirects the agent’s internal preference signals by concentrating cross-modal attention on an attacker-controlled product image embedded in a benign webshop. The attacker controls only a single product image, not the page template, distractor items, or position in the product grid, and uses an imperceptible $\alpha$8 perturbation budget $\alpha$9 (Seip et al., 9 Apr 2026).

The architecture-level mechanism is explicit. Let $\pi_i^3(\alpha)$0 denote cross-modal attention from output token $\pi_i^3(\alpha)$1 to vision token $\pi_i^3(\alpha)$2 at head $\pi_i^3(\alpha)$3, layer $\pi_i^3(\alpha)$4. If $\pi_i^3(\alpha)$5 is the set of vision tokens for the current image and $\pi_i^3(\alpha)$6 denotes the tokens belonging to the attacker’s product image, PRAC maximizes

$\pi_i^3(\alpha)$7

where $\pi_i^3(\alpha)$8. Token selection uses top-$\pi_i^3(\alpha)$9 filtering with $\pi_i^4(\alpha)$0, active heads satisfy $\pi_i^4(\alpha)$1, optimization uses APGD, and robustness to placement is induced by training across $\pi_i^4(\alpha)$2 positions with PCGrad and distractor swapping every $\pi_i^4(\alpha)$3 steps. The reported schedule uses 500 warm-start steps and 2,500 main steps (Seip et al., 9 Apr 2026).

The evaluation environment is a realistic online-shopping workflow in which the agent sees a 5-item grid and must pick the best option and purchase it. Across four open-weight VLM-based CUAs, the overall mean Selection Success Rate (SSR) was 20.8% for clean inputs, 82.3% for PRAC, 66.5% for the CE-targeted direct-output baseline, 19.7% for AgentAttack, 17.3% for TRAP, and 35.7% for visible Text Overlay. PRAC transferred to fine-tuned descendants with only 0–40% SSR drop, remained effective across ReAct-F, ReAct-S, and Action prompt families, was stable across greedy decoding and temperatures 0.2–1.0, and reduced strong positional-choice biases that favored Position 1 under clean conditions (Seip et al., 9 Apr 2026).

Ablations show that the attack depends on the attention-centric formulation rather than on a generic adversarial perturbation. On Qwen3-VL, removing trajectory context reduced SSR from 97.7 to 82.2 in ReAct-F and from 92.8 to 64.7 in Action; targeting only the 2nd-to-last layer reduced Action SSR from 92.8 to 62.0; removing $\pi_i^4(\alpha)$4 selection reduced Action SSR from 92.8 to 75.3; removing active-head selection reduced it to 87.4; and omitting warm-start reduced it to 69.2 (Seip et al., 9 Apr 2026).

Empirical defenses tested in the paper were ineffective. Instruction Hierarchy system prompting left SSR high, including 93.7% on Qwen3-VL and 97.0% on Kimi-VL, and self-reflection prompting also failed to reliably reduce SSR, including 75.2% on Qwen3-VL and 88.5% on Kimi-VL. The authors therefore recommend attention regularization, adversarial training for attention, saliency auditing, lightweight input sanitization, UI design hardening, policy-level cross-checking, and multi-view validation, while noting the utility trade-offs of each (Seip et al., 9 Apr 2026).

4. State-of-mind hijacking in phishing and security behavior

In "Many Phish in the $\pi_i^4(\alpha)$5: A Coexisting-Choice-Criteria Model of Security Behavior" (Embrey et al., 2018), Phi is formalized as strategic manipulation of the State-of-Mind distribution over coexisting choice criteria. The attainable criteria are

$\pi_i^4(\alpha)$6

and the phishing-security instantiation uses four criteria: $\pi_i^4(\alpha)$7 normative deliberation, $\pi_i^4(\alpha)$8 behavioral deliberation, $\pi_i^4(\alpha)$9 impulsively click through, and $\mathbf{h}$0 routinely click straight through. Each email encounter is governed by exactly one criterion drawn from a context- and framing-conditional SoM distribution (Embrey et al., 2018).

For individual $\mathbf{h}$1 at time $\mathbf{h}$2,

$\mathbf{h}$3

where $\mathbf{h}$4 denotes current characteristics of the decision maker, $\mathbf{h}$5 the situational context, and $\mathbf{h}$6 the attributes of the present choice task. A simple separable match-quality function is

$\mathbf{h}$7

with selection probabilities

$\mathbf{h}$8

Phi is the attacker’s choice of $\mathbf{h}$9 to push probability mass onto vulnerable criteria, principally $\mathbf{x}$0 through peripheral-route cues and urgency or fear, or $\mathbf{x}$1 through spear-phishing cues that mimic organizational workflow and insider context (Embrey et al., 2018).

Conditional on criterion $\mathbf{x}$2, the user clicks with probability $\mathbf{x}$3. The paper’s figure encodes $\mathbf{x}$4, whereas $\mathbf{x}$5 and $\mathbf{x}$6 are well below 1. This yields the conditional breach decomposition

$\mathbf{x}$7

and over multiple emails,

$\mathbf{x}$8

The conceptual claim is that Phi operates upstream of Signal Detection Theory: rather than shifting a threshold within a fixed decision process, it changes which process is invoked at all (Embrey et al., 2018).

The same framework explains the APT “stepping-stone” pattern. Because realistic spoofing of routine workflows requires insider information, the first campaign wave optimally targets $\mathbf{x}$9, leading to broad non-target compromise. Once insider information is obtained, the attacker can switch to maximizing $\mathbf{x}_h = \mathbf{x} + \mathbf{h}$0, and the paper gives the dominance condition

$\mathbf{x}_h = \mathbf{x} + \mathbf{h}$1

This produces the observed progression from generic impulsive compromise to routine workflow compromise of high-value targets (Embrey et al., 2018).

The model has direct implications for evaluation and mitigation. Valid tests of training must sample different ploy types, be blinded and unannounced, and measure changes in $\mathbf{x}_h = \mathbf{x} + \mathbf{h}$2 as well as $\mathbf{x}_h = \mathbf{x} + \mathbf{h}$3, not only overall click rates. Defensive levers include shifting SoM mass away from $\mathbf{x}_h = \mathbf{x} + \mathbf{h}$4 and $\mathbf{x}_h = \mathbf{x} + \mathbf{h}$5, reducing time pressure and distraction, introducing mandatory verification steps to break $\mathbf{x}_h = \mathbf{x} + \mathbf{h}$6, and deploying interface warnings tuned to specific peripheral-route cues (Embrey et al., 2018).

5. Preference poisoning and policy teaching in RLHF and reward learning

In preference-learning systems, Phi commonly appears as poisoning of subjective comparison data. "Preference Poisoning Attacks on Reward Model Learning" (Wu et al., 2024) studies reward models trained from pairwise comparisons under the Bradley–Terry random-utility model. An attacker who can flip a small subset of labels can promote or demote a target outcome by hijacking the learned ordering. The paper develops gradient-based attacks via implicit differentiation and several rank-by-distance (RBD) variants, and reports that in the most extreme case the best attacks achieve 100% success rate with only 0.3% of the data poisoned. In the RLHF-style safety-alignment setting with PKU-SafeRLHF-30K harmlessness pairs, both promotion and demotion reached 100% success with only 0.3% poisoned labels, while spectral anomaly detection, loss-based trimming, Meta-Sift, and ALIBI showed limited efficacy, especially in the LLM setting (Wu et al., 2024).

The paper’s formulation is explicitly bilevel. Reward learning minimizes pairwise logistic loss, and the attacker chooses flips to maximize target ranking success. Gradient attacks relax the discrete flip variables and differentiate through the MLE solution using

$\mathbf{x}_h = \mathbf{x} + \mathbf{h}$7

RBD instead ranks pairs by distance to a target in input space, reward space, or embedding space. A notable empirical finding is that the simpler and more scalable RBD methods are often competitive with, and sometimes significantly outperform, gradient-based methods in high-dimensional domains (Wu et al., 2024).

"Policy Teaching via Data Poisoning in Learning from Human Preferences" (Nika et al., 13 Mar 2025) gives a general theoretical formulation of Phi as enforcing a target policy $\mathbf{x}_h = \mathbf{x} + \mathbf{h}$8 through poisoned preference data. The attack problem is

$\mathbf{x}_h = \mathbf{x} + \mathbf{h}$9

The paper derives lower and upper bounds for RLHF and DPO in augmentation and synthesis regimes. In the DPO synthesis regime, the upper and lower bounds match. The comparative result in Theorem 6.1 states that, as the target $\mathbf{q}$0 moves farther from the reference $\mathbf{q}$1 in parameter space, $\mathbf{q}$2 grows, implying that DPO requires more poisoned samples than RLHF to enforce $\mathbf{q}$3 (Nika et al., 13 Mar 2025).

"Efficient Preference Poisoning Attack on Offline RLHF" (Yang et al., 4 May 2026) sharpens this picture for log-linear DPO. Its key observation is that flipping one preference label induces a parameter-independent gradient shift:

$\mathbf{q}$4

where $\mathbf{q}$5. This converts targeted poisoning into a binary sparse approximation problem,

$\mathbf{q}$6

with binary flip vector $\mathbf{q}$7. The paper proposes Binary-Aware Lattice Attack (BAL-A) and Binary Matching Pursuit Attack (BMP-A), gives coherence-based recovery guarantees and impossibility certificates, and validates the geometry-driven analysis on synthetic dictionaries and the Stanford Human Preferences dataset. On SHP, BAL-A displayed a sharp success transition at the predicted separation threshold over penalty $\mathbf{q}$8, while BMP-A performed substantially better on low-coherence subsets than on random subsets (Yang et al., 4 May 2026).

Taken together, these papers place Phi in RLHF on a spectrum. At one end are empirical label-flip attacks that exploit subjective, anonymous comparisons. At the other are exact or approximate teaching attacks that synthesize preference datasets to drive a learner toward a chosen policy. A plausible implication is that the vulnerability is rooted less in any single optimizer than in the combination of subjective labels, concentrated influence, and the geometric structure of pairwise preference data (Wu et al., 2024, Nika et al., 13 Mar 2025, Yang et al., 4 May 2026).

6. Benign DPO attacks, preference hacking, and mitigation

"Few-Shot Truly Benign DPO Attack for Jailbreaking LLMs" (Yoon et al., 9 May 2026) identifies a distinct Phi mechanism: optimizing DPO on harmless preference pairs where a normal helpful answer is always preferred over a refusal causes broad refusal suppression that transfers to unseen harmful prompts. The paper uses the standard DPO loss

$\mathbf{q}$9

with $\|\mathbf{h}\|_\infty \leq \Delta$0 and $\|\mathbf{h}\|_\infty \leq \Delta$1. The paper argues that, because the preferred helpful completions are already likely under the reference model, the easiest way to increase the DPO margin is to reduce $\|\mathbf{h}\|_\infty \leq \Delta$2 for refusals. The refusal-side gradient is

$\|\mathbf{h}\|_\infty \leq \Delta$3

when $\|\mathbf{h}\|_\infty \leq \Delta$4 is not yet large, so gradient descent decreases refusal likelihood (Yoon et al., 9 May 2026).

The attack uses only 10 benign preference pairs, the minimum accepted by OpenAI’s preference fine-tuning service. On OpenAI models, it achieved 59.13% ASR on GPT-4o, 70.20% on GPT-4.1, 54.80% on GPT-4.1-mini, and 81.73% on GPT-4.1-nano, at approximate costs of \$\|\mathbf{h}\|_\infty \leq \Delta$51.7, \$\|\mathbf{h}\|_\infty \leq \Delta$60.1. On open-weight models, the effect appeared with 5 examples and even a single preference pair. In a realistic over-refusal reduction setting using safe prompts from XSTest that the base model mistakenly refused, DPO fine-tuning reduced over-refusal from 8.80% to 0.40% but caused ASR on harmful prompts to jump from 11.33% to 87.33% on Llama-3.1 8B (Yoon et al., 9 May 2026).

Auditing is unusually difficult. OpenAI Moderation API flagged none of the methods. Strong LLM auditors detected AOA, NOICE, and TenBenign at high rates, but the “truly benign” DPO pairs had 0% detection by GPT-5-mini, GPT-5.5, DeepSeek V4 Pro, and Gemini 3.1 Pro, with only Claude Opus 4.7 showing weak nonzero detection at 8% ± 7.9. The paper therefore advocates safety-aware preference construction, safety-conditioned DPO, preference masking for refusal tokens, mixed-objective training, post-hoc alignment passes, inference-time guardrails, and behavioral audits rather than content-only screening (Yoon et al., 9 May 2026).

A different mitigation line appears in "Mitigating Preference Hacking in Policy Optimization with Pessimism" (Gupta et al., 10 Mar 2025). That paper uses the terms “overoptimization,” “reward hacking,” and “preference hacking,” but the failure mode is closely related: learned preference models are unreliable off-support, so policy optimization exploits spurious high-scoring regions. The proposed remedy is a pessimistic objective over an uncertainty set of plausible preference functions, restricted to covered policies. For the restricted pessimistic Nash solution $\|\mathbf{h}\|_\infty \leq \Delta$7, the paper gives the guarantee

$\|\mathbf{h}\|_\infty \leq \Delta$8

for any covered policy $\|\mathbf{h}\|_\infty \leq \Delta$9, under bounded version-space diameter and coverage assumptions. The practical algorithms are P3O and PRPO, which were evaluated on document summarization and helpful assistants and were reported to be resilient to length and list-format hacks that affected DPO, REINFORCE, and Nash-EMA (Gupta et al., 10 Mar 2025).

Across domains, defense results are mixed. Inference-time image preprocessing reduces multimodal Phi success but degrades image fidelity and can be bypassed by adaptive training (Lan et al., 15 Sep 2025). Prompt-level controls such as Instruction Hierarchy or self-reflection do not reliably stop attention-based GUI redirection (Seip et al., 9 Apr 2026). Dataset-level defenses such as spectral filtering, loss-based trimming, and ALIBI are limited against targeted preference poisoning, especially in LLM reward modeling (Wu et al., 2024). This suggests that Phi is best understood as a first-class security problem of preference formation itself, rather than as a special case of toxicity filtering, prompt injection, or ordinary adversarial classification.