Cross-Modal Preference Steering

• CPS is a framework that combines preference optimization and adversarial manipulation to align AI model outputs with human and system-defined goals across multiple modalities.
• It employs techniques such as nonlinear mapping, hierarchical supervision, and representation-level steering to capture complex cross-modal relationships and mitigate biases.
• Empirical results indicate that CPS enhances retrieval accuracy, reduces hallucinations, and boosts content selection efficiency in multimodal intelligence applications.

Cross-Modal Preference Steering (CPS) refers to strategies, architectures, and algorithms designed to steer, align, or control model outputs and decisions based on human or system preferences that span multiple modalities—commonly vision, language, audio, and interaction. CPS encompasses both preference optimization (enhancing alignment with desired outcomes or judgments) and adversarial manipulation (exploiting biases or vulnerabilities in multimodal AI agents). The field advances multimodal intelligence systems by integrating preference reasoning, cross-modal representation learning, hierarchical supervision, and model steering at various granularity levels. Recent research demonstrates that CPS is central to improving alignment, robustness, transparency, and safety in vision-LLMs, multimodal LLMs, and real-world agentic systems.

1. Foundations and Motivations

The heterogeneity of data modalities (e.g., images, texts) creates unique challenges in information retrieval, decision-making, and preference modeling. Early studies approached cross-modal tasks as ranking problems, learning shared embedding spaces to measure similarities between modalities. Much of this work focused on aligning representations linearly, which limits the capacity to capture complex inter-modal relationships (Luo et al., 2017). Cross-Modal Preference Steering emerged from the necessity to accommodate intricate semantic correspondences and to improve generalization in multimedia retrieval, multimodal reasoning, and content selection tasks.

Motivations for CPS include:

• Achieving robust semantic alignment between modalities (image, text, audio).
• Mitigating hallucinations, context confusion, and bias in multimodal models.
• Enabling interpretable, scalable model control for alignment and safety.

2. Core Methodologies and Optimization Frameworks

Fundamental CPS methodologies span several key approaches:

Shared Embedding Spaces and Nonlinear Mapping

• Nonlinear mapping functions are used to embed heterogeneous features into a common space ℰ ⊆ ℝᵈ: For images, $h(x) = \sigma(W_1 x + b_1)$ and for text, $g(z) = \sigma(W_2 z + b_2)$, with cross-modal similarity measured via $S(x, z) = h(x)^{\top} g(z)$ (Luo et al., 2017).
• Nonlinear embeddings better model intricate inter-modal relationships compared to linear projections.

Preference Optimization Paradigms

• Direct Preference Optimization (DPO): Leverages preference pairs to optimize model outputs toward preferred responses. Losses typically take the form:

$$L(\theta) = -\log{\sigma\left( \frac{\log p_\theta(\text{preferred}) - \log p_\theta(\text{rejected})}{\beta} \right)}$$

• Bi-directional Preference Optimization (BiPO): Steers model activations using continuous vectors optimized over preference pairs, supporting both promotion and suppression of behaviors (Cao et al., 28 May 2024).
• Hierarchical Preference Optimization: Multi-level supervision across response-, segment-, and token-levels, enabling fine-grained correction of hallucinations and misalignment (Fu et al., 28 Jan 2025, Li et al., 28 May 2025, Fu et al., 1 Oct 2025).
• Self-Paced Learning with Diversity (SPLD): Gradually trains by selecting "easy" rankings from diverse queries, enforcing diversity to prevent overfitting and improve generalization (Luo et al., 2017).

Representation-Level Steering

• Modality preference is measurable in latent representations, and can be actively steered via activation addition in specific layers (e.g., $h' = h + s_\ell^t$) to amplify reliance on vision or text (Zhang et al., 27 May 2025).

3. Hierarchical and Multi-Faceted Preference Steering

Recent advances demonstrate that CPS benefits from hierarchical decomposition:

• Context-to-Cue Direct Preference Optimization (CcDPO): Integrates global context with fine-grained cues, using sequence- and region-level supervision to combat hallucinations in multi-image tasks (Li et al., 28 May 2025).
• Multi-faceted Cross-modal DPO (MCM-DPO): Aggregates losses over single, pairwise, and multi-modal dimensions ($$\mathcal{L}_\text{MCM-DPO} = \lambda\,\mathcal{L}_\text{single} + \alpha\,\mathcal{L}_\text{pair} + \gamma\,\mathcal{L}_\text{multi}$$), providing nuanced constraint for alignment (Fu et al., 1 Oct 2025).
• Hierarchical Token and Segment-Level Losses: Distinct loss objectives for tokens and segments establish granular control over alignment and hallucination reduction (Fu et al., 28 Jan 2025).

The use of large-scale preference datasets (e.g., MultiScope-42k, TAlt, PAlt) facilitates scalable multi-level optimization, yielding improved performance in both single- and multi-image settings.

4. CPS in Model Steering and Personalized Alignment

CPS encompasses a variety of interpretable, training-free, and plug-and-play steering techniques:

• Residual-Based Steering (PaLRS): Extracts steering vectors from mean differences in residual streams based on preference pairs, adding these vectors during inference for rapid, data-efficient alignment (Cava et al., 28 Sep 2025).
• Confident Direction Steering (CONFST): Uses classifier-selected confident activation vectors from user history to form steering directions, capable of aligning multiple user preferences and avoiding explicit instruction (Song et al., 4 Mar 2025).
• Feature Steering with Sparse Autoencoders (FSRL): Trains adapters to modulate interpretable SAE features, offering transparent control over style and alignment-related latent concepts (Ferrao et al., 16 Sep 2025).

These methods allow for flexible adjustment of model behavior without full retraining, supporting real-time personalization and efficient adaptation to evolving preference signals.

5. Cross-Modal Adversarial Manipulation

CPS also describes adversarial techniques aimed at manipulating agentic systems:

• Cross-Modal Content Optimization: Jointly optimizes visual and textual channels through imperceptible perturbations and RLHF-induced bias exploitation, using ensemble surrogate models and crop aggregation for robust black-box attacks (Jiang et al., 4 Oct 2025).
• AUV-Fusion: Integrates user-interaction-derived embeddings with visually plausible perturbations, injecting these into VAE diffusion models to steer recommendations in Visual-Aware Recommender Systems while maintaining stealth (Ling et al., 30 Jul 2025).

The effectiveness and low detectability of these attacks underscore vulnerabilities in VLM-based agentic applications and highlight the need for next-generation multi-modal defense strategies.

6. Applications and Empirical Results

CPS yields substantial empirical gains across several domains:

Task / Domain	Empirical Improvement / Outcome	Reference(s)
Multimedia Retrieval	+3–5% mAP with SPLD, fast convergence, higher robustness to query diversity	(Luo et al., 2017)
Visual Instruction Tuning	Surpasses Vicuna and LLaVA on MT-Bench, boosts MM-Vet and LLaVA-Bench scores, low alignment tax	(Li et al., 16 Feb 2024)
Hallucination Mitigation	>50% hallucination reduction on Object HalBench with hierarchical multi-modal optimization	(Fu et al., 28 Jan 2025)
Fine-Grained Retrieval	MAPLE narrows modality gap, improves Recall@1, excels on nuanced cross-modal benchmarks	(Zhao et al., 8 Jun 2025)
Agentic Content Selection	CPS raises target selection rate from 12.5% to 50–71% in web agent tasks, 70% lower detection	(Jiang et al., 4 Oct 2025)
Alt-text Generation	MCM-DPO boosts ROUGE-L (from 32.71 to 39.54) and CIDEr (from 157.73 to 207.98) on PAlt	(Fu et al., 1 Oct 2025)

Approaches such as hierarchical DPO, confident steering, and rDPO consistently outperform traditional fine-tuning and RLHF methods, demonstrating enhanced alignment, reduced hallucination rates, efficiency, and transparency.

7. Implications and Future Directions

The proliferation of CPS research leads to several implications:

• Safety and Robustness: As model alignment via RLHF and preference optimization grows more sophisticated, corresponding vulnerabilities in agentic systems are exposed, necessitating robust, multi-modal defense frameworks.
• Transparency and Diagnostics: Interpretable preference steering (via SAE features or compositional preference models) provides diagnostic capability to analyze and troubleshoot alignment artifacts, such as a preference for stylistic features over explicit alignment concepts (Ferrao et al., 16 Sep 2025, Go et al., 2023).
• Generalization and Scalability: Methods leveraging modularity, hierarchical optimization, and plug-and-play steering demonstrate scalable generalization to new modalities, tasks, and user populations.
• Cross-Modal Extension: CPS patterns are applicable to diverse domains—robotics (with cross-modality attention for skill segmentation (Jiang et al., 20 Apr 2025)), interactive retrieval, personalized dialogue, and machine translation.

A plausible implication is that future research may focus on dynamic, real-time CPS systems capable of both defending against multimodal adversaries and adapting to complex, evolving user preferences. Further integration of interpretable representation learning, reinforcement learning adapters, and multi-level supervision is expected to drive advances in safe, nuanced, and transparent cross-modal intelligence.