Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs (2510.09201v1)

Abstract: LLMs have shown remarkable success, and their multimodal expansions (MLLMs) further unlock capabilities spanning images, videos, and other modalities beyond text. However, despite this shift, prompt optimization approaches, designed to reduce the burden of manual prompt crafting while maximizing performance, remain confined to text, ultimately limiting the full potential of MLLMs. Motivated by this gap, we introduce the new problem of multimodal prompt optimization, which expands the prior definition of prompt optimization to the multimodal space defined by the pairs of textual and non-textual prompts. To tackle this problem, we then propose the Multimodal Prompt Optimizer (MPO), a unified framework that not only performs the joint optimization of multimodal prompts through alignment-preserving updates but also guides the selection process of candidate prompts by leveraging earlier evaluations as priors in a Bayesian-based selection strategy. Through extensive experiments across diverse modalities that go beyond text, such as images, videos, and even molecules, we demonstrate that MPO outperforms leading text-only optimization methods, establishing multimodal prompt optimization as a crucial step to realizing the potential of MLLMs.

• The paper introduces multimodal prompt optimization that jointly refines textual and non-textual prompts to achieve significant performance gains.
• It employs alignment-preserving exploration and prior-inherited Bayesian UCB selection to efficiently navigate the multimodal search space with reduced evaluation costs.
• Experimental results across 10 datasets demonstrate superior generalizability and robust improvements over text-only methods.

Multimodal Prompt Optimization for MLLMs: Expanding the Search Space Beyond Text

Motivation and Problem Definition

The paper introduces the problem of multimodal prompt optimization (MPO), addressing a critical limitation in current prompt optimization paradigms for Multimodal LLMs (MLLMs). While MLLMs are architecturally capable of ingesting and reasoning over diverse modalities (text, images, video, molecules), existing automatic prompt optimization (APO) methods remain confined to the textual domain. This restriction underutilizes the expressive capacity of MLLMs, as non-textual signals often provide richer, more direct context for many tasks. The authors formalize the multimodal prompt as a tuple $$(\bm{t}, \bm{m}) \in \mathcal{T} \times \mathcal{M}$$, where $\bm{t}$ is the textual prompt and $\bm{m}$ is the non-textual prompt, and define the optimization objective as maximizing expected task performance over this joint space.

Figure 1: Existing prompt optimization approaches are limited to text, whereas multimodal prompt optimization leverages both textual and non-textual context for MLLMs.

The Multimodal Prompt Optimizer (MPO) Framework

MPO is a unified framework for joint optimization of multimodal prompts, consisting of two principal components:

1. Alignment-Preserving Exploration: Updates textual and non-textual prompts in a coupled manner, guided by a single semantic feedback signal derived from failure analysis. This ensures cross-modal consistency and avoids semantic misalignment between modalities.
2. Prior-Inherited Bayesian UCB Selection: Utilizes the performance of parent prompts as informative priors to warm-start the candidate selection process, mitigating the cold-start inefficiency of standard bandit-based or uniform allocation strategies.

   Figure 2: MPO consists of alignment-preserving exploration for prompt refinement and prior-inherited Bayesian UCB for efficient candidate selection.

Alignment-Preserving Exploration

The exploration module operates by:

• Cohesive Backpropagation: Analyzing a failure set $\mathcal{F}$ to generate unified feedback $\nabla_{\bm{p}}$ that encodes cross-modal weaknesses.
• Joint Multimodal Update: Using the feedback to simultaneously refine the textual prompt and generate modality-specific conditions for non-textual prompt revision via a generator $g$ (e.g., text-to-image, text-to-molecule).
• Exploration Operators: Three operators—generation, edit, and mix—systematically expand, refine, and recombine non-textual prompts, enabling broad and expressive search in the multimodal space.

Prior-Inherited Bayesian UCB Selection

The selection strategy models each candidate's expected score as a Beta distribution, initializing child prompts' priors from their parent's posterior mean. This warm-start mechanism leverages the empirically strong parent-child score correlation (Pearson $r=0.88$), reducing wasted evaluations on low-potential candidates and accelerating best-arm identification. Theoretical analysis guarantees non-increasing identification cost under informative priors.

Experimental Results

MPO is benchmarked across 10 datasets spanning image, video, and molecular modalities, and compared against manual prompting, few-shot prompting, and leading text-only APO baselines (APE, OPRO, EvoPrompt, PE2, ProTeGi, SEE). The results demonstrate:

• Consistent and significant performance gains over all baselines in every modality.
• Superior generalizability across diverse backbone models (Qwen2.5-VL, Gemma3, InternVL-3.5, GPT-4.1 nano), optimizer models, and modality-specific generators, including lightweight open-source variants.

  Figure 3: MPO generalizes across base models, optimizer models, and modality-specific generators; cross-modal alignment correlates strongly with performance gain.

Cross-Modal Alignment and Modality Synergy

Ablation studies confirm that joint optimization of textual and non-textual components yields higher alignment scores and larger performance gains than sequential or random prompt combinations. The full multimodal prompt outperforms single-modality ablations, indicating a synergistic effect rather than mere additive improvement.

Operator Contributions and Qualitative Analysis

Each exploration operator (generation, edit, mix) contributes distinct improvements: generation explores novel regions, edit fine-tunes local features, and mix synthesizes complementary strengths. Combining all three operators within MPO achieves the highest accuracy.

Figure 4: The image prompt optimization process for CUB bird species classification demonstrates the role of each operator in refining multimodal context.

Selection Efficiency and Training Dynamics

MPO's prior-inherited Bayesian UCB achieves target performance with only 30% of the evaluation budget required by uniform allocation, and 42–70% less than standard UCB or prior-free variants. Training curves show that MPO continues to improve beyond the plateau of text-only methods, escaping local optima and approaching the global optimum in the multimodal space.

Figure 5: Prior-inherited Bayesian UCB selection strategy yields substantial evaluation budget savings compared to alternatives.

Figure 6: MPO exhibits sustained improvement over ProTeGi during optimization on CUB.

Hidden State Analysis

PCA visualization of MLLM hidden states reveals that multimodal prompts shift model representations into distinct regions, enabling richer reasoning pathways inaccessible to text-only prompts.

Computational and Resource Considerations

MPO incurs a marginal increase in computational cost due to modality-specific generator calls, but this is manageable with lightweight generators. The performance improvement is unattainable by text-only methods, justifying the additional expense for practical deployment.

Implications and Future Directions

The introduction of multimodal prompt optimization marks a paradigm shift in leveraging MLLMs for real-world tasks. By expanding the search space and coupling modalities, MPO unlocks the full expressive capacity of MLLMs, enabling superior contextual grounding and reasoning. The framework is robust to backbone and generator variations, and its efficiency in candidate selection makes it scalable to large combinatorial spaces.

Theoretically, the work establishes the importance of cross-modal alignment and informative priors in optimization over multimodal domains. Practically, it provides a blueprint for automatic prompt engineering in applications ranging from medical imaging to drug discovery.

Future research may explore:

• Extension to additional modalities (audio, sensor data)
• Integration with gradient-based soft prompt tuning for hybrid discrete-continuous optimization
• Automated discovery of optimal operator scheduling and meta-prompting strategies
• Application to agentic and interactive MLLM systems requiring dynamic multimodal context adaptation

Conclusion

This work formalizes and solves the problem of multimodal prompt optimization for MLLMs, demonstrating that joint optimization of textual and non-textual prompts, coupled with efficient selection via prior-inherited Bayesian UCB, yields substantial and consistent improvements over text-only methods. The framework is generalizable, efficient, and establishes multimodal prompt optimization as a key direction for advancing the capabilities and practical utility of MLLMs.