Multimodal Mixture-of-Bayesian Prompt Ensembles

• The method employs a cluster-conditioned Bayesian weighting scheme to calibrate multimodal text-to-image judgments.
• It integrates multiple semantically equivalent prompts to mitigate bias, overconfidence, and domain-specific variations.
• Empirical evaluations show that MMB improves F1 scores and reduces calibration errors compared to single and averaged prompt baselines.

Multimodal Mixture-of-Bayesian Prompt Ensembles (MMB) denotes a multimodal-aware prompt ensembling method in which prompt-conditioned predictions are aggregated through a Bayesian mixture whose weights depend on visual structure in the input, rather than on a single global prompt prior. In its primary formulation, MMB is used to calibrate multimodal LLMs (MLLMs) acting as judges for text-to-image (TTI) systems: a judge receives a caption and a pair of images, returns prompt-conditioned probabilities $p(y\mid x,a)$, and combines them through cluster-conditioned prompt weights and image-dependent cluster responsibilities (Slyman et al., 10 Sep 2025). More broadly, the term is also used in adjacent work as a conceptual target for combining Bayesian prompt uncertainty, multimodal specialization, and prompt ensembles (Derakhshani et al., 2022, Bendou et al., 21 Nov 2025).

1. Problem setting and conceptual scope

The canonical MMB setting is MLLM-as-a-judge for TTI evaluation. The input $x$ comprises a caption and either one image or two images; the output is a judgment such as which image is preferred. The calibration target is explicit: for predicted label $\hat y$ and confidence $f(\hat y\mid x)=p$, calibration requires

$P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$

In this setting, calibration matters because high-confidence judgments may be automated, whereas low-confidence cases may be deferred (Slyman et al., 10 Sep 2025).

The motivation for MMB is the combination of bias, overconfidence, and domain dependence in multimodal judging. The judge may favor particular styles or generators, may be extremely confident when wrong, and may respond differently to small prompt changes across photographic, abstract, or synthetic image domains. Standard prompt ensembling methods from text-only settings assume that prompt quality is globally stable, but the TTI case violates that assumption because different prompts are better for different image types (Slyman et al., 10 Sep 2025).

In a narrower historical sense, MMB names the image-cluster-conditioned Bayesian ensemble introduced for TTI judgment calibration. In a broader interpretive sense, related work suggests an MMB family of methods in which prompts are treated as latent or sampled variables, multimodal inputs induce prompt specialization, and final predictions are formed by averaging or mixing over prompt-conditioned predictors rather than selecting a single prompt (Derakhshani et al., 2022, Bendou et al., 21 Nov 2025).

2. Bayesian formulation

MMB extends Bayesian Prompt Ensembles (BPE) by replacing a single global prompt distribution with an image-conditioned mixture over prompt distributions. Let $\boldsymbol a=\{a_1,\dots,a_N\}$ denote a discrete prompt set, and let $D=\{(x_j,y_j^*)\}_{j=1}^M$ denote validation data. In the BPE formulation, the prompt posterior is approximated by a variational distribution $q(a)$ with weights $w_a$, yielding the optimization problem

$$\operatorname*{arg\,max}_{\mathbf w} \sum_a w_a\Bigl[\sum_{j=1}^M \log p(y_j^*\mid x_j,a)\Bigr] - \sum_a w_a\log w_a.$$

The first term favors prompts with high validation likelihood; the second is an entropy regularizer induced by the variational objective under a uniform prior (Slyman et al., 10 Sep 2025).

MMB introduces a latent group variable $x$0 representing an image cluster. Conditional prompt distributions are then group specific, with weights $x$1. Under the assumptions $x$2 and $x$3, and with uniform $x$4, the objective becomes

$x$5

Each sample contributes to every group proportionally to its soft cluster assignment $x$6, so the model learns cluster-specific prompt reliabilities rather than a single global prompt ranking (Slyman et al., 10 Sep 2025).

Inference is the corresponding nested mixture: $x$7 The outer mixture is over image groups, and the inner mixture is over prompts within each group. In effect, MMB is a mixture of Bayesian prompt ensembles rather than a flat prompt average (Slyman et al., 10 Sep 2025).

This Bayesian structure has clear antecedents. “Bayesian Prompt Learning for Image-LLM Generalization” models a residual prompt as a latent variable with Gaussian prior and optimizes an ELBO, with inference performed by averaging over samples from the learned prompt posterior (Derakhshani et al., 2022). “Repulsive Bayesian Prompt Learning” recasts multimodal prompt tuning as posterior sampling with SGHMC and averages over prompt samples via

$x$8

which makes the ensemble interpretation explicit (Bendou et al., 21 Nov 2025). These formulations do not define the cluster-conditioned TTI judge of MMB, but they supply the probabilistic prompt machinery on which MMB naturally builds.

3. Multimodal conditioning and mixture mechanism

The multimodal component of MMB is implemented through image embeddings and soft clustering. The image encoder $x$9 is CLIP ViT-B/16 with laion2b_s34b_b88k weights. A large unlabeled image set from the generators under evaluation is embedded and clustered with spherical k-means using FAISS. The distance is

$\hat y$0

and the centroids are $\hat y$1 (Slyman et al., 10 Sep 2025).

Soft group assignment is defined by

$\hat y$2

with cosine similarity and temperature $\hat y$3. As $\hat y$4, assignments become hard; as $\hat y$5, they become uniform, which effectively collapses the model toward a global BPE-like ensemble (Slyman et al., 10 Sep 2025).

The prompt pool contains 100 semantically equivalent prompts. The prompts vary by persona and rubric, including formulations such as “landscape artist,” “graphic designer,” “photographer,” “art historian,” and “AI ethics specialist,” and also by ordering and formatting variants. For each prompt $\hat y$6 and each input $\hat y$7, GPT-4o returns a preference distribution $\hat y$8 over the two candidate images. Those prompt-conditioned probabilities are the basic likelihood terms in the MMB objective (Slyman et al., 10 Sep 2025).

Optimization of the cluster-specific weights $\hat y$9 is performed with L-BFGS using learning rate $f(\hat y\mid x)=p$0, history size $f(\hat y\mid x)=p$1, at most $f(\hat y\mid x)=p$2 iterations, and strong Wolfe line search. Clustering uses $f(\hat y\mid x)=p$3 samples, $f(\hat y\mid x)=p$4 random initializations, and $f(\hat y\mid x)=p$5 iterations. These steps are offline with respect to the black-box judge: MMB changes prompt weighting, not model parameters (Slyman et al., 10 Sep 2025).

4. Empirical behavior in TTI judgment calibration

The principal evaluation uses HPSv2 and MJBench-Bias. On HPSv2, the test set contains $f(\hat y\mid x)=p$6 groups of $f(\hat y\mid x)=p$7 images, yielding $f(\hat y\mid x)=p$8 pairwise comparisons. Metrics include Expected Calibration Error (ECE), Max Calibration Error (MCE), negative log-likelihood (NLL), Brier score, Cohen’s Kappa, Accuracy, F1, ROC-AUC, and AUC-PR (Slyman et al., 10 Sep 2025).

In the representative HPSv2 setting with $f(\hat y\mid x)=p$9 prompts and $P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$0 validation samples, the single random prompt baseline (“Std. (single)”) yields NLL $P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$1, Brier $P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$2, Kappa $P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$3, Acc $P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$4, ROC $P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$5, and F1 $P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$6. The “Best (single)” prompt yields $P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$7. The unweighted average ensemble yields $P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$8. BPE yields $P\bigl(y^*=\hat y \mid f(\hat y\mid x)=p\bigr)=p.$9. MMB yields the best values on all six reported metrics: NLL $\boldsymbol a=\{a_1,\dots,a_N\}$0, Brier $\boldsymbol a=\{a_1,\dots,a_N\}$1, Kappa $\boldsymbol a=\{a_1,\dots,a_N\}$2, Acc $\boldsymbol a=\{a_1,\dots,a_N\}$3, ROC $\boldsymbol a=\{a_1,\dots,a_N\}$4, and F1 $\boldsymbol a=\{a_1,\dots,a_N\}$5 (Slyman et al., 10 Sep 2025).

The calibration advantage is similarly explicit in ECE. In the same $\boldsymbol a=\{a_1,\dots,a_N\}$6-prompt, $\boldsymbol a=\{a_1,\dots,a_N\}$7-sample setting, ECE is $\boldsymbol a=\{a_1,\dots,a_N\}$8 for the random single prompt, $\boldsymbol a=\{a_1,\dots,a_N\}$9 for the best single prompt, $D=\{(x_j,y_j^*)\}_{j=1}^M$0 for simple averaging, $D=\{(x_j,y_j^*)\}_{j=1}^M$1 for BPE, and $D=\{(x_j,y_j^*)\}_{j=1}^M$2 for MMB (Slyman et al., 10 Sep 2025). The paper’s teaser reports that MMB improves F1 by $D=\{(x_j,y_j^*)\}_{j=1}^M$3 relative to Avg. and reduces ECE by $D=\{(x_j,y_j^*)\}_{j=1}^M$4 relative to Avg., with all differences significant at $D=\{(x_j,y_j^*)\}_{j=1}^M$5 under permutation testing (Slyman et al., 10 Sep 2025).

On MJBench-Bias, the test pairs are deliberately constructed so that no systematic preference should exist, making overconfidence the primary failure mode. In an example configuration with $D=\{(x_j,y_j^*)\}_{j=1}^M$6 prompts and $D=\{(x_j,y_j^*)\}_{j=1}^M$7 samples, average confidence is $D=\{(x_j,y_j^*)\}_{j=1}^M$8 for the random single prompt, $D=\{(x_j,y_j^*)\}_{j=1}^M$9 for the best single prompt, $q(a)$0 for Avg., $q(a)$1 for BPE, and $q(a)$2 for MMB. Lower is better in this evaluation, so Avg. is most cautious, while MMB is markedly less overconfident than BPE or the single-prompt baselines while retaining stronger HPSv2 discrimination (Slyman et al., 10 Sep 2025).

Cluster count ablations indicate that MMB is not highly sensitive to moderate variation in $q(a)$3. With $q(a)$4 prompts and $q(a)$5 validation samples on HPSv2, $q(a)$6 gives ECE $q(a)$7, NLL $q(a)$8, Brier $q(a)$9, and F1 $w_a$0; $w_a$1 gives $w_a$2; $w_a$3 gives $w_a$4; $w_a$5 and $w_a$6 remain essentially unchanged. The reported interpretation is mild improvement from $w_a$7 to $w_a$8, followed by saturation (Slyman et al., 10 Sep 2025).

Qualitative cluster analysis shows prompt specialization aligned with visual content. Pastel landscapes weight “landscape artist” most highly; sci-fi and galaxy renders weight “graphic designer”; photographic sheep images weight “photographer”; some human-centered clusters assign highest weight to an “AI ethics specialist” persona. The paper also notes non-cohesive clusters whose prompt weights remain near uniform (Slyman et al., 10 Sep 2025).

5. Antecedents and adjacent research lines

Several neighboring lines of work supply the components from which an MMB-style system can be understood.

Research line	Main mechanism	Relation to MMB
Bayesian prompt distributions (Derakhshani et al., 2022)	Residual prompt latent variable, Gaussian prior, ELBO, Monte Carlo prompt averaging	Provides the Bayesian prompt posterior and sample-based ensemble core
Repulsive Bayesian prompt learning (Bendou et al., 21 Nov 2025)	Cyclical SGHMC with representation-space repulsion and uniform averaging over samples	Makes multimodal prompt posteriors and mode diversity explicit
Multimodal prompt experts (Wu et al., 2024)	V-Prompt, L-Prompt, VL-Prompt with Block-Aware Prompt Fusion	Supplies modality-specific and unified prompt specialists in a unified VLM
Dynamic prompt experts for fusion (Jiang et al., 2024)	Prompt experts routed by multimodal pairing priors	Shows instance-wise routing over multimodal prompt experts
Multimodal prompt optimization (Choi et al., 10 Oct 2025)	Text-plus-non-text prompt pairs with prior-inherited Bayesian UCB	Provides Bayesian selection over multimodal prompts rather than prompt parameters
Multi-prompt decoding (Heineman et al., 2024)	Prompt bank plus MBR over pooled candidates	Treats prompts as an ensemble dimension at inference
Mixture of Prompts for LLM adaptation (Dun et al., 2023)	Prompt experts and smart gating under task heterogeneity	Shows mixture-of-prompts as a heterogeneity-management mechanism

Taken together, these works suggest that MMB is best understood not as a single isolated trick but as the intersection of three technical commitments. The first is Bayesian treatment of prompts, either through variational posteriors over prompt embeddings (Derakhshani et al., 2022) or sampled prompt posteriors with ensemble averaging (Bendou et al., 21 Nov 2025). The second is multimodal specialization, implemented either as explicit text/image/unified prompt experts in a unified vision-language backbone (Wu et al., 2024) or as dynamic prompt experts routed by multimodal pairing priors (Jiang et al., 2024). The third is prompt-level mixture or ensemble inference, whether via prompt banks in MBR decoding (Heineman et al., 2024), Bayesian UCB over multimodal prompt pairs (Choi et al., 10 Oct 2025), or cluster-conditioned weighting in MMB itself (Slyman et al., 10 Sep 2025).

A plausible implication is that “MMB” can denote both a concrete TTI judge-calibration method and a broader design pattern: multimodal inputs induce prompt specialization, Bayesian structure regularizes or samples prompt uncertainty, and final prediction averages across prompt-conditioned hypotheses rather than trusting a single prompt. That interpretation is stated directly or indirectly in several of the related papers’ “design insights for MMB” sections (Wu et al., 2024, Bendou et al., 21 Nov 2025).

6. Limitations and open directions

The primary MMB paper identifies three direct limitations. First, performance depends on clustering hyperparameters such as $w_a$9 and $$\operatorname*{arg\,max}_{\mathbf w} \sum_a w_a\Bigl[\sum_{j=1}^M \log p(y_j^*\mid x_j,a)\Bigr] - \sum_a w_a\log w_a.$$0. Second, MMB adds computation through CLIP embedding and k-means clustering, even though these steps are offline. Third, the method remains black-box with respect to the judge: it only changes prompt weighting and cannot correct intrinsic model biases in GPT-4o or any other base MLLM (Slyman et al., 10 Sep 2025).

The same paper proposes several extensions. One is to move beyond explicit clustering toward a learned image-conditioned mapping from $$\operatorname*{arg\,max}_{\mathbf w} \sum_a w_a\Bigl[\sum_{j=1}^M \log p(y_j^*\mid x_j,a)\Bigr] - \sum_a w_a\log w_a.$$1 directly to prompt weights, which would make the mixture more like a conventional mixture-of-experts. Others include applying the framework to VQA, content moderation, and ordinal scoring, and combining it with improved prompt design such as chain-of-thought prompting or RLHF-style judge prompt optimization (Slyman et al., 10 Sep 2025).

Related Bayesian prompt-learning work points to deeper unresolved questions. “Bayesian Prompt Learning for Image-LLM Generalization” uses a single Gaussian posterior and explicitly notes richer alternatives such as normalizing flows, energy-based models, and diffusion models as prompt distributions (Derakhshani et al., 2022). “Repulsive Bayesian Prompt Learning” notes that posterior samples are used with fixed $$\operatorname*{arg\,max}_{\mathbf w} \sum_a w_a\Bigl[\sum_{j=1}^M \log p(y_j^*\mid x_j,a)\Bigr] - \sum_a w_a\log w_a.$$2 and without explicit mode segmentation, and sketches extensions based on clustering posterior samples into modes and learning mixture weights (Bendou et al., 21 Nov 2025). This suggests that a next-generation MMB could replace cluster-conditioned prompt weights learned from validation likelihood with a more explicit posterior over multimodal prompt modes.

A final open issue concerns the location of uncertainty. In the TTI judge formulation, uncertainty enters through prompt-conditioned prediction variance and cluster-conditioned averaging. In multimodal prompt-expert systems such as MoPE-BAF and MoPE, the uncertainty is still implicit in routing or attention rather than explicitly Bayesian (Wu et al., 2024, Jiang et al., 2024). A plausible implication is that future MMB variants will need to decide whether uncertainty should live in prompt parameters, routing distributions, cluster assignments, or all three simultaneously.