Bimodal Bias Merging (BBM) Overview

• Bimodal Bias Merging (BBM) is a technique that merges two distinct bias modes—via model checkpoints in LLMs or spatial maps in medical imaging—to mitigate bias.
• It employs various interpolation methods such as SLERP, Linear, and Nearswap to balance bias reduction with the retention of downstream utility or localization accuracy.
• The approach reveals trade-offs where parameter alignment and interpolation strength are critical for optimizing bias mitigation without overly degrading model performance.

Bimodal Bias Merging (BBM) is a term used in two distinct 2025 research settings. In one usage, BBM denotes a parameter-space procedure for merging two language-model checkpoints that embody different “bias modes” in order to attenuate social bias without unduly harming general capability (Shirafuji et al., 2 Dec 2025). In the other, BBM denotes a post-processing method for phrase grounding in medical vision-language diffusion models that aligns and merges image bias and text bias maps with a phrase activation map to increase localization certainty (Nützel et al., 16 Jul 2025). The shared terminology reflects a common operation—merging structured biases or priors—but the objects being merged, the mathematical operators, and the evaluation criteria differ substantially.

1. Dual usage and definitional scope

The two published meanings of BBM can be summarized as follows.

Setting	What is merged	Primary objective
Social bias mitigation in LLMs	A pre-trained base model and a bias-inverted counterpart	Reduce bias while preserving downstream performance
Medical phrase grounding	Image bias, text bias, and the original phrase activation map	Refine localization by enhancing jointly supported regions

In the social-bias setting, BBM is instantiated as merging a pre-trained base model with its bias-inverted counterpart, constructed via task arithmetic; the reported findings are stated to transfer directly to merging any two models with opposing biases or a base–debiased pair (Shirafuji et al., 2 Dec 2025). In the medical-grounding setting, BBM is explicitly a post-processing step applied after token-level cross-attention maps have been extracted from a Stable Diffusion v1.5 U-Net conditioned by a frozen CXR-BERT text encoder (Nützel et al., 16 Jul 2025).

A common misconception is that BBM names a single standardized algorithm. The literature here shows the opposite: the acronym labels two method families with different mathematical objects. In one case, the relevant variables are flattened model parameters $w_A, w_B \in \mathbb{R}^d$ and interpolation weights $t$ or $\alpha$; in the other, the relevant variables are spatial maps such as $P_{\mathrm{comb}}(p)$, $B_{\mathrm{image}}(p)$, and $B_{\mathrm{text}}(p)$ defined on a $64 \times 64$ latent grid.

2. BBM for social bias mitigation via checkpoint merging

In the model-merging survey, BBM is defined as merging two checkpoints that embody different “bias modes” in order to attenuate social bias without unduly harming general capability (Shirafuji et al., 2 Dec 2025). The paper constructs a bias-inverted model using continual pretraining on biased data to amplify bias, then subtracts that “bias vector” from the base:

$$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$

Before merging, $\theta_{\mathrm{bias}}^{\mathrm{inv}}$ is normalized to have the same global norm as $\theta_{\mathrm{LLM}}$ so that the scaling hyperparameter remains interpretable. The merged parameters are then defined by

$t$0

with $t$1 in the study, and $t$2 denoting one of seven merging operators.

The seven surveyed operators are Linear, Karcher Mean, SLERP, NuSLERP, TIES, DELLA, and Nearswap. Linear interpolation treats parameters as Euclidean vectors and averages them:

$t$3

SLERP performs spherical linear interpolation on unit-normalized weights using the inter-weight angle

$t$4

and interpolates along the hypersphere rather than along a Euclidean line. NuSLERP applies spherical interpolation per tensor or per layer with local normalization and potentially layer-specific interpolation weights $t$5. Karcher Mean computes a Riemannian center of mass on the sphere by minimizing $t$6 subject to $t$7, with updates expressed through the Riemannian log and exponential maps. TIES and DELLA operate in delta space relative to a reference checkpoint $t$8, using thresholding, sign resolution, sparsity, or magnitude-aware sampling to reduce interference. Nearswap performs similarity-aware interpolation that strengthens interpolation where checkpoints agree and weakens it where they differ.

The survey evaluates these operators on 13 open-weight checkpoints drawn from the GPT, LLaMA, and Qwen families: GPT-2 small, medium, large, XL, GPT-Neo-2.7B, LLaMA-2-7B, LLaMA-3-8B, LLaMA-3.1-8B, LLaMA-3.2-1B, LLaMA-3.2-3B, Qwen2-0.5B, Qwen2-1.5B, and Qwen2-7B. Because all merges are between a pre-trained model and its bias-inverted counterpart, layer-wise tensor shapes match one-to-one and no neuron permutation is required. Merging is implemented with the mergekit toolkit using default hyperparameters besides $t$9.

Bias inversion data come from StereoSet intrasentence, made “bias-only” by filling cloze blanks with the stereotypical option. Continual pretraining is run for 30 epochs with max_length 512, batch size 64, weight decay 0.01, warmup ratio 0.1, and family-specific learning-rate schedules. Bias is then evaluated on BBQ, BOLD, and HONEST, while downstream utility is evaluated on eight SuperGLUE tasks—BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, and WSC—using the EleutherAI LM Evaluation Harness.

3. Empirical trade-offs and operating regimes in LLM BBM

The central empirical finding is a bias–utility trade-off: increasing $\alpha$0 consistently reduces measured bias but tends to degrade downstream accuracy (Shirafuji et al., 2 Dec 2025). The majority of methods show near-linear bias reduction as $\alpha$1 increases from $\alpha$2 to $\alpha$3. The paper also reports that some algorithms over-shoot and produce anti-stereotypical tendencies, including BBQ scores approaching $\alpha$4.

Among the seven operators, Linear, SLERP, and Nearswap are identified as the consistent winners. They reduce bias across BBQ, BOLD, and HONEST while largely preserving SuperGLUE. The reported best operating point is SLERP at moderate interpolation, $\alpha$5–$\alpha$6, which produces more bias reduction than Linear with similar downstream scores. Nearswap also consistently reduces bias with limited downstream loss, particularly at moderate $\alpha$7, and is described as especially helpful when the two models partially agree.

The less stable set comprises Karcher Mean, NuSLERP, TIES, and DELLA. These methods are reported to over-mitigate bias more often and to substantially degrade some SuperGLUE abilities. The largest losses appear in reading comprehension and commonsense or causal reasoning: ReCoRD drops by $\alpha$8 to $\alpha$9 relative to the pre-trained baseline, BoolQ by $P_{\mathrm{comb}}(p)$0 to $P_{\mathrm{comb}}(p)$1, COPA by $P_{\mathrm{comb}}(p)$2 to $P_{\mathrm{comb}}(p)$3, and CB by $P_{\mathrm{comb}}(p)$4 to $P_{\mathrm{comb}}(p)$5. By contrast, the paper summarizes smaller average declines for the more stable set: Linear at roughly $P_{\mathrm{comb}}(p)$6 to $P_{\mathrm{comb}}(p)$7, SLERP preserving downstream performance comparably to Linear at $P_{\mathrm{comb}}(p)$8–$P_{\mathrm{comb}}(p)$9, and Nearswap around $B_{\mathrm{image}}(p)$0 on average.

These results are interpreted in the paper through the geometry of the merge. SLERP’s advantage is that global spherical interpolation respects the geometry of the weight space when the two bias modes are near a common hypersphere, preserving a global meaning of $B_{\mathrm{image}}(p)$1 across all layers. Linear is robust and simple but can under-correct bias compared to SLERP. Nearswap emphasizes aligned parameters and reduces destructive interference where parameters diverge. Conversely, per-layer normalization and per-tensor interpolation in NuSLERP can introduce local imbalances, and the selectivity of TIES or DELLA can overshoot debiasing in this setting.

Across model families, the survey reports broadly similar curves within families but also notes anomalies, such as LLaMA-2-7B behaving differently from other LLaMAs. No consistent correlation is found between parameter count and bias score. A plausible implication is that alignment lineage and the geometry of the interpolation path matter more than sheer scale for this form of BBM.

4. BBM for phrase grounding in medical vision-language diffusion models

In the medical grounding paper, BBM is introduced as a post-processing method that refines cross-attention-based activation maps for phrase grounding by explicitly aligning and merging two priors obtained from the same text-to-image diffusion model (Nützel et al., 16 Jul 2025). The two priors are defined as follows. The image bias is what the model “expects” spatially from the image channel alone, operationalized via the [start] token attention when the U-Net is conditioned on the noisy ground-truth image. The text bias is what the model “expects” spatially from the text channel alone, operationalized by running the model with Gaussian noise as image input and using the lexical tokens’ attention.

BBM is placed after attention extraction and phrase-map aggregation in the Generate-to-Ground pipeline. The reported sequence is: train a Stable Diffusion v1.5 U-Net on chest X-rays with a frozen CXR-BERT text encoder injected into all cross-attention layers; at test time, extract token-level cross-attention maps across timesteps, layers, and heads while conditioning on the radiology report and repeatedly denoising the ground-truth image; aggregate token maps into a phrase activation map; apply BBM to compute image bias and text bias and merge them with the phrase activation map; and finally threshold the refined activation map to generate a mask and then a bounding box.

Cross-attention is defined per timestep $B_{\mathrm{image}}(p)$2, layer $B_{\mathrm{image}}(p)$3, and head $B_{\mathrm{image}}(p)$4 by

$B_{\mathrm{image}}(p)$5

where $B_{\mathrm{image}}(p)$6 is the image query and $B_{\mathrm{image}}(p)$7 is the text key. Attention is averaged across heads and then upsampled to a common $B_{\mathrm{image}}(p)$8 latent resolution. Because the paper reports that CXR-BERT yields cross-resolution consistency, it averages all layers and all timesteps by default.

The unrefined phrase activation map is

$B_{\mathrm{image}}(p)$9

where $B_{\mathrm{text}}(p)$0 is the set of phrase-relevant tokens after lexical filtering with SciSpaCy. The image bias is

$B_{\mathrm{text}}(p)$1

and the text bias is

$B_{\mathrm{text}}(p)$2

where $B_{\mathrm{text}}(p)$3 is min–max normalization to $B_{\mathrm{text}}(p)$4.

5. Mathematical rule, thresholding, and reported performance in medical BBM

The interaction map used by BBM is the Hadamard product

$B_{\mathrm{text}}(p)$5

A scalar confidence score $B_{\mathrm{text}}(p)$6 measures alignment between the two biases through the structural similarity index:

$B_{\mathrm{text}}(p)$7

The paper gives three interpolation variants. The linear form is

$B_{\mathrm{text}}(p)$8

The quadratic Bézier form is

$B_{\mathrm{text}}(p)$9

The default BBM rule is a mixture Bézier interpolation with control point

$64 \times 64$0

followed by

$64 \times 64$1

This formulation is used to bias the refinement toward the intersection of the two biases and to suppress spurious activation outside jointly supported regions.

To generate a binary mask, the paper fits a 2-component Gaussian Mixture Model to the values of $64 \times 64$2 and thresholds at the posterior-equality point $64 \times 64$3, defined by

$64 \times 64$4

The final mask is $64 \times 64$5, and the tight bounding box of the largest connected component is returned. The reported default hyperparameters are to average all timesteps, all cross-attention layers, and all heads, with $64 \times 64$6 in reported results, and to apply no morphological post-processing in the core setup.

On the MS-CXR test set, the paper reports weighted-average results across diseases and three seeds. BioViL achieves mIoU $64 \times 64$7 and CNR $64 \times 64$8; BioViL-L achieves mIoU $64 \times 64$9 and CNR $$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$0; CLIP_LDM achieves mIoU $$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$1 and CNR $$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$2; CXR-BERT_LDM without BBM achieves mIoU $$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$3 and CNR $$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$4; and CXR-BERT_LDM with BBM achieves mIoU $$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$5 and CNR $$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$6 (Nützel et al., 16 Jul 2025). The key reported effect of BBM is therefore an increase in CNR of approximately $$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$7 on average, while the mIoU change is small and can slightly decrease after thresholding because BBM tends to sharpen confident regions and can shrink masks.

The paper’s Appendix further separates the interpolation variants: Linear Bézier yields mIoU $$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$8 and CNR $$\theta_{\mathrm{bias}}^{\mathrm{inv}} = 2\theta_{\mathrm{LLM}} - \theta_{\mathrm{bias}}.$$9; Quadratic Bézier yields mIoU $\theta_{\mathrm{bias}}^{\mathrm{inv}}$0 and CNR $\theta_{\mathrm{bias}}^{\mathrm{inv}}$1; and Mixture Bézier yields mIoU $\theta_{\mathrm{bias}}^{\mathrm{inv}}$2 and CNR $\theta_{\mathrm{bias}}^{\mathrm{inv}}$3. The reported interpretation is that linear interpolation improves contrast but can over-expand activations, quadratic gating tightens masks and gives the best mIoU, and the mixture formulation balances both and is therefore used as the default BBM.

6. Failure modes, implementation constraints, and broader interpretation

The two BBM literatures are unified less by shared code than by a common use of “bias” as an operational prior that can be estimated and then recombined. In the LLM survey, the prior is encoded in checkpoint parameters, and successful BBM depends on alignment between a base model and its bias-inverted counterpart, matched global norms, and moderate interpolation weights (Shirafuji et al., 2 Dec 2025). In the medical grounding work, the priors are spatial attention maps derived from the same diffusion model under two different conditioning regimes, and successful BBM depends on the agreement between $\theta_{\mathrm{bias}}^{\mathrm{inv}}$4 and $\theta_{\mathrm{bias}}^{\mathrm{inv}}$5, as quantified by SSIM (Nützel et al., 16 Jul 2025). This suggests a broader methodological pattern: BBM is not a single algorithm but a family of bias-alignment procedures in which interpolation is most reliable when the merged objects are already structurally compatible.

The main limitations are also domain-specific. For LLM BBM, findings are derived from merging a base model with its bias-inverted counterpart created via task arithmetic on StereoSet stereotypes; the paper explicitly notes that merging two independently trained, differently biased models may require additional alignment and may show different sensitivities. It also reports that larger $\theta_{\mathrm{bias}}^{\mathrm{inv}}$6 values, especially $\theta_{\mathrm{bias}}^{\mathrm{inv}}$7, erase bias but devastate downstream ability. For medical BBM, the method relies on well-aligned biases; severe domain shifts that disrupt either text or image bias reduce $\theta_{\mathrm{bias}}^{\mathrm{inv}}$8 and automatically down-weight the bias interaction, falling back toward $\theta_{\mathrm{bias}}^{\mathrm{inv}}$9. The paper also notes that lexical filtering does not handle semantic negation, that Pneumothorax is a failure case across methods, and that small or thin structures can be missed if one bias is weak.

Several practical cautions recur across the two usages. In model merging, the recommended default is to start with Linear and SLERP, with SLERP at $\theta_{\mathrm{LLM}}$0–$\theta_{\mathrm{LLM}}$1 preferred for a better bias–utility balance, and to avoid Karcher Mean, NuSLERP, TIES, and DELLA unless there is strong task-specific validation. In phrase grounding, the recommended default is Mixture Bézier BBM with per-image SSIM weighting, 2-component GMM thresholding, and normalization of $\theta_{\mathrm{LLM}}$2, $\theta_{\mathrm{LLM}}$3, $\theta_{\mathrm{LLM}}$4, and $\theta_{\mathrm{LLM}}$5 to $\theta_{\mathrm{LLM}}$6 before merging. In both cases, excessive or poorly aligned merging is reported to be damaging: in one domain to reading comprehension and causal reasoning, and in the other to mask extent and thin-structure localization.

Within contemporary arXiv usage, then, “Bimodal Bias Merging” refers to two technically unrelated but conceptually analogous operations: checkpoint interpolation across opposing bias modes in LLMs, and confidence-gated fusion of image and text priors in medical phrase grounding. The term is therefore best understood contextually, with its precise meaning determined by whether the underlying objects are model parameters or spatial activation maps.