---
title: 'Bi-VLM: Diverse Bi-directional Vision–Language Models'
url: https://www.emergentmind.com/topics/bi-vlm
type: topic
---

# Bi-VLM: Diverse Bi-directional Vision–Language Models

to=arxiv_search.search  qq彩票  大发快三是不是 下载彩神争霸াদি 平台直属 ＿影音先锋 code  手机上天天中彩票json
{"query":"all:\"Bi-VLM\" OR ti:\"Bi-VLM\" OR abs:\"Bi-VLM\"","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}
to=arxiv_search.search _一本道  微信里的天天中彩票json
{"query":"id:2602.03151 OR id:2601.14133 OR id:2501.07769 OR id:2509.18763 OR id:2509.21723 OR id:2305.12231","max_results":10,"sort_by":"submittedDate","sort_order":"descending"}
Bi-VLM is a label used in recent arXiv literature for several distinct vision–language research programs rather than a single standardized model family. In one line of work, it denotes bi-directional vision–language modeling through missing-modality feature restoration and alignment in a frozen CLIP latent space; in another, it denotes a two-brain VLM architecture for embodied control; in another, bi-directional modality interaction prompt learning inside frozen CLIP; and in another, an ultra-low-precision post-training quantization pipeline for VLMs. Related formulations extend comparable ideas to vision-language-anchored bimanual manipulation and bi-level vision-language graph matching for text-guided medical segmentation [2602.03151; 2601.14133; 2501.07769; 2509.18763; 2509.21723; 2305.12231].

## 1. Terminological scope and recurring design motifs

A common source of confusion is that the label does not denote one canonical architecture in the cited literature. Instead, recent uses cluster around several technical motifs: bidirectional restoration between image and text feature spaces, asymmetric coordination between two VLM streams, prompt-level bi-directional interaction inside frozen backbones, and distribution-aware quantization for efficient deployment.

| Usage | Core mechanism | Setting |
|---|---|---|
| Missing-modality Bi-VLM | Conditional diffusion, DMG, CMML | Foundation VLM robustness |
| TwinBrain Bi-VLM | Frozen Left Brain, trainable Right Brain, AsyMoT | Embodied VLA control |
| BMIP-style bi-directional interaction | Layered prompt replacement with attention-derived weights | Few-shot CLIP adaptation |
| Quantization Bi-VLM | Gaussian-quantile saliency-aware hybrid PTQ | Ultra-low-bit VLM deployment |

Across these uses, several structural themes recur. One is the preservation of a frozen semantic backbone while adding a lightweight or specialized mechanism around it: frozen CLIP encoders in missing-modality restoration and BMIP, a frozen generalist VLM in TwinBrainVLA, and post-training rather than retraining-oriented compression in ultra-low-bit Bi-VLM. Another is explicit asymmetry: the available modality conditions restoration of the missing one, the Left Brain semantically anchors the Right Brain without receiving gradients, and saliency-aware quantization treats outlier and inlier weights differently. This suggests that, in current usage, “Bi-VLM” often marks an attempt to preserve a stable semantic prior while introducing a second pathway, direction, or precision regime for adaptation.

## 2. Bi-directional feature restoration for missing-modality inference

In "Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration," Bi-VLM is realized as a mid-stage training module inserted between frozen CLIP dual encoders and downstream heads. The visual encoder $E_v$ and text encoder $E_t$ remain frozen, while an enhanced diffusion transformer operates in the VLM feature space to restore the missing feature $x_0 \in \{h_v, h_t\}$ from the available conditional feature $c \in \{h_t, h_v\}$. The forward noising process is
$$
q(x_t \mid x_{t-1}) = N(x_t; \sqrt{\alpha_t} x_{t-1}, (1-\alpha_t) I),
$$
and the reverse conditional process is
$$
p_\theta(x_{t-1} \mid x_t, c) = N(x_{t-1}; \mu_\theta(x_t, t, c), \Sigma_\theta(x_t, t, c)).
$$
The corresponding clean estimate is
$$
\hat{x}_0 = \frac{x_t - \sqrt{1-\bar{\alpha}_t}\,\epsilon_\theta(x_t, t, c)}{\sqrt{\bar{\alpha}_t}}.
$$
This formulation supports both text-to-vision and vision-to-text restoration, with denoising trajectories anchored on the frozen CLIP manifold.

Two mechanisms define the paper’s bi-directional character. Dynamic Modality Gating (DMG) adaptively fuses conditional and restored content, conceptually written as
$$
\alpha = \sigma(W[h_c; r] + b), \qquad f = \alpha \odot r + (1-\alpha) \odot h_c,
$$
and implemented inside DiT blocks through learnable attention-pooled gate vectors and layer-wise gates $Z_{\text{attn}}$ and $Z_{\text{mlp}}$. Cross-Modal Mutual Learning (CMML) then forces restored features to be valid conditions in the opposite direction. The total loss is
$$
L_{\text{total}} = L_{\text{mutual}} + L_{\text{base},i2t} + L_{\text{base},t2i}.
$$
During inference, if text is missing, the model restores $\hat{h}_t$ from $h_v$; if image is missing, it restores $\hat{h}_v$ from $h_t$; the available and restored features are then concatenated and fed to a lightweight decoder.

The reported evaluation frames this Bi-VLM as a robustness mechanism for missing-modality deployment rather than a replacement for the underlying foundation model. Benchmarks are MM-IMDb (F1-Macro), N24News (Accuracy), MMHS11K (Accuracy), and Food101 (Accuracy). At 70% missing, the paper reports MM-IMDb (missing image) F1-M +7.07% over the best baseline, MMHS11K (missing image) ACC +8.16%, and N24News ACC +3.88% for image missing, with Food101 showing small but consistent gains. Performance degrades only ~3% from 10%→90% missing in MM-IMDb, whereas baselines drop >10%. Ablations further report that replacing DMG with AdaLN or simple concat degrades performance, removing CMML weakens robustness and increases semantic drift, performance scales positively from Food101 (68K) to COCO (445K) to CC3M (3M), gains rise from 16→20 DiT layers with diminishing returns beyond 20, and accuracy plateaus at ~50 DDIM steps, with latency ~22.94 ms under complete modalities and ~678–707 ms under missing modalities for 50 DDIM steps [2602.03151].

## 3. Dual-VLM asymmetry for embodied action

In "TwinBrainVLA: Unleashing the Potential of Generalist VLMs for Embodied Tasks via Asymmetric Mixture-of-Transformers," Bi-VLM denotes a two coordinated VLM design. The architecture pairs a frozen “Left Brain” VLM that retains robust general visual reasoning with a fully trainable “Right Brain” VLM specialized for embodied perception and proprioceptive grounding. Both streams are isomorphic stacks initialized from the same checkpoint, using Qwen2.5-VL-3B-Instruct or Qwen3-VL-4B-Instruct, but they receive different token sequences: the Left Brain processes visual and text tokens only, while the Right Brain additionally ingests state tokens $\phi(s)$ derived from proprioceptive state $s \in \mathbb{R}^{d_s}$.

Coordination is implemented through the Asymmetric Mixture-of-Transformers (AsyMoT). At each layer $l$, the Right Brain forms joint keys and values by concatenating its own projections with stop-gradient copies from the Left Brain,
$$
K_{\text{joint}} = [sg(K_L^l); K_R^l], \qquad V_{\text{joint}} = [sg(V_L^l); V_R^l],
$$
and computes
$$
\text{Attn}_R^l = \text{softmax}\!\left(\frac{Q_R^l K_{\text{joint}}^\top}{\sqrt{d_k}}\right)V_{\text{joint}}.
$$
The asymmetry is strict: $\theta_L$ is frozen, $\nabla \theta_L = 0$, and no explicit scalar gating network is introduced. This architecture structurally isolates the semantic generalist from action gradients, while allowing the specialist stream to query stable semantic representations.

Action generation is delegated to a Diffusion Transformer policy head trained with flow matching. The policy models
$$
\frac{d a_t}{dt} = v_\psi(a_t, t, H_R),
$$
with straight-path interpolation $a_t = (1-t)a_0 + ta_1$ and loss
$$
\mathcal{L}_{FM}(\psi) = \mathbb{E}_{t,a_0,a_1}\left[\|v_\psi(a_t,t,H_R) - (a_1-a_0)\|^2\right].
$$
Training freezes the Left Brain and optimizes only the Right Brain VLM, the state encoder, and the DiT policy head, using 40k steps, 16× NVIDIA H100 GPUs, batch size 16 per device, AdamW, lr = 1e−5, cosine annealing, gradient clipping (norm 1.0), and DeepSpeed ZeRO-2.

Empirically, TwinBrainVLA is evaluated on SimplerEnv and RoboCasa. On SimplerEnv, TwinBrainVLA + Qwen2.5-VL-3B-Instruct reaches 58.4% average success, while TwinBrainVLA + Qwen3-VL-4B-Instruct reaches 62.0%, surpassing Isaac-GR00T-N1.6 (57.1%) by +4.9%. On RoboCasa GR1 Tabletop, the corresponding averages are 53.5% and 54.6%, with the Qwen3-VL-4B-Instruct version outperforming Isaac-GR00T-N1.6 (47.6%) by +7.0%, QwenGR00T (47.8%) by +6.8%, and QwenPI (43.9%) by +10.7%. The paper argues semantic preservation by architectural design rather than by post-training VQA or captioning benchmarks; explicit external VLM evaluations are not reported. Limitations include the same-architecture constraint for Left/Right pairing, simulation-first evaluation, and the higher inference cost of running both VLMs plus DiT ODE steps [2601.14133].

## 4. Bi-directional modality interaction in prompt learning

"BMIP: Bi-directional Modality Interaction Prompt Learning for VLM" uses the idea of Bi-VLM at the prompt-learning level inside a frozen CLIP backbone. The model keeps the image encoder $f$ and text encoder $g$ frozen and learns layered prompts, projection heads, and attention-derived weighting layers. Deep language prompts $\{P_i\}_{i=0}^J$ are inserted into the first $J$ layers of the text encoder, and deep vision prompts $\{\tilde{P}_i\}_{i=0}^J$ are inserted into the first $J$ layers of the image encoder. BMIP does not add new cross-attention modules; instead, it reads the native self-attention outputs of each branch and uses them to derive prompt substitution weights.

The central interaction block defines
$$
w_v = L_v(A_v), \qquad w_l = L_l(A_l),
$$
and then performs bi-directional prompt replacement:
$$
\tilde{P}_{i}' = w_v * \tilde{P}_{i} + (1 - w_v) * F_v(P_{i}),
$$
$$
P_i' = \left[w_l * P_i + (1 - w_l) * F_l(\tilde{P}_i)\right].
$$
Thus the vision prompt absorbs transformed language information and the language prompt absorbs transformed vision information, with the balance learned from attention outputs. The paper positions this against single-modal prompt learning and uni-directional prompt transfer, arguing that simple aggregation or one-way transfer neglects the alignment effects resulting from two-way interaction.

The training objective remains the standard VLM classification objective under few-shot finetuning with frozen encoders,
$$
p(y=i \mid I)=\frac{\exp(\cos(x,z_i)/\tau)}{\sum_{j=1}^N \exp(\cos(x,z_j)/\tau)},
$$
with gradients flowing through the prompts, projection heads, and attention-weight generators. BMIP also proposes open-world generalization, defined as simultaneous evaluation on an unknown distribution composed of both base and novel classes, alongside cross-dataset transfer and domain generalization.

Quantitatively, for open-world generalization on ViT-B/16, BMIP reports average HM/Acc of 79.04 / 72.17, compared with MaPLe 78.22 / 71.76, CoCoOp 74.72 / 67.67, CoOp 72.14 / 65.57, and CLIP 70.84 / 63.92. Representative dataset-level results include EuroSAT HM 86.10 ± 1.58 for BMIP versus 81.43 ± 0.53 for MaPLe, Flowers102 HM 83.86 ± 1.70 versus 82.78 ± 0.69, FGVC-Aircraft HM 37.25 ± 0.93 versus 35.29 ± 0.58, and SUN397 HM 79.02 ± 0.24 versus 79.58 ± 0.13. In cross-dataset transfer, BMIP reports average accuracy 66.86 versus MaPLe 66.30, and in domain generalization an overall average 62.50 versus 62.36 and OOD average 60.40 versus 60.28. Aggregation ablations report BMIP at 79.04 / 72.17, above IVLP, CoCoOp, MaPLe, $\text{MaPLe}^\dagger$, Addition, Attention, and Joint, supporting the paper’s claim that attention-guided learnable substitution is more effective than simple aggregation. The method is also composable: MaPLe+BMIP improves HM/Acc from 78.22 / 71.76 to 79.03 / 72.54, PromptSRC+BMIP from 79.67 / 73.43 to 80.03 / 73.97, and CoPrompt+BMIP from 78.99 / 71.48 to 79.54 / 72.35 [2501.07769].

## 5. Bi-VLM as ultra-low-precision post-training quantization

In "Bi-VLM: Pushing Ultra-Low Precision Post-Training Quantization Boundaries in Vision-Language Models," the term denotes a post-training quantization method for VLM deployment under hardware constraints. The method assumes approximate Gaussian layerwise weight statistics and partitions each weight matrix into an outlier subset and multiple inlier subsets using Gaussian quantiles. For a layer $W_l \in \mathbb{R}^{m \times n}$, the standardized magnitude variable is
$$
t(w) = \frac{|w| - \mu_l}{\sigma_l},
$$
with thresholds derived from $\Phi^{-1}(\cdot)$ and salient percentile $p_l^{sal}$. The saliency metric is
$$
s(w) = \frac{(|w|-\mu_l)_+}{\sigma_l},
$$
which labels weights as salient when they exceed the learned tail threshold.

Quantization is hybrid and saliency-aware. Salient entries receive 2-bit non-uniform quantization with row-wise scalers,
$$
W_l^{sal,quant} = a^{sal} \odot B_l^{sal},
$$
whereas each inlier subset is strictly binarized,
$$
W_l^{uns,quant}(k) = a_l^{uns}(k)\cdot B_l^{uns}(k).
$$
The layerwise reconstruction objective is
$$
\min_Q \; \|W_l^{sal} - a^{sal} \odot B_l^{sal}\|_F^2 + \sum_{k=1}^K \|W_l^{uns}(k) - a_l^{uns}(k) B_l^{uns}(k)\|_F^2,
$$
and the optimal scalar for an inlier subset is
$$
a_l^{uns}(k)^* = \frac{\langle W_l^{uns}(k), B_l^{uns}(k)\rangle}{\|B_l^{uns}(k)\|_F^2}.
$$
Layerwise saliency search uses Brent’s method under bounded salient proportions, with empirical caps of ≈5% in vision and ≈1% in the LM. After quantization, the paper further prunes image tokens based on attention mass and reports 90%–99% image token redundancy in quantized models.

The method is evaluated on Llama 3.2-Vision instruction 11B, LLaVA-One-Vision 7B, and Qwen2.5-VL-7B-Instruct over MME, MMMU, ScienceQA-IMG, and VizWiz-VQA, using PTQ with 64 calibration samples. The abstract reports that, for the language model part of the VLM, Bi-VLM outperforms the SOTA by 3%–47% on the visual question answering task in terms of four different benchmarks and three different models, and for the overall VLM it outperforms the SOTA by 4%–45%. The detailed results state, for example, that on Llama 3.2-Vision 11B the language part improves over SOTA by 4%–47% and the whole VLM by 8%–45%; on LLaVA-One-Vision 7B the gains are +3%–20% for the language part and +4%–19% for the whole model; and on Qwen2.5-VL-7B-Instruct they are +5%–10% and +4%–12%, respectively. Storage analysis gives
$$
L_{\text{model}} = L_B + L_a,
$$
with average storage ≈ 1.014 bits/weight for $N_b=2$ and $p_l^{sal,max}\approx 1\%$ in the LM, implying $CR \approx 16/1.014 \approx 15.8\times$ for 16-bit FP weights and $\approx 31.6\times$ for 32-bit FP weights. The paper identifies the vision encoder as highly sensitive, the adaptor as low sensitivity, and the LM as considerably sensitive in the 1–1.1 bit regime, and notes that text-token pruning harms accuracy much more than image-token pruning [2509.18763].

## 6. Adjacent formulations: bimanual manipulation and bi-level graph matching

Related work broadens the technical neighborhood around the label. "VLBiMan: Vision-Language Anchored One-Shot Demonstration Enables Generalizable Robotic Bimanual Manipulation" defines a framework for bimanual manipulation in which a single precisely labeled human demonstration is decomposed into invariant anchors and adjustable components, with adaptation driven by Florence-2 and SAM2 segmentation, stereo back-projection, and geometric feasibility constraints. The demonstration trajectory $\tau = \{(s_t,a_t)\}_{t=1..T}$ is partitioned into invariant primitives $A$ and adjustable components $C$ using the bind indicator and object geometry tolerance, while scene adaptation is computed from
$$
\Delta x = p_{\text{new}} - p_{\text{demo}}, \qquad \Delta \theta = \angle(v_{\text{new}}, v_{\text{demo}}), \qquad \Delta T = [R(\Delta \theta), \Delta x].
$$
Execution uses progressive IK refinement,
$$
q^{(n+1)} = IK(T_g^{(n)}), \qquad T_g^{(n)} = SplineInterp(T_{start}, T_{goal}, n),
$$
and dynamic collision compensation
$$
\tilde{x}_{goal} = x_{goal} + \delta_{base}u_{\parallel} + \delta_z u_z.
$$
On six primary tasks with new placements and same objects, VLBiMan reports average 85.0%, versus ReKep+ 65.0%, ReKep 43.3%, Robot-ABC 36.7%, MAGIC 45.8%, and Mechanisms 28.3%; with novel instances the figure is 78.3%. Under dynamic interference, the averages are 70.0% for same objects and 59.2% for novel instances. Long-horizon tasks reach 52.5% and 41.3% without interference, and cross-embodiment transfer on a humanoid dual-arm robot reaches 83.8% and 76.3% without interference. Ablations report that removing IK refinement drops the six-task novel-instance-with-interference result from 59.2% to 29.2%, removing collision avoidance to 34.2%, replacing the paper’s grasp adaptation with AnyGrasp to 31.7%, and downgrading VLMs to SAM+DINOv2 to 35.8% [2509.21723].

A second adjacent line is "Bi-VLGM : Bi-Level Class-Severity-Aware Vision-Language Graph Matching for Text Guided Medical Image Segmentation," which is not titled Bi-VLM but is explicitly a bi-level vision-language matching framework. It augments HRNetV2 with a word-level VLGM module for local-class alignment and a sentence-level VLGM module for global-severity alignment. Soft correspondence matrices are produced by AIS over GCN-embedded visual and textual graphs, with losses
$$
L_{local}^{gt} = L_{CE}(\hat{X}_{local}, X_{local}^{gt}) + L_{QC}(E_I,E_T,\hat{X}_{local}),
$$
$$
L_{global}^{gt} = L_{CE}(\hat{X}_{global}, X_{global}^{gt}) + L_{QC}(E_V,E_S,\hat{X}_{global}),
$$
encoder objective
$$
L_E = \lambda_a L_{local}^{gt} + \lambda_b L_{global}^{gt},
$$
and segmentation objective
$$
L_G = \lambda_c L_{Dice} + \lambda_d L_{local}^p + \lambda_e L_{global}^p.
$$
Severity-aware prompts are constructed from lesion area ratios using thresholds $t_1 = 0.06$ and $t_2 = 0.12$. On IDRiD, Bi-VLGM reports mAUPR 68.12%, mF 66.71%, and mIoU 51.37%; on DDR it reports mIoU 31.78% and mF 47.29%. Ablations show mIoU rising from 47.74 for HRNetV2 to 50.00 with word-level only, 50.50 with sentence-level only, and 51.37 with both. A comparison to contrastive loss reports VLGM mIoU 51.37 versus 49.46. These neighboring formulations show that the broader “bi-” vision-language design space also includes explicit decomposition into two alignment levels, two arms, or two control regimes, not only two modalities or two model branches [2305.12231].

Source: https://www.emergentmind.com/topics/bi-vlm