BreezeCLIP: Efficient Vision-Language Encoder

• BreezeCLIP is a dual-encoder that efficiently aligns visual and textual features, optimized for resource-constrained visual question answering.
• It employs a hybrid architecture combining CNN and Transformer elements with inverted bottleneck designs to reduce computation while maintaining effectiveness.
• Integrated with Q-Gated Cross-Modal Fusion and a lightweight LLaMA decoder, BreezeCLIP achieves a strong balance between efficiency and accuracy on benchmarks like GQA and VQAv2.

Searching arXiv for the specified paper and closely related context. BreezeCLIP is a compact vision-language encoder introduced within the lightweight multimodal LLM framework BcQLM, or BreezeCLIP-enhanced Q-Gated Multimodal LLM. It is designed for end-to-end visual question answering under resource-constrained deployment settings, where energy efficiency, computational scalability, and environmental sustainability are treated as primary engineering constraints. In the reported system, BreezeCLIP provides the visual and textual representations that are subsequently fused by Q-Gated Cross-Modal Fusion modules and decoded by a lightweight LLaMA-3.2-1B component. The full BcQLM system has 1.2 billion parameters, while the BreezeCLIP encoder itself has 31 million parameters (Xiang et al., 10 Sep 2025).

1. System identity and model boundary

BreezeCLIP is not the full multimodal LLM; it is the dual-encoder vision-language component inside BcQLM. The paper defines BcQLM as a lightweight MLLM framework for end-to-end visual question answering, with BreezeCLIP serving as “a compact yet powerful vision-language encoder optimised for efficient multimodal understanding.” The overall pipeline is explicitly modular: BreezeCLIP produces visual patch embeddings $F_I$, token embeddings $F_T$, and a pooled global question vector $\bar F_T$; these are passed into an 8-head Q-GCAM fusion module; the fused features are projected by an adapter into pseudo-text tokens; those pseudo tokens are concatenated with question tokens and then processed by a LLaMA-3.2-1B decoder to generate an answer (Xiang et al., 10 Sep 2025).

This separation is central to interpreting the architecture. A common source of confusion is to treat BreezeCLIP and BcQLM as interchangeable names, but the paper distinguishes them sharply: BreezeCLIP is the encoder, whereas BcQLM is the full system that combines BreezeCLIP, Q-GCAM, an adapter, and a lightweight decoder. The modularity also underpins the paper’s claim that the design is extensible beyond the immediate visual question answering setting.

2. Dual-encoder architecture

BreezeCLIP is described as a dual-encoder transformer designed to extract and align visual and textual features with minimal computational cost. Its text encoder has 24 Transformer layers, explicitly noted as the same depth as BERT-Large, and uses an inverted-bottleneck design in each layer. Token embeddings, with vocabulary size reduced via factorised embedding, are projected down to a lower dimension for multi-head attention and feed-forward operations and then restored to the original hidden size. Residual connections and layer normalization follow each sub-layer (Xiang et al., 10 Sep 2025).

The image encoder combines convolutional and Transformer stages. It begins with a convolutional stem followed by MobileNetV2-style inverted-residual bottleneck blocks for early local feature extraction. It then applies three bottleneck Transformer blocks at decreasing spatial resolutions, specified as $14\times 14$ and $7\times 7$. These three Transformer blocks contain 2, 4, and 3 self-attention layers respectively, for a total of 9 layers. Within each block, a $3\times 3$ convolution captures local context, a $1\times 1$ convolution reduces channels, features are flattened into non-overlapping patches for global self-attention, then reshaped back and merged via $1\times 1$ convolutions (Xiang et al., 10 Sep 2025).

The architecture therefore combines mobile-style convolutional efficiency with selective global self-attention. This suggests that BreezeCLIP is positioned as a hybrid encoder rather than a purely ViT-style or purely CNN-style backbone. The text side similarly uses a compression-oriented design through factorised embedding and inverted bottlenecks, indicating that parameter reduction is pursued on both modalities rather than only on vision.

3. Q-Gated Cross-Modal Fusion

After BreezeCLIP’s encoders, BcQLM inserts Q-Gated Cross-Modal Fusion modules when combining the encoder with a lightweight LLaMA-3.2-1B decoder. The paper states that Q-GCAM fuses flattened image patch tokens, used as queries, with question tokens, used as keys and values, in a dynamic, question-aware fashion. Given visual patch embeddings $F_I\in\mathbb{R}^{B\times N\times d}$ and token embeddings $F_T\in\mathbb{R}^{B\times T\times d}$, the cross-attention stage is

$F_T$0

$F_T$1

The gate is computed from the global question vector and each visual token. The paper defines

$F_T$2

and then computes a scalar gate $F_T$3 as

$F_T$4

The gated fusion stage is

$F_T$5

$F_T$6

The paper also gives a simplified notation:

$F_T$7

$F_T$8

In that interpretation, $F_T$9 down-weights uninformative modalities per question, while $\bar F_T$0 and $\bar F_T$1 are small projection or FFN functions. The stated purpose is to provide fine-grained, question-aware blending without full cross-attention in every layer (Xiang et al., 10 Sep 2025).

Methodologically, Q-GCAM is the mechanism that converts BreezeCLIP from a dual encoder into a question-conditioned multimodal reasoning front end for the decoder. The ablation data indicate that this fusion stage is not incidental: it is the primary contributor to the gain from the BreezeCLIP-only baseline to the full BcQLM configuration.

4. Distillation objective and training regimen

BreezeCLIP is trained to inherit semantic alignment from a large CLIP ViT-B/32 teacher. The paper specifies a two-part objective consisting of contrastive loss and distillation loss. The contrastive term on paired student embeddings $\bar F_T$2 is

$\bar F_T$3

with temperature $\bar F_T$4 and weight $\bar F_T$5.

The distillation loss on L2-normalized teacher and student embeddings is

$\bar F_T$6

with normalization factor $\bar F_T$7.

The total loss is

$\bar F_T$8

where $\bar F_T$9 and $14\times 14$0 are set equal to 1 in experiments. The paper states that $14\times 14$1, $14\times 14$2, and $14\times 14$3 were chosen via grid search on GQA validation to balance alignment strength and training stability (Xiang et al., 10 Sep 2025).

The pretraining dataset for BreezeCLIP is GQA scene-graph descriptions converted to text-image pairs. Downstream fine-tuning uses GQA, VQAv2, and VizWiz. Pretraining with contrastive and distillation losses uses Adam with learning rate $14\times 14$4, StepLR decay every 10 epochs with $14\times 14$5, 64 epochs, batch size 32, and gradient clipping 1.0. Fusion and decoder training freezes BreezeCLIP while tuning Q-GCAM and LLaMA-3.2-1B with AdamW, learning rate $14\times 14$6, StepLR with step_size 5 and $14\times 14$7, 15 epochs, batch size 32, and gradient clipping 1.0 (Xiang et al., 10 Sep 2025).

This training protocol separates representation learning from multimodal generation. A plausible implication is that freezing BreezeCLIP during the second stage helps preserve the distilled vision-language alignment while allowing the cross-modal and decoder layers to specialize for task-level answer generation.

5. Efficiency profile and benchmarked performance

The paper frames BreezeCLIP and BcQLM primarily through the trade-off between model compactness and task performance. BreezeCLIP has 31 million parameters, described as an 80% reduction from 151 million for CLIP, while the full BcQLM system has 1.2 billion parameters including the LLaMA-1B decoder. For a batch of 20 on an RTX 4070 Ti, BcQLM is reported at $14\times 14$8 FLOPs per sample, 127.1 ms per sample latency, and 4.99 GiB peak memory. The comparison systems are Qwen2.5-VL-3B at $14\times 14$9 FLOPs, 253.1 ms, and 7.15 GiB, and Gemma3-4B at $7\times 7$0 FLOPs, 273.9 ms, and 7.43 GiB (Xiang et al., 10 Sep 2025).

At input resolution $7\times 7$1, BcQLM achieves 60.8 on GQA, 71.0 on VQAv2, and 49.5 on VizWiz. At $7\times 7$2, it achieves 62.4 on GQA, 78.7 on VQAv2, and 56.1 on VizWiz. The paper states that the 62.4 result on GQA is second best among published systems and that, despite 10× fewer parameters than many baselines, BcQLM matches or outperforms standard-size MLLMs on these tasks (Xiang et al., 10 Sep 2025).

Setting	Dataset	Accuracy (%)
224×224	GQA	60.8
224×224	VQAv2	71.0
224×224	VizWiz	49.5
336×336	GQA	62.4
336×336	VQAv2	78.7
336×336	VizWiz	56.1

These results place BreezeCLIP in a deployment-oriented regime rather than in the maximal-scale regime typical of many MLLMs. The performance gains with higher resolution further suggest that the encoder retains useful spatial sensitivity even under aggressive parameter constraints.

6. Ablations, deployment, and practical interpretation

The reported ablations on GQA validation isolate the contribution of BreezeCLIP and Q-GCAM. The configuration “CLIP + LLaMA” scores 58.8%, “BreezeCLIP + LLaMA” scores 59.2%, and “BreezeCLIP + Q-GCAM + LLaMA” scores 60.8%. Two alternatives perform worse: “Q-GCAM′ (Token Balance)” at 58.7% and “Q-GCAM″ (Visual Query)” at 55.7%. The paper summarizes this as a $7\times 7$3 gain for Q-GCAM over brute-force BreezeCLIP (Xiang et al., 10 Sep 2025).

The decoder unfreezing study reports that unfreezing ratios of 5%, 50%, and 100% show stable semantic correctness even with only 5% parameters tuned, while full unfreezing refines fine details. In deployment terms, the paper argues that the BreezeCLIP encoder and Q-GCAM are lightweight enough to run on edge GPUs or NPUs, and that the LLaMA-1B decoder is smaller than typical 7B or 13B LLMs. Partial freezing of LLaMA, down to 5% unfreeze, is presented as a further reduction in memory and compute overhead (Xiang et al., 10 Sep 2025).

The application scenarios listed are on-device visual question answering for assistive technology, including use by visually impaired users; mobile or embedded diagnostics in healthcare for point-of-care image QA; remote education tools interpreting diagrams or experimental setups; and disaster-relief robotics or drones requiring visual understanding with minimal latency and no cloud connectivity. These examples are consistent with the system’s stated emphasis on practical hardware constraints. More broadly, BreezeCLIP can be understood as the representation-learning substrate that enables those downstream deployment claims: without the compact encoder and the question-aware fusion stage, the overall system would not exhibit the reported efficiency envelope.