UniBind: Unified Multi-Modal Learning

• UniBind is a unified multi-modal representation framework that leverages LLM-generated text clouds to create a balanced, modality-agnostic embedding space across seven data types.
• It constructs a rich knowledge base by indexing category-level and instance-level multi-modal descriptions using frozen CLIP-style encoders for precise semantic alignment.
• UniBind significantly enhances recognition and retrieval performance across 14 benchmarks while reducing trainable parameters by up to 90% compared to image-centric approaches.

UniBind is a LLM-augmented framework for unified and balanced multi-modal representation learning across seven diverse data modalities: images, text, audio, point cloud, thermal, video, and event data. It addresses the limitations of prior CLIP-style and image-centered binding methods, such as ImageBind, which produce embedding spaces biased toward images and inadequately capture inter-modality semantics. UniBind introduces a modality-agnostic alignment center, utilizing clouds of text embeddings generated by LLMs and multi-modal LLMs to create a high-fidelity, semantically rich, and balanced embedding space shared across modalities. The result is a system that can be flexibly integrated into existing CLIP-style architectures, significantly boosting recognition and retrieval performance, while requiring substantially fewer trainable parameters (Lyu et al., 2024).

1. Motivation and Theoretical Basis

Conventional multi-modal representation methods, particularly variants of CLIP and ImageBind [Girdhar et al. 2023], use RGB images as a fixed alignment center, requiring all modalities to align against them via contrastive learning. This "image-centric" paradigm leverages established image–text datasets but induces bias, diminishing the representational quality for non-image modalities. Additionally, standard category-name prompts (e.g., "A photo of a [class]") used as text centers are insufficient for capturing the deep semantics inherent in multi-modal data, leading to sub-optimal modality mixing and inter-class discrimination. UniBind addresses these issues by decoupling the alignment center from any specific modality and employing rich, LLM-generated text clouds as class-wise anchors, thereby ensuring balanced modality representation and enhanced semantic grounding [(Lyu et al., 2024), Fig 1].

2. Knowledge Base Construction

UniBind constructs a comprehensive text-embedding knowledge base for each dataset, consisting of two key components:

• Category-level Descriptions: For each class $C_i$, one or more LLMs (such as GPT-4 or LLaMA) are prompted to generate up to 1,000 paraphrases or descriptive sentences $T^1_{C_i}, T^2_{C_i}, \dots, T^n_{C_i}$ of length ≤77 tokens.
• Instance-level Multi-modal Descriptions: Each individual sample from every modality (e.g., an image $I_i$, audio clip $A_i$, etc.) is processed by a multi-modal LLM (e.g., BLIP-2, LLaMA-Adapter) to generate a concise, modality-aware textual description ($T_{I_i}, T_{P_i}, \dots$), which captures nuances like shape, thermal pattern, or event dynamics.

Every description is indexed in the knowledge base as $$\{\text{ID}, \text{Category}, \text{Description}, \text{Source}\}$$ (see Fig 5). This dual-sourcing of text—one from LLMs and another from multi-modal LLMs—enhances semantic diversity and fidelity compared to single-source or category-name-only approaches.

3. LLM-Augmented Class-wise Embedding Centers

UniBind abstracts away from a single-modal center by creating a "cloud" of top-$K$ text embeddings per class, serving as a modality-agnostic alignment center.

• All descriptions are encoded by a frozen CLIP-style text encoder $F^T$ into $z=F^T(T)\in\mathbb{R}^d$.
• For each class $C_j$, the system computes cosine similarities between all candidate descriptions and the class-prompt embedding $p_j=F^T(\text{“A photo of a }C_j\text{”})$.
• The top $K$ (empirically $K=50$) highest-scoring descriptions are selected:

$$\{T^k_{C_j}\}_{k=1}^K=\arg\max_{T\in\mathcal{KB}_{C_j}} \cos(F^T(T),p_j)$$

These embeddings, $EC_j = \{z^1_{C_j}, \ldots, z^K_{C_j}\}$, constitute the class-wise anchor. This representation encodes considerably finer semantic distinctions than a prompt mean, which is demonstrated empirically by enhanced cluster separation (Fig 4a, Fig 10).

4. Unified LLM-Augmented Contrastive Learning

Each modality encoder, $F_m$, is frozen and extended with a lightweight, trainable linear projection head $\phi_m$. For a batch of modality-$m$ samples $\{M_i\}$ and their associated descriptions $\{T_{M_i}\}$, positive pairs $(\phi_m(F_m(M_i)), F^T(T_{M_i}))$ are constructed, and negatives are drawn from other batch samples.

The contrastive objective for modality $m$ is:

$$\mathcal{L}_m = -\frac{1}{N}\sum_{i=1}^N \log\frac{ \exp\bigl(\phi_m(F_m(M_i))^\top\,z_{M_i}/\tau\bigr) }{ \sum_{j=1}^N \exp\bigl(\phi_m(F_m(M_i))^\top\,z_{M_j}/\tau\bigr) }$$

where $z_{M_j}=F^T(T_{M_j})$ and $\tau$ is the temperature (typically 0.07). The total loss aggregates across all modalities:

$$\mathcal{L}_{\rm total} = \sum_{m\in\{\text{img, audio, ...}\}} \mathcal{L}_m$$

This ensures equidistant alignment of all modalities to the same text cloud anchors, preventing image-dominated representational spaces.

5. System Integration and Training Protocols

UniBind is compatible with CLIP-style architectures such as CLIP, ImageBind, E-CLIP, AudioCLIP, PointBind, and PointCLIP. Integration entails freezing all backbone encoders, appending each with a modality-specific projection head ($\phi_m$), and leveraging the frozen text encoder for both knowledge-base indexing and loss computation.

Parameter Efficiency: Only $\phi_m$ and minimal cloud selection layers are trained, reducing the trainable parameter count by approximately 90% compared to full-model fine-tuning ($<$10% are updated).

Protocols:

• Zero-shot: The projection heads are frozen; predictions use the maximal cosine similarity over the class text cloud:

$$\hat{y} = \arg\max_j\; \max_{z\in EC_j} \cos(\phi_m(F_m(M)), z)$$

• Fine-tuning: Only $\{\phi_m\}$ are updated via $\mathcal{L}_{\rm total}$ for 10–20 epochs, using AdamW with learning rates $5 \times 10^{-5}$–$5 \times 10^{-3}$ and batch sizes 64–1024.

Training only the projection heads preserves pre-trained knowledge in the backbones and avoids catastrophic drift.

6. Empirical Performance and Ablations

UniBind demonstrates statistically significant improvements across seven modalities and 14 benchmarks, including ImageNet-1K, Places-365, Caltech-101, ESC-50, Urban-S, ModelNet-40, ShapeNet, LLVIP, RGB-T, MSR-VTT, UCF-101, N-Caltech-101, and N-ImageNet.

Setting	Mean Gain	Notable Result
Zero-Shot (top-1)	+6.36% vs. prior art	Table 2
Fine-Tuning (ImageNet)	+6.75% (e.g. on PointBind)	Table 2, Fig 3
Retrieval (Recall@20)	+17.96% (event–image)	Table 4

No additional modality-specific pre-training or architectural changes are required; gains result solely from the richer, cloud-based, modality-agnostic alignment strategy.

Ablation findings:

• LLM-augmented Contrastive Learning (LCL): Omitting LCL yields poorly mixed clusters; its inclusion improves retrieval Recall@20 by +1.55–+17.96%, depending on modality pair and architecture.
• Embedding Center Localization (ECL): Using a single prompt for alignment costs up to –8.28% top-1 accuracy (e.g., PointBind on N-Cal). ECL with $K=50$ achieves sharper, better-separated modality clusters.
• Knowledge Base Mix: Combining both LLM and multi-modal LLM outputs yields better results than either alone. $K=50$ offers optimal balance between coverage and noise.

The synergy of all three components—rich knowledge base, ECL, LCL—is critical for the observed improvements.

7. Significance and Context within Multi-Modal Representation Learning

UniBind's modality-agnostic, LLM-augmented binding offers a unified resolution to the trade-offs imposed by prior image-centered architectures. By leveraging large, diverse clouds of text descriptions, the framework more effectively captures category semantics and modality-specific nuances, thus promoting balanced cluster formation and improving both within-modality and cross-modal retrieval and recognition. Its compatibility with diverse existing architectures, substantial parameter savings, and strong empirical gains underscore its practical significance in the field of multi-modal learning (Lyu et al., 2024). The framework highlights the value of integrating modern LLM outputs with multi-modal systems and sets a new benchmark for balanced, scalable, and semantically grounded multi-modal representation.