CodeBind: Multimodal Representation Alignment

• CodeBind is a multimodal alignment framework that decomposes modality embeddings into shared semantic and specific branches using a unified compositional codebook design.
• It incrementally aligns target modalities with bridging modalities through compositional vector quantization, addressing challenges like incomplete pairing and representation bias.
• Empirical evaluations across nine modalities demonstrate significant gains in retrieval, classification, and fusion performance, validating its selective alignment approach.

CodeBind is a multimodal representation alignment framework that optimizes multimodal representation spaces through a modality-shared-specific codebook design. It is intended for large multimodal LLMs and robotic perception systems, and it aligns text, image, video, audio, depth, thermal, tactile, 3D point cloud, and EEG without requiring fully paired data across all modalities. Rather than enforcing hard alignment of entire embeddings, CodeBind decomposes features into shared components for semantic consistency and specific components for modality-unique details, quantizes them through a unified compositional codebook design, and incrementally aligns target and bridging modalities (Chen et al., 18 May 2026).

1. Problem setting and rationale

CodeBind addresses multimodal representation alignment under three coupled constraints: cross-modal information discrepancy, incomplete pairing, and representation bias. The paper argues that traditional alignment methods often learn suboptimal alignment spaces because different modalities do not carry the same information, fully paired data for uncommon modalities are rare, and data-rich modalities such as vision and text can dominate the geometry of a common latent space. In that setting, hard alignment tends to preserve only a “least common denominator” semantic core while suppressing modality-specific information such as thermal signatures, tactile cues, depth structure, or EEG-specific response patterns (Chen et al., 18 May 2026).

The framework is motivated by the needs of large multimodal LLMs and robotics. In both cases, one wants a single semantic interface across highly unequal sensing channels, but not at the cost of erasing what is unique to each modality. CodeBind therefore treats full embedding equality as undesirable. Its central premise is that robust multimodal alignment should maximize agreement only on the modality-invariant part of a representation while reserving separate capacity for modality-private content.

A recurrent misconception in multimodal alignment is that a single tightly shared continuous space is always preferable. CodeBind rejects that assumption. The paper’s position is that semantic consistency and modality preservation are distinct requirements, and that collapsing them into one objective produces avoidable losses in retrieval, classification, and fusion quality.

2. Decoupled representations and the shared-specific codebook design

The core representational step is a decomposition of each modality embedding into a shared branch and a specific branch. For a sample $x_i^\mathcal{M}$, the encoder output is

$z_i^\mathcal{M} = E(x_i^\mathcal{M}),$

which is then split into

$$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$

The shared embedding is intended to capture modality-invariant semantics; the specific embedding preserves modality-unique details. Both branches are quantized:

$$\hat{z}^\mathcal{M}_{\text{shared}} = Q(z^\mathcal{M}_{\text{shared}}), \qquad \hat{z}^\mathcal{M}_{\text{spec}} = Q(z^\mathcal{M}_{\text{spec}}).$$

The quantization design is asymmetric. CodeBind uses one universal shared codebook, $\mathcal{C}_{\text{shared}}$, for the shared branch of all modalities within a target-bridge training path, and one modality-specific codebook, $\mathcal{C}_{\text{spec}}^\mathcal{M}$, for the specific branch of each modality (Chen et al., 18 May 2026).

This architecture is the paper’s main departure from hard alignment. Instead of pushing entire embeddings into one common space, CodeBind aligns only the shared branch and uses the specific branch to absorb the residual information that should not be forced into the common semantic basis. The paper frames this with mutual-information language: traditional alignment maximizes $I(z^{\mathcal{M}_1}, z^{\mathcal{M}_2})$, whereas CodeBind aims to make $$I(y; z^\mathcal{M}) \approx I(y; z^\mathcal{M}_{\text{shared}})$$ and maximize $$I(z^{\mathcal{M}_1}_{\text{shared}}, z^{\mathcal{M}_2}_{\text{shared}})$$.

A further detail is that text is treated specially: it uses only a shared embedding and has no specific branch or reconstruction. This reflects the role of text as a semantic bridge rather than a modality whose private low-level structure must be reconstructed.

3. Bridging modalities, incremental alignment, and compositional quantization

CodeBind follows a bridging-modality alignment setup similar in spirit to ImageBind and ViT-Lens. A frozen vision-LLM, such as OpenCLIP, provides bridging modalities, primarily text and image/vision. Target modalities such as audio, depth, thermal, tactile, EEG, video, and point cloud are then aligned to that space. The important supervision assumption is partial pairing rather than full multimodal tuples: image-text, image-depth, image-thermal, image-audio, and comparable pairwise datasets are sufficient (Chen et al., 18 May 2026).

The training process is incremental and multi-path. Each target modality is aligned with one or more bridging modalities concurrently, while the bridging modalities themselves remain semantically aligned. This avoids the requirement for fully paired all-to-all data. A technical nuance is that codebooks are not globally unified across every target modality jointly. Each target-bridge combination uses its own set of shared and specific codebooks. Thus, “unified” refers to the shared semantic basis within a target-bridge path rather than one universal codebook spanning every dataset.

Quantization is implemented through compositional vector quantization. A $d$-dimensional embedding is partitioned into $z_i^\mathcal{M} = E(x_i^\mathcal{M}),$0 subvectors,

$z_i^\mathcal{M} = E(x_i^\mathcal{M}),$1

and each subvector is quantized independently. The resulting effective capacity is

$z_i^\mathcal{M} = E(x_i^\mathcal{M}),$2

This design is used to obtain high-fidelity discrete representations without an enormous monolithic codebook. The paper presents compositional VQ as more scalable and more robust than standard VQ, especially under severe modality heterogeneity (Chen et al., 18 May 2026).

Codebook maintenance is also explicit. The shared codebook is initialized using K-means on text subvectors, specific codebooks are initialized similarly on modality-specific embeddings, and all subvectors and codevectors are normalized to unit magnitude. Codevectors are updated by exponential moving average, and underused entries are reinitialized from current minibatch anchors to prevent codebook collapse.

4. Objectives, optimization, and inference behavior

CodeBind is trained with a multi-objective loss that combines alignment, reconstruction, disentanglement, quantization, and codebook regularization. Cross-modal alignment is imposed on quantized shared embeddings using a symmetric InfoNCE objective. For a target modality $z_i^\mathcal{M} = E(x_i^\mathcal{M}),$3 and bridge modality $z_i^\mathcal{M} = E(x_i^\mathcal{M}),$4, the paper writes

$z_i^\mathcal{M} = E(x_i^\mathcal{M}),$5

with a symmetric reverse-direction term forming $z_i^\mathcal{M} = E(x_i^\mathcal{M}),$6 (Chen et al., 18 May 2026).

To preserve full modality information, the framework reconstructs each modality input from the concatenation of quantized shared and specific embeddings through a transformer reconstruction decoder:

$z_i^\mathcal{M} = E(x_i^\mathcal{M}),$7

Disentanglement is encouraged by an orthogonality term,

$z_i^\mathcal{M} = E(x_i^\mathcal{M}),$8

and a uniformity loss on specific embeddings. Quantization is stabilized by a commitment loss,

$z_i^\mathcal{M} = E(x_i^\mathcal{M}),$9

The appendix adds codebook-level regularizers, including Cross-Modal Code Matching, codevector contrastive regularization, and codevector uniformity, along with adaptive loss balancing based on EMA of loss magnitudes (Chen et al., 18 May 2026).

At inference time, the architecture becomes simpler. For cross-modal classification and retrieval, only the shared embeddings are used; the specific branch can be discarded and the reconstruction decoder is omitted. For fine-grained intra-modal tasks or multimodal fusion, the specific embeddings can be reincorporated, typically by concatenation or summation. Reported implementation details include compositional VQ with 1024 shared and 256 specific codevectors, codevector dimension 8, learning rate $$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$0, LoRA rank 4 for target encoders, an 8-layer ViT reconstruction decoder, and training on 8 NVIDIA RTX 3090 GPUs.

5. Evaluation across nine modalities

The empirical study spans nine modalities: text, image, video, audio, depth, thermal, tactile, 3D point cloud, and EEG. Evaluation covers classification and retrieval across ImageNet-1K, Places365, Kinetics-400, MSR-VTT, NYU-D, SUN-D, AudioSet, ESC, Clotho, AudioCaps, VGGSound, LLVIP, FLIR_v2, Touch-and-Go material / hard-soft / rough-smooth, ImageNet-EEG, ShapeNet55, and ModelNet40 (Chen et al., 18 May 2026).

Two principal instantiations are reported. CodeBind-IB augments ImageBind, and CodeBind-VL augments ViT-Lens. The paper also compares against broader systems including FreeBind, OmniBind, LanguageBind, OneLLM, OneEncoder, and UniAlign. Its headline claim is state-of-the-art or highly competitive multimodal classification and retrieval without relying on fully paired multimodal tuples or extensive pseudo-pair generation.

The reported gains over ImageBind are broad. Selected examples for CodeBind-IB include ImageNet 77.7 $$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$1 79.3, Places365 45.4 $$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$2 55.5, K400 50.5 $$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$3 54.4, NYU-D 54.0 $$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$4 59.3, SUN-D 35.1 $$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$5 45.7, AudioSet 17.6 $$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$6 21.1, ESC 66.9 $$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$7 71.0, LLVIP 63.4 $$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$8 95.5, FLIR_v2 46.6 $$z^\mathcal{M}_{\text{shared}}, \qquad z^\mathcal{M}_{\text{spec}}.$$9 97.2, TAG-M 24.2 $$\hat{z}^\mathcal{M}_{\text{shared}} = Q(z^\mathcal{M}_{\text{shared}}), \qquad \hat{z}^\mathcal{M}_{\text{spec}} = Q(z^\mathcal{M}_{\text{spec}}).$$0 42.6, and IN-EEG 18.4 $$\hat{z}^\mathcal{M}_{\text{shared}} = Q(z^\mathcal{M}_{\text{shared}}), \qquad \hat{z}^\mathcal{M}_{\text{spec}} = Q(z^\mathcal{M}_{\text{spec}}).$$1 33.1. The thermal and tactile gains are especially large (Chen et al., 18 May 2026).

The reported gains over ViT-Lens are similarly broad. For CodeBind-VL, the paper lists NYU-D 68.5 $$\hat{z}^\mathcal{M}_{\text{shared}} = Q(z^\mathcal{M}_{\text{shared}}), \qquad \hat{z}^\mathcal{M}_{\text{spec}} = Q(z^\mathcal{M}_{\text{spec}}).$$2 71.1, SUN-D 52.2 $$\hat{z}^\mathcal{M}_{\text{shared}} = Q(z^\mathcal{M}_{\text{shared}}), \qquad \hat{z}^\mathcal{M}_{\text{spec}} = Q(z^\mathcal{M}_{\text{spec}}).$$3 54.8, AudioSet 26.7 $$\hat{z}^\mathcal{M}_{\text{shared}} = Q(z^\mathcal{M}_{\text{shared}}), \qquad \hat{z}^\mathcal{M}_{\text{spec}} = Q(z^\mathcal{M}_{\text{spec}}).$$4 29.2, VGGSound 31.7 $$\hat{z}^\mathcal{M}_{\text{shared}} = Q(z^\mathcal{M}_{\text{shared}}), \qquad \hat{z}^\mathcal{M}_{\text{spec}} = Q(z^\mathcal{M}_{\text{spec}}).$$5 39.5, ESC 75.9 $$\hat{z}^\mathcal{M}_{\text{shared}} = Q(z^\mathcal{M}_{\text{shared}}), \qquad \hat{z}^\mathcal{M}_{\text{spec}} = Q(z^\mathcal{M}_{\text{spec}}).$$6 78.8, IN-EEG $$\hat{z}^\mathcal{M}_{\text{shared}} = Q(z^\mathcal{M}_{\text{shared}}), \qquad \hat{z}^\mathcal{M}_{\text{spec}} = Q(z^\mathcal{M}_{\text{spec}}).$$7, and ModelNet40 $$\hat{z}^\mathcal{M}_{\text{shared}} = Q(z^\mathcal{M}_{\text{shared}}), \qquad \hat{z}^\mathcal{M}_{\text{spec}} = Q(z^\mathcal{M}_{\text{spec}}).$$8. Against broader large-scale or synthetic-data-heavy baselines, the framework is often competitive despite using naturally paired data rather than extensive pseudo-pair generation or massive crowdsourced multimodal corpora.

6. Ablations, interpretation, and limitations

The ablation results attribute the performance gains to three interacting components: the codebook, the shared/specific decoupling, and reconstruction. The paper reports that adding the codebook alone already improves alignment over ImageBind, and adding decoupling and reconstruction yields further gains. A shared codebook for shared embeddings outperforms separate codebooks, with +3.2% on NYU-D and +6.9% on FLIR_v2. Compositional VQ also outperforms standard VQ by +10.8%, +5.7%, and +16.1% on three ablation datasets (Chen et al., 18 May 2026).

The codebook analysis is central to the paper’s interpretation of why CodeBind works. On FLIR_v2, compositional design increases shared codebook usage from 75.6% to 100%, with much higher perplexity, which the paper reads as reduced collapse and improved utilization. Class-wise codevector usage patterns are similar for paired modalities within a category, suggesting that semantic consistency is being enforced at the code level rather than only through global embedding similarity.

The specific branch is not treated as a residual nuisance channel. Fine-grained retrieval and probing analyses indicate that specific embeddings are better at predicting non-semantic attributes such as lighting, texture, camera geometry, scene environment, and color tone. In multimodal fusion, adding specific embeddings improves AVE event classification; with dense text descriptions, CodeBind reaches 97.3%, versus ImageBind’s 94.4%. This suggests that the framework’s “partial alignment” is not merely a regularization device but a representational claim about what multimodal correspondence should preserve.

The limitations are equally explicit. The authors note that interpreting modality-specific information in uncommon modalities remains difficult; category names are used rather than dense descriptions during alignment for fair comparison; the framework introduces additional training complexity through codebooks, multiple losses, and the reconstruction decoder; reconstruction adds training overhead although it is dropped at inference; and codebooks are trained per target-bridge path rather than as one globally shared codebook over every target modality jointly (Chen et al., 18 May 2026).

In sum, CodeBind redefines multimodal alignment as a problem of selective agreement rather than total collapse into a single latent space. Its main contribution is the claim that multimodal systems should align only the semantic invariants, preserve modality-private structure explicitly, and use a shared discrete basis to stabilize cross-modal comparison under incomplete pairing and severe modality imbalance.