RLBind: Robust Cross-Modal Alignment

• RLBind is a unified framework that aligns multimodal embeddings using text as an anchor, ensuring adversarial invariance across sensors.
• It employs a two-stage training process—unsupervised adversarial fine-tuning followed by supervised cross-modal alignment—to enhance both clean accuracy and robustness.
• Demonstrated improvements on image, audio, thermal, and video tasks highlight its potential for safer robot perception and reliable multi-sensor integration.

RLBind, introduced in "RLBind: Adversarial-Invariant Cross-Modal Alignment for Unified Robust Embeddings" (Lu, 17 Sep 2025), is a two-stage adversarial-invariant cross-modal alignment framework for unified robust embeddings. Built on LanguageBind, it treats text as the anchor modality and refines the encoders for image, audio, thermal/infrared, and video so that clean and adversarial samples remain semantically aligned in a shared embedding space. The framework targets a joint robustness–generalization problem: prior defenses can harden the visual encoder locally, but they often overlook broader cross-modal correspondence and may degrade zero-shot transfer, which is problematic for robot perception, cross-modal retrieval, and other settings that depend on a stable unified representation.

1. Problem setting and motivation

Unified multi-modal encoders map diverse sensory streams into a shared embedding space in which different modalities can be compared by simple similarity measures. In the setting addressed by RLBind, such spaces support zero-shot recognition, cross-modal retrieval, and downstream robot perception by fusing cameras, microphones, thermal sensors, and language. LanguageBind is a recent Bind-style encoder in which all modalities are explicitly aligned to a fixed language encoder via an InfoNCE contrastive loss, using text as the anchor modality (Lu, 17 Sep 2025).

The central failure mode is robustness under deployment. The visual branch is described as particularly vulnerable to adversarial perturbations and natural corruptions, including noise, blur, low light, thermal artifacts, and sensor failures. In robotics, this vulnerability is safety-relevant because vision often dominates perception; if it fails, the robot may collide, mis-grasp, or misinterpret scenes. The unified embedding must therefore remain reliable under attack if cross-modal fusion and zero-shot generalization are to be preserved.

A common misconception is that robustness in such systems can be treated as a purely vision-side issue. RLBind is motivated by the observation that existing defenses largely focus on vision-only CLIP-style encoders: supervised adversarial training such as TeCoA can degrade zero-shot performance and overfit to specific domains, while FARE in RobustCLIP improves robustness by aligning clean and perturbed image embeddings but only touches the visual encoder and does not restore LanguageBind-like multi-modal correspondence. For LanguageBind specifically, the reported behavior is that image and video accuracy collapses under adversarial attacks, and RobustCLIP-type defenses can disrupt the unified embedding and hurt zero-shot cross-modal generalization (Lu, 17 Sep 2025).

2. Position within Bind-style encoders and architectural assumptions

RLBind operates on top of LanguageBind rather than replacing its pretraining procedure. The base model is a text-anchored unified encoder in which all non-text modalities are aligned to language. RLBind keeps that text-centric design and refines it for adversarial invariance. This is significant because the text anchor serves as a semantic center across classes and modalities, allowing robustness objectives to be formulated in terms of class-wise similarity to text embeddings rather than only in terms of within-modality feature matching (Lu, 17 Sep 2025).

Architecturally, the text encoder is fixed throughout training and is specified as ViT-L/14 from OpenCLIP. The non-text encoders are also ViT-L/14-based. The image encoder is fine-tuned from OpenCLIP ViT-L/14; audio and video encoders are fully fine-tuned ViT-L/14 models in LanguageBind; and the thermal encoder uses a LoRA-based parameter-efficient fine-tuning structure in LanguageBind that RLBind can preserve where desired. All encoders map into a common embedding dimension $d$, and embeddings are normalized for contrastive learning and similarity computation.

RLBind inherits LanguageBind’s contrastive InfoNCE alignment as the base geometry. It does not re-train the base LanguageBind from scratch. Instead, it applies an unsupervised adversarial fine-tuning stage followed by a supervised cross-modal adversarial-invariant alignment stage. This design isolates robustness refinement from the original large-scale contrastive pretraining and makes the method a post hoc robustness framework for text-anchored multi-modal encoders.

3. Stage 1: unsupervised adversarial fine-tuning

The first stage applies FARE to each non-text modality encoder. The adversary is norm-bounded in $\ell_\infty$, with perturbation radius $\varepsilon \in \{2/255, 4/255\}$, and adversarial examples are generated by APGD. For a clean input $x_m$ in modality $m$, the adversarial example $z_m$ satisfies $\|z_m - x_m\|_\infty \le \varepsilon$ and is chosen to maximize the distance between the current encoder output and the reference embedding from the original pre-trained LanguageBind encoder (Lu, 17 Sep 2025).

The Stage 1 objective for modality $m$ is

$$L_{\mathrm{FARE},m}(\phi_m, x_m) = \max_{\|z_m - x_m\|_{\infty} \le \varepsilon} \bigl\|\phi_m(z_m) - \phi_{\mathrm{Org},m}(x_m)\bigr\|_{2}^{2}.$$

The rationale is that if adversarial and original embeddings remain close in $\ell_2$ norm, then their cosine similarities to text also remain close. This allows the encoder to be hardened without labels and without explicitly using text supervision during Stage 1. In RLBind, this principle is extended beyond images to all non-anchor modalities, including audio, thermal, and video.

Stage 1 uses unlabeled data from the target datasets: ImageNet-1k, ESC-50, LLVIP, and MSR-VTT. The text encoder remains frozen, while non-text encoders are either fully fine-tuned or LoRA-tuned. Training is conducted for 1 epoch. The reported implementation uses 8 GPUs with 32GB each, and full experiment sets take approximately 3 days. The output of this stage is a robustness-enhanced multi-modal encoder whose adversarial embeddings remain close to original clean embeddings. However, the paper explicitly notes that Stage 1 alone can degrade clean accuracy and does not explicitly enforce cross-modal adversarial invariance (Lu, 17 Sep 2025).

4. Stage 2: text-anchored cross-modal adversarial invariance

The second stage is the defining contribution of RLBind. It introduces supervised cross-modal alignment using class-wise text anchors. For a labeled dataset with $\ell_\infty$0 classes, each class $\ell_\infty$1 has a text embedding $\ell_\infty$2, collected into an anchor matrix $\ell_\infty$3. For a modality $\ell_\infty$4, the encoder produces a clean embedding $\ell_\infty$5 and an adversarial embedding $\ell_\infty$6. RLBind then enforces that clean and adversarial samples induce similar similarity profiles to all text anchors (Lu, 17 Sep 2025).

Several correspondence functions are considered: dot product, scaled dot product, cosine closeness, normalized Euclidean distance, bilinear mapping, and a non-linear MLP mapping. The empirical conclusion is that simple dot product yields the best robustness–accuracy trade-off, with bilinear mapping also performing well. More complex MLP mappings tend to overfit at the available data scale.

For each class, clean and adversarial similarity scores are computed against the text anchors, and these scores are assembled into class-score vectors. Adversarial-invariant cross-modal alignment requires the clean and adversarial vectors to remain close for all classes. RLBind studies two alignment strategies. The first is point-wise alignment using $\ell_\infty$7 distance with $\ell_\infty$8. The second is distribution-level alignment using symmetric KL divergence after softmax with temperature $\ell_\infty$9. The reported trade-off is that KL alignment often yields stronger robustness but can slightly reduce clean accuracy; for a balanced robustness–generalization profile, RLBind uses dot-product correspondence with point-wise $\varepsilon \in \{2/255, 4/255\}$0 alignment.

Stage 2 also includes supervised classification on both clean and adversarial inputs. Its objective is

$\varepsilon \in \{2/255, 4/255\}$1

where $\varepsilon \in \{2/255, 4/255\}$2 is either the point-wise $\varepsilon \in \{2/255, 4/255\}$3 term or the symmetric KL term, and $\varepsilon \in \{2/255, 4/255\}$4 balances classification and alignment. This stage is supervised, uses APGD adversarial examples again, and fine-tunes the Stage 1 encoders for one epoch. Its purpose is not merely to classify adversarial samples correctly, but to make the cross-modal geometry itself adversarial-invariant (Lu, 17 Sep 2025).

5. Training procedure, datasets, and empirical profile

The high-level RLBind training algorithm has four steps. First, pre-trained LanguageBind encoders are loaded, the text encoder is frozen, and non-text encoders are configured for full fine-tuning or LoRA tuning. Second, Stage 1 performs FARE fine-tuning on clean inputs and APGD adversarial variants for each modality. Third, Stage 2 generates fresh adversarial examples around labeled clean inputs, computes clean and adversarial embeddings, compares them to the class-wise text anchors, and updates the non-text encoders using clean cross-entropy, adversarial cross-entropy, and cross-modal alignment loss. Fourth, the result is a robust text-anchored encoder with adversarial-invariant cross-modal alignment (Lu, 17 Sep 2025).

Evaluation is conducted on ImageNet-1k for image, ESC-50 for audio, LLVIP for thermal/infrared, and MSR-VTT for video. Stage 1 uses the full training sets. Stage 2 uses sampled training sets with 10 examples per class, or all examples if fewer, due to memory constraints and for fair evaluation. The attack is APGD under an $\varepsilon \in \{2/255, 4/255\}$5 constraint, with perturbation strengths $\varepsilon \in \{2/255, 4/255\}$6 and $\varepsilon \in \{2/255, 4/255\}$7. The metrics are clean accuracy and robust accuracy under attack.

The baseline LanguageBind model is highly brittle under attack. Under $\varepsilon \in \{2/255, 4/255\}$8, image robust accuracy is reported as $\varepsilon \in \{2/255, 4/255\}$9 and video robust accuracy as $x_m$0, which the paper characterizes as effective collapse. Stage 1 FARE substantially improves robustness but can incur nontrivial clean-accuracy degradation, especially on thermal data. Stage 2 RLBind then improves both robustness and clean accuracy relative to both the baseline and the FARE-only intermediate model.

Modality	LanguageBind clean / robust @ $x_m$1	RLBind4 clean / robust @ $x_m$2
Image	74.07 / 9.12, 2.84	77.39 / 51.62, 33.02
Audio	94.75 / 19.45, 14.25	92.50 / 34.25, 18.00
Thermal	81.66 / 23.86, 21.22	85.00 / 85.00, 85.00
Video	42.74 / 1.21, 0.13	51.58 / 38.95, 32.63

The RLBind2 variant also shows strong gains. For image, it reaches $x_m$3 clean accuracy and robust accuracies of $x_m$4 and $x_m$5. For thermal, RLBind2 reaches $x_m$6 clean accuracy and $x_m$7 robust accuracy at both perturbation strengths. For video, it improves from a baseline robust accuracy of $x_m$8 at $x_m$9 to $m$0.

The ablation studies isolate the contribution of Stage 2. Relative to FARE2, RLBind2 improves image clean accuracy by $m$1 and robust accuracy by $m$2 at $m$3 and $m$4 at $m$5; for thermal, the gains are $m$6 clean and $m$7, $m$8 robust. Relative to FARE4, RLBind4 improves image clean accuracy by $m$9 and robust accuracy by $z_m$0, $z_m$1; thermal by $z_m$2 clean and $z_m$3, $z_m$4; and video by $z_m$5 clean and $z_m$6, $z_m$7. These reported deltas support the paper’s conclusion that cross-modal adversarial-invariant alignment is essential rather than merely auxiliary.

A separate loss-function ablation clarifies the robustness–accuracy trade-off. Clean cross-entropy alone produces high clean accuracy, exemplified by $z_m$8 in RLBind2, but robust accuracy collapses to $z_m$9 and $\|z_m - x_m\|_\infty \le \varepsilon$0. Adding adversarial cross-entropy improves robustness to $\|z_m - x_m\|_\infty \le \varepsilon$1 and $\|z_m - x_m\|_\infty \le \varepsilon$2, at the cost of reducing clean accuracy to $\|z_m - x_m\|_\infty \le \varepsilon$3. Adding cross-modal alignment then yields $\|z_m - x_m\|_\infty \le \varepsilon$4 clean accuracy with $\|z_m - x_m\|_\infty \le \varepsilon$5 and $\|z_m - x_m\|_\infty \le \varepsilon$6 robust accuracy, which the paper interprets as alleviating the robustness–accuracy trade-off.

The parameterization study compares LoRA against full fine-tuning. Fully fine-tuned RLBind2 reports $\|z_m - x_m\|_\infty \le \varepsilon$7 clean accuracy and $\|z_m - x_m\|_\infty \le \varepsilon$8, $\|z_m - x_m\|_\infty \le \varepsilon$9 robust accuracy; fully fine-tuned RLBind4 reports $m$0 clean and $m$1, $m$2 robust. LoRA-tuned RLBind4 reports $m$3 clean and $m$4, $m$5 robust. The result suggests that RLBind improves robustness under both full and parameter-efficient training regimes (Lu, 17 Sep 2025).

6. Robotics relevance, limitations, and future directions

RLBind is explicitly motivated by robot perception stacks in which RGB cameras, thermal cameras, microphones, depth sensors, and language interfaces must be integrated into a unified semantic space. In such systems, robust visual embeddings are important for navigation, manipulation, and human-robot interaction, while adversarial-invariant cross-modal alignment preserves the semantic coherence needed for zero-shot understanding, cross-modal retrieval, and fusion across heterogeneous sensors. The reported claim is that RLBind provides a practical path toward safer multi-sensor perception stacks for embodied robots in navigation, manipulation, and other autonomy settings (Lu, 17 Sep 2025).

The limitations are also explicit. Evaluation is restricted to norm-bounded $m$6 APGD attacks; physical patch attacks, temporal adversarial video attacks, and missing-modality attacks are not tested. Experiments are conducted on public benchmarks rather than the full proprietary VIDAL-10M dataset used to train LanguageBind. The framework assumes reliable text anchors and labeled data for Stage 2, so noisy or mismatched text descriptions may weaken alignment. It also incurs substantial computational overhead because APGD-based adversarial training is expensive.

Future directions named or implied in the paper include scaling to VIDAL-10M, extending to additional modalities such as depth and point clouds, exploring adaptive anchors and distribution-aware objectives, and broadening robustness to temporal attacks, multimodal joint attacks, and missing or corrupted modalities. Integration with reinforcement learning or online adaptation on robots is also identified as a promising direction, in which the multi-modal encoder becomes part of a perception–decision loop rather than a standalone representation module.