---
title: Shared Cross-Modal Adapter (SCMA)
url: https://www.emergentmind.com/topics/shared-cross-modal-adapter-scma
type: topic
---

# Shared Cross-Modal Adapter (SCMA)

Shared Cross-Modal Adapter (SCMA) denotes a parameter-efficient multimodal adaptation pattern in which a lightweight adapter transformation is shared across modalities, and in some formulations across layers or clients, so that a frozen or largely frozen backbone can be specialized without duplicating full model parameters. The term is used explicitly in "Q-CLIP: Unleashing the Power of Vision-Language Models for Video Quality Assessment through Unified Cross-Modal Adaptation" [2508.06092], where identical adapter parameters are applied to both visual and textual branches and reused across inserted layers. Closely related formulations appear as a globally shared projection that aligns cross-modal features in "pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models" [2507.05394] and as a shallow shared feature bridge between modalities in "DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter" [2508.01592]. In this sense, SCMA is best understood as a family of sharing-centric adapter designs within the broader literature on cross-modal adapter tuning.

## 1. Conceptual lineage within adapter-based multimodal transfer

The immediate lineage of SCMA lies in earlier cross-modal adapter work that sought to retain the benefits of large pre-trained vision-language backbones while reducing the cost and overfitting risk of full fine-tuning. "Cross-Modal Adapter for Text-Video Retrieval" [2211.09623] introduced a Cross-Modal Adapter for CLIP-based text-video retrieval that was explicitly designed for the multi-modal domain and allowed early cross-modal interactions between CLIP's two encoders. Its reported benefits were that it reduced **99.6\%** of fine-tuned parameters, saved approximately 30\% of training time, and fixed all pre-trained parameters so that the pre-trained model could be shared across datasets; it achieved superior or comparable performance to fully fine-tuned methods on MSR-VTT, MSVD, VATEX, ActivityNet, and DiDeMo.

Subsequent work expanded the same parameter-efficient agenda beyond text-video retrieval. "Towards Versatile and Efficient Visual Knowledge Integration into Pre-trained Language Models with Cross-Modal Adapters" [2305.07358] proposed X-adapter as a plug-and-play module inserted into frozen PLMs, with separate V-expert and T-expert sub-modules for integrating image and text representations from a VLM. "Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-Language Models" [2404.12588] introduced XMAdapter, which established cache models for both image and text modalities and fused their affinities through a dynamically adjusted ratio. These works did not all use the label SCMA, but they collectively established the premise that explicit cross-modal interaction can be concentrated in a small trainable module rather than dispersed across a fully fine-tuned backbone. This suggests that SCMA inherits the core adapter logic of frozen foundation encoders plus lightweight multimodal specialization.

## 2. Core architectural principle: sharing as the alignment bottleneck

In its most explicit form, SCMA is a shared residual refinement module inserted symmetrically into both branches of a vision-language model. In Q-CLIP, SCMA is placed in both visual and textual branches, with two variants: E-SCMA, aligned with encoder layers, and P-SCMA, applied after projection modules. The same architecture is shared across both modalities, and for E-SCMA the same parameters are reused at all inserted layers to avoid parameter explosion [2508.06092]. The encoder-side update is described as
$$
V_k' = E_k^v(V_k), \qquad T_k' = E_k^t(T_k)
$$
$$
\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))
$$
$$
\Delta V_k' = LayerNorm(\Delta V_k), \qquad \Delta T_k' = LayerNorm(\Delta T_k)
$$
$$
V_{k+1} = V_k' + \alpha_k \Delta V_k', \qquad T_{k+1} = T_k' + \beta_k \Delta T_k'.
$$
Here the shared pathway is not merely a parameter-saving device; it is intended to reduce feature distribution discrepancies between visual and textual branches and improve precise cross-modal alignment.

A closely related but structurally different formulation appears in pFedMMA. There, each modality has its own down-projection and up-projection, but both branches pass through the same shared projection matrix $W_{js}$, which is also the only component synchronized across clients in the federated setting [2507.05394]. The modality-specific adapters are
$$
\mathcal{A}_j^{(I)}(\boldsymbol{z}_j^{(I)}) = W_{ju}^{(I)} \cdot \delta\left(W_{js} \cdot \delta\left(W_{jd}^{(I)} \cdot \boldsymbol{z}_j^{(I)}\right)\right),
$$
$$
\mathcal{A}_j^{(T)}(\boldsymbol{z}_j^{(T)}) = W_{ju}^{(T)} \cdot \delta\left(W_{js} \cdot \delta\left(W_{jd}^{(T)} \cdot \boldsymbol{z}_j^{(T)}\right)\right).
$$
This design makes the shared module a cross-modal alignment bottleneck while leaving the surrounding projections modality-specific and client-private.

## 3. Major implementation variants

The SCMA pattern has been instantiated in several distinct operational regimes. The common feature is parameter sharing across modalities; the differences concern whether sharing is extended across layers, tasks, or clients.

| Work | Task context | Shared component |
|---|---|---|
| "Q-CLIP: Unleashing the Power of Vision-Language Models for Video Quality Assessment through Unified Cross-Modal Adaptation" [2508.06092] | Video Quality Assessment | One set of SCMA parameters shared across visual and textual branches and reused across inserted layers |
| "pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-Language Models" [2507.05394] | Personalized federated VLM adaptation | Globally shared projection $W_{js}$ inside modality-specific adapters; only this component is communicated and aggregated |
| "DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter" [2508.01592] | Spatio-temporal multimodal tracking | Shallow adapter with shared FC layers used by both modality branches; outputs are added to the other branch |

In Q-CLIP, SCMA is part of a fully VLMs-based framework for VQA and is coupled with five learnable quality-level prompts that guide the model in perceiving subtle quality variations [2508.06092]. In pFedMMA, the shared projection is the globally synchronized component that supports generalization across heterogeneous clients, while modality-specific projections remain local for personalization [2507.05394]. In DMTrack, the shallow adapter is explicitly parameter-shared between RGB and an auxiliary modality and acts as a “feature bridge” before a deeper, non-shared adapter performs pixel-wise attention-based refinement [2508.01592]. These are not identical architectures, but all instantiate the same underlying principle: shared low-dimensional transformation as the locus of cross-modal coupling.

## 4. Parameter efficiency, freezing, and system-level consequences

A defining property of SCMA-style methods is that they relocate adaptation into a very small trainable subspace. In Q-CLIP, SCMA contains only about **0.14 million** trainable parameters in total, while the VLM backbone, encoders, and projection heads remain frozen [2508.06092]. The reported parameter-efficiency comparisons are unusually large: relative to MBVQA, CLIP-VQA, Q-Align, and FAST-VQA, Q-CLIP is reported as **666× fewer**, **3964× fewer**, **58,556× fewer**, and **197×** more parameter-efficient, respectively. In the earlier Cross-Modal Adapter for text-video retrieval, the comparable claim was a **99.6\%** reduction in fine-tuned parameters and approximately 30\% savings in training time, again with all pre-trained parameters fixed [2211.09623].

In federated settings, sharing also has a communication-theoretic role. pFedMMA communicates only the small, low-dimensional shared projection matrix $W_{js}$ between clients and server, while up-projection and down-projection layers remain purely local [2507.05394]. This asymmetric optimization strategy ties SCMA-like sharing to a specific systems objective: improving global generalization without forcing full parameter synchronization.

A plausible implication is that SCMA should not be reduced to “small adapters” alone. The stronger claim made by this line of work is that the *location* and *scope* of sharing determine whether parameter efficiency is accompanied by alignment, generalization, and deployability.

## 5. Empirical behavior and ablation evidence

The most direct empirical evidence for SCMA comes from ablation studies on whether sharing is actually beneficial. In Q-CLIP, adding SCMA to only one branch produced limited gains, whereas applying it to both branches improved performance more substantially; explicit parameter sharing across branches improved performance further, and adding layer sharing gave the best result on LSVQ [2508.06092]. The reported values were: visual only, SROCC \(=0.866\) and PLCC \(=0.864\); text only, SROCC \(=0.837\) and PLCC \(=0.838\); both without sharing, SROCC \(=0.875\) and PLCC \(=0.878\); both shared, SROCC \(=0.885\) and PLCC \(=0.886\); shared plus layer-sharing, **SROCC \(=0.895\)** and **PLCC \(=0.897\)**. The same study reports that more E-SCMA layers further boost performance and that t-SNE visualizations show better separation of modalities and quality levels.

DMTrack provides complementary evidence that sharing is useful at the foundational alignment stage but need not be universal. Removing the shallow shared adapter reduced performance from 60.3 to 59.5 on LasHeR, from 62.4 to 62.1 on VisEvent, and from 64.7 to 62.4 on DepthTrack, with an average drop of \(-1.13\) [2508.01592]. In a second ablation, the progressive organization **SA + DA** outperformed **SA + SA**, **DA + DA**, and **DA + SA**, with the reported scores 60.3, 62.4, and 64.7 for LasHeR, VisEvent, and DepthTrack, respectively. This is a precise indication that a shared shallow bridge and a non-shared deeper refinement are not interchangeable.

Broader parameter-efficient multimodal adaptation results point in the same direction even when the module is not named SCMA. XMAdapter reported that cross-modal fusion with both text and image caches outperformed prior adapter-based methods across 15 benchmark datasets, and its ablation study found the best result at \(\gamma = 0.7\), rather than at a unimodal extreme [2404.12588]. Although XMAdapter is a cache-based cross-modal adapter rather than a shared branch-wise residual module, it reinforces the more general empirical lesson that explicit use of both modalities is advantageous.

## 6. Relation to adjacent adapter paradigms and recurring misconceptions

SCMA is not synonymous with all cross-modal adapters. UniCrossAdapter for radiology report generation distributes lightweight adapters across both the visual and textual branches of CLIP after each of the last three blocks, keeps all CLIP parameters frozen, and uses separate unimodal adaptation, cross-modal adaptation, and multi-scale fusion stages [2503.15940]. Its detailed comparison is explicit: a hypothetical SCMA would use a single set of cross-modal adapter weights shared across all layers and possibly between modalities, whereas UniCrossAdapter is layerwise non-shared and includes separate unimodal and cross-modal branches. The authors state that this provides greater expressivity per layer at marginal parameter cost. In this literature, therefore, SCMA is one design point within a larger space of multimodal PEFT architectures.

A common misconception is that “shared” implies that every adapter parameter should be universal across modalities and layers. The surveyed evidence does not support such a single prescription. Q-CLIP reports gains from both branch-wise and layer-wise sharing [2508.06092]; pFedMMA shares only the central projection while keeping surrounding projections local and modality-specific [2507.05394]; DMTrack shares only the shallow adapter and leaves the deep adapter unshared [2508.01592]. Conversely, the continual learning framework in "Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation" [2511.06723] treats SCMA as a baseline-style concept in which a single adapter module may be shared across all modalities and possibly all layers or blocks, and argues that a mixture-of-experts cross-modality adapter is more flexible for continual learning because it allows specialization and freezing of experts across tasks. This suggests that SCMA is best characterized as a *sharing principle*—typically a single shared bottleneck or bridge for cross-modal alignment—rather than a single canonical topology.

The broader adapter literature reinforces this interpretation. X-adapter, for example, showed that plug-and-play modules with V-expert and T-expert branches can inject aligned image and text knowledge from VLMs into frozen PLMs while updating only added parameters [2305.07358]. The existence of such alternatives clarifies the specific commitment of SCMA: not merely multimodal fusion, but multimodal fusion under explicit parameter sharing.

Source: https://www.emergentmind.com/topics/shared-cross-modal-adapter-scma