Shared Cross-Modal Adapter (SCMA)

• The paper demonstrates that SCMA is a parameter-efficient multimodal adaptation approach that shares lightweight adapter parameters across modalities and layers, enabling specialization of a frozen backbone.
• It introduces a shared residual module that aligns visual and textual branches by reducing feature discrepancies and acting as an alignment bottleneck.
• Empirical studies report significant gains, with SCMA achieving up to 666× fewer tuned parameters while improving performance metrics such as SROCC and PLCC.

Shared Cross-Modal Adapter (SCMA) denotes a parameter-efficient multimodal adaptation pattern in which a lightweight adapter transformation is shared across modalities, and in some formulations across layers or clients, so that a frozen or largely frozen backbone can be specialized without duplicating full model parameters. The term is used explicitly in "Q-CLIP: Unleashing the Power of Vision-LLMs for Video Quality Assessment through Unified Cross-Modal Adaptation" (Mi et al., 8 Aug 2025), where identical adapter parameters are applied to both visual and textual branches and reused across inserted layers. Closely related formulations appear as a globally shared projection that aligns cross-modal features in "pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-LLMs" (Ghiasvand et al., 7 Jul 2025) and as a shallow shared feature bridge between modalities in "DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter" (Li et al., 3 Aug 2025). In this sense, SCMA is best understood as a family of sharing-centric adapter designs within the broader literature on cross-modal adapter tuning.

1. Conceptual lineage within adapter-based multimodal transfer

The immediate lineage of SCMA lies in earlier cross-modal adapter work that sought to retain the benefits of large pre-trained vision-language backbones while reducing the cost and overfitting risk of full fine-tuning. "Cross-Modal Adapter for Text-Video Retrieval" (Jiang et al., 2022) introduced a Cross-Modal Adapter for CLIP-based text-video retrieval that was explicitly designed for the multi-modal domain and allowed early cross-modal interactions between CLIP's two encoders. Its reported benefits were that it reduced 99.6\% of fine-tuned parameters, saved approximately 30\% of training time, and fixed all pre-trained parameters so that the pre-trained model could be shared across datasets; it achieved superior or comparable performance to fully fine-tuned methods on MSR-VTT, MSVD, VATEX, ActivityNet, and DiDeMo.

Subsequent work expanded the same parameter-efficient agenda beyond text-video retrieval. "Towards Versatile and Efficient Visual Knowledge Integration into Pre-trained LLMs with Cross-Modal Adapters" (Zhang et al., 2023) proposed X-adapter as a plug-and-play module inserted into frozen PLMs, with separate V-expert and T-expert sub-modules for integrating image and text representations from a VLM. "Cross-Modal Adapter: Parameter-Efficient Transfer Learning Approach for Vision-LLMs" (Yang et al., 2024) introduced XMAdapter, which established cache models for both image and text modalities and fused their affinities through a dynamically adjusted ratio. These works did not all use the label SCMA, but they collectively established the premise that explicit cross-modal interaction can be concentrated in a small trainable module rather than dispersed across a fully fine-tuned backbone. This suggests that SCMA inherits the core adapter logic of frozen foundation encoders plus lightweight multimodal specialization.

2. Core architectural principle: sharing as the alignment bottleneck

In its most explicit form, SCMA is a shared residual refinement module inserted symmetrically into both branches of a vision-LLM. In Q-CLIP, SCMA is placed in both visual and textual branches, with two variants: E-SCMA, aligned with encoder layers, and P-SCMA, applied after projection modules. The same architecture is shared across both modalities, and for E-SCMA the same parameters are reused at all inserted layers to avoid parameter explosion (Mi et al., 8 Aug 2025). The encoder-side update is described as

$V_k' = E_k^v(V_k), \qquad T_k' = E_k^t(T_k)$

$$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$

$$\Delta V_k' = LayerNorm(\Delta V_k), \qquad \Delta T_k' = LayerNorm(\Delta T_k)$$

$$V_{k+1} = V_k' + \alpha_k \Delta V_k', \qquad T_{k+1} = T_k' + \beta_k \Delta T_k'.$$

Here the shared pathway is not merely a parameter-saving device; it is intended to reduce feature distribution discrepancies between visual and textual branches and improve precise cross-modal alignment.

A closely related but structurally different formulation appears in pFedMMA. There, each modality has its own down-projection and up-projection, but both branches pass through the same shared projection matrix $W_{js}$, which is also the only component synchronized across clients in the federated setting (Ghiasvand et al., 7 Jul 2025). The modality-specific adapters are

$$\mathcal{A}_j^{(I)}(\boldsymbol{z}_j^{(I)}) = W_{ju}^{(I)} \cdot \delta\left(W_{js} \cdot \delta\left(W_{jd}^{(I)} \cdot \boldsymbol{z}_j^{(I)}\right)\right),$$

$$\mathcal{A}_j^{(T)}(\boldsymbol{z}_j^{(T)}) = W_{ju}^{(T)} \cdot \delta\left(W_{js} \cdot \delta\left(W_{jd}^{(T)} \cdot \boldsymbol{z}_j^{(T)}\right)\right).$$

This design makes the shared module a cross-modal alignment bottleneck while leaving the surrounding projections modality-specific and client-private.

3. Major implementation variants

The SCMA pattern has been instantiated in several distinct operational regimes. The common feature is parameter sharing across modalities; the differences concern whether sharing is extended across layers, tasks, or clients.

Work	Task context	Shared component
"Q-CLIP: Unleashing the Power of Vision-LLMs for Video Quality Assessment through Unified Cross-Modal Adaptation" (Mi et al., 8 Aug 2025)	Video Quality Assessment	One set of SCMA parameters shared across visual and textual branches and reused across inserted layers
"pFedMMA: Personalized Federated Fine-Tuning with Multi-Modal Adapter for Vision-LLMs" (Ghiasvand et al., 7 Jul 2025)	Personalized federated VLM adaptation	Globally shared projection $W_{js}$ inside modality-specific adapters; only this component is communicated and aggregated
"DMTrack: Spatio-Temporal Multimodal Tracking via Dual-Adapter" (Li et al., 3 Aug 2025)	Spatio-temporal multimodal tracking	Shallow adapter with shared FC layers used by both modality branches; outputs are added to the other branch

In Q-CLIP, SCMA is part of a fully VLMs-based framework for VQA and is coupled with five learnable quality-level prompts that guide the model in perceiving subtle quality variations (Mi et al., 8 Aug 2025). In pFedMMA, the shared projection is the globally synchronized component that supports generalization across heterogeneous clients, while modality-specific projections remain local for personalization (Ghiasvand et al., 7 Jul 2025). In DMTrack, the shallow adapter is explicitly parameter-shared between RGB and an auxiliary modality and acts as a “feature bridge” before a deeper, non-shared adapter performs pixel-wise attention-based refinement (Li et al., 3 Aug 2025). These are not identical architectures, but all instantiate the same underlying principle: shared low-dimensional transformation as the locus of cross-modal coupling.

4. Parameter efficiency, freezing, and system-level consequences

A defining property of SCMA-style methods is that they relocate adaptation into a very small trainable subspace. In Q-CLIP, SCMA contains only about 0.14 million trainable parameters in total, while the VLM backbone, encoders, and projection heads remain frozen (Mi et al., 8 Aug 2025). The reported parameter-efficiency comparisons are unusually large: relative to MBVQA, CLIP-VQA, Q-Align, and FAST-VQA, Q-CLIP is reported as 666× fewer, 3964× fewer, 58,556× fewer, and 197× more parameter-efficient, respectively. In the earlier Cross-Modal Adapter for text-video retrieval, the comparable claim was a 99.6\% reduction in fine-tuned parameters and approximately 30\% savings in training time, again with all pre-trained parameters fixed (Jiang et al., 2022).

In federated settings, sharing also has a communication-theoretic role. pFedMMA communicates only the small, low-dimensional shared projection matrix $W_{js}$ between clients and server, while up-projection and down-projection layers remain purely local (Ghiasvand et al., 7 Jul 2025). This asymmetric optimization strategy ties SCMA-like sharing to a specific systems objective: improving global generalization without forcing full parameter synchronization.

A plausible implication is that SCMA should not be reduced to “small adapters” alone. The stronger claim made by this line of work is that the location and scope of sharing determine whether parameter efficiency is accompanied by alignment, generalization, and deployability.

5. Empirical behavior and ablation evidence

The most direct empirical evidence for SCMA comes from ablation studies on whether sharing is actually beneficial. In Q-CLIP, adding SCMA to only one branch produced limited gains, whereas applying it to both branches improved performance more substantially; explicit parameter sharing across branches improved performance further, and adding layer sharing gave the best result on LSVQ (Mi et al., 8 Aug 2025). The reported values were: visual only, SROCC $=0.866$ and PLCC $$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$0; text only, SROCC $$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$1 and PLCC $$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$2; both without sharing, SROCC $$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$3 and PLCC $$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$4; both shared, SROCC $$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$5 and PLCC $$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$6; shared plus layer-sharing, SROCC $$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$7 and PLCC $$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$8. The same study reports that more E-SCMA layers further boost performance and that t-SNE visualizations show better separation of modalities and quality levels.

DMTrack provides complementary evidence that sharing is useful at the foundational alignment stage but need not be universal. Removing the shallow shared adapter reduced performance from 60.3 to 59.5 on LasHeR, from 62.4 to 62.1 on VisEvent, and from 64.7 to 62.4 on DepthTrack, with an average drop of $$\Delta V_k = Up_k(FFN(Down_k(V_k))), \qquad \Delta T_k = Up_k(FFN(Down_k(T_k)))$$9 (Li et al., 3 Aug 2025). In a second ablation, the progressive organization SA + DA outperformed SA + SA, DA + DA, and DA + SA, with the reported scores 60.3, 62.4, and 64.7 for LasHeR, VisEvent, and DepthTrack, respectively. This is a precise indication that a shared shallow bridge and a non-shared deeper refinement are not interchangeable.

Broader parameter-efficient multimodal adaptation results point in the same direction even when the module is not named SCMA. XMAdapter reported that cross-modal fusion with both text and image caches outperformed prior adapter-based methods across 15 benchmark datasets, and its ablation study found the best result at $$\Delta V_k' = LayerNorm(\Delta V_k), \qquad \Delta T_k' = LayerNorm(\Delta T_k)$$0, rather than at a unimodal extreme (Yang et al., 2024). Although XMAdapter is a cache-based cross-modal adapter rather than a shared branch-wise residual module, it reinforces the more general empirical lesson that explicit use of both modalities is advantageous.

6. Relation to adjacent adapter paradigms and recurring misconceptions

SCMA is not synonymous with all cross-modal adapters. UniCrossAdapter for radiology report generation distributes lightweight adapters across both the visual and textual branches of CLIP after each of the last three blocks, keeps all CLIP parameters frozen, and uses separate unimodal adaptation, cross-modal adaptation, and multi-scale fusion stages (Chen et al., 20 Mar 2025). Its detailed comparison is explicit: a hypothetical SCMA would use a single set of cross-modal adapter weights shared across all layers and possibly between modalities, whereas UniCrossAdapter is layerwise non-shared and includes separate unimodal and cross-modal branches. The authors state that this provides greater expressivity per layer at marginal parameter cost. In this literature, therefore, SCMA is one design point within a larger space of multimodal PEFT architectures.

A common misconception is that “shared” implies that every adapter parameter should be universal across modalities and layers. The surveyed evidence does not support such a single prescription. Q-CLIP reports gains from both branch-wise and layer-wise sharing (Mi et al., 8 Aug 2025); pFedMMA shares only the central projection while keeping surrounding projections local and modality-specific (Ghiasvand et al., 7 Jul 2025); DMTrack shares only the shallow adapter and leaves the deep adapter unshared (Li et al., 3 Aug 2025). Conversely, the continual learning framework in "Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation" (Chee et al., 10 Nov 2025) treats SCMA as a baseline-style concept in which a single adapter module may be shared across all modalities and possibly all layers or blocks, and argues that a mixture-of-experts cross-modality adapter is more flexible for continual learning because it allows specialization and freezing of experts across tasks. This suggests that SCMA is best characterized as a sharing principle—typically a single shared bottleneck or bridge for cross-modal alignment—rather than a single canonical topology.

The broader adapter literature reinforces this interpretation. X-adapter, for example, showed that plug-and-play modules with V-expert and T-expert branches can inject aligned image and text knowledge from VLMs into frozen PLMs while updating only added parameters (Zhang et al., 2023). The existence of such alternatives clarifies the specific commitment of SCMA: not merely multimodal fusion, but multimodal fusion under explicit parameter sharing.