AdaRing: Efficient Vision-Language Adaptation

• AdaRing is a fine-tuning framework for large vision-language models that uses cross-layer tensor ring decomposition to enable ultra-light adaptation.
• It employs diverse rank-driven adapters, coupling fine-grained and coarse-grained modules via a learnable combinator to balance discriminative power and generalization.
• Empirical results on 11 benchmarks show state-of-the-art performance with up to 90% reduction in trainable parameters compared to traditional methods.

Searching arXiv for AdaRing and related PEFT/VLM context. AdaRing is a parameter-efficient fine-tuning framework for large vision-LLMs such as CLIP that combines cross-layer tensor ring decomposition, diverse rank-driven adapters, and a generalization-aware training objective to achieve ultra-light adaptation on downstream tasks (Huang et al., 16 Aug 2025). It is situated within the broader parameter-efficient fine-tuning paradigm for frozen pre-trained models, alongside adapter tuning and low-rank update methods such as LoRA (Hu et al., 2021), but it is distinguished by explicitly modeling cross-layer redundancy among adapters and by coupling adapters of different effective capacity within each transformer layer (Huang et al., 16 Aug 2025).

1. Position within vision-language adaptation

Large vision-LLMs, including CLIP, are pre-trained on massive image-text corpora and later adapted to downstream tasks such as classification, retrieval, or VQA (Radford et al., 2021). In standard parameter-efficient fine-tuning, the backbone is frozen and only inserted modules such as adapters, prompts, or low-rank factors are optimized. The data identify three common scaling strategies in CLIP-style adapter tuning: inserting adapters into every layer, using multiple adapters per layer, and applying low-rank matrix decomposition per adapter (Huang et al., 16 Aug 2025).

AdaRing is motivated by two limitations of these strategies. First, independently decomposing each adapter per layer yields a limited compression rate because redundancy across layers is ignored, so the parameter count grows linearly with depth. Second, homogeneous adapters with the same rank or structure across layers have limited representational capacity because they tend to learn similar features, which can weaken generalization across varied tasks (Huang et al., 16 Aug 2025). AdaRing addresses both limitations by stacking adapters across layers into a single tensorized object and decomposing that object with tensor ring decomposition, while also introducing fine-grained and coarse-grained adapters that cooperate through a learnable combinator (Huang et al., 16 Aug 2025).

A naming ambiguity occasionally arises because the literature also contains the edge-training system "RingAda" (Li et al., 27 Feb 2025) and the algebraic acronym ADR associated with Auslander-Dlab-Ringel constructions (Conde et al., 2017). In the present usage, AdaRing denotes the vision-language fine-tuning framework introduced in "AdaRing: Towards Ultra-Light Vision-Language Adaptation via Cross-Layer Tensor Ring Decomposition" (Huang et al., 16 Aug 2025).

2. Cross-layer tensor ring decomposition

The technical core of AdaRing is the reformulation of layerwise adapter weights as a single high-order tensor across both feature dimensions and the layer dimension (Huang et al., 16 Aug 2025). In ordinary low-rank adapter tuning, each layer $l$ has an adapter weight matrix

$$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$

AdaRing stacks these matrices across layers and reshapes them into

$\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$

with

$$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$

The layer index becomes an explicit tensor mode, allowing cross-layer structure to be factorized jointly rather than independently (Huang et al., 16 Aug 2025).

Given a $d$-order tensor $\bm{\mathcal{A}}$, tensor ring decomposition represents it by 3-order cores $\bm{\mathcal{G}^j}$ with ring ranks $R_0,\dots,R_{d-1}$, satisfying $R_0 = R_d$: $\bm{\mathcal{A}_{(i_1, \dots, i_d)} = \sum_{r_0 = r_d,\, r_1,\dots,r_{d-1}}^{R_0,\dots,R_{d-1}} \bm{\mathcal{G}^1}_{(r_0, i_1, r_1)} \, \bm{\mathcal{G}^2}_{(r_1, i_2, r_2)} \cdots \bm{\mathcal{G}^d}_{(r_{d-1}, i_d, r_d)}. \tag{1}$ AdaRing applies this decomposition directly to the stacked adapter tensor, obtaining $$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$0 tensor-ring cores: $$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$1 The core $$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$2 is special because it handles the layer dimension $$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$3, and its ranks control the degree of cross-layer compression (Huang et al., 16 Aug 2025).

This formulation suggests that AdaRing compresses not only intra-layer low-rank structure but also inter-layer redundancy. The data further note that CLIP transformer layers exhibit cosine similarities greater than 80% even across distant layers, which provides an empirical rationale for cross-layer sharing (Huang et al., 16 Aug 2025).

3. Shared cores, layer slices, and forward computation

AdaRing separates the tensor-ring representation into layer-shared tensor cores and a layer-specific slice (Huang et al., 16 Aug 2025). The shared cores are

$$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$4

which do not depend on $$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$5 and encode common transformation structure across all adapterized layers. For each layer $$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$6, the layer-specific component is the slice

$$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$7

In the AdaRing design, the shared cores capture global adapter structure reused across layers, while $$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$8 provides a low-dimensional per-layer refinement (Huang et al., 16 Aug 2025).

For a layer-$$\bm{A}_l \in \mathbb{R}^{I \times O}, \quad l = 1,\dots,L.$$9 adapter $\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$0, the input vector $\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$1 is first tensorized into $\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$2, then multiplied through the shared cores and the layer slice $\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$3, producing $\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$4, and finally vectorized back to $\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$5: $\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$6 The adapter output is then added to the frozen backbone output in the usual adapter or LoRA pattern (Huang et al., 16 Aug 2025).

The trainable parameter count for the tensor-ring adapter in one encoder is

$\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$7

By contrast, a standard per-layer matrix low-rank adapter has

$\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$8

The first expression isolates the layer-dependent term to a low-rank factor involving $\bm{\mathcal{A} \in \mathbb{R}^{I_1 \times \cdots \times I_p \times L \times O_1 \times \cdots \times O_q},$9, while the remaining shared-core terms are independent of the number of layers (Huang et al., 16 Aug 2025). This is the principal mechanism by which AdaRing attains strong compression.

4. Diverse rank-driven adapters and combinational cooperation

AdaRing does not rely on a single homogeneous adapter family. Instead, it uses two TR-structure adapters per layer: a fine-grained adapter with relatively large layer ranks and a coarse-grained adapter with extremely small layer ranks, potentially rank $$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$0 (Huang et al., 16 Aug 2025). The paper’s Observation 3 states that higher-rank adapters have stronger discriminative capacity on base tasks, while lower-rank adapters are more generalizable and preserve behavior closer to the frozen CLIP backbone (Huang et al., 16 Aug 2025).

For the $$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$1-th visual encoder layer, the output is

$$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$2

and for the textual encoder,

$$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$3

The mixing coefficients $$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$4 are generated by a combinator implemented as a single learnable linear layer (Huang et al., 16 Aug 2025).

This design yields an explicit division of labor. The fine-grained branch emphasizes discriminative adaptation for seen classes, whereas the coarse-grained branch preserves CLIP-like generalization for unseen classes (Huang et al., 16 Aug 2025). The combinator makes the interaction data-adaptive, which the data describe as cooperation between adapters. A plausible implication is that AdaRing operationalizes a capacity-generalization tradeoff at the level of per-sample adapter mixing rather than only through static hyperparameter choice.

5. Training objective, initialization, and integration into CLIP

AdaRing is integrated into CLIP by fully freezing both the visual encoder $$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$5 and the textual encoder $$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$6, while inserting two TR-structure adapters per transformer layer in both encoders (Huang et al., 16 Aug 2025). The rest of CLIP remains untouched. The training setting reported in the data uses few-shot adaptation with 16 samples per class, ViT-B/16 CLIP, batch size 128 for ImageNet and 16 for other datasets, and 10 epochs per experiment (Huang et al., 16 Aug 2025).

The optimization objective combines a standard CLIP-style classification loss with a regularizer that keeps adapted visual embeddings close to those produced by frozen CLIP: $$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$7

$$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$8

$$I = \prod_{j=1}^p I_j, \quad O = \prod_{j=1}^q O_j.$$9

Here, $d$0 is the preservation ratio (Huang et al., 16 Aug 2025).

The empirical behavior of $d$1 is explicitly characterized in the data. Increasing $d$2 improves novel-task accuracy up to a point, while an excessively large $d$3 sacrifices base-task discrimination. The reported setting chooses $d$4 as a balance (Huang et al., 16 Aug 2025). This suggests that AdaRing’s generalization-aware training is not merely an auxiliary penalty but a primary mechanism for maintaining the contribution of the coarse-grained branch.

Initialization is also structured around preserving the frozen model at the start of training. Layer-specific slices $d$5 are initialized to zeros so that the initial adapted model matches frozen CLIP, while the layer-shared cores are initialized from a Gaussian distribution to promote diverse update directions and avoid collapse (Huang et al., 16 Aug 2025).

6. Empirical results and parameter efficiency

AdaRing is evaluated on 11 image classification datasets: ImageNet, Caltech101, OxfordPets, StanfordCars, OxfordFlowers, Food101, FGVCAircraft, EuroSAT, UCF101, DTD, and SUN397 (Huang et al., 16 Aug 2025). The reported metrics are base accuracy, novel accuracy, and harmonic mean. Across these benchmarks, the method is described as achieving state-of-the-art performance while reducing average training parameters by 90% (Huang et al., 16 Aug 2025).

The reported average over the 11 datasets is as follows:

Method	Base	Novel	HM
AdaRing	87.16	81.12	84.03
MMA	83.20	76.80	79.87
PromptKD	86.96	80.73	83.73

These values show that AdaRing matches or surpasses PromptKD and clearly outperforms MMA on both base and novel tasks while being far more parameter-efficient than MMA (Huang et al., 16 Aug 2025). Across 9 of 11 datasets, AdaRing ranks first on both base and novel tasks (Huang et al., 16 Aug 2025).

The OxfordFlowers comparison is especially emphasized in the data as an illustration of compression. MMA uses about $d$6M trainable parameters, whereas AdaRing uses about $d$7M trainable parameters, corresponding to roughly a 90% reduction while improving performance (Huang et al., 16 Aug 2025). On EuroSAT, AdaRing reports Base $d$8, Novel $d$9, HM $\bm{\mathcal{A}}$0, while MMA reports Base $\bm{\mathcal{A}}$1, Novel $\bm{\mathcal{A}}$2, HM $\bm{\mathcal{A}}$3, which the data summarize as $\bm{\mathcal{A}}$4 on base and $\bm{\mathcal{A}}$5 on novel relative to MMA (Huang et al., 16 Aug 2025). On FGVCAircraft, AdaRing reports Base $\bm{\mathcal{A}}$6, Novel $\bm{\mathcal{A}}$7, HM $\bm{\mathcal{A}}$8, compared with MMA at Base $\bm{\mathcal{A}}$9, Novel $\bm{\mathcal{G}^j}$0, HM $\bm{\mathcal{G}^j}$1 (Huang et al., 16 Aug 2025).

The ablations included in the data are methodologically significant. Cross-layer TRD alone reduces parameters by about 68% relative to MMA without hurting performance, while adding diverse adapters further improves performance under the same or lower parameter budgets. The data also state that diverse adapters outperform homogeneous adapters at the same parameter budget (Huang et al., 16 Aug 2025). These observations isolate the contributions of cross-layer tensorization and adapter diversity rather than conflating them.

7. Interpretation, scope, and limitations

AdaRing’s conceptual contribution lies in combining two distinct regularities: cross-layer redundancy and rank-dependent specialization. The first is addressed by a shared tensor-ring basis across layers; the second by coupling fine-grained and coarse-grained adapters with a learnable mixer and a preservation regularizer (Huang et al., 16 Aug 2025). In practical terms, the method is intended for settings with strict parameter or memory constraints, where both few-shot performance on seen classes and generalization to unseen classes are important (Huang et al., 16 Aug 2025).

The design is described as generic in the sense that TRD and cross-layer tensorization can be applied to transformer-based vision or language encoders, and there is no CLIP-specific assumption beyond standard adapter insertion (Huang et al., 16 Aug 2025). The paper itself focuses on CLIP, so stronger claims about transfer to other architectures would be inferential rather than demonstrated.

Several limitations are explicitly suggested in the data. If ranks are set too low across the board, performance degrades rapidly, especially on complex datasets. If $\bm{\mathcal{G}^j}$2 is set too high, generalization is overemphasized and base-task accuracy declines. Efficient tensor-ring implementation may also be non-trivial and may require careful engineering in deployment (Huang et al., 16 Aug 2025). These caveats indicate that AdaRing’s efficiency gains depend on disciplined rank selection, preservation-ratio tuning, and a sufficiently optimized tensor implementation.

A common misconception would be to view AdaRing as merely a tensorized version of LoRA. The evidence in the data points to a broader construction: cross-layer tensor ring decomposition addresses compression by sharing structure across layers, while the dual-adapter scheme and generalization-aware loss address the representation-diversity problem that homogeneous low-rank adapters leave unresolved (Huang et al., 16 Aug 2025). In that sense, AdaRing is not only a compression mechanism but also a coordination mechanism for heterogeneous low-capacity and higher-capacity adaptation paths.