VRGAdapter: Probabilistic Adapter for VLMs

• VRGAdapter is a graph-based textual tuning method that represents each class as a Gaussian distribution to capture semantic diversity.
• It employs probabilistic message propagation with separate updates for means and variances, enabling uncertainty-aware refinement of class prototypes.
• Empirical results demonstrate significant improvements in few-shot and out-of-distribution classification over deterministic adapters.

Searching arXiv for VRGAdapter and closely related graph-adapter literature to ground the article in current papers. Vertex Random Graph Adapter (VRGAdapter) is a textual adaptation method for fine-tuning frozen vision-LLMs (VLMs), introduced for downstream few-shot image classification and related transfer settings. It replaces the deterministic class node used in standard graph-based textual adapters with a random variable, so that graph-based adaptation is performed over a class graph whose vertices are distributions rather than fixed embeddings. In the formulation introduced in "Beyond Graph Model: Reliable VLM Fine-Tuning via Random Graph Adapter" (Jiang et al., 14 Jul 2025), each class is represented by a Gaussian-distributed text embedding estimated from multiple large-language-model-generated descriptions; graph propagation then updates both the mean and variance of each class representation, after which a reparameterized sample is fused with the original class mean to form the final textual prototype used for classification.

1. Concept and problem formulation

VRGAdapter is motivated by two limitations attributed to prior adapter-based text-side tuning for frozen VLMs. First, existing methods such as CLIP-Adapter, TaskRes, and GraphAdapter typically refine a deterministic text representation for each class, even though a class can admit multiple semantically distinct but valid descriptions. Second, although graph-based adapters already exploit inter-class relationships, they do so with deterministic node states rather than uncertainty-aware or diversity-aware class representations (Jiang et al., 14 Jul 2025).

The method therefore treats each class text representation not as a single vector but as a distribution. The paper’s central claim is that class descriptions vary in attributes, context, phrasing, and granularity, and that this variation contains discriminative semantic information that should be retained rather than averaged away. In this design, the “random graph” designation is specific: randomness is attached to the class vertices, while edges remain deterministic similarities computed from node means. The method is thus “vertex-random” in the sense that the graph nodes are Gaussian-distributed embeddings.

The resulting pipeline is explicitly class-centric. For each class, an LLM generates multiple descriptions; all descriptions are encoded by the frozen CLIP text encoder; a Gaussian is fitted to the resulting embedding set; a class graph is built from pairwise semantic similarities; graph propagation updates class-wise means and variances; and a sampled adapted textual prototype is fused with the original class mean to obtain the classifier prototype. The paper also states that this formulation is more general than traditional graph-based adapters and encompasses deterministic GraphAdapter as a special case (Jiang et al., 14 Jul 2025).

2. Vertex Random Knowledge Graph

The graph substrate used by VRGAdapter is the Vertex Random Knowledge Graph (VRKG). For a dataset with $C$ classes, the method prompts an LLM to generate $M$ descriptions for each class,

$$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$

and encodes them with the frozen CLIP text encoder $f_T$,

$\mathbf{T}_i^m = f_T(\mathbf{P}_i^m).$

Each class node $v_i$ is represented by a Gaussian random variable

$$\mathcal{H}_i \sim \mathcal{N}\big(\boldsymbol{\mu}_i, \operatorname{diag}(\boldsymbol{\sigma}_i)\big), \quad i \in [1,C],$$

with empirical mean and diagonal variance

$$\boldsymbol{\mu}_i = \frac{1}{M} \sum_{m=1}^M \mathbf{T}_i^m,$$

$$\boldsymbol{\sigma}_i = \frac{1}{M} \sum_{m=1}^M (\mathbf{T}_i^m- \boldsymbol{\mu}_i) \odot (\mathbf{T}_i^m - \boldsymbol{\mu}_i).$$

This construction is the core mechanism by which intra-class semantic diversity is retained: the mean captures the semantic center of the description set, while the variance captures dispersion across descriptions (Jiang et al., 14 Jul 2025).

Edges encode inter-class semantic relationships. The adjacency matrix is deterministic and defined from cosine similarity between class means,

$$\mathbf{A}_{ij} = \cos(\boldsymbol{\mu}_i, \boldsymbol{\mu}_j), \quad i,j \in [1,C].$$

The degree matrix is

$M$0

The method therefore combines two distinct information channels within a single graph object: intra-class semantic diversity through $M$1, and inter-class structure through $M$2. The paper explicitly notes that a fully probabilistic alternative with random edges would be possible in principle, but was not adopted because message passing on graphs with random edges would be more complex.

3. Probabilistic message propagation and adapter output

VRGAdapter applies a Gaussian-based graph convolution that propagates means and variances separately across $M$3 graph layers. With initialization

$M$4

the updates are

$M$5

$M$6

Here $M$7 includes node $M$8 and its neighbors, and the method uses separate learnable matrices $M$9 and $$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$0 for center and variance dynamics. In implementation, the paper uses ELU for means and ReLU for variances (Jiang et al., 14 Jul 2025).

This propagation mechanism distinguishes VRGAdapter from deterministic graph adapters. In a standard deterministic GNN-style textual adapter, node state is a single vector and only its location is updated. In VRGAdapter, propagation updates both semantic center and semantic spread. The final node state is therefore a context-aware distribution representation, not merely a context-aware point estimate. The paper’s qualitative interpretation is that related classes can make a class representation more certain when their semantics agree, or sustain broader uncertainty when semantic neighborhoods are heterogeneous.

After $$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$1 layers, the model samples an adapted textual prototype from the refined Gaussian by reparameterization. The resulting sample $$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$2 is then fused with the original class mean using the residual formulation

$$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$3

and the final text classifier matrix is

$$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$4

The residual form is presented as a way to preserve the original CLIP-aligned semantic prior while injecting graph-adapted stochastic refinement. A notable special case is $$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$5, for which the empirical variance collapses toward zero and the model reduces toward deterministic message passing, making GraphAdapter a special case of VRGAdapter (Jiang et al., 14 Jul 2025).

4. Training regime, UMF extension, and implementation profile

VRGAdapter is trained in the frozen-VLM regime. The backbone encoders remain fixed, and the trainable components are the Gaussian graph convolution parameters $$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$6, $$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$7, and, when the auxiliary fusion module is used, the auxiliary residual classifiers $$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$8. The training objective is standard cross-entropy on the fused prediction. The paper explicitly states that it does not introduce additional KL or explicit distribution regularization terms for the Gaussian node distributions; uncertainty parameters are learned indirectly through downstream classification supervision (Jiang et al., 14 Jul 2025).

A supplementary component, Uncertainty-guided Multi-branch Fusion (UMF), is introduced to improve reliability through adaptive ensembling of multiple pre-trained visual models. UMF is not identical to VRGAdapter itself: VRGAdapter adapts the text branch, whereas UMF combines predictions from three branches, namely the frozen CLIP branch and two auxiliary branches instantiated in experiments as MoCo and DINO. Auxiliary branches construct prototype classifiers from few-shot support data, while fusion weights are modulated by a kurtosis-based confidence score. This makes UMF an uncertainty-aware ensemble layer on top of the textual adapter rather than the defining component of the adapter.

The implementation settings reported in the paper are concrete. The optimizer is AdamW with learning rate $$[\mathbf{P}_i^1, \mathbf{P}_i^2, \ldots,\mathbf{P}_i^M] = \text{LLM}(\text{``Describe class } i \text{''}),$$9, cosine decay, 50 epochs, batch size 256, $f_T$0 descriptions per class, $f_T$1 graph layers, hidden feature dimension 16, residual blend coefficient $f_T$2, ELU for means, ReLU for variances, and a single NVIDIA RTX 3090. Main experiments use CLIP with ResNet-50; auxiliary visual models are MoCo (ResNet-50) and DINO (ResNet-50). Additional backbone studies include CLIP RN50, CLIP RN101, CLIP ViT-B/32, and CLIP ViT-B/16. On 16-shot ImageNet-1K, the paper reports that VRGAdapter trains in 40 minutes for 50 epochs, has 6.31M trainable parameters, and achieves 71.35% accuracy with RN50, compared with GraphAdapter’s 40-minute training time, 4.14M parameters, and 65.70% accuracy (Jiang et al., 14 Jul 2025).

5. Empirical evaluation and ablation findings

The method is evaluated on 11 few-shot classification datasets: ImageNet-1K, StanfordCars, Caltech101, UCF101, Flowers102, Food101, DTD, EuroSAT, FGVCAircraft, OxfordPets, and SUN397. It is also tested for out-of-distribution generalization on ImageNet-V2 and ImageNet-Sketch. With a ResNet-50 CLIP backbone on ImageNet-1K, VRGAdapter reports 63.88, 65.53, 67.54, 69.31, and 71.35 for 1-shot, 2-shot, 4-shot, 8-shot, and 16-shot settings, respectively. Under the same protocol, GraphAdapter reports 61.50, 62.32, 63.12, 64.23, and 65.70, yielding absolute gains of $f_T$3, $f_T$4, $f_T$5, $f_T$6, and $f_T$7. Compared with AMU-Tuning, VRGAdapter is reported to improve by $f_T$8, $f_T$9, $\mathbf{T}_i^m = f_T(\mathbf{P}_i^m).$0, $\mathbf{T}_i^m = f_T(\mathbf{P}_i^m).$1, and $\mathbf{T}_i^m = f_T(\mathbf{P}_i^m).$2 (Jiang et al., 14 Jul 2025).

Across the 11 datasets, the reported average accuracies are 68.81, 71.04, 75.20, 78.10, and 81.25 for 1-shot through 16-shot evaluation. The paper states that these averages exceed strong baselines including Tip-Adapter-F, GraphAdapter, CaFo, and AMU-Tuning. In individual datasets, the gains are described as especially large on DTD, EuroSAT, Flowers102, and UCF101. For out-of-distribution evaluation after training on 16-shot ImageNet-1K, the RN50 backbone attains 71.35 on ImageNet-1K, 62.41 on ImageNet-V2, and 41.23 on ImageNet-Sketch; AMU-Tuning gives 70.02, 58.64, and 40.04. With ViT-B/16, VRGAdapter attains 76.78, 68.60, and 51.78, compared with AMU-Tuning’s 74.98, 65.42, and 50.37.

The ablation studies isolate the contribution of the probabilistic text-side design. On ImageNet-1K, a CLIP prototype baseline gives 59.42, 60.50, 61.69, 62.89, and 64.46; adding VRGAdapter only gives 62.49, 62.92, 63.57, 64.60, and 66.03; adding auxiliary branches only gives 60.27, 62.53, 64.97, 67.47, and 69.84; adding AUX + UMF gives 60.77, 63.10, 65.48, 67.73, and 70.05; and the full system gives 63.88, 65.53, 67.54, 69.31, and 71.35. Direct comparison with GraphAdapter under the same CLIP+MoCo+DINO setting yields, on DTD, 54.08, 56.15, 63.89, 67.73, and 72.87 for GraphAdapter versus 56.80, 59.99, 65.43, 68.91, and 74.00 for VRGAdapter. Hyperparameter ablation reports that increasing $\mathbf{T}_i^m = f_T(\mathbf{P}_i^m).$3 from 2 to 50 steadily improves performance, with gains saturating after about 20; $\mathbf{T}_i^m = f_T(\mathbf{P}_i^m).$4 works best; $\mathbf{T}_i^m = f_T(\mathbf{P}_i^m).$5 gives the best uncertainty sensitivity for kurtosis in UMF; and $\mathbf{T}_i^m = f_T(\mathbf{P}_i^m).$6 best balances CLIP and auxiliary branches (Jiang et al., 14 Jul 2025).

6. Terminological context, misconceptions, and limitations

A recurrent misconception is to identify VRGAdapter with a generic random graph generator. In the VLM paper, the “random graph” designation does not refer to stochastic edge generation or a graph process over time. The randomness resides in the class-node representations, which are modeled as Gaussian random variables, while graph edges are deterministic cosine similarities computed from class means. In this sense, the method is closer in spirit to the broader probabilistic notion of vertex-random graph constructions, where randomness is attached to vertices and adjacency is induced from vertex-associated latent variables, than to arc-random models (Bahadır et al., 2016). A plausible implication is that the name “vertex-random” is conceptually aligned with latent node randomness, but the formal object used by VRGAdapter is an adapter over class-embedding distributions rather than a general isomorphism-invariant random graph model.

The method should also not be conflated with dynamic random graph processes whose vertex updates are insertion and deletion events. "Dynamic random graphs with vertex removal" studies a Dynamic Random Graph with Vertex Removal (DRGVR), a mathematically explicit vertex-based birth-death graph process in which a new vertex is introduced with probability $\mathbf{T}_i^m = f_T(\mathbf{P}_i^m).$7, a vertex is removed with probability $\mathbf{T}_i^m = f_T(\mathbf{P}_i^m).$8, and new edges are formed with probability inversely proportional to the current number of vertices (Díaz et al., 2022). That model is relevant only by analogy: it also attaches the principal source of structural change to vertices, but it is a random graph process, not an adapter for VLM text tuning.

The limitations of VRGAdapter are mostly architectural rather than theoretical. The node distribution is restricted to a diagonal Gaussian, so multimodal or correlated semantics are not explicitly represented. Edge construction is deterministic and based only on class means, so uncertainty is not propagated through the topology itself. The method depends on the quality and diversity of LLM-generated descriptions. It adds moderate complexity and trainable parameters relative to simpler adapters. The paper also notes that there is no explicit regularizer preventing pathological variance estimates beyond what downstream classification supervision induces. These points delimit the scope of the method: it is a probabilistic graph-based textual adapter for frozen VLMs, not a full probabilistic graph model of class semantics or a dynamic graph-learning framework.