LatHAdapter: Hierarchical Fine-Tuning for VLMs

• The paper introduces the LatHAdapter framework that leverages learnable attribute prompts and hyperbolic embedding to capture one-to-many semantic relationships.
• It integrates two modules—Attribute-aware Text Refiner (ATR) and Hyperbolic Hierarchical Learning (HHL)—to refine class embeddings and support hierarchical feature learning.
• Empirical results across 11 benchmarks show improvements in few-shot, cross-domain, and base-to-new generalization, validating its efficiency with minimal parameter overhead.

The Latent Hierarchical Adapter (LatHAdapter) framework represents a class of adapter-based fine-tuning strategies for pre-trained Vision-LLMs (VLMs), designed to improve downstream few-shot and transfer learning by leveraging the latent semantic hierarchies present in data. Its core innovation lies in combining learnable attribute prompts with hyperbolic geometric modeling, thereby capturing the inherent one-to-many semantic associations among categories, intermediary attributes, and images (Zhao et al., 15 Aug 2025).

1. Architectural Overview

LatHAdapter operates on top of a frozen or lightly tuned vision-language foundation such as CLIP, introducing two principal modules:

• Attribute-aware Text Refiner (ATR): This module refines class-level text embeddings by propagating information through a graph involving both class and learnable attribute prompt nodes. For a batch of $B$ image feature vectors $\{\mathbf{v}_i\}_{i=1}^B$ and $C$ class embeddings $\{\mathbf{t}_j\}_{j=1}^C$, together with $N$ learnable attribute prompts $\{\mathbf{a}_k\}_{k=1}^N$, the ATR uses normalized cosine similarity to form an affinity matrix $S$. A single normalized graph convolution step (controlled by $\beta$) aggregates feature information, producing refined class embeddings $\hat{\mathbf{T}}$:

$$\hat{\mathbf{F}} = \mathbf{F} + \beta D^{-1/2} S D^{-1/2} \mathbf{F}, \quad \hat{\mathbf{T}} = \hat{\mathbf{F}}_{1:C, :}.$$

• Hyperbolic Hierarchical Learning (HHL): Refined class embeddings, attributes, and image features are jointly projected into a Poincaré ball of curvature $-c$. Prototype (class) nodes are located near the center, attributes at mid-radii, and images near the boundary, supporting rich, tree-like semantic structures.

This adapter structure can be attached as a light-weight plug-in to most existing parameter-efficient fine-tuning (PEFT) strategies without substantial parameter overhead, requiring only $N \times d$ new parameters for attribute prompts.

2. Hyperbolic Embedding and Operations

LatHAdapter projects all relevant vectors into the Poincaré ball $\mathbb{D}_c^d$ via the exponential map at the origin:

$$\exp_0^c(\mathbf{x}) = \tanh\left(\sqrt{c}\|\mathbf{x}\|\right) \frac{\mathbf{x}}{\sqrt{c}\|\mathbf{x}\|}.$$

Letting $\hat{\mathbf{t}}_j$, $\mathbf{a}_k$, and $\mathbf{v}_i$ denote the refined class, attribute, and image vectors respectively, their hyperbolic projections are:

$$\tilde{\mathbf{t}}_j = \exp_0^c(\hat{\mathbf{t}}_j), \quad \tilde{\mathbf{a}}_k = \exp_0^c(\mathbf{a}_k), \quad \tilde{\mathbf{v}}_i = \exp_0^c(\mathbf{v}_i).$$

Key hyperbolic operations include:

• Möbius addition: Used to formulate anchor-positive/negative semantic relationships,

$$\mathbf{u} \oplus_c \mathbf{z} = \frac{(1+2c\langle \mathbf{u},\mathbf{z}\rangle + c\Vert\mathbf{z}\Vert^2)\mathbf{u} + (1-c\Vert\mathbf{u}\Vert^2)\mathbf{z}}{1+2c\langle \mathbf{u},\mathbf{z}\rangle + c^2\Vert\mathbf{u}\Vert^2\Vert\mathbf{z}\Vert^2}.$$

• Hyperbolic distance (Poincaré distance):

$d_H(\mathbf{u}, \mathbf{z}) = \frac{2}{\sqrt{c}} \arctanh\left(\sqrt{c}\,\|-\mathbf{u} \oplus_c \mathbf{z}\|\right).$

Hyperbolic geometry supports volume growth and hierarchical, one-to-many relationships more natively than Euclidean space, enabling fine-grained disentangling of class-attribute-instance relationships.

3. Hierarchical Loss Formulation

LatHAdapter’s training objective enforces both discriminative classification and latent hierarchy consistency:

• Euclidean contrastive classification: Standard cross-entropy based on cosine similarity between $\mathbf{v}_i$ and $\hat{\mathbf{t}}_j$.
• Hyperbolic classification: Cross-entropy based on Poincaré distances $d_H(\tilde{\mathbf{v}}_i, \tilde{\mathbf{t}}_j)$.
• Image-attribute regularization ($\mathcal{L}_{v-a}$): For positive image pairs and a negative image, the model locates their lowest common ancestor (LCA) attribute node, identified as the attribute closest in hyperbolic distance to both images. Triplet regularization attracts positives and repels negatives via the LCA.
• Attribute-class regularization ($\mathcal{L}_{a-t}$): Analogous to $\mathcal{L}_{v-a}$ but at the class-attribute level.

The unified objective is:

$$\mathcal{L} = \mathcal{L}_{\mathrm{ecls}} + \mathcal{L}_{\mathrm{hcls}} + \mathcal{L}_{v-a} + \mathcal{L}_{a-t}.$$

Such a combined loss encourages formation of a semantic tree: class prototypes branch to learned attribute pivots, which themselves connect to image instances distributed near the Poincaré ball boundary.

4. Experimental Setup and Key Results

LatHAdapter has been evaluated on a comprehensive suite of few-shot and generalization tasks (Zhao et al., 15 Aug 2025):

• Base-to-New Generalization: Evaluated on 11 datasets (e.g., ImageNet, Caltech101, OxfordPets), using H-mean of base and novel class accuracy. Integrating LatHAdapter into MaPLe, PromptSRC, and PromptKD yields consistent improvements of $+0.73$ to $+1.12$ pp in H-mean; for all 11 datasets, H is improved by $\sim$1–1.5 points.
• Few-Shot Classification: On 1/2/4/8/16 shot settings, PromptSRC+LatHAdapter shows average gains of $+2.39$ pp at 1-shot, $+1.60$ pp at 16-shot; MaPLe+LatHAdapter yields $+4.68$ pp at 1-shot and $+1.52$ pp at 16-shot.
• Cross-Domain and Transfer: When trained on 16-shot ImageNet and tested zero-shot on other datasets, LatHAdapter consistently achieves $+0.4$–$0.6$ pp gains on cross-dataset targets and $+0.3$–$0.5$ pp on domain-shifted versions of ImageNet.

Baselines include CoOp, CoCoOp, CLIP-Adapter, Graph-Adapter, MaPLe, LoRA, TIP-Adapter, MMA, PromptSRC, and PromptKD.

5. Ablations and Analysis

Controlled ablation studies illuminate the contribution of each component:

• ATR module alone (without HHL): Adding the Attribute-aware Text Refiner (no hyperbolic hierarchy) may not improve H-mean, and may decrease it slightly (PromptSRC+ATR, H: $79.67 \rightarrow 79.28$).
• Full LatHAdapter (ATR + HHL): Restoration and improvement are observed ($+0.73$ pp to H: 80.40), directly attributable to hierarchical hyperbolic loss. Base-class and novel-class gains are both present ($+1.01/+1.19$).
• Hyperparameters: Optimal $\beta$ for ATR is around 0.1; excess weights overemphasize attribute nodes. Datasets with more classes benefit from $20\!-\!24$ attribute prompts; smaller datasets are optimal at $8\!-\!12$ prompts.
• Over-parameterization: More attribute prompts raise base-class accuracy, though excessive prompts may slightly diminish generalization to novel classes.

6. Discussion and Implications

Hyperbolic geometry’s exponential volume growth and hierarchical layout properties underpin LatHAdapter’s ability to capture one-to-many category-attribute-image relationships, outperforming Euclidean approaches that tend to collapse such diversity into flat clusters. The plug-and-play nature of LatHAdapter, with minimal parameter overhead, allows straightforward integration into PEFT pipelines.

Observed strengths:

• Consistent empirical improvements (+0.7 to +4.7 pp gains) across >15 benchmarks.
• Fine-grained, data-driven discovery of latent attributes beneficial to both base and new categories.
• Wraps around diverse PEFT strategies (e.g., MaPLe, PromptSRC, PromptKD) with no disruption to backbone architecture.

Limitations include sensitivity to the curvature ($c$), regularization margins ($\sigma$), and prompt count ($N$), as well as the latent, potentially uninterpretable nature of discovered attributes. Hyperbolic operations incur slight additional computation, though inference efficiency remains satisfactory on GPU hardware. A plausible implication is that improving interpretability and aligning discovered attributes to human concepts could further extend LatHAdapter’s utility in open-set and cross-domain adaptation scenarios (Zhao et al., 15 Aug 2025).