HAC: Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA

Abstract: Recent advances in representation learning have shown that hyperbolic geometry can offer a more expressive alternative to the Euclidean embeddings used in CLIP models, capturing hierarchical structures and leading to better-organized representations. However, current hyperbolic CLIP variants are trained entirely from scratch, which is computationally expensive and resource-intensive. In this work, we propose HAC (Hyperbolic Adaptation of CLIP), a parameter-efficient framework that enables pretrained CLIP models to transition into hyperbolic space via lightweight fine-tuning. We apply HAC to Visual Question Answering (VQA), where models must interpret visual elements and align them with textual queries. Notably, HAC's training is performed on a dataset with no overlap with any VQA benchmark, resulting in a strict zero-shot evaluation paradigm that underscores HAC's task-agnostic adaptability. We evaluate HAC across a diverse suite of VQA benchmarks spanning General, Reasoning, and OCR categories. Both HAC-S (small) and HAC-B (medium) consistently surpass Euclidean baselines and prior hyperbolic approaches, with HAC-B delivering up to a +1.9 point average improvement over CLIP-B on reasoning-intensive tasks. Our code is available at https://github.com/fdibiton/HAC

• The paper presents HAC, a parameter-efficient hyperbolic adaptation of CLIP that re-engineers Euclidean embeddings into the Lorentz model for enhanced zero-shot VQA.
• It employs modular fine-tuning methods, such as sequential adapters and LoRA, updating only key parameters to transform visual and textual representations efficiently.
• Empirical results across multiple VQA benchmarks show significant performance gains and hierarchical disentanglement, validating hyperbolic geometry as a robust inductive bias.

Parameter-Efficient Hyperbolic Adaptation of CLIP for Zero-Shot VQA

Introduction and Motivation

This work introduces HAC (Hyperbolic Adaptation of CLIP), a framework for parameter-efficient geometric adaptation of pretrained CLIP models, enabling their deployment in hyperbolic space for zero-shot Visual Question Answering (VQA). Prior hyperbolic CLIP approaches necessitate training from scratch, incurring substantial computational cost and large data requirements. HAC bridges this gap by employing lightweight fine-tuning modules that reorganize Euclidean embeddings into hyperbolic geometry, thus capturing hierarchical and relational structure inherent in vision-language tasks, especially VQA.

The theoretical motivation stems from the limitations of Euclidean spaces—which provide uniform distances and polynomial volume growth—making them suboptimal for representing hierarchies. Hyperbolic geometry, with negative curvature and exponential expansion, naturally encodes such semantic hierarchies, where specific instances are located farther from the origin, clustered under general concepts. This matches VQA needs, where object-scene and compositional relations must be modeled robustly for effective reasoning.

HAC Architecture and Geometric Adaptation

HAC adapts the dual-encoder CLIP architecture (vision Transformer + textual Transformer) using localized, highly parameter-efficient modules. Adapters, low-rank updates (LoRA), bias and LayerNorm tuning are compared as plug-in adaptation modules. These modules wrap or inject into a subset of Transformer blocks, keeping the pretrained backbone frozen. Only a restricted portion of parameters are updated during adaptation: adaptation modules, projection heads, and final LayerNorms of both encoders—yielding significant savings in both training time and resource consumption.

HAC adapts the original Euclidean representations by mapping them into the Lorentz model of hyperbolic space: each (L2-normalized) embedding is augmented with a zero time component, scaled by learnable projection factors, and then projected onto the hyperboloid by the exponential map. Hierarchical entailment is imposed via entailment cones, promoting radial separation of general and specific instances within the manifold.

Figure 1: Overview of HAC enabling geometric adaptation of pretrained CLIP models through parameter-efficient fine-tuning. Frozen CLIP blocks are wrapped by adaptation modules introducing new trainable parameters conducive to geometric reorganization.

For a selected Transformer block, the adaptation module $\mathcal{U}_\ell$ can implement several efficient adaptations: sequential adapters, residual adapters, or low-rank matrix adjustments. All options are benchmarked to determine optimal strategies across varying model capacities.

Figure 2: HAC Adapted Transformer Blocks: Adaptation modules apply lightweight geometric transformations (adapters, LoRA, etc.) to selected block submodules, with all other parameters kept frozen.

Training Objectives and Zero-Shot VQA

HAC’s training objective is inherited from prior compositional hyperbolic CLIP formulations but specifically adapted to the adaptation scenario. The loss is a weighted sum of a Hierarchical Compositional Contrastive loss, which aligns both full-scene and object-region representations, and a Hierarchical Compositional Entailment loss, directly encoding object-scene hierarchies through entailment cones. Purely parameter-efficient modules reshape the embedding space without full-model retraining, optimizing both geometric fidelity and data efficiency.

At inference, zero-shot VQA is formulated as candidate answer matching. For each candidate, the model computes the hyperbolic distance between the image and question-answer pair embedding in the Lorentz space, choosing the answer with the minimal (equivalently, most negative) Lorentzian geodesic distance. Importantly, HAC’s fine-tuning is performed on data disjoint from all benchmarked VQA tasks, ensuring strict zero-shot evaluation and strong claims of task transferability.

Empirical Results

HAC is evaluated across six VQA benchmarks, including general, reasoning, and OCR-centric datasets. Results show consistent improvement in zero-shot VQA performance over both Euclidean CLIP baselines and prior hyperbolic CLIP models. Notably, HAC matches or exceeds the performance of the fully hyperbolic HyCoCLIP model (which requires full training and larger datasets) on several VQA tasks, with parameter counts and computational complexity reduced by over an order of magnitude.

On more challenging, reasoning-intensive datasets, the HAC-B (medium-scale, ViT-B based) model with LoRA adaptation delivers up to a +1.9 point average improvement over the strong CLIP-B baseline. Adapter selection is critical: LoRA is optimal for larger model variants, while sequential adapters perform best on smaller models, with bias and LayerNorm tuning being less effective for challenging tasks or larger capacities.

Learned Hyperbolic Structure

Visualization of learned representations using horospherical projections reveals that HAC internalizes object-scene hierarchies: object-centered (local) embeddings systematically occupy regions closer to the hyperbolic origin, while full-scene (global) descriptions are located at larger radii. This radial separation aligns with theoretical predictions for hierarchical hyperbolic geometry, demonstrating HAC’s capacity for geometric disentanglement through minimal parameter updates.

Figure 3: HoroPCA 2D projection of embeddings from best HAC-B model, evidencing organization of object-box (general) and full-scene (specific) representations along the expected hyperbolic hierarchy.

Ablation studies show that removal of the entailment cone loss collapses the learned hierarchy, emphasizing its structural importance for usable hyperbolic representations in vision-LLMs.

Figure 4: HAC-S with and without the compositional entailment loss; the loss ablation disrupts radial (hierarchical) separation of objects and scenes within hyperbolic space.

Practical and Theoretical Implications

Practically, HAC enables the deployment of hyperbolic geometry in strong pretrained vision-language encoders with negligible additional cost. The method is highly modular: PEFT strategies can be selected for a spectrum of hardware budgets, and adaptation can be confined to the final layers for maximal effect. Theoretically, HAC demonstrates that geometric organization—distinct from semantic learning—can be transferred independently via localized adaptation. The empirical improvements on reasoning and compositional tasks suggest that hyperbolic geometry provides inductive biases beneficial for zero-shot VQA, supporting efficient abstraction and generalization.

This also points to a future where large-scale vision-LLMs can be efficiently re-geometrized for specialized tasks (e.g., hierarchical reasoning, compositional generalization) without catastrophic forgetting, thereby marrying the strengths of foundation model pretraining and non-Euclidean manifold optimization.

Conclusion

HAC establishes that parameter-efficient geometric adaptation is a viable path for equipping pretrained vision-LLMs with hyperbolic hierarchical structure, boosting zero-shot transfer in VQA. The combination of strong empirical results, clear geometric interpretability, and resource efficiency indicates that future advances in modular, geometry-aware adaptation are likely to expand the capabilities and applicability of foundation models in visual reasoning and beyond.