HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding

Abstract: CLIP (Contrastive Language-Image Pre-training) has become a de facto paradigm for image-text alignment, but it struggles with long-context descriptions (>77 tokens) due to absolute positional encoding and pretraining on short captions. In long contexts, sentences are often reordered, summarized, or partially omitted. Although prior works extend CLIP with longer positional encodings, they often suffer from degraded image-text alignment under such text perturbations. We attribute this limitation to the Euclidean contrastive objective, which enforces strict one-to-one matching and lacks explicit mechanisms for modeling hierarchical relationships between global context and its constituent elements. To address this issue, we propose HyFL-CLIP, a hyperbolic fine-tuning framework that distills the well-established text-image alignment learned in Euclidean CLIP into hyperbolic space via cross-manifold similarity distillation, leveraging its geometry to capture hierarchical and entailment relations. Our method models hierarchical semantics by linking summarized token-wise features, long-context descriptions, constituent short textual components, and images, capturing part-whole relationships via hyperbolic entailment with Einstein midpoint aggregation. Experiments on diverse benchmarks, including long-context cross-modal retrieval, cross-modal retrieval with caption perturbations, intra-modality retrieval, and short-text cross-modal retrieval, show that HyFL-CLIP achieves more robust long-context understanding. In particular, it yields up to 19.5% improvement in long-text cross-modal retrieval under textual perturbations over the best prior method. We also show HyFL-CLIP can be seamlessly integrated into other model frameworks by applying it to Stable Diffusion XL (SDXL).

• The paper introduces HyFL-CLIP, a framework that fine-tunes CLIP into hyperbolic space to robustly model hierarchical and long-context semantics.
• It employs cross-manifold similarity distillation, hyperbolic geodesic contrastive loss, and Einstein midpoint aggregation to enforce structured image-text alignment.
• Empirical results show up to 19.5% improvement in retrieval accuracy under perturbations while maintaining strong short-caption and generative performance.

HyFL-CLIP: Hyperbolic Fine-Tuning of CLIP for Robust Long-Context Understanding

Motivation and Limitations in Long-Context Image-Text Alignment

The CLIP family and its Euclidean descendants establish text-image alignment via a contrastive paradigm, yet these models degrade significantly on descriptions exceeding their original context window (e.g. $>77$ tokens), even when perturbations retain overall semantics through sentence reorderings or partial omissions. The core limitation arises from the point-to-point Euclidean contrastive objective, which is intrinsically misaligned with the hierarchical and inclusion relationships that characterize complex, compositional language. As demonstrated in long-context and perturbed-caption retrieval, standard and even extended-CLIP variants fail to maintain robust image-text alignment; token dominance emerges and semantic information is distributed non-uniformly, inducing brittleness to small modifications Figure 1.

Figure 1: Robust long-context text-image retrieval is fundamentally degraded under sentence order shuffling in prior approaches, with models missing key semantic elements, highlighting a lack of robust hierarchical modeling.

HyFL-CLIP Architecture: Explicit Hierarchical Modeling in Hyperbolic Space

HyFL-CLIP introduces a fine-tuning framework that distills pretrained Euclidean text-image alignment from CLIP into a Lorentz-model hyperbolic manifold, inheriting core semantic structure and introducing consistent modeling for hierarchical and part-whole relationships. The architecture is defined by three principal innovations: (i) cross-manifold similarity distillation anchored on short captions, (ii) hyperbolic geodesic contrastive learning, and (iii) hierarchical entailment loss leveraging Einstein midpoint aggregation. The final design further regularizes channel occupancy via a hyperbolic radius entropy penalty.

Figure 2: Schematic of HyFL-CLIP, integrating cross-manifold Euclidean$\rightarrow$hyperbolic distillation, hierarchical entailment via Einstein midpoint aggregation, hyperbolic geodesic contrastive loss, and radius entropy regularization.

Cross-Manifold Similarity Distillation

The knowledge transfer is initialized from Open-CLIP representations, projecting annotated (short caption, image) pairs to Lorentz space via the exponential map. CLIP's robust local geometric similiarity landscape is preserved by matching the distributions over image candidates (conditioned on a text) between the Euclidean cosine and hyperbolic negative-geodesic distance. Training proceeds to minimize the cross-entropy between these distributions under separate temperatures, providing a soft alignment anchor for the hyperbolic embedding space.

Hyperbolic Geodesic Contrastive Loss

Building on the transferred structure, HyFL-CLIP utilizes both long and short captions in a bidirectional geodesic InfoNCE-style loss, using the Lorentzian geodesic metric. This ensures that long-context image-text pairs are tightly coupled in the hyperbolic geometry, supporting the modeling of fine-grained distinctions while maintaining principal retrieval accuracy.

Hierarchical Entailment via Einstein Midpoint Aggregation

To address part-whole and hierarchical semantic relations, token-wise features (text and image modality) are reduced via a similarity-weighted Einstein midpoint, with attention weights proportional to each token's semantic relevance to the cross-modal representation. The global embedding is then constrained to lie within the entailment cone of this token-aggregate, providing a geometric envelope robust to local perturbations and reflecting inclusion hierarchies. Aperture hyperparameters scale the margin for entailment, trading off precision for robustness.

Entropy Regularization

To reduce embedding collapse and prevent degenerate radii, the framework introduces a radius-entropy regularizer, enforcing balanced allocation of hyperbolic embedding radii and thereby stabilizing long-context representation topology.

Empirical Performance: Robustness and Hierarchical Generalization

Long-Context Cross-Modal Retrieval and Perturbation Robustness

HyFL-CLIP demonstrates strong performance gains (up to 19.5% improvement) in Top-1 text-image retrieval accuracy on DOCCI, Urban-1k, Long-DCI, and DCI when evaluated under diverse textual perturbations (word/sentence drop, order shuffling, subsampling). Notably, it exhibits only minor performance drops in adversarial conditions that critically harm Euclidean and previous hierarchical-CLIP baselines, indicating actual robustness to hierarchical or partial input loss.

Consistency in Short-Caption and Intra-Modal Retrieval

Despite explicit hyperbolic tuning for long-context settings, HyFL-CLIP maintains performance parity or superiority on COCO and Flickr30K benchmarks for short-caption retrieval. Intra-text retrieval experiments (COCO, nocaps, IMDB, 20NG) confirm improved alignment of hierarchical text features within the same modality, with increased mAP and Pr@R.

Qualitative Analysis and Token Attribution

HoroPCA visualizations demonstrate concentrated and semantically stable embeddings under challenging caption perturbations, while attention weight maps show a more discriminative attribution to tokens aligned with visual semantics compared to the baseline Figure 3.

Figure 3: HoroPCA visualization of text summary tokens, long-text embeddings, and image embeddings, revealing the structured hyperbolic clustering, and the comparative token-contribution (semantic) weights.

Integration with Generative Models

HyFL-CLIP's text encoder seamlessly replaces the baseline in Stable Diffusion XL, with generated images exhibiting finer semantic detail alignment (e.g., object counts, detailed local attributes) on both short and long captions compared to Euclidean-pretrained backbones Figure 4.

Figure 4: HyFL-CLIP-integrated SDXL captures more faithful fine-grained scene semantics, attributed to improved hyperbolic long-text feature integration.

Ablations and Comparative Analyses

Systematic removal of the hierarchical entailment loss or the distillation term induces measurable declines in both robustness and global retrieval accuracy. Figure 5 visualizes the cluster degradation and diminished alignment when components are omitted, empirically validating the necessity of each.

Figure 5: Embedding distributions under full model vs. ablations, showing the role of hierarchical entailment and cross-manifold distillation in maintaining structured alignment.

Embedding analysis under perturbation (Figures 6 and 7) quantifies the stability improvement: perturbed long captions remain tightly grouped around their original reference in hyperbolic space, and their rank variance in image-text similarity search is drastically reduced compared to the Euclidean baseline.

Figure 6: Visualization of altered (perturbed) captions’ embeddings, demonstrating the robust semantic proximity in HyFL-CLIP compared to the Euclidean baseline.

Figure 7: Rank distribution of caption perturbations in the image-text similarity search space; hyperbolic entailment concentrates related variants near top ranks.

Token contribution analyses (Figures 9, 10, 11; 16–18) further illustrate improved, more semantically-consistent token weighting—HyFL-CLIP yields attribution that prioritizes contextually salient and visually relevant tokens, in contrast with flat or fragmented Euclidean token attributions.

Practical and Theoretical Implications

HyFL-CLIP provides a tractable, compute-efficient fine-tuning route (2-epoch vs. 10–20 epochs for comparable baselines) to adapt powerful pre-trained Euclidean VLMs for robust long-context understanding, without full retraining. The hybrid approach enables efficient transfer of Euclidean knowledge while leveraging the structural advantages of hyperbolic geometry for modeling hierarchy, abstraction, and semantic entailment.

On the theoretical side, HyFL-CLIP demonstrates the utility of explicit geometric entailment constraints over naive contrastive alignment for robustly modeling compositional, hierarchical semantics—an enduring problem in vision-language modeling—showcasing hyperbolic space not just as a curiosity, but as a tool for practical, generalizable alignment in real-world, non-i.i.d. conditions.

Future Perspectives

The hybrid cross-manifold methodology established here lays a foundation for generalization of multimodal (and monomodal) foundation models beyond rigid compositional limits. Direct extensions include integration into large multimodal LLMs, incorporation in more complex structured prediction tasks, and broader applications in compositional or hierarchical generative pipelines. The findings motivate theoretical and practical investigation into further exploitation of Riemannian geometry for semantic representation learning.

Conclusion

HyFL-CLIP systematically addresses the long-standing pathology of brittle, order-dependent long-context representation in foundation VLMs. By cross-manifold distillation from pretrained Euclidean models and explicit hierarchical entailment in Lorentz space, it achieves state-of-the-art robustness to caption perturbation, enhanced part-whole modeling, and strong generalization in generative integration scenarios, all with only marginal computational overhead relative to standard fine-tuning practice. This work represents a substantial advancement in the geometric modeling of hierarchical multimodal semantics for vision-language systems (2607.00428).