HLFormer: Hyperbolic Video Retrieval

• The paper introduces a novel framework that leverages Lorentz-model hyperbolic attention and partial order constraints to enhance retrieval from untrimmed videos.
• It integrates parallel hyperbolic and Euclidean attention blocks with an adaptive fusion module to precisely localize temporal content in videos.
• Ablation studies confirm that components like the Mean-Guided Adaptive Interaction Module and L_pop loss critically improve cross-modal semantic alignment.

HLFormer is a neural architecture for Partially Relevant Video Retrieval (PRVR) that introduces hyperbolic geometry to better model the hierarchical and partial semantic relationships between untrimmed video content and text queries. By addressing the geometric limitations of traditional Euclidean embeddings, HLFormer leverages Lorentz-model hyperbolic attention and partial order constraints to improve temporal localization and cross-modal retrieval performance. The framework incorporates hybrid attention mechanisms and a novel adaptive fusion module, setting new benchmarks for PRVR effectiveness (Li et al., 23 Jul 2025).

1. Problem Motivation and Hyperbolic Embedding Foundations

PRVR seeks to retrieve relevant segments from untrimmed videos given text descriptions that may correspond only to a portion of the video content. Existing approaches predominantly rely on Euclidean representations, which distort hierarchies inherent in video data due to insufficient capacity for tree-like structures. In Euclidean space, volume grows polynomially with radius, causing embeddings of semantically distant video moments to cluster together when mapping hierarchical data such as frames, clips, and composite moments. This phenomenon, termed "crowding," impedes the separation of salient video events from background content.

Hyperbolic spaces—with constant negative curvature (K<0) and exponentially expanding volume—admit low-distortion representations of trees with large branching factors in low dimensions. This property is exploited in HLFormer via the Lorentzian (hyperboloid) model, defined as

$$L^n = \{x \in \mathbb{R}^{n+1} \mid \langle x, x \rangle_L = -1, x_0 > 0\}; \quad \langle x, y \rangle_L = -x_0 y_0 + x_s^T y_s,$$

where $x_s$ denotes spatial coordinates. Embedding video moments in $L^n$ preserves their hierarchical semantics, offering better discrimination for temporal retrieval tasks.

2. Model Architecture: Parallel Hyperbolic and Euclidean Attention

HLFormer encodes video features and text queries through a two-branch design:

2.1 Lorentz Attention Block (LAB):

• Projects Euclidean features ($X_\text{in}^E \in \mathbb{R}^{M \times d}$) to Lorentz space via exponential maps and learnable scaling.
• Applies Lorentz-linear layers that maintain the hyperboloidal constraint.
• Computes queries, keys, and values in Lorentz space; attention scores are based on squared Lorentzian distances with a Gaussian temporal mask.
• Each output is a Lorentzian centroid, later mapped back to Euclidean space for further fusion.

2.2 Euclidean Attention Block (EAB):

• Implements conventional scaled dot-product attention with Gaussian time weighting in $\mathbb{R}^{M \times d}$, facilitating comparability with hyperbolic attentions.

2.3 Hybrid Multi-block Pipeline:

• Input features undergo $N_\mathcal{L}$ LABs (hyperbolic) and $N_E$ EABs (Euclidean), producing $N_O = N_\mathcal{L} + N_E$ sequence outputs for subsequent dynamic combination.

3. Mean-Guided Adaptive Interaction Module (MAIM)

The MAIM module adaptively fuses the multi-scale outputs from different attention blocks at each time step. It computes a global context vector as the mean of all outputs and applies cross-attention from this guide to each block output. Learnable fusion weights are produced by a linear transformation of the cross-attention outputs, followed by per-time-step softmax normalization:

$$\tilde{w}_{i, t} = \frac{\exp(w_{i,t}/\tau)}{\sum_{k=1}^{N_O} \exp(w_{k,t}/\tau)},\quad \tilde{X}_t = \sum_{i=1}^{N_O} \tilde{w}_{i, t} X_{\sigma_i}(t).$$

This dynamic fusion enables HLFormer to select the most informative spatio-temporal patterns within each video for PRVR, compensating for the rigidities of static or concatenation-based fusion.

4. Partial Order Preservation via Lorentzian Cones

To enforce the semantic entailment "text $\prec$ video" in retrieval (i.e., text queries describe parts or generalizations of video content), HLFormer introduces the Partial Order Preservation Loss ($x_s$0). Sentence ($x_s$1) and video embeddings ($x_s$2) are projected to Lorentz space. An entailment cone with half-aperture $x_s$3 anchors the video embedding, requiring the text embedding to fall within this cone:

$x_s$4

where $x_s$5 is the video–text exterior angle. This loss constrains embeddings hierarchically, aligning representations with the partial relevance structure central to PRVR.

The overall HLFormer loss aggregates $x_s$6 (standard cross-modal retrieval), $x_s$7 (sentence query diversity), and $x_s$8, the latter weighted by tunables $x_s$9.

5. Training Configuration and Implementation Details

HLFormer uses a diverse set of benchmarks for evaluation, including ActivityNet Captions, TVR, and Charades-STA datasets, employing both ResNet152 (frame) and I3D (clip) features for videos, and RoBERTa encodings for text (projected to $L^n$0). The architecture deploys 8 attention blocks ($L^n$1, $L^n$2 Lorentz, $L^n$3 Euclid), with 4 heads per block.

Empirically important training aspects include:

• Adam optimizer, batch size 128, 100 epochs, learning rate schedule as in MS-SL.
• Numerical stability via learnable scales in exponential/log maps and clipping Lorentz inner products.
• Implementation and code base at https://github.com/lijun2005/ICCV25-HLFormer.

6. Experimental Results and Ablation Analysis

HLFormer establishes new state-of-the-art performance on PRVR metrics, evaluated by SumR ($L^n$4R@K):

Dataset	HLFormer SumR	Best Baseline	Δ Improvement
ActivityNet	154.9	147.6	+7.3
Charades-STA	78.7	74.7	+4.0
TVR	187.7	179.9	+7.8

Ablation studies demonstrate that removal of the gaze or glance branch, omitting $L^n$5, and replacing MAIM fusion with static or naive alternatives all yield significant degradation in retrieval scores. Visualizations using UMAP or HoroPCA confirm that hyperbolic attention blocks form tighter intra-moment clusters and separate relevant moments from background more effectively. The $L^n$6 loss additionally arranges text embeddings within video entailment cones, corroborating semantic order alignment.

7. Algorithmic Workflow

A high-level procedural sketch is as follows:

1. Extract text and video features via pretrained encoders and project to the target latent space.
2. For each temporal granularity (frames, clips), process inputs through multiple LABs and EABs.
3. Fuse block outputs dynamically with MAIM.
4. Compute cross-modal matching scores.
5. Apply $L^n$7, $L^n$8, and $L^n$9 losses for optimization, with unified embeddings projected into Lorentz space for final matching and partial order enforcement.
6. Iterate over training steps as in conventional deep learning pipelines (Li et al., 23 Jul 2025).

HLFormer demonstrates that integrating hyperbolic geometry within deep video retrieval models yields improved hierarchical modeling, more effective partial relevance grounding, and robust cross-modal retrieval capability, advancing the state of PRVR methodologies.