HyFI: Hyperbolic Feature Interpolation for Brain-Vision Alignment

Abstract: Recent progress in artificial intelligence has encouraged numerous attempts to understand and decode human visual system from brain signals. These prior works typically align neural activity independently with semantic and perceptual features extracted from images using pre-trained vision models. However, they fail to account for two key challenges: (1) the modality gap arising from the natural difference in the information level of representation between brain signals and images, and (2) the fact that semantic and perceptual features are highly entangled within neural activity. To address these issues, we utilize hyperbolic space, which is well-suited for considering differences in the amount of information and has the geometric property that geodesics between two points naturally bend toward the origin, where the representational capacity is lower. Leveraging these properties, we propose a novel framework, Hyperbolic Feature Interpolation (HyFI), which interpolates between semantic and perceptual visual features along hyperbolic geodesics. This enables both the fusion and compression of perceptual and semantic information, effectively reflecting the limited expressiveness of brain signals and the entangled nature of these features. As a result, it facilitates better alignment between brain and visual features. We demonstrate that HyFI achieves state-of-the-art performance in zero-shot brain-to-image retrieval, outperforming prior methods with Top-1 accuracy improvements of up to +17.3% on THINGS-EEG and +9.1% on THINGS-MEG.

• The paper introduces HyFI, a hyperbolic geodesic interpolation method that fuses semantic and perceptual visual features while compressing them to better match the limited, entangled information in EEG and MEG signals.
• HyFI achieves state-of-the-art 200-way retrieval, raising Top-1 accuracy from 50.9% to 68.2% on THINGS-EEG and from 26.7% to 35.8% on THINGS-MEG, with statistically significant gains across five runs.
• Ablations show that hyperbolic geometry and feature interpolation provide complementary benefits, while remaining challenges include modest cross-subject accuracy, backbone dependence, and validation beyond retrieval tasks.

Motivation and problem setting

HyFI addresses zero-shot brain-to-image retrieval from non-invasive neural recordings (EEG and MEG), where the goal is to map brain activity into a shared embedding space with visual stimuli so that the image corresponding to the perceived stimulus can be retrieved among unseen candidates. The authors identify two structural limitations of prior dual-pathway decoding frameworks, which typically align brain signals separately with semantic features (CLIP embeddings) and perceptual features (e.g., VAE-derived representations). First, the modality gap: neural signals carry substantially less information than image embeddings, owing to attentional bottlenecks, limited visual working memory, and the low signal-to-noise ratio of EEG/MEG recordings. Second, feature entanglement: perceptual and semantic information is encoded interactively in neural activity rather than in independent channels, so aligning each feature type through a separate pathway is mismatched with how the brain represents stimuli.

Method

The framework operates in the Lorentz (hyperboloid) model of hyperbolic space with learnable curvature $\kappa$. Semantic and perceptual views of each image are produced by two augmentations: fovea blur (simulating peripheral vision to preserve semantics) and Gaussian blur (suppressing high-frequency detail to emphasize coarse perceptual attributes such as color and shape). These are encoded by a frozen pre-trained vision-language backbone (default CLIP-RN50), projected by learnable linear layers, and lifted onto the hyperboloid via the exponential map at the time origin, with learnable scalars $\alpha_v$, $\alpha_b$ controlling embedding norms. The core operation is geodesic interpolation between the semantic feature $\mathbf{z}_v^s$ and the perceptual feature $\mathbf{z}_v^p$:

$$\hat{\mathbf{z}}_v = \exp^{\kappa}_{\mathbf{z}_v^{s}}\left(t \cdot \log^{\kappa}_{\mathbf{z}_v^{s}}(\mathbf{z}_v^{p})\right),$$

where the interpolation coefficient $t = \sigma(W_t f_v(\mathbf{x}_v^s))$ is predicted per-image by a learned linear head. Alignment is then performed with a symmetric hyperbolic contrastive loss between interpolated visual embeddings and brain encoder outputs.

The central geometric claim is that this interpolation simultaneously fuses the two feature types and compresses their information content. Rewriting the Lorentz geodesic as a weighted combination,

$$\gamma_{\mathbf{p} \to \mathbf{q}}(t) = \frac{\sinh((1-t)\beta)}{\sinh(\beta)}\mathbf{p} + \frac{\sinh(t\beta)}{\sinh(\beta)}\mathbf{q}, \quad \beta = \sqrt{\kappa}\, d_{\mathbb{L}}(\mathbf{p},\mathbf{q}),$$

the coefficients are strictly smaller than their Euclidean counterparts $(1-t)$ and $t$ for $\alpha_v$0, and their sum is strictly less than one. The appendix proves that the spatial component of the hyperbolic interpolation therefore lies strictly closer to the origin than Euclidean interpolation (except in degenerate collinear cases), and since $\alpha_v$1, proximity to the origin directly bounds representational capacity. This is consistent with prior observations that points near the hyperbolic origin encode more abstract concepts [ganea2018hyperbolic; khrulkov2020hyperbolic]. The compression effect is thus presented as a principled mechanism for matching the limited information content of brain signals while reflecting their entangled encoding of perceptual and semantic attributes.

Main results

On 200-way zero-shot retrieval, HyFI sets state-of-the-art results on both benchmarks, averaged over 5 runs with reported statistical significance ($\alpha_v$2):

Benchmark	Metric	UBP (prior SOTA)	HyFI	Improvement
THINGS-EEG (intra-subject)	Top-1	50.9%	68.2%	+17.3
THINGS-EEG (intra-subject)	Top-5	79.7%	91.9%	+12.2
THINGS-MEG (intra-subject)	Top-1	26.7%	35.8%	+9.1
THINGS-MEG (intra-subject)	Top-5	55.2%	64.6%	+9.4

The gains hold across all ten EEG subjects individually (e.g., subject 8 reaches 78.9% Top-1 versus 58.6% for UBP). In the harder inter-subject (leave-one-out) setting, HyFI improves average Top-1 from 12.4% to 15.1% on THINGS-EEG and from 2.2% to 3.2% on THINGS-MEG, though it is not uniformly best per subject—for instance, ATM-S outperforms HyFI on subjects 4 and 10 in Top-5 accuracy.

Ablations and analyses

The ablation isolates the contributions of geometry and interpolation. Hyperbolic alignment alone raises THINGS-EEG Top-1 from 49.4% to 54.3%; Euclidean (CLIP-space) interpolation alone reaches 59.7%; combining both yields 68.2%. The same ordering holds on MEG (23.1 → 28.8 → 25.2 → 35.8), indicating that fusion and geometric compression contribute complementary benefits.

Encoder generality is demonstrated over eleven vision backbones (CLIP RN50/RN101, five ViT variants, MERU, HyCoCLIP) and four brain encoders (ShallowNet, EEGNet, TSConv, EEGProject); HyFI improves every combination. Notably, CNN-based CLIP backbones—particularly RN50—align better with neural signals than larger ViTs, and lightweight models outperform deeper ones, suggesting compact representations are more compatible with neural data. The augmentation ablation confirms fovea blur as the best semantic transform and Gaussian blur as the best perceptual one (68.2% Top-1 versus 65.3% for Gaussian blur applied without fovea blur).

Two analyses support the mechanism. Distance-from-origin visualization shows interpolated embeddings contract toward the time origin in hyperbolic space but not in CLIP space, while EEG embeddings sit farther out, consistent with their higher variability requiring less-constrained regions. The learned coefficient $\alpha_v$3 concentrates below 0.5, indicating a semantic-dominant mixture; qualitatively, images with low $\alpha_v$4 are iconic category exemplars (banana–fruit, cheetah–mammal), whereas high-$\alpha_v$5 images exhibit salient low-level attributes. An additional ablation shows computing $\alpha_v$6 from EEG features degrades performance substantially (54.5% versus 68.2%), implying the coefficient should reflect stimulus properties rather than noisy neural input.

Limitations and open questions

Several caveats qualify these results. The inter-subject gains, while consistent, are modest in absolute terms (15.1% Top-1 on EEG, 3.2% on MEG), leaving cross-subject generalization largely unresolved. The method's benefit depends on augmentation sensitivity of the vision backbone: the appendix shows the semantic/perceptual dissociation induced by fovea and Gaussian blur is pronounced in CLIP-RN50 but weak or absent in models such as CLIP-ViT-L/14, so the approach's effectiveness is partly contingent on encoder choice rather than purely geometric properties. The compression argument rests on the assumption that reduced distance from the origin corresponds to information content appropriate for brain signals—an interpretation borrowed from prior hierarchical-embedding work rather than directly measured here. Finally, evaluation is restricted to 200-way zero-shot retrieval on THINGS benchmarks; whether hyperbolic interpolation transfers to full image reconstruction or other decoding targets remains untested.

Conclusion

HyFI reframes brain-vision alignment as a problem of matching both the entanglement and the limited capacity of neural representations, using hyperbolic geodesic interpolation to fuse and compress semantic and perceptual visual features within a single pathway. The large improvements over prior state of the art (+17.3% Top-1 on THINGS-EEG, +9.1% on THINGS-MEG), together with consistent gains across encoder combinations and supporting geometric analysis, establish hyperbolic interpolation as an effective alignment mechanism for information-asymmetric modalities. The main open questions concern cross-subject robustness, dependence on augmentation-sensitive backbones, and extension beyond retrieval-based evaluation.