HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoning

Abstract: Vision-LLMs (VLMs) struggle with compositional reasoning that requires understanding inter-object relationships. A natural remedy is to inject explicit scene graph triplets $\langle s, p, o \rangle$ from an off-the-shelf scene graph generator (SGG), but we show this backfires: discrete text labels collide with the continuous visual modality, degrading GQA accuracy from 60.38\% to 58.86\%. We propose \textbf{HyperVis}, which bypasses the SGG semantic bottleneck entirely. From $N$ class-agnostic region proposals, we compute a dense $O(N<sup>2)$ visual relation tensor via spatially-biased cross-attention, project it onto a Lorentz hyperboloid, and enforce hierarchy through spatial physics, namely IoA-driven entailment cones and exterior-angle repulsion. We discover that HyperVis contributes in two complementary ways: (1) as a \emph{training-time regularizer}, the hyperbolic relational losses shape LoRA representations that improve generative VQA (GQA 61.03\% vs.\ 57.21\% for LoRA fine-tuning without relational losses, recovering and surpassing the baseline); and (2) as an \emph{inference-time relational encoder}, hyperbolic prefix tokens boost discriminative compositional scoring (SugarCrepe 79.94\%, $+$6.25pp over baseline). The learned curvature stabilises at $κ{=}4.0$, an order of magnitude above prior hyperbolic VLMs where $κ$ typically collapses toward zero, indicating that continuous visual features genuinely require the exponential volume of strongly curved space. A controlled Euclidean ablation confirms this decomposition: the relational pipeline regularises LoRA comparably in flat space (GQA 60.81\%), but the compositionality gain is specifically hyperbolic (SugarCrepe $+$4.58pp over Euclidean), with entailment loss ${\sim}6{\times}$ higher in Euclidean training. Codes are available at TBA.

• The paper introduces HyperVis, a method using continuous latent visual relational graphs on the Lorentz hyperboloid to enhance compositional reasoning.
• It leverages spatially-biased cross-attention and IoA-driven geometric constraints to preserve relational and hierarchical structure.
• Empirical results on generative VQA and compositional benchmarks demonstrate improved performance over discrete scene graph approaches.

HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoning

Overview

"HyperVis: Continuous Latent Visual Relational Graphs on the Lorentz Hyperboloid for Compositional Reasoning" (2606.06100) presents a novel architectural paradigm for enabling compositional and relational reasoning in vision-LLMs (VLMs). The authors introduce HyperVis, an explicit continuous relational graph computation over class-agnostic visual features that is embedded in hyperbolic space, specifically the Lorentz model, and shaped via IoA-driven geometric constraints. This framework is proposed as an alternative to the textual scene graph injection approaches that rely on off-the-shelf scene graph generators (SGGs) and exploit discrete predicate vocabularies.

Motivation and Limitations of Prior Work

Contemporary VLMs such as BLIP-2 and LLaVA demonstrate limited compositional and relational reasoning, as evidenced by suboptimal performance on benchmarks such as SugarCrepe and Winoground, which directly probe relational and compositional understanding. While integrating SGG triplets into VLMs is a common attempt to enhance relational capacity, the authors rigorously demonstrate that this approach is counterproductive. Injecting discrete SGG predicates into a VLM not only fails to address the modality gap but further degrades accuracy (GQA: 58.86% vs. 60.38% baseline), attributed to rigid predicate vocabularies, error propagation, and loss of rich spatial-visual cues.

HyperVis: Methodological Innovation

HyperVis forgoes explicit semantic annotation and instead operates entirely within the visual modality. The pipeline can be decomposed as follows:

• Region Proposal and Feature Extraction: From each image, $N=36$ class-agnostic region proposals are extracted, utilizing raw visual features without class predictions.
• Dense Visual Relational Tensor: All pairwise ($O(N^2)$) relations are computed via spatially-biased cross-attention, incorporating signed relative geometry ($\Delta_{ij}$) to preserve directionality.
• Hyperbolic Embedding: Relation features are projected to the Lorentz hyperboloid using the exponential map, with per-region aggregation via the Einstein midpoint, preserving geometric fidelity.
• IoA-driven Hierarchical Constraints: Spatial containment is enforced through binary IoA thresholds. If $IoA(a \to b) > \tau_{in}$, the embedding of $a$ is constrained to lie within the entailment cone of $b$; for $IoA(a \to b) < \tau_{out}$, angular repulsion is enforced.
• Hyperbolic Top-K Gating: The most salient relations (Top-4) are selected by geodesic proximity to a trainable hyperbolic query and injected into the VLM as relational prefix tokens.
• Integration and Optimization: Only LoRA adapters and the specialized relational module are trained; all VLM backbone parameters are frozen. Training employs Riemannian optimization, with rigorous attention to numerical stability, including custom backward passes for $\arccos$/$\arccosh$ and isolated optimizer groups for $\kappa$.

Empirical Results and Analysis

Generative VQA (GQA)

HyperVis demonstrates a dual improvement over traditional approaches:

• LoRA-only Fine-tuning: Fine-tuning only with LoRA adapters on GQA degrades performance (from 60.38% to 57.21%), due to overfitting on surface patterns and loss of relational reasoning capacity.
• LoRA + HyperVis (No Prefix Injection at Inference): Including hyperbolic relational losses fully recovers and surpasses the baseline (61.03%), indicating that dense continuous relational constraints act as effective regularization during LoRA finetuning.
• Textual SGG Injection: The textual triplet injection consistently underperforms, confirming the detrimental effects of the modality gap.

Compositional Reasoning (SugarCrepe, Winoground)

• SugarCrepe: When relational prefix tokens are retained at inference, HyperVis achieves 79.94%, a +6.25pp improvement over baseline LLaVA-1.5-7B, and a +4.58pp gain over the Euclidean ablation. Gains are amplified in evaluation categories stressing relational and hierarchical containment (e.g., Add-Object, Add-Attribute).
• Winoground: HyperVis' gains are less pronounced due to high benchmark variance, but a consistent trend is observed favoring strong compositionality.
• Dual Evaluation Protocol: The decoupling of training and inference impact allows precise dissection of contributions: hyperbolic relational losses regularize LoRA during training for generative VQA, while inference-time prefix tokens inject explicit compositional representations for scoring tasks.

Curvature Dynamics and Geometry

• Learned Curvature: Contrary to prior hyperbolic VLMs, which exhibit a collapse of $O(N^2)$0 toward zero (flat Euclidean geometry), HyperVis achieves a stable, high curvature regime ($O(N^2)$1), empirically demonstrating the necessity of exponential volume for disentangling heavily overlapping visual feature representations.
• Geometry Ablation: Euclidean ablations show that while LoRA regularization benefit is geometry-agnostic, compositional performance (SugarCrepe) is specifically dependent on hyperbolic geometry; entailment cones degrade to imprecise half-space partitions in Euclidean space, sharply diminishing fine-grained compositional accuracy.

Ablations and Robustness

• Prefix Token Count: Increasing $O(N^2)$2 beyond 4 degrades compositional performance, consistent with redundancy and potential attention conflicts.
• Loss Component Ablation: Both entailment and angular loss are required for optimal performance; the former primarily drives compositional improvements, and the latter is essential to maintain high curvature.
• Spatial Encoding: Directional deltas are essential; magnitude-only encoding destroys the relational directionality needed for image-side discrimination.

Comparison to Self-Prompting Schemes

The HyperVis pipeline outperforms chain-of-thought scene graph prompting (CCoT) approaches ([mitra2024ccot]) both in GQA and compositional evaluation, while being structurally more efficient (single forward pass, no textual predicate dependency).

Theoretical and Practical Implications

The compelling evidence for high hyperbolic curvature fundamentally challenges the prevailing understanding around the curvature bottleneck, with direct ramifications for the design of future VLMs. By demonstrating that hierarchical relational structures can be induced via purely geometric cues, HyperVis enables VLMs to acquire compositional reasoning skills without reliance on brittle external semantic parsers. The hyperbolic framework efficiently resolves the feature overlap inherent to visual regions, introducing a scalable and modular mechanism for representing visual hierarchy. This framework generalizes to broader modalities: extension to video via spatiotemporal tubes and integration into embodied reasoning systems appear directly feasible.

Future Directions

Open challenges and promising research paths include: resolution of the generating/scoring tradeoff induced by L2-norm mismatch in hyperbolic prefix tokens, direct extension to video and embodied visual reasoning, adaptive region proposal schemes to mitigate missed/occluded entities, and further optimization for scalability with large $O(N^2)$3.

Conclusion

HyperVis introduces an explicit, geometric alternative to predicate-based relational reasoning within VLMs, leveraging continuous visual scene graphs embedded in the Lorentz hyperboloid and regularized through IoA-driven angular and entailment constraints. The method achieves both superior compositional reasoning and stable high-curvature manifold configuration, empirically establishing the superiority of continuous, hyperbolically structured representations for visual relational reasoning (2606.06100).