Cone Attention Mechanisms

• Cone Attention is a geometry-aware method that replaces dot-product similarity with cone-structured relations to explicitly encode hierarchy.
• The hyperbolic formulation uses shadow cones in Poincaré space to capture latent tree structures, while the Lorentzian approach models causal, part-whole relationships via light-cone membership.
• Empirical results show improved parameter efficiency and performance across tasks in translation and vision, underscoring distinct inductive biases in each formulation.

Searching arXiv for papers directly related to "Cone Attention" and hierarchy-aware attention mechanisms. Cone Attention denotes a family of geometry-aware attention mechanisms in which similarity is determined by cone-structured relations rather than by a Euclidean dot product. In current arXiv usage, the term primarily refers to two distinct constructions: a hyperbolic formulation based on entailment or shadow cones, where two points are associated by the depth of their lowest common ancestor in a latent hierarchy, and a Lorentzian formulation based on light-cone membership, where hierarchy is treated as a temporal coordinate and attention is biased by causal structure (Tseng et al., 2023, Kartik et al., 25 Mar 2026).

1. Scope and central idea

Both major uses of the term replace the standard similarity score in attention with a cone-derived notion of compatibility, but they do so for different geometric reasons. The hyperbolic formulation targets latent tree structure and partial orders. The Lorentzian formulation targets asymmetric causality in hierarchies, especially part-whole structure in vision. In both cases, the aim is to make hierarchy explicit in the attention kernel rather than to rely on dot products to recover it implicitly (Tseng et al., 2023, Kartik et al., 25 Mar 2026).

Formulation	Geometry	Association principle
Hyperbolic cone attention	Poincaré half-space	Depth of lowest common ancestor in a cone-induced hierarchy
Lorentzian cone attention	Minkowski spacetime	Lorentzian distance plus forward light-cone membership

In the hyperbolic case, cone attention is presented as a drop-in replacement for dot-product attention in transformers and related architectures. In the Lorentzian case, the cone mechanism is embedded in Worldline Slot Attention, where each object is represented not by a single slot but by a worldline of slots sharing the same spatial position and differing in a hierarchy coordinate interpreted as time. The shared feature of both families is that the receptive field is no longer isotropic in an ordinary Euclidean latent space.

A common misconception is to equate cone attention with ordinary causal masking in sequence models. The literature distinguishes them sharply. In transformer terminology, causal attention usually refers to triangular masks over sequence indices; cone attention instead uses geometric cones in hyperbolic space or spacetime, and the restriction is expressed by the similarity itself rather than by a purely combinatorial mask (Tseng et al., 2023, Kartik et al., 25 Mar 2026).

2. Hyperbolic cone attention

The hyperbolic formulation uses the Poincaré upper half-space model

$$\mathsf{H}^d = (\mathcal{U}^d, g_u), \qquad \mathcal{U}^d = \{x\in\mathbb{R}^d : x_d > 0\},$$

with metric $g_u(x)=g_e/x_d^2$. Height $x_d$ is interpreted as hierarchical depth: smaller $x_d$ is closer to leaves, larger $x_d$ is closer to the root at infinity. The construction uses shadow cones, in penumbral or umbral form, to define a partial order on points in hyperbolic space (Tseng et al., 2023).

Given two embedded points $u,v\in\mathsf{H}^d$, the core object is the lowest common ancestor $\sup_2(u,v)$, defined as the root of the lowest cone containing both points. Cone attention then uses a hierarchy-aware similarity of the form

$K(u,v) = f\big(H(\sup_2(u,v), \mathcal{S})\big),$

where $\mathcal{S}$ is the light source in the shadow-cone construction and $f$ is monotonically decreasing. In the infinite-setting construction actually used for attention, the similarity is implemented directly from the Euclidean height of $g_u(x)=g_e/x_d^2$0, producing closed-form kernels for both penumbral and umbral cones (Tseng et al., 2023).

For infinite-setting umbral cones, the lowest-common-ancestor height is

$g_u(x)=g_e/x_d^2$1

and the similarity is

$g_u(x)=g_e/x_d^2$2

The penumbral case is piecewise and depends on whether a cone exists that contains both points. A notable special case is that when all points lie on the same horosphere, cone attention reduces to a Laplacian kernel in the horizontal coordinates (Tseng et al., 2023).

The attention operator itself preserves the standard softmax-and-value-aggregation pattern: $g_u(x)=g_e/x_d^2$3 Only the similarity changes. Queries and keys are first mapped from Euclidean space into hyperbolic space. Two maps are used in practice: $g_u(x)=g_e/x_d^2$4 for the umbral case, and

$g_u(x)=g_e/x_d^2$5

for the penumbral case. Values remain Euclidean, and positional encodings are handled as in the baseline transformer (Tseng et al., 2023).

Conceptually, this realizes attention over a latent tree. Similarity is high when two tokens share a deep lowest common ancestor and low when they diverged near the root. The paper also emphasizes a transitivity-like structure absent from dot products: if two pairs have high cone similarity through a shared intermediate token, the third similarity cannot be arbitrarily small in the same way Euclidean inner products allow (Tseng et al., 2023).

3. Lorentzian light-cone attention

A distinct formulation appears in Worldline Slot Attention for hierarchical object discovery in vision. Here each object is represented by a worldline

$g_u(x)=g_e/x_d^2$6

with a learned spatial center $g_u(x)=g_e/x_d^2$7 replicated across fixed hierarchy times $g_u(x)=g_e/x_d^2$8, $g_u(x)=g_e/x_d^2$9, and $x_d$0. In the experiments, $x_d$1 objects and $x_d$2 hierarchy levels are used. The temporal coordinate is not physical time but an abstract hierarchy coordinate corresponding roughly to object, part, and subpart levels (Kartik et al., 25 Mar 2026).

Features and slots live in Lorentzian $x_d$3 with Minkowski inner product

$x_d$4

and proper-time distance

$x_d$5

The causal structure is then encoded by light cones. For a slot $x_d$6, the future light cone is

$x_d$7

Lower-$x_d$8 slots are more abstract and have wider future cones; higher-$x_d$9 slots are more specific and have narrower cones (Kartik et al., 25 Mar 2026).

The cone score used in attention is defined from temporal gap $x_d$0, spatial distance $x_d$1, and an adaptive horizon $x_d$2: $x_d$3 This score favors forward-in-time, timelike relations and strongly penalizes backward attention and spacelike separation. Cone width is level dependent, with base horizons

$x_d$4

modulated by local density $x_d$5 via

$x_d$6

Sparse regions therefore get wider cones, and dense regions get narrower ones (Kartik et al., 25 Mar 2026).

The resulting attention logit combines $x_d$7 with $x_d$8, using $x_d$9 and temperature $x_d$0, followed by a softmax over slots. The authors describe this as a cone-shaped, causally directed attention mechanism in spacetime. Attention is biased from low $x_d$1 to high $x_d$2; backward attention is not hard-masked but heavily discouraged (Kartik et al., 25 Mar 2026).

This formulation is not merely a new kernel. The three level-slots of an object share the same spatial coordinate $x_d$3, and updates are performed at the level of the shared center. After each attention iteration, evidence from the three levels is aggregated before a GRU update, enforcing multi-level sharing across the worldline (Kartik et al., 25 Mar 2026).

4. Architectural integration and algorithmic form

The two cone-attention traditions differ substantially in how invasive the geometric modification is. In the hyperbolic formulation, cone attention is explicitly designed as a drop-in replacement for dot-product attention. Query, key, and value projections remain standard; the only altered component is the similarity function. Multi-head attention, batching, positional encodings, and Euclidean value aggregation remain unchanged. The computational complexity stays $x_d$4, although the reported PyTorch implementation incurs a 10–20% slowdown per training step because the kernel uses norms, square roots, exponentials, and other elementwise operations rather than a single batched matrix multiplication (Tseng et al., 2023).

In the Lorentzian formulation, the cone mechanism is inseparable from the architecture. Worldline binding makes levels non-exchangeable, because the mapping from slot index to hierarchy level is fixed by the temporal coordinates. Feature times are learned from local density using

$x_d$5

so sparse points receive lower $x_d$6 and dense points higher $x_d$7. The model then iterates three times: computing horizons, Lorentzian distances, cone scores, attention weights, per-slot aggregates, level-wise aggregation inside each worldline, GRU updates of $x_d$8, and reconstruction of slots $x_d$9. The loss is

$u,v\in\mathsf{H}^d$0

with no explicit hierarchy loss. The reported model size is only $u,v\in\mathsf{H}^d$1 parameters, and the hierarchy is claimed to emerge from geometry plus worldline binding rather than from direct supervision (Kartik et al., 25 Mar 2026).

These design differences matter for interpretation. In one line of work, cone attention is a similarity operator compatible with existing attention blocks. In the other, it is a geometric prior governing an entire latent object model. This suggests that the term names a design family rather than a single operator.

5. Empirical behavior

The empirical literature separates cleanly along the two geometric choices. Hyperbolic cone attention is evaluated as a replacement for dot-product attention in existing architectures. Lorentzian cone attention is evaluated inside a purpose-built worldline architecture for hierarchical object discovery.

For hyperbolic cone attention, reported gains span graph learning, machine translation, vision transformers, language modeling, and diffusion. On IWSLT’14 De→En with a fairseq transformer_iwslt_de_en model, the reported BLEU scores are approximately 34.56 for dot product, 35.56 for penumbral cone attention, and 35.07 for umbral cone attention. On DeiT-Ti trained on ImageNet-1K for 500 epochs, top-1/top-5 accuracy moves from 73.65/91.99 for dot product to 74.34/92.38 for penumbral and 74.46/92.54 for umbral attention. On WikiText-103 with adaptive inputs and context windows $u,v\in\mathsf{H}^d$2, perplexity changes from 20.86/19.22 for dot product to 20.72/19.01 for penumbral, while umbral is worse on that task at 21.16/19.59. For DiT-B/4 on ImageNet-1K at 400K steps, FID-50K improves from 68.9 to 67.7 or 67.6 (Tseng et al., 2023).

A recurrent result in that paper is parameter efficiency. On IWSLT’14 De–En, cone attention with token embedding dimension $u,v\in\mathsf{H}^d$3 is reported to match or exceed dot-product performance at $u,v\in\mathsf{H}^d$4, reducing parameters from 39.5M to 31.2M, a 21% reduction. On DeiT-Ti at 300 epochs, dot product with $u,v\in\mathsf{H}^d$5 gives 72.05/91.17, while penumbral cone attention with $u,v\in\mathsf{H}^d$6 gives 72.25/91.20 (Tseng et al., 2023).

For Lorentzian cone attention in vision, the central comparison is between LoCo, Hyperbolic WL, Euclidean WL, and Euclidean Std, all with $u,v\in\mathsf{H}^d$7K parameters, the same architecture otherwise, three attention iterations, and the same optimizer. The reported Object ARI / Level Accuracy results are:

Dataset	LoCo	Hyperbolic WL	Euclidean WL	Euclidean Std
Toy	0.508 / 0.539	0.151 / 0.395	0.515 / 0.078	0.403 / 0.328
Sprites	0.618 / 0.479	0.171 / 0.345	0.475 / 0.079	0.445 / 0.335
CLEVR	0.227 / 0.661	0.195 / 0.534	0.001 / 0.078	0.003 / 0.359

The corresponding average level accuracies are 0.559 for LoCo, 0.425 for Hyperbolic WL, 0.078 for Euclidean WL, and 0.341 for Euclidean Std, with random chance for three levels given as approximately 0.33. The Euclidean worldline model is reported to collapse deterministically to 0.078 level accuracy with standard deviation $u,v\in\mathsf{H}^d$8, despite good object clustering on some datasets. The Lorentzian model, by contrast, achieves 0.479–0.661 level accuracy and is described as stable across seeds, with standard deviation $u,v\in\mathsf{H}^d$9 on CLEVR (Kartik et al., 25 Mar 2026).

The interpretation advanced in the vision work is specific: worldline binding without causal geometry is not sufficient, and hyperbolic geometry, while better than the Euclidean baseline, remains weaker than Lorentzian geometry because it encodes radial tree structure rather than asymmetric causal structure (Kartik et al., 25 Mar 2026).

6. Distinctions, limitations, and recurring debates

The most important distinction is terminological. “Cone Attention” does not designate a single canonical mechanism. One research line uses hyperbolic cones to measure lowest-common-ancestor depth; the other uses Lorentzian light cones to enforce asymmetric, forward causal neighborhoods. Their shared vocabulary should not obscure the fact that they encode different inductive biases and solve different modeling problems (Tseng et al., 2023, Kartik et al., 25 Mar 2026).

A second recurring debate concerns what kind of hierarchy is being modeled. The hyperbolic formulation treats hierarchy as latent tree structure and uses cone depth as a proxy for divergence in that tree. The Lorentzian formulation explicitly argues that visual hierarchies are causal rather than tree-like: parts depend asymmetrically on wholes, and this asymmetry is not naturally captured by radial branching alone. The reported comparison between Hyperbolic WL and LoCo is presented as evidence for that claim in the point-cloud setting (Kartik et al., 25 Mar 2026).

The two literatures also report different limitations. Hyperbolic cone attention is slower than dot-product attention by 10–20% per training step in the reported implementation, and hyperbolic embeddings show somewhat greater sensitivity to initialization. Umbral attention underperforms dot product and penumbral attention on WikiText-103, which the paper relates to the non-convexity of umbral cones. Gains on diffusion are smaller than on more overtly hierarchical tasks (Tseng et al., 2023).

The Lorentzian formulation has a different set of restrictions. It assumes a density-to-hierarchy correlation, uses a fixed three-level depth, and is evaluated on 2D synthetic point clouds rather than raw images. The authors identify integration with CNN or ViT encoders as future work. They also make the level mapping intentionally non-permutation-invariant, so level accuracy measures whether learning respects the built-in temporal ordering rather than whether a model can post hoc permute slots to fit labels (Kartik et al., 25 Mar 2026).

Taken together, these works indicate that cone attention is best viewed as a geometric strategy for enforcing non-Euclidean relational structure inside attention. A plausible implication is that the appropriate cone depends on the structure one wants to privilege: lowest-common-ancestor depth for latent trees, or asymmetric causal reachability for part-whole hierarchies.