Geometry-Guided Gated Transformer (G3T)

• The paper demonstrates that G3T learns per-token geometric preferences via a lightweight gating mechanism that routes tokens through three geometry-specific attention branches.
• It employs a routing MLP to compute convex mixture weights over Euclidean, hyperbolic, and spherical sublayers, enabling adaptive handling of mixed relational structures.
• Empirical evaluations show approximately 10% improvements in MRR and Hits@10 over fixed-geometry baselines with minimal computational overhead.

Searching arXiv for the specified paper to ground the article in the cited source. Geometry-Guided Gated Transformer (G3T) is a transformer architecture that performs per-token routing across three geometry-specific attention branches—Euclidean, hyperbolic, and spherical—through a lightweight, differentiable gating mechanism. In the source formulation, G3T is a renamed presentation of the Curvature-Adaptive Transformer (CAT), intended to emphasize the gating perspective rather than the curvature-adaptive label (Lin et al., 2 Oct 2025). The central premise is that standard transformer attention implicitly assumes Euclidean geometry, whereas relational data may exhibit mixed geometric structure; G3T addresses this by learning both token-wise geometric preferences and non-Euclidean curvature parameters within a single model (Lin et al., 2 Oct 2025).

1. Architectural definition

G3TBlock takes an input tensor $X \in \mathbb{R}^{B \times N \times d}$. A small routing MLP computes per-token logits over three expert branches, Euclidean, Hyperbolic, and Spherical, according to

$$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$

A softmax over the last dimension yields convex weights $$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$. In parallel, $X$ is fed into three geometry-specific attention sublayers that produce $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$. The final output is the gated mixture

$$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$

G3T can be stacked or interleaved with feed-forward and residual layers exactly like a standard transformer (Lin et al., 2 Oct 2025).

This design places G3T within a family of mixture-style architectures, but its experts are not ordinary subnetworks. Each branch corresponds to a distinct geometric manifold and uses geometry-specific operations optimized for that manifold. A plausible implication is that G3T should be understood less as a conventional mixture-of-experts model than as a mixture-of-geometries transformer, where the gating variable expresses geometric preference rather than only computational specialization.

2. Token-wise gating and mixture formation

The per-token gating mechanism is defined directly on the input representation $X\in\mathbb{R}^{B\times N\times d}$. The routing logits are

$$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$

The softmax gates are

$$\alpha_{b,n,g} = \frac{\exp(l_{b,n,g})}{\sum_{h\in\{E,H,S\}}\exp(l_{b,n,h})}, \qquad \text{for } g\in\{E,H,S\}.$$

The mixture output is

$$Y = \sum_{g\in\{E,H,S\}} \alpha_{[...,g]} \odot Y^{(g)},$$

where $$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$0 denotes element-wise scaling broadcast over the feature dimension (Lin et al., 2 Oct 2025).

The gating is therefore convex and token-local. Each token receives a distribution over the three geometries rather than a hard assignment. This matters because the source description frames the routing weights as interpretable curvature preferences: at any point, $$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$1 quantifies the model’s geometric bias per token, while averaging $$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$2 over tokens and batches yields a global “usage” score of each geometry (Lin et al., 2 Oct 2025). High $$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$3 signals hierarchical (tree-like) structures, high $$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$4 indicates cyclic or periodic patterns, and high $$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$5 corresponds to locally flat relations. One can also correlate $$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$6 with relation types or token positions to discover structure (Lin et al., 2 Oct 2025).

A common misconception would be to treat the gating as merely an ensembling heuristic. The formulation instead ties routing to geometric inductive bias: the gate is not selecting arbitrary branches, but distributing mass among manifolds associated with distinct relational structure.

3. Geometry-specific self-attention branches

The Euclidean branch uses standard scaled dot-product attention. With branch-specific projections $$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$7,

$$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$8

$$l = W_g \cdot X + b_g,\qquad l \in \mathbb{R}^{B \times N \times 3}.$$9

$$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$0

and

$$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$1

The hyperbolic branch operates in a Poincaré ball of curvature $$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$2. The formulation maps projected inputs to the manifold using the exponential map at the origin:

$$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$3

The hyperbolic distance is

$$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$4

Attention weights are defined via negative distance:

$$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$5

Aggregation uses Möbius scalar multiplication and addition,

$$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$6

followed by stacking and the same residual-plus-FFN pattern:

$$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$7

The spherical branch operates on the unit sphere of curvature $$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$8. Inputs are lifted to $$\alpha = (\alpha_E,\alpha_H,\alpha_S)\in\mathbb{R}^{B\times N\times 3}$$9 by appending zero, then mapped via the exponential map at $X$0:

$X$1

Using the spherical inner product $X$2,

$X$3

The branch performs weighted sum on the sphere via geodesic-weighted combination and then applies the log map:

$X$4

These are assembled into $X$5, followed by the same residual and feed-forward structure to obtain $X$6 (Lin et al., 2 Oct 2025).

Taken together, the three branches implement distinct relational priors. The Euclidean branch corresponds to flat interactions, the hyperbolic branch to negative-curvature structure, and the spherical branch to positive-curvature structure. This suggests that G3T is designed for heterogeneous geometric regimes within the same input rather than for a globally homogeneous manifold assumption.

4. Learnable curvature and optimization behavior

G3T treats the hyperbolic curvature $X$7 and spherical curvature $X$8 as learnable scalars. To ensure positivity, each is parameterized via softplus:

$X$9

Gradients flow through exponential and logarithmic maps, through $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$0, and thus update $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$1 via backpropagation. In practice, $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$2 and $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$3 are initialized, for example with $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$4, and updated per training step with the same optimizer, AdamW (Lin et al., 2 Oct 2025).

This coupling of gating and curvature learning is structurally important. The source summary states that G3T learns both the curvature and the gating jointly, enabling specialization to hierarchical relations via negative curvature, cyclic structures via positive curvature, and flat interactions via zero curvature, all within one forward pass (Lin et al., 2 Oct 2025). A plausible implication is that the model does not merely choose among pre-specified geometries; it also tunes the strength of the non-Euclidean departures represented by the hyperbolic and spherical branches.

The optimization recipe reported for reproduction includes standard AdamW, learning rate approximately $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$5, dropout approximately $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$6, and label smoothing $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$7. It also recommends initializing curvature $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$8 so that $$Y^{(E)},Y^{(H)},Y^{(S)} \in \mathbb{R}^{B\times N\times d}$$9 and applying an entropy regularization

$$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$0

annealed from $$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$1 to encourage early mixing (Lin et al., 2 Oct 2025).

5. Empirical results and ablation findings

The reported experiments evaluate knowledge-graph completion on FB15k-237 and WN18RR, with approximately $$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$2M and approximately $$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$3M parameters per model. The reported results are as follows (Lin et al., 2 Oct 2025):

Model	FB15k-237	WN18RR
Fixed-E	MRR $$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$4, Hits@10 $$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$5	MRR $$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$6, Hits@10 $$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$7
Fixed-H	MRR $$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$8, Hits@10 $$Y_{b,n,:} = \alpha_{b,n,E}\,Y^{(E)}_{b,n,:} + \alpha_{b,n,H}\,Y^{(H)}_{b,n,:} + \alpha_{b,n,S}\,Y^{(S)}_{b,n,:}.$$9	MRR $X\in\mathbb{R}^{B\times N\times d}$0, Hits@10 $X\in\mathbb{R}^{B\times N\times d}$1
Fixed-S	MRR $X\in\mathbb{R}^{B\times N\times d}$2, Hits@10 $X\in\mathbb{R}^{B\times N\times d}$3	MRR $X\in\mathbb{R}^{B\times N\times d}$4, Hits@10 $X\in\mathbb{R}^{B\times N\times d}$5
G3T (ours)	MRR $X\in\mathbb{R}^{B\times N\times d}$6, Hits@10 $X\in\mathbb{R}^{B\times N\times d}$7	MRR $X\in\mathbb{R}^{B\times N\times d}$8, Hits@10 $X\in\mathbb{R}^{B\times N\times d}$9

The accompanying abstract characterizes these outcomes as approximately $$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$0 improvements in MRR and Hits@10 over fixed-geometry baselines, with minimal overhead consisting of a $$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$1 parameter increase and comparable inference time (Lin et al., 2 Oct 2025). The summary likewise describes the result as approximately $$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$2 relative gains in relational reasoning benchmarks over any fixed-geometry counterpart (Lin et al., 2 Oct 2025).

The ablation findings identify two specific phenomena. First, removing entropy regularization on $$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$3 early in training causes routing to collapse to Euclidean and MRR drops by approximately $$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$4–$$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$5. Second, turning off one branch, for example removing the spherical branch, yields intermediate performance between full G3T and the best fixed geometry, confirming that each branch contributes (Lin et al., 2 Oct 2025).

These findings narrow the interpretation of the gains. The reported improvement is not presented as arising solely from increased parameter count; rather, the ablations attribute part of the effect to nontrivial routing behavior and branch complementarity. This suggests that the geometric mixture itself, and not only architectural expansion, is operationally significant.

6. Efficiency, implementation, and reproducibility

The implementation notes state that G3T adds only $$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$6—approximately $$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$7 per token—on top of three parallel attention heads, for approximately $$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$8 total overhead (Lin et al., 2 Oct 2025). For runtime, FB15k-237 inference latency on A10 is reported as fixed-E $$l = W_g\,X + b_g,\qquad l\in\mathbb{R}^{B\times N\times 3}.$$9 versus G3T $$\alpha_{b,n,g} = \frac{\exp(l_{b,n,g})}{\sum_{h\in\{E,H,S\}}\exp(l_{b,n,h})}, \qquad \text{for } g\in\{E,H,S\}.$$0, including three attentions, and still faster than running three separate models (Lin et al., 2 Oct 2025).

The recommended software options are geoopt for exponential, logarithmic, and Möbius operations, or custom CUDA kernels; the feed-forward layers may be shared across branches to save memory (Lin et al., 2 Oct 2025). For evaluation, the reported protocol is masked (filtered) ranking, with at least five seeds (Lin et al., 2 Oct 2025).

These implementation details clarify an important practical point. The architecture uses three geometry-specific branches in parallel, so “minimal overhead” in the abstract should not be conflated with unchanged branch-level computation. The source instead specifies the trade-off precisely: a modest parameter increase, measurable additional latency relative to a single Euclidean branch, and improved efficiency relative to deploying three separate models.

7. Interpretation, scope, and relation to CAT

The source explicitly states that G3T is a detailed, self-contained description of the Geometry-Guided Gated Transformer, closely following the design of “CAT: Curvature-Adaptive Transformers for Geometry-Aware Learning,” with the name changed to emphasize the gating view (Lin et al., 2 Oct 2025). In that sense, G3T is not a distinct architecture from CAT in the underlying technical design; it is a renamed presentation of the same core idea.

The CAT abstract frames the broader motivation: transformers implicitly assume Euclidean geometry in their attention mechanisms, which can limit effectiveness on data with non-Euclidean structure; recent extensions to hyperbolic and spherical spaces require committing to a single geometry a priori, reducing flexibility when data exhibits mixed geometric properties (Lin et al., 2 Oct 2025). CAT, and equivalently G3T, is proposed as a scalable and interpretable foundation for mixture-of-geometry architectures across language, vision, and multimodal domains (Lin et al., 2 Oct 2025). Because the detailed experimental evidence provided here is confined to knowledge graph completion benchmarks, any broader claim should be read as architectural scope rather than demonstrated benchmark coverage in those other domains.

The interpretability claim is correspondingly narrow and concrete. The routing network provides interpretable curvature preferences, and the learned gates can be averaged or correlated with structural factors such as relation types or token positions (Lin et al., 2 Oct 2025). A plausible implication is that G3T occupies an intermediate position between pure performance-oriented routing architectures and explicitly analyzable inductive-bias models: its gate values serve simultaneously as control signals and as probes of geometric structure.

Within this framing, G3T is best understood as a transformer variant for heterogeneous relational geometry. Its defining features are soft per-token routing, manifold-specific attention operators, jointly learned curvature, and interpretable geometric usage patterns. The reported evidence supports the claim that learned geometric adaptation can outperform any single fixed geometry in the tested relational reasoning setting (Lin et al., 2 Oct 2025).