MöbiusAttention Variants in Transformers

• MöbiusAttention variants are advanced attention mechanisms that use complex-valued Möbius transformations to capture richer geometric inter-token relationships.
• They incorporate non-linear projective mappings to warp token embeddings into complex space, enabling enhanced modeling of cyclical and spatial patterns compared to standard attention.
• Architectural variants like MöbiusBERT and MobRoFormer achieve competitive performance on GLUE benchmarks with fewer parameters and similar training durations.

MöbiusAttention refers to a class of attention mechanisms for Transformer architectures that replaces the standard linear query computation with non-linear Möbius (projective) transformations in complex space. These variants incorporate geometric operations capable of mapping between lines, circles, and other conic geometries, enabling a richer modeling of inter-token relationships than standard linear approaches. Two primary architectures, MöbiusBERT and MobRoFormer, implement these mechanisms in BERT and RoFormer frameworks respectively, demonstrating empirical gains or parity with reduced model size and training overhead (Halacheva et al., 2024).

1. Möbius Transformation in Attention Mechanisms

The Möbius transformation is defined over the Riemann sphere, or complex projective line $\mathbb{CP}^1$, where any $x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$ is represented in homogeneous coordinates as $[x:1]$. The general form is

$$f(x)=\frac{a x + b}{c x + d}, \quad a,b,c,d \in \mathbb{C},\; ad-bc \neq 0.$$

In MöbiusAttention, each token–position embedding pair is represented as $\rho_i = w_i + i\, p_{w_i}$ with $w_i,p_{w_i} \in \mathbb{R}^d$. The query vector is produced by applying an independent Möbius transformation to each complex coordinate: $$\mathcal M_{q_j}(\rho_{ij}) = \frac{a_{q_j} \rho_{ij} + b_{q_j}}{c_{q_j} \rho_{ij} + d_{q_j}}, \quad j=1,\ldots,d,$$ where the Möbius maps are parameterized by four complex scalars per feature dimension. This mapping warps input geometry, supporting transformations among lines, circles, and broader conic types (parabolic, elliptic, loxodromic, hyperbolic). Keys and values in MöbiusAttention maintain complex-linearity: $$K_j(\rho_{ij}) = w_{k,j}\, \rho_{ij}, \quad V_j(\rho_{ij}) = w_{v,j}\, \rho_{ij}, \quad w_{k,j}, w_{v,j} \in \mathbb{C}.$$

2. Layer Formulation: MöbiusAttention vs. Standard Attention

Standard scaled dot-product attention operates on real-valued projections: $$\mathrm{Attention}(Q,K,V) = \operatorname{softmax} \left( \frac{Q K^\top}{\sqrt{d}} \right) V,\quad Q = X W_Q, K = X W_K, V = X W_V\,.$$ In contrast, a MöbiusAttention head uses the complex embedding $\rho \in \mathbb{C}^{N\times d}$ as input, computes queries using the Möbius transformation, and projects keys and values linearly in complex space. The unnormalized attention scores are obtained as $x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$0 (conjugate transpose). The output is then

$x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$1

Multilayer heads can combine $x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$2 real-valued (standard) heads and $x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$3 complex-valued (Möbius) heads, with outputs concatenated and projected to the next layer.

3. Architectural Variants: MöbiusBERT and MobRoFormer

MöbiusBERT (“framed” design)

• 11 transformer blocks: blocks 2–10 standard BERT, blocks 1 and 11 use mixed heads (6 vanilla, 6 Möbius).
• Block 1 input forms $x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$4 from token and positional embeddings.
• Block 11 input: real channel from block 10 output, imaginary from original token embeddings.
• Feature reduction via optional linear-before-Möbius (“T” variant), where a real linear map reduces parameters.
• Orthogonal initialization variant (“Ortho”) initializes Möbius matrices in $x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$5.
• Model: $x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$6104M parameters (vs. 110M for BERT), unchanged pre-training time (26 h).

MobRoFormer

• RoFormer backbone with 10 layers, 12 heads.
• First and last blocks hybridize 6 vanilla RoFormer heads with 6 Möbius heads.
• Identical input channel treatment as MöbiusBERT.
• Explored “H” (mixed heads) and “H+T” (mixed + linear-before-Möbius); parameter count $x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$7113–114M vs. 110M for RoFormer.

4. Algorithmic Workflow: Mixed-Head Block

The computation in a mixed-head MöbiusAttention block follows:

$x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$9

5. Empirical Comparisons

Empirical evaluation on the GLUE benchmark reveals that MöbiusAttention variants achieve competitive performance with fewer or similar parameters and with no increase in pre-training time. Major results include:

Model	Layers	Params (M)	GLUE Avg.
BERT-Base	12	110	83.64
RoFormer	10	110	83.49
MöbiusBERT	11	~104	up to 83.85
MobRoFormer	10	113–114	83.79

Both MöbiusBERT and MobRoFormer outperform their respective baselines on core tasks such as MNLI, QQP, QNLI, and RTE, despite equal or smaller parameter counts and identical pre-training duration (Halacheva et al., 2024).

6. Theoretical Properties, Benefits, and Limitations

MöbiusAttention introduces several theoretical and practical properties relative to standard self-attention:

• Benefits:
  • Inserts a powerful, non-linear geometry via Möbius transformation at the core of the attention operation, enabling the modeling of complex inter-token spatial relationships (including mappings that warp lines to circles).
  • Natively complex-valued representations encode cyclical and positional patterns relevant to many sequence modeling tasks.
  • Realizes empirical gains or maintains parity even with parameter reduction.
  • Modular design allows for selective mixing of Möbius and standard heads, balancing increase in expressivity against risk of overfitting.
• Drawbacks:
  • Each Möbius head introduces four complex scalars ($x\in \hat{\mathbb{C}}=\mathbb{C} \cup \{\infty\}$8 eight real parameters) per dimension, incurring higher parameter counts and compute in pure Möbius configurations.
  • Susceptible to overfitting if Möbius attention is deployed in all layers; the "framed" architecture mitigates this by limiting deployment to select blocks.
  • Operations in complex arithmetic increase runtime overhead unless efficiently implemented.
  • Current evaluation is restricted to GLUE; broader validation on large-scale NLP or vision tasks remains for future work.

A plausible implication is that MöbiusAttention provides an approach for embedding richer geometric priors in attention-based models, supporting configurations that may trade off expressivity, efficiency, and generalization capacity in novel ways (Halacheva et al., 2024).