Geometric Algebra Attention Networks

• Geometric Algebra Attention Networks are neural architectures that encode tokens as multivectors and motors, ensuring rotation, translation, and permutation equivariance.
• They use algebraic primitives like the geometric product, wedge, and sandwich actions to model invariant features and complex spatial relationships.
• Empirical evaluations show these networks excel in tasks from crystal structure identification to protein generation and vision, offering robust and efficient performance.

Geometric Algebra Attention Networks are neural architectures in which attention, message passing, or both are organized around geometric or Clifford algebraic structure rather than flat vector similarities alone. Across the literature, this label covers rotation- and permutation-equivariant models for small point clouds built from geometric products and attention reductions over tuples (Spellings, 2021), E(3)-equivariant Transformers whose inputs, outputs, and hidden states live in the projective geometric algebra $G_{3,0,1}$ (Brehmer et al., 2023), vision backbones in which the Clifford geometric product $uv = u \cdot v + u \wedge v$ replaces the conventional attention-plus-FFN decomposition (Ji, 11 Jan 2026), and protein generators that extend invariant point attention with projective geometric algebra and higher order message passing (Wagner et al., 2024). A broader comparative literature places these systems alongside geometry-grounded attention mechanisms defined on matrix Lie groups, lattice symmetries, and Boolean manifolds, but the strict geometric-algebra line is characterized by multivectors, motors, sandwich actions, and algebraic bilinear operations as primary computational objects (Musialski, 18 Jun 2026, Atzeni et al., 2023, Shi et al., 11 Nov 2025).

1. Conceptual scope and representative systems

In the narrow sense, a Geometric Algebra Attention Network uses a Clifford or projective geometric algebra to represent tokens, hidden states, or geometric features, and then builds attention from equivariant linear maps, geometric products, wedge or join operations, and frame changes induced by sandwich products. In the wider usage that appears in adjacent papers, the phrase can also cover architectures whose goal is similar—namely, to make attention structurally geometric and symmetry-aware—even when the carrier space is not itself a Clifford algebra. This broader framing is explicit in comparisons between Clifford-based methods and Lie-algebra or lattice-symmetry attention constructions (Musialski, 18 Jun 2026, Atzeni et al., 2023).

System	Geometric carrier	Core interaction
GAAN (Spellings, 2021)	Multivectors in $\mathrm{Cl}(3)$ from tuples of relative position vectors	Geometric products plus attention reductions over tuples
GATr (Brehmer et al., 2023)	Multivectors in projective geometric algebra $G_{3,0,1}$	Multivector self-attention and geometric bilinear layers
CliffordNet (Ji, 11 Jan 2026)	Real-valued channels interpreted through Clifford interaction branches	Clifford geometric product with sparse rolling
CFA / GAFL (Wagner et al., 2024)	PGA residue-frame features and motors	IPA-style scores plus GeometricBilinear and higher order products
GAI-GS (Shen et al., 18 May 2026)	Space-time algebra features inside a GATr encoder	Multivector-valued interaction operators interpreted as sandwich products

These systems differ markedly in what they call a token. GAANs start from point tuples and derive multivector invariants from relative position vectors; GATr places the hidden state itself in projective geometric algebra; CliffordNet keeps ordinary channel tensors but interprets the interaction as an implicit scalar-plus-bivector geometric product; CFA augments scalar IPA features with PGA-valued messages over residue frames; GAI-GS uses a Geometric Algebra Transformer as a scene tokenizer inside a 3D Gaussian splatting pipeline (Spellings, 2021, Brehmer et al., 2023, Ji, 11 Jan 2026, Wagner et al., 2024, Shen et al., 18 May 2026).

2. Multivectors, motors, and algebraic primitives

The common algebraic substrate is the multivector. In the geometric algebra of $\mathbb{R}^3$, a multivector decomposes into scalar, vector, bivector, and trivector parts, giving an 8-dimensional real vector space. GAAN exploits exactly this decomposition: scalar and trivector components, together with vector and bivector norms, are used as rotation-invariant quantities extracted from products of input vectors. This provides a systematic route to distances, angles, areas, and volumes without hand-coding separate formulas (Spellings, 2021).

Projective geometric algebra $G_{3,0,1}$ extends this picture by introducing the null basis vector $e_0$. GATr uses $G_{3,0,1}$ as a 16-dimensional vector-space representation in which points, lines, planes, directions, translations, rotations, and more general Euclidean motions all live in the same algebra. In that setting, the sandwich action

$$\rho_u(x)= \begin{cases} u x u^{-1}, & u\ \text{even},\ u \hat{x} u^{-1}, & u\ \text{odd}, \end{cases}$$

realizes the $E(3)$ action on multivectors, while the geometric product, dual, and join supply the basic equivariant primitives. GATr emphasizes that this gives a faithful representation of the full Euclidean group rather than only an $uv = u \cdot v + u \wedge v$0 representation with translations handled externally (Brehmer et al., 2023).

CliffordNet takes a different route. It does not explicitly keep full multivectors as separate grade objects; instead, the geometric product is embedded as a scalar branch and a bivector branch,

$uv = u \cdot v + u \wedge v$1

where ordinary real-valued feature channels and local context vectors are used as proxies for the algebraic operands. The paper calls this algebraic completeness under the geometric product, because both the generalized inner product and the exterior wedge product are modeled explicitly rather than collapsing the interaction to a single scalar similarity (Ji, 11 Jan 2026).

CFA relies on the same projective algebra as GATr but foregrounds rigid motions as motors. In its PGA formulation, planes are grade-1 elements, lines are grade-2 elements, points are grade-3 elements, and Euclidean motions are represented by motors $uv = u \cdot v + u \wedge v$2 acting through

$uv = u \cdot v + u \wedge v$3

It further uses the join

$uv = u \cdot v + u \wedge v$4

and the infinity norm $uv = u \cdot v + u \wedge v$5 to expose point-line-plane incidence and metric relations inside the attention block (Wagner et al., 2024).

GAI-GS uses a related algebraic vocabulary in a space-time algebra setting, where attention is interpreted as a multivector-valued interaction operator acting by sandwich product,

$uv = u \cdot v + u \wedge v$6

In that formulation, reflections, diffractions, and transmissions are all treated as algebraic transformations of rays or scene features rather than as separate hand-engineered operators (Shen et al., 18 May 2026).

3. Attention constructions and message-passing patterns

The earliest explicit GA attention formulation in the dataset is the small-point-cloud GAAN. For a tuple $uv = u \cdot v + u \wedge v$7, it forms the geometric product

$uv = u \cdot v + u \wedge v$8

extracts rotation-invariant features

$uv = u \cdot v + u \wedge v$9

builds tuple features $\mathrm{Cl}(3)$0 by combining those invariants with tuple-associated values, and then computes attention weights

$\mathrm{Cl}(3)$1

followed by the aggregation

$\mathrm{Cl}(3)$2

A companion vector-output construction replaces invariant scalar outputs by scalar-weighted combinations of covariant vectors derived from the same products, which is how the architecture emits frame-consistent geometric outputs such as refined coordinates (Spellings, 2021).

GATr reinterprets the full Transformer stack in multivector form. Queries, keys, and values are themselves multivectors, equivariant linear maps have a constrained grade-aware form, and the attention rule is

$\mathrm{Cl}(3)$3

The inner product is the invariant $\mathrm{Cl}(3)$4 inner product, and the MLP block is replaced by geometric bilinear layers combining the geometric product and an equivariant join. GATr also adds distance-aware attention features so that Euclidean distances between point trivectors can enter the logits without sacrificing equivariance (Brehmer et al., 2023).

CliffordNet dispenses with explicit all-to-all attention matrices. Its dual-stream block constructs a detail stream and a context stream, then applies sparse rolling across channels to sample diagonals of the full channel interaction matrix. For each shift $\mathrm{Cl}(3)$5, it computes a scalar branch

$\mathrm{Cl}(3)$6

and a bivector branch

$\mathrm{Cl}(3)$7

concatenates them, projects them back to the base channel dimension, and injects the result through a Gated Geometric Residual. In the No-FFN variant, this geometric block is the entire interaction layer, which the paper presents as a local, linear-time, attention-like alternative to explicit self-attention (Ji, 11 Jan 2026).

CFA preserves IPA’s scalar attention scores but replaces the value path with PGA-valued message construction. Residue-frame motors $\mathrm{Cl}(3)$8 are used to transport multivector features between local frames, and pairwise messages are built as

$\mathrm{Cl}(3)$9

Because GeometricBilinear is bilinear, attention-weighted sums can be moved inside the operator, yielding efficient pairwise aggregation. CFA then adds a GeometricManyBodyProduct layer so that products of aggregated two-body messages generate implicit three-body structure, a design explicitly motivated as higher order message passing (Wagner et al., 2024).

GAI-GS places a GATr-based Multi-view Tokenizer inside a scene mapping network for wireless modeling. Its GA encoder yields per-token embeddings $G_{3,0,1}$0, $G_{3,0,1}$1, and a global CLS token, while attention is interpreted as

$G_{3,0,1}$2

with $G_{3,0,1}$3 functioning as a multivector-valued interaction operator. The resulting GA features condition attenuation, signal amplitude, and residual updates for Gaussian rotation, scaling, opacity, and spherical harmonics (Shen et al., 18 May 2026).

4. Symmetry structure and equivariance

A central reason for using geometric algebra in attention is that symmetry can be encoded structurally rather than only statistically. GAAN is explicitly designed to be translation-, rotation-, and permutation-equivariant or invariant by construction. Translation invariance is obtained by building geometric products only from relative coordinates; rotation invariance comes from using scalar and trivector components and vector and bivector norms as invariant features; permutation equivariance follows from shared tuple-local maps and attention reductions over tuples (Spellings, 2021).

GATr generalizes this program to full Euclidean equivariance. Because $G_{3,0,1}$4 carries a faithful representation of $G_{3,0,1}$5, points, directions, and rigid motions can all appear as hidden states while transforming correctly under rotations, translations, and reflections. The paper proves that its linear maps, geometric product, equivariant join, LayerNorm, and gated nonlinearity all commute with the $G_{3,0,1}$6 action, so equivariance is preserved layer by layer rather than retrofitted by output symmetrization (Brehmer et al., 2023).

CFA inherits SE(3)-equivariant reasoning from IPA but enlarges the internal geometric state. Attention scores remain invariant because they are built from scalar dot products, pair features, and Euclidean distances between query and key points expressed in the global frame. The PGA message path remains equivariant because neighbor features are transported into a canonical local frame by motor conjugation, combined there through geometric bilinears, and converted back into backbone updates by predicted motors (Wagner et al., 2024).

Not every architecture in the family pursues the same level of formal symmetry. CliffordNet is explicitly framed as a vision backbone for 2D images with strictly linear complexity $G_{3,0,1}$7, and the paper positions it as “geometry as computation” rather than as a strictly $G_{3,0,1}$8-equivariant Transformer. Its inductive bias comes from the coexistence of coherence and structure branches and from local Laplacian-style context, not from a theorem of Euclidean equivariance (Ji, 11 Jan 2026).

A recurring misconception is that any geometry-aware attention mechanism is automatically a geometric algebra attention network in the strict sense. The comparative discussion around Lie-Algebra Attention makes the distinction precise: GA attention “put[s] the algebra in the representation of tokens,” whereas Lie-Algebra Attention “put[s] the tokens directly on the group manifold, and read[s] off all geometry from there.” The same comparison notes that GA methods usually operate on multivectors in a flat ambient algebra, while Lie-Algebra Attention uses bare matrix Lie group elements and derives pairwise invariants from $G_{3,0,1}$9 and its logarithm (Musialski, 18 Jun 2026).

5. Empirical domains and reported results

The empirical record is heterogeneous because the architectures target different regimes. GAAN was evaluated on crystal structure identification, MD17 molecular force regression, and protein coarse-grain backmapping. On crystal structure identification it reported $\mathbb{R}^3$0 accuracy, compared with a spherical-harmonic baseline at $\mathbb{R}^3$1. On MD17 it significantly outperformed SchNet but trailed specialized equivariant architectures such as GemNet-Q and NequIP. In protein backmapping, the equivariant GAANs converged faster and reached lower training errors than naïve transformers trained with random rotation augmentation (Spellings, 2021).

GATr was evaluated on $\mathbb{R}^3$2-body dynamics, wall-shear-stress estimation on arterial meshes, and robotic motion planning. On randomly oriented arterial meshes it achieved $\mathbb{R}^3$3 mean approximation error, compared with approximately $\mathbb{R}^3$4 for GEM-CNN, about $\mathbb{R}^3$5 for a Transformer baseline, and about $\mathbb{R}^3$6 for PointNet++. In robotic planning it reached about $\mathbb{R}^3$7 normalized reward, versus about $\mathbb{R}^3$8 for Transformer-Diffuser and about $\mathbb{R}^3$9 for the reproduced Diffuser baseline. The scaling study further reported that GATr and a standard Transformer have essentially identical large-$G_{3,0,1}$0 scaling, while SE(3)-Transformer and SEGNN run out of memory at much smaller problem sizes (Brehmer et al., 2023).

CliffordNet targets 2D vision rather than point-cloud or frame-equivariant tasks, but it is one of the clearest demonstrations of a GA interaction replacing conventional attention-style mixing. Its Nano variant achieved $G_{3,0,1}$1 on CIFAR-100 with $G_{3,0,1}$2M parameters, matching ResNet-18 with $G_{3,0,1}$3 fewer parameters; CliffordNet-Fast reached $G_{3,0,1}$4 with $G_{3,0,1}$5M parameters and no FFN; CliffordNet-Base reached $G_{3,0,1}$6 with $G_{3,0,1}$7M parameters. Under the same recipe, a ViT-Tiny with $G_{3,0,1}$8M parameters and $G_{3,0,1}$9 achieved $e_0$0, which the paper uses to argue that a geometrically complete interaction can subsume much of the usual attention-plus-FFN workload (Ji, 11 Jan 2026).

CFA was tested inside GAFL for protein backbone generation. On PDB-scale generation, GAFL reported designability $e_0$1 versus $e_0$2 for FrameFlow, helix content $e_0$3 versus $e_0$4, and strand content $e_0$5 versus $e_0$6. On small proteins it reported designability around $e_0$7 and strand content close to the PDB reference, whereas several strong baselines under-produced strands. An ablation on SCOPe-128 showed that retrained FrameFlow reached about $e_0$8 designability, adding PGA features without higher order terms increased this to about $e_0$9, and the full CFA block reached about $G_{3,0,1}$0 (Wagner et al., 2024).

GAI-GS provides an application-specific but informative case study. On multiple real-world indoor wireless datasets, the paper reports Room 1 RSSI MAE at $G_{3,0,1}$1 GHz improving from $G_{3,0,1}$2 dB for WRF-GS to $G_{3,0,1}$3 dB for GAI-GS, and spatial-spectrum SSIM improving from $G_{3,0,1}$4 for WRF-GS and $G_{3,0,1}$5 for NeRF-APT to $G_{3,0,1}$6 for GAI-GS. The gains are attributed to GA-informed interaction modeling inside the scene tokenizer rather than to changes in the rendering equation alone (Shen et al., 18 May 2026).

6. Adjacent paradigms, misconceptions, and open directions

The immediate neighborhood of Geometric Algebra Attention Networks includes several architectures that are geometric and algebraic in spirit but differ in carrier space and symmetry mechanism. Lie-Algebra Attention is the clearest contrast: it defines a token as a bare matrix Lie group element, uses the invariant

$G_{3,0,1}$7

and scores pairs by the negative squared algebra norm. The paper explicitly contrasts this with GA attention by noting that GA tokens are usually multivectors in a Clifford algebra over flat $G_{3,0,1}$8, whereas Lie-Algebra Attention places tokens directly on a matrix Lie group manifold (Musialski, 18 Jun 2026).

LatFormer sits in another neighboring tradition. It constrains attention masks to realize group actions on hypercubic lattices, showing that for any transformation of the hypercubic lattice there exists a binary attention mask implementing that action. Attention weights are multiplied by soft masks generated by a convolutional Lattice Mask Expert, so the symmetry prior enters as a restriction on admissible attention patterns rather than through multivector-valued hidden states (Atzeni et al., 2023).

The gate-level Boolean evolutionary geometric attention network is a more radical discrete variant. It operates on a Boolean field over a 2D manifold, uses XNOR-based Boolean Query–Key similarity, Boolean RoPE, and reaction–diffusion logic kernels, and treats attention as a hard 0/1 gate on neighborhood diffusion. The paper is explicit that this is not geometric algebra, but it presents the model as a discrete prototype from which GA-based attention mechanisms could inherit ideas about metric-like similarity, geometric predicates, and local manifold-aware processing (Shi et al., 11 Nov 2025).

A different kind of adjacent work studies the geometry of attention function spaces rather than geometric data. “Geometry of Lightning Self-Attention” analyzes unnormalized self-attention as a polynomial map, characterizes generic fibers, computes neuromanifold dimensions, and describes singular and boundary points. It does not study Clifford-valued models, but it establishes that identifiability, fiber structure, and dimension can be addressed rigorously for attention parametrizations. A plausible implication is that comparable algebraic-geometric questions may become relevant for future geometric-algebra attention parameterizations as their symmetries and gauge freedoms become better formalized (Henry et al., 2024).

Several open directions recur across the literature. GAAN points to higher-order tuple products, graph-restricted tuple selection, and multivector-valued propagation as natural extensions (Spellings, 2021). GATr highlights higher-dimensional and alternative-signature algebras, conformal or Lorentzian variants, and universal approximation questions (Brehmer et al., 2023). CliffordNet leaves explicit global attention and larger-scale datasets such as ImageNet-1K and 21K for future work (Ji, 11 Jan 2026). GAI-GS identifies the absence of a direct standard-transformer ablation, signature choice, and hierarchical sparsity for larger scenes as open issues (Shen et al., 18 May 2026). CFA, finally, shows that PGA and higher-order products improve protein generation, but the persistent over-representation of helices in very long proteins indicates that richer geometric interaction alone does not eliminate all structural biases (Wagner et al., 2024).

Taken together, these works present Geometric Algebra Attention Networks not as a single architecture but as a design family. The recurrent pattern is stable: choose a geometric algebra that faithfully represents the objects and transformations of interest; encode tokens, states, or messages as multivectors or motors; use invariant scores and algebraically structured value transformations; and let equivariance emerge from the algebraic action rather than from ad hoc feature engineering. The literature differs on whether this program is implemented as explicit self-attention, attention-like local interaction, or hybrid message passing, but it converges on the view that geometric algebra can serve as a native computational language for attention in geometry-rich domains (Brehmer et al., 2023, Wagner et al., 2024).