Equivariant Geometric Transformer

Updated 7 July 2026

Equivariant Geometric Transformers are architectures that integrate symmetry constraints, ensuring that transformations on the input yield predictable, equivariant outputs.
They achieve equivariance by representing geometric tokens in spaces with group actions, and by constructing attention and feed-forward layers that respect these symmetries.
Their applications span diverse domains such as 3D reconstruction, molecular modeling, point-cloud registration, and autonomous driving while addressing computational efficiency.

Equivariant Geometric Transformer (EGT) denotes a class of transformer-like geometric learning architectures in which the representation space, attention mechanism, and feed-forward maps are constrained by a symmetry group, so that transforming the input induces a prescribed transformation of the output rather than an arbitrary change. In recent work, the term is used broadly rather than as a single canonical design: it includes $SE(3)$ -equivariant transformers on ray space for neural rendering and $3D$ reconstruction, anchor-based equivariant transformers for low-overlap point-cloud registration, E(3)/SE(3)-equivariant molecular and protein models, geometric-algebra transformers for Euclidean and Lorentz symmetries, and $SE(2)$ -equivariant traffic models (Xu et al., 2022, Lin et al., 2024, Zhang et al., 21 Mar 2025, Spinner et al., 2024, Xu et al., 1 Apr 2026).

1. Conceptual scope

Across the literature, an EGT is defined less by a single fixed layer formula than by a recurring design principle: geometric tokens are represented in a space carrying a group action, and transformer operations are built so that the action commutes with the network. In some instances the tokens are rays or points; in others they are atoms, residues, multivectors, or anchor-indexed point features. What is shared is the requirement that geometry enters the architecture through symmetry-compatible constructions rather than through unconstrained coordinate concatenation (Xu et al., 2022, Kong et al., 2023, Haan et al., 2023).

This breadth is visible in the range of concrete instantiations. The “Equivariant Geometric Transformer” in ray space is defined between homogeneous spaces of $SE(3)$ , with rays treated as geometric tokens and attention built from equivariant kernels (Xu et al., 2022). SE3ET is presented as an Equivariant Geometric Transformer for low-overlap registration, where point features live on points $\times$ anchors and rotations act by anchor permutations (Lin et al., 2024). GET extends the idea to a “geometric graph of sets” for universal $3D$ molecular interaction learning, while L-GATr and DriveGATr use geometric algebra over spacetime and $2D$ projective geometry, respectively, to realize Lorentz- and $SE(2)$ -equivariant transformers (Kong et al., 2023, Brehmer et al., 2024, Xu et al., 1 Apr 2026).

Representation	Symmetry	Representative EGT
Ray space $\mathcal{R}\cong SE(3)/(SO(2)\times \mathbb{R})$	$SE(3)$	ray-space light-field transformer (Xu et al., 2022)
Points $3D$0 anchors	discretized $3D$1 via anchor permutations	SE3ET (Lin et al., 2024)
Irreps on residue graphs	$3D$2/$3D$3	E$3D$4former (Zhang et al., 21 Mar 2025)
Scalar/vector streams	$3D$5	MEET (Jiao et al., 23 Jun 2026)
Geometric algebra multivectors	$3D$6 or Lorentz	GATr/L-GATr (Haan et al., 2023, Spinner et al., 2024)
Projective geometric algebra tokens	$3D$7	DriveGATr (Xu et al., 1 Apr 2026)

A recurrent misconception is that “EGT” names one architecture in the same way that a specific model family does. The literature instead uses it as a structural category. This suggests that the unifying notion is not the particular parameterization of attention, but the combination of transformer-style global interaction with explicit geometric equivariance.

2. Symmetry groups and representation spaces

The mathematical core of an EGT is the choice of representation space on which the symmetry group acts. The ray-space construction makes this explicit by modeling oriented rays $3D$8 in Plücker coordinates and observing that $3D$9 acts transitively as

$SE(2)$ 0

so that ray space is the homogeneous space

$SE(2)$ 1

Fields on $SE(2)$ 2 are then sections of associated bundles, with group action

$SE(2)$ 3

where the twist $SE(2)$ 4 accounts for the stabilizer-induced gauge change (Xu et al., 2022).

Other EGTs realize equivariance through discrete or tensorial representations rather than homogeneous spaces. SE3ET uses E2PN features

$SE(2)$ 5

with $SE(2)$ 6 octahedral anchors, and equivariance takes the form

$SE(2)$ 7

so rotation acts by permutation of the anchor index while translation acts through spatial neighborhoods (Lin et al., 2024).

Irreps-based protein and molecular EGTs instead decompose features by degree. In E $SE(2)$ 8former, node states are organized into $SE(2)$ 9 irreducible representations of degree $SE(3)$ 0, with $SE(3)$ 1 scalars, $SE(3)$ 2 vectors, and $SE(3)$ 3 higher-order tensors; Equiformer-style tensor products then preserve equivariance by Clebsch–Gordan rules (Zhang et al., 21 Mar 2025). MEET adopts a reduced type-0/type-1 decomposition,

$SE(3)$ 4

with scalars invariant and vectors transforming as $SE(3)$ 5 under $SE(3)$ 6 (Jiao et al., 23 Jun 2026).

Geometric-algebra EGTs encode the group action directly inside the algebra. In the GATr blueprint, Euclidean, projective, and conformal geometric algebras define multivector feature spaces and equivariant linear maps tailored to $SE(3)$ 7 (Haan et al., 2023). L-GATr represents tokens as multivectors in $SE(3)$ 8, with Lorentz action

$SE(3)$ 9

and grade projections supplying invariant subspaces under the Lorentz group (Spinner et al., 2024). DriveGATr performs the same maneuver for $\times$ 0, where rigid $\times$ 1 motions are encoded as sandwich products $\times$ 2 on $\times$ 3 projective geometric algebra multivectors (Xu et al., 1 Apr 2026).

3. Attention and equivariant computation

In an EGT, attention is not merely augmented with geometric features; it is redefined so that logits are invariant and value transport is equivariant. The ray-space EGT makes this explicit. Queries are built by equivariant linear maps, keys and values by equivariant convolution kernels, and for each field type the inner product $\times$ 4 is invariant because the irreps are unitary. As a result, the softmax weights are invariant while the weighted sum of values is equivariant, yielding an $\times$ 5-equivariant transformer on $\times$ 6 and $\times$ 7 (Xu et al., 2022).

SE3ET uses a different but conceptually parallel construction. Its Equivariant Self-Attention is computed separately for each anchor index, so a global rotation simply permutes anchor channels and the entire computation is permuted consistently. Cross-cloud interaction is handled by three variants: ICA on invariant pooled features, ACA on joint point/anchor indices, and RCA on discretized relative rotations $\times$ 8. In all cases, equivariance is preserved because anchor permutations induced by rotation are respected by the attention structure itself (Lin et al., 2024).

Scalar/vector EGTs often recover geometric attention through invariant dot products and distances. MEET defines head-wise logits

$\times$ 9

where the dot product is built from scalar channels and flattened vector channels, and the distance term is implemented by query/key augmentation rather than a separate $3D$0 bias tensor. Because the logits depend only on scalar features, vector inner products, and pairwise distances, they are $3D$1-invariant; the value update remains equivariant because vector outputs are linear combinations of equivariant vectors with invariant coefficients (Jiao et al., 23 Jun 2026).

A closely related perspective appears in Platonic Transformers. There, features are lifted to functions on a finite group $3D$2 of reference frames, and every equivariant linear layer is a group convolution

$3D$3

Frame-dependent RoPE attention then becomes formally equivalent to a dynamic group convolution, so attention can be interpreted as learning adaptive geometric filters while retaining the standard transformer computation graph (Islam et al., 3 Oct 2025).

These variants illustrate a general principle: logits are typically built from invariant contractions, whereas the values retain the nontrivial representation. This suggests that “equivariant attention” in the EGT literature is usually a precise separation between invariant routing and equivariant content transport, rather than a direct analogue of unconstrained attention on raw coordinates.

4. Major application domains

EGTs have been specialized to a wide range of geometric tasks. In multi-view vision, the ray-space EGT supports two application pipelines. For equivariant $3D$4 reconstruction, multi-view radiance samples are mapped to an SDF $3D$5 through view-wise equivariant feature extraction, ray-to-point equivariant convolution, and ray-to-point cross-attention. For generalized neural rendering, ray-to-ray equivariant convolution produces features along target rays, ray-to-ray cross-attention aggregates information from neighboring source rays, and self-attention along the ray supports density prediction before volumetric rendering (Xu et al., 2022).

In point-cloud registration, SE3ET targets rigid registration of partial point clouds under arbitrary transformations and low overlap. The architecture uses an E2PN equivariant point-convolution backbone, equivariant self-attention on superpoints, and either invariant or equivariant cross-attention to produce robust coarse and fine correspondence features before RANSAC or local-to-global registration (Lin et al., 2024).

In biomolecular learning, GET frames an arbitrary $3D$6 complex as a geometric graph of sets, where each block contains a variable-size atom set $3D$7. Its bilevel attention simultaneously models sparse block-level interactions and dense atom-level cross-attention, while preserving E(3)-equivariance and retaining fine-grained information across proteins, small molecules, and RNA/DNAs (Kong et al., 2023). E$3D$8former targets noisy protein structures by combining Equiformer-style equivariant transformer layers with an energy-aware graph and an equivariant high-tensor-elastic selective SSM, and it is evaluated on inverse folding and binding site prediction with both experimental and AlphaFold-predicted structures (Zhang et al., 21 Mar 2025). MEET serves as an E(3)-equivariant backbone for atomistic peptide–protein complexes and is used inside a VAE encoder, sequence decoder, structure decoder, and block-latent diffusion denoiser for target-specific peptide design (Jiao et al., 23 Jun 2026).

In high-energy physics, L-GATr uses multivectors in $3D$9 and Lorentz-equivariant transformer blocks for amplitude regression, jet classification, and Lorentz-equivariant generative modeling via continuous normalizing flows and flow matching (Spinner et al., 2024). A subsequent large-scale study reports state-of-the-art performance for amplitude regression, jet classification, and the first Lorentz-equivariant generative network benchmarked across LHC tasks (Brehmer et al., 2024). In autonomous driving, DriveGATr encodes agents and map elements as multivectors in $2D$0 projective geometric algebra and combines agent–map attention, agent self-attention, and causal temporal attention in an $2D$1-equivariant transformer for behavior modeling on the Waymo Open Motion Dataset (Xu et al., 1 Apr 2026).

The breadth of these applications is not incidental. It reflects the fact that EGTs are most natural where the task is governed by known geometric symmetries but still benefits from transformer-style global context.

5. Efficiency, global context, and scaling

A central tension in the EGT literature is the cost of combining equivariance with global interaction. Several recent models explicitly target this bottleneck. MEET rewrites distance-aware attention so that geometric biases are encoded directly in augmented queries and keys; with FlashAttention-style kernels, peak activation memory becomes

$2D$2

linear in atom count, and the reported synthetic alanine-chain benchmark shows about $2D$3 MB for MEET versus more than $2D$4 MB for EPT at $2D$5 residues (Jiao et al., 23 Jun 2026).

Geometric Hyena replaces equivariant self-attention by equivariant long convolution. Its vector and geometric long convolutions preserve rotation and translation equivariance while reducing the global-context operator to sub-quadratic complexity; the paper reports that it processes the geometric context of $2D$6 tokens $2D$7 faster than the equivariant transformer and allows $2D$8 longer context within the same budget (Moskalev et al., 28 May 2025). Clebsch-Gordan Transformer reaches a similar objective from the irreps side: it replaces quadratic self-attention with Clebsch–Gordan convolution, achieving $2D$9 token complexity and $SE(2)$ 0 harmonic complexity while supporting high-order irreps and optional permutation equivariance (Howell et al., 28 Sep 2025).

Other models prioritize retaining standard transformer efficiency. Platonic Transformer imposes translation equivariance and discrete rotational equivariance through group-structured lifting and weight sharing, while preserving the exact architecture and computational cost of a standard Transformer (Islam et al., 3 Oct 2025). DriveGATr is motivated by the observation that explicit pairwise relative positional encodings add an additional quadratic cost in the number of agents; by encoding scene elements as projective geometric algebra multivectors and using standard attention between multivectors, it avoids that extra cost while maintaining $SE(2)$ 1-equivariance (Xu et al., 1 Apr 2026).

These developments show that “equivariant transformer” no longer implies a single computational profile. Some EGTs accept quadratic global attention; others recover global context through long convolutions, group convolutions, or efficient structured kernels. A plausible implication is that future EGT design will be driven as much by scaling laws and systems constraints as by the symmetry formalism itself.

6. Variants, limitations, and open directions

One recurrent misconception is that equivariance claims are uniform across the literature. In practice they differ substantially in scope. Some models are exact only for coordinate-frame changes on fixed continuous geometric signals: the ray-space formulation explicitly notes that equivariance is exact with respect to coordinate-frame changes on a fixed continuous light field, while sparse multi-view sampling and newly revealed surfaces break exact equivariance (Xu et al., 2022). Some models discretize the symmetry group: in SE3ET, equivariance is exact for the discretized octahedral group and approximate for arbitrary $SE(2)$ 2 rotations (Lin et al., 2024). Platonic Transformers are equivariant to finite subgroups of $SE(2)$ 3, not full continuous $SE(2)$ 4 (Islam et al., 3 Oct 2025).

Architectural limitations are equally model-specific. E $SE(2)$ 5former uses a Lennard–Jones potential with fixed parameters $SE(2)$ 6 and $SE(2)$ 7 for all residues and tasks, and its energy-aware graph still depends on an initial coarse radius $SE(2)$ 8 (Zhang et al., 21 Mar 2025). MEET keeps coordinates fixed in the backbone and uses only type-0 and type-1 features rather than higher-order tensor irreps (Jiao et al., 23 Jun 2026). DriveGATr is designed for self-driving scenes with permutation symmetry over tokens but still relies on a specific projective-geometric encoding of scene elements (Xu et al., 1 Apr 2026). Geometric-algebra EGTs emphasize expressive equivariant operations, but the algebra itself does not necessarily realize every representation family one might want for a task (Spinner et al., 2024).

Open directions recur across papers. The ray-space formulation points to extending equivariant attention and convolution on ray space to detection, segmentation, and pose estimation, and to other groups and homogeneous spaces (Xu et al., 2022). E $SE(2)$ 9former suggests learned or more physically grounded energy models, dynamic SSMs beyond layer depth, larger complexes, and multimodal combinations with protein LLMs (Zhang et al., 21 Mar 2025). MEET points toward combining more expressive $\mathcal{R}\cong SE(3)/(SO(2)\times \mathbb{R})$ 0 representations with memory-efficient attention, adding explicit local sparsity to reduce FLOPs, and integrating joint coordinate and feature updates while preserving linear activation memory (Jiao et al., 23 Jun 2026). CGT frames the remaining challenge from the irreps perspective: global equivariant attention at $\mathcal{R}\cong SE(3)/(SO(2)\times \mathbb{R})$ 1 is now available, but pushing harmonic complexity below $\mathcal{R}\cong SE(3)/(SO(2)\times \mathbb{R})$ 2 and preserving permutation symmetry in long-convolutional settings remain open problems (Howell et al., 28 Sep 2025).

Taken together, these results indicate that the term “Equivariant Geometric Transformer” now names a mature but internally diverse research area. Its defining idea is stable: encode geometry in a representation space on which the symmetry group acts, and make transformer operations commute with that action. Its concrete realizations, however, span homogeneous spaces, anchor permutations, irreps, geometric algebras, and long-convolutional hybrids, each with different trade-offs in exactness, expressivity, and scalability.