LieTransformer: Equivariant Self-Attention

• LieTransformer is an architecture for group equivariant self-attention that lifts features from homogeneous spaces to the Lie group domain for symmetry-aware processing.
• It employs LieSelfAttention layers that combine content-based and location-based interactions via relative group displacements to maintain structural equivariance.
• Empirical evaluations on tasks like shape counting, molecular regression, and Hamiltonian dynamics show improved accuracy, data efficiency, and robustness over non-equivariant models.

Searching arXiv for the target paper and closely related work to ground the article and citations. {"query":"arXiv LieTransformer Equivariant self-attention for Lie Groups (Hutchinson et al., 2020) related LieConv SE3-Transformer equivariant attention", "max_results": 10} LieTransformer is an architecture for group equivariant self-attention on arbitrary Lie groups and their discrete subgroups, introduced to extend symmetry-aware deep learning beyond equivariant convolutions to Transformer-style attention (Hutchinson et al., 2020). It operates on data defined on homogeneous spaces, lifts features to the group domain, applies LieSelfAttention layers that remain equivariant under the regular representation, and then uses group pooling to obtain invariant outputs. The model was presented as a general recipe for combining the inductive bias of symmetry with the non-linear modeling capacity of self-attention, and was evaluated on shape counting in point clouds, molecular property regression on QM9, and Hamiltonian particle dynamics (Hutchinson et al., 2020).

1. Origins and problem formulation

LieTransformer arises from the observation that group equivariant neural networks improve generalisation performance and data efficiency through principled parameter sharing, but that most of this literature had concentrated on equivariant convolutions. In the linear setting, equivariant maps on homogeneous spaces are characterized as convolutions, which made convolutional constructions the dominant paradigm. The LieTransformer work shifts the focus to self-attention, treating it not as an exception to symmetry-based design but as another operator that can be made equivariant.

The relevant symmetry class is broad. A Lie group $G$ is treated as a set of symmetry transformations, such as rotations $SO(2)$, translations $T(n)$, or permutations. A group action is written as $\rho(g)\,v$, and equivariance of a map $\Phi:V_1 \to V_2$ means

$$\Phi[\rho_1(g)f] = \rho_2(g)\Phi[f] \quad \forall g \in G.$$

Invariance is the special case in which the output action is trivial. Within this framework, LieTransformer targets data for which the correct inductive bias is not merely permutation invariance over sets, but equivariance or invariance under a geometric symmetry group.

A common misconception is to treat LieTransformer as a conventional Transformer with a geometric positional encoding. Its construction is more specific: the architecture is designed for features on homogeneous spaces with an explicit group action, and its equivariance claim is structural rather than empirical.

2. Lifting to the group and the regular representation

A central technical device in LieTransformer is lifting. Input data are represented as pairs $(x_i,\mathtt{f}_i)$, where $x_i$ are locations and $\mathtt{f}_i$ are features. Rather than applying attention directly on the original domain $\mathcal{X}$, the model maps features from $SO(2)$0 to the group $SO(2)$1 itself, so that the regular representation can be used as the organizing principle for equivariant layer design.

The lifting map is written as

$SO(2)$2

where $SO(2)$3 is the coset mapping a reference point $SO(2)$4 to $SO(2)$5, and $SO(2)$6 is the stabilizer subgroup of $SO(2)$7. This converts features on a homogeneous space into features on the group domain. In that lifted domain, standard equivariant constructions become available, including the group convolution

$SO(2)$8

The significance of lifting is architectural rather than cosmetic. It allows the attention mechanism to be defined in terms of relative group displacement $SO(2)$9, which is the key quantity needed for equivariance under left translation. This suggests that LieTransformer should be understood as a regular-representation Transformer rather than as a Euclidean Transformer with symmetry-aware features.

3. LieSelfAttention

The core layer is LieSelfAttention. For lifted features $T(n)$0, with finite $T(n)$1, the layer computes attention at each group element $T(n)$2 by combining content-based and location-based terms over all $T(n)$3 in the domain or a local neighborhood.

Its algorithmic structure has three components. First, it computes a content-based interaction $T(n)$4 between $T(n)$5 and $T(n)$6. Second, it computes a location-based interaction $T(n)$7, so the geometric dependence enters only through relative group displacement. Third, it combines these terms, optionally normalizes them, and aggregates value vectors: $T(n)$8 A multi-head version is defined in the standard manner by using per-head parameterization and combining the resulting heads.

The equivariance argument follows from the dependence on $T(n)$9. Under simultaneous left action by $\rho(g)\,v$0, relative displacement is unchanged: $\rho(g)\,v$1 With the corresponding transformation rule for the feature part, the full attention mechanism is equivariant with respect to the regular representation. The formal statement given is that for all $\rho(g)\,v$2,

$\rho(g)\,v$3

This construction generalizes the usual attention pattern in a precise way: the attention kernel depends jointly on feature content and relative group position, but only through quantities compatible with symmetry. In that sense, LieSelfAttention is a non-linear equivariant operator, not merely a convolutional surrogate.

4. Architecture, invariance, and approximation regime

The full LieTransformer architecture stacks LieSelfAttention layers together with equivariant layer normalization and pointwise MLPs. After the stack, a final $\rho(g)\,v$4-pooling stage averages over group elements to obtain group-invariant outputs. The resulting model is therefore equivariant throughout its internal processing and invariant at the readout stage.

The paper emphasizes that the construction applies to arbitrary Lie groups and their discrete subgroups acting on homogeneous spaces. This generality is important because much prior equivariant work had been tailored to specific groups or to convolutional operators. In LieTransformer, the basic recipe is modular: lift, apply equivariant self-attention blocks, then pool.

That generality has direct computational consequences. Lifting is memory intensive because each input point may be lifted to multiple group elements, increasing representation size multiplicatively with the number of samples over the stabilizer subgroup $\rho(g)\,v$5. Self-attention then scales quadratically in the number of group elements, although subsampling or local neighborhoods can reduce the cost. For groups with continuous stabilizer $\rho(g)\,v$6, Monte Carlo sampling is required; in that regime, equivariance holds in expectation rather than exactly, and increasing the number of samples reduces the error. The work also notes numerical and optimization difficulties associated with Monte Carlo estimation and certain group parameterizations.

These caveats clarify another common misunderstanding. LieTransformer is not presented as a universally cheaper substitute for equivariant convolutions. Its contribution is broader operator expressivity and strong inductive bias under symmetry, with computational overhead that can be substantial.

5. Empirical behavior across tasks

The experimental program was designed to demonstrate generality across distinct symmetry groups and data modalities (Hutchinson et al., 2020).

Task	Symmetry	Main finding
2D shape counting in point clouds	$\rho(g)\,v$7	Higher accuracy and maintained performance under transformations
QM9 molecular property prediction	$\rho(g)\,v$8	Competitive with specialized equivariant models; attention often outperforms LieConv for smaller data
Hamiltonian particle trajectories	$\rho(g)\,v$9	More data efficient than baselines; orders-of-magnitude MSE gains over non-equivariant models

In 2D shape counting, LieTransformer-$\Phi:V_1 \to V_2$0 and LieTransformer-$\Phi:V_1 \to V_2$1 were compared with SetTransformer. The LieTransformer variants achieved higher accuracy and maintained performance under $\Phi:V_1 \to V_2$2 transformations, whereas SetTransformer performance dropped under $\Phi:V_1 \to V_2$3 augmentation. The significance is that invariance was not being supplied only through data augmentation; it was built into the architecture.

On QM9 molecular property prediction, LieTransformer was compared with non-equivariant graph models, specialized equivariant models such as SchNet, Cormorant, DimeNet, and SE3-Transformer, and the lifting-based convolutional baseline LieConv. The reported outcome was that LieTransformer matched or outperformed general-purpose equivariant models and was competitive with specialized ones. The best results were obtained when the true $\Phi:V_1 \to V_2$4 symmetry was used rather than only $\Phi:V_1 \to V_2$5 translations plus augmentation. This suggests that the correctness of the symmetry group can matter at least as much as architectural scale.

For Hamiltonian dynamics, the model learned a Hamiltonian from rollouts of interacting particles with springs and was used for future trajectory prediction. Here LieTransformer was described as much more data efficient, superior or equal with fewer training examples, and better than LieConv especially in data-starved regimes and for longer rollouts. The paper reports orders-of-magnitude improvements in MSE over non-equivariant fully connected and graph-network baselines.

6. Theoretical legacy, related frameworks, and distinctions

Later work on equivariant non-linear maps on homogeneous spaces places LieTransformer in a broader mathematical framework. In particular, a universal non-linear equivariant integral-operator construction was shown to subsume $\Phi:V_1 \to V_2$6-CNNs, conventional and relative-position attention, and LieTransformers, with generalized steerability constraints characterizing admissible non-linear equivariant operators (Nyholm et al., 29 Apr 2025). Within that view, LieTransformer is not an isolated architecture but a concrete specialization of a general theory of equivariant non-linear maps.

This later perspective sharpens the original contribution. LieTransformer established that self-attention can be made equivariant for arbitrary Lie groups and discrete subgroups; the later framework shows how such layers fit into a universal operator family. A plausible implication is that LieTransformer’s main historical role is methodological: it demonstrated that attention belongs inside the symmetry-constrained design space, rather than outside it.

The name can also be confused with later “Lie” attention models that use a different ontology for tokens. For example, Lie-Algebra Attention places the token itself on a matrix Lie group and defines attention scores from the algebra norm of a relative pose, $\Phi:V_1 \to V_2$7, with no external feature payload (Musialski, 18 Jun 2026). LieTransformer differs fundamentally: it begins with features on a homogeneous space, lifts them to the group, and performs equivariant self-attention in the regular representation. The two approaches are related by their concern with symmetry, but they are not interchangeable formulations.

Taken together, these developments position LieTransformer as an early general architecture for equivariant attention on Lie groups: broad in scope, mathematically principled, empirically competitive across several scientific tasks, and limited primarily by the cost of lifting and attention over expanded group domains.