Lie-Algebra Attention Mechanism

• Lie-Algebra Attention is an attention mechanism where tokens are group elements from a matrix Lie group, using intrinsic Lie-algebra norms for scoring.
• It leverages the geometry of the Lie group via principal logarithm maps and a block-weighted Frobenius inner product, ensuring strict invariance and equivariance.
• Empirical results show that this method matches or outperforms learned MLP kernels across SE(2), SO(3), and Aff(2) tasks with 50–80× fewer parameters.

Lie-Algebra Attention is an attention mechanism in which each token is a bare element $g_i$ of a matrix Lie group $G \subset GL(m, \mathbb{R})$, with no auxiliary features or external representations attached. Scoring between tokens leverages the canonical geometry of $G$, specifically the unique Lie-algebra norm of the relative pose in the group, leading to an intrinsically equivariant and invariant framework for attention that requires neither learned proximity kernels nor representation-theoretic constructions. The approach generalizes attention to arbitrary matrix Lie groups—even non-compact or non-abelian groups such as the affine groups $Aff(n)$ with scale and shear—bypassing earlier restrictions found in irreducible representation (irrep) and surjective exponential map (surjective-exp) based architectures. This methodology is exemplified in "The Token Is a Group Element: On Lie-Algebra Attention over Matrix Lie Groups" (Musialski, 18 Jun 2026).

1. Formal Definitions and Core Construction

Let $G$ be a matrix Lie group (closed subgroup of $GL(m, \mathbb{R})$). Its Lie algebra $\mathfrak{g}$ is the tangent space at the identity, equipped with the commutator bracket. In Lie-Algebra Attention, the input tokens $\{g_i\}_{i=1}^N$ are group elements $g_i\in G$.

For two tokens $i$ and $G \subset GL(m, \mathbb{R})$0, form the group-relative pose as $G \subset GL(m, \mathbb{R})$1. On a principal logarithm chart $G \subset GL(m, \mathbb{R})$2, the logarithm map is well-defined and yields $G \subset GL(m, \mathbb{R})$3.

A block-weighted Frobenius inner product on $G \subset GL(m, \mathbb{R})$4 is introduced. Decompose an orthonormal basis of $G \subset GL(m, \mathbb{R})$5 into $G \subset GL(m, \mathbb{R})$6 irreducible blocks (for instance, by the action of $G \subset GL(m, \mathbb{R})$7). Set positive weights $G \subset GL(m, \mathbb{R})$8, and let $G \subset GL(m, \mathbb{R})$9 be the block-diagonal matrix applying each $G$0 on its block. The inner product and associated norm are

$G$1

for $G$2.

The attention score between tokens $G$3 and $G$4 is then defined as

$G$5

where $G$6 is a temperature parameter. This serves as a canonical negative squared-norm proximity kernel on $G$7.

2. Equivariance and Invariance Properties

The Lie-Algebra Attention mechanism is strictly invariant under the diagonal (left) group action $G$8 for any $G$9: the relative pose $Aff(n)$0 and thus $Aff(n)$1 are unchanged, making $Aff(n)$2 exactly invariant.

For equivariant token update, any transformation $Aff(n)$3, with $Aff(n)$4 independent of left-multiplication of the input, preserves $Aff(n)$5-equivariance. If $Aff(n)$6 with $Aff(n)$7 computed from invariant features, updates preserve equivariance and induce the correct cocycle law: the updated relative pose between $Aff(n)$8 and $Aff(n)$9 is $G$0, making group composition and chaining consistent by construction.

3. Relationship to Alternative Approaches

<table> <thead> <tr><th>Method</th><th>Token Type</th><th>Applicability to $G$1</th></tr> </thead> <tbody> <tr> <td>Irrep-based (e.g., tensor-field networks, SE(3)-Transformer)</td> <td>$G$2 with $G$3-representation $G$4</td> <td>Not applicable (fails for non-compact, non-unitary groups)</td> </tr> <tr> <td>Surjective-exp (LieConv-style)</td> <td>Lift $G$5 via log, convolve over $G$6</td> <td>Not applicable (exponential not surjective on $G$7)</td> </tr> <tr> <td>Frame or point-based (e.g., AlphaFold IPA)</td> <td>Auxiliary local frames + distance kernels</td> <td>Only translation + rotation; not scale/shear</td> </tr> <tr> <td>LieTransformer (Hutchinson et al., 2020)</td> <td>$G$8 with learned MLP kernel</td> <td>Not applicable (surjective-exp constraint fails)</td> </tr> <tr> <td>Lie-Algebra Attention</td> <td>Bare $G$9</td> <td>Applies to all $GL(m, \mathbb{R})$0 with principal log</td> </tr> </tbody> </table>

Lie-Algebra Attention eliminates the need for explicit vector representations, irreps, Clebsch-Gordan products, or learned kernels. The attention kernel is uniquely fixed by the Lie group's geometry and admits closed-form evaluation for all matrix Lie groups where a principal logarithm chart is defined, specifically including full affine groups $GL(m, \mathbb{R})$1 where previous approaches do not apply (Musialski, 18 Jun 2026).

4. Empirical Performance and Benchmarks

Lie-Algebra Attention was compared against:

• A learned MLP kernel ($GL(m, \mathbb{R})$2) on the same invariant $GL(m, \mathbb{R})$3
• Classical vector-token attention ($GL(m, \mathbb{R})$4) on absolute coordinates

Tasks: Sequence completion (one missing token in $GL(m, \mathbb{R})$5 sequences) over $GL(m, \mathbb{R})$6, $GL(m, \mathbb{R})$7, and $GL(m, \mathbb{R})$8. Key performance results:

• On $GL(m, \mathbb{R})$9: Lie-Algebra Attention ($\mathfrak{g}$0) used only 36 kernel parameters vs $\mathfrak{g}$1's 1,932 (%%%%60$GL(m, \mathbb{R})$661%%%% reduction). Test pose RMSE 0.003 ($\mathfrak{g}$4) vs 0.005 ($\mathfrak{g}$5). Invariance error $\mathfrak{g}$6 for $\mathfrak{g}$7, $\mathfrak{g}$8 for $\mathfrak{g}$9 (vector-token).
• On $\{g_i\}_{i=1}^N$0: $\{g_i\}_{i=1}^N$1 had 24 parameters, $\{g_i\}_{i=1}^N$2 1,932; pose errors $\{g_i\}_{i=1}^N$3 for both; invariance error $\{g_i\}_{i=1}^N$4 ($\{g_i\}_{i=1}^N$5), $\{g_i\}_{i=1}^N$6 ($\{g_i\}_{i=1}^N$7).
• On $\{g_i\}_{i=1}^N$8: $\{g_i\}_{i=1}^N$9 60 parameters vs $g_i\in G$0 3,084; pose error $g_i\in G$1 (G, C, indistinguishable); invariance error $g_i\in G$2 ($g_i\in G$3), $g_i\in G$4 ($g_i\in G$5).

Closed-form algebra-norm attention matches or outperforms the learned MLP kernel with 50–80$g_i\in G$6 fewer parameters, holds exact invariance, and operates in group regimes inaccessible to previous attention methods (Musialski, 18 Jun 2026).

5. Computational Complexity and Implementation

For each token pair, $g_i\in G$7 is computed using closed-form or series expansions (e.g., Rodrigues, Cayley–Hamilton) for block-triangular algebras such as $g_i\in G$8 or $g_i\in G$9. The per-pair computational cost is $i$0 but is negligible for the practical $i$1 encountered.

The block-weighted norm, if $i$2 is stored in block coordinates, is $i$3.

Softmax normalization proceeds as standard: $i$4. Space and time scaling is $i$5 in both storage and compute, as with standard attention. The only additional cost is $i$6 evaluations of the matrix logarithm and block-norm, which remains $i$7 overhead for $i$8 (Musialski, 18 Jun 2026).

6. Significance and Extensions

Lie-Algebra Attention demonstrates that the structure of matrix Lie groups admits a closed-form, unique proximity kernel for attention—circumventing kernel learning and representation-theoretical obstacles. Strict invariance and exact cocycle condition are satisfied to machine precision. Potential extensions include generalization to arbitrary principal logarithm charts and handling more complex group actions or associated homogeneous spaces. The methodology absorbs and supersedes previous approaches that used vector- or feature-tensor tokens and enables attention models for regimes (such as full affine symmetry) previously inaccessible, as well as sharp reductions in kernel parameter count without loss of empirical accuracy.

Equivariant self-attention architectures such as the LieTransformer (Hutchinson et al., 2020) use a learned kernel (often an MLP) on a group-invariant or logarithm-based coordinate, still requiring surjective exponential maps and hence are not applicable to all matrix Lie groups (specifically, they are excluded for $i$9). The approach in Lie-Algebra Attention achieves full generality for matrix Lie groups with a well-defined principal logarithm, positioning it as a canonical construction in the landscape of group-theoretic attention mechanisms.