--- title: Lorentz-Equivariant Geometric Algebra Transformer url: https://www.emergentmind.com/topics/lorentz-equivariant-geometric-algebra-transformer-l-gatr type: topic --- # Lorentz-Equivariant Geometric Algebra Transformer The Lorentz-Equivariant Geometric Algebra Transformer (L-GATr) is a neural architecture that combines geometric algebra representations of four-dimensional Minkowski space with exactly Lorentz-equivariant operations, embedded within the scalable Transformer backbone. It is designed for learning tasks in high-energy physics, notably regression, classification, and generative modeling of particle collision data, where Lorentz symmetry is fundamental. L-GATr encodes each particle’s state as a multivector in the 16-dimensional Clifford algebra $\mathrm{Cl}(1,3)$, and defines all neural operations—including linear layers, attention, and normalization—so as to remain equivariant under $\mathrm{SO}^+(1,3)$ transformations, ensuring that predictions properly reflect the underlying physical symmetries of relativistic kinematics [2405.14806, 2411.00446, 2502.16630, 2508.14898]. ## 1. Geometric Algebra Foundations and Lorentz Equivariance L-GATr is built on the Clifford algebra $\mathrm{Cl}(1,3)$, where any element $x$ may be written as $x = \sum_{k=0}^4 \langle x \rangle_k$, with grades corresponding to scalars ($k=0$), four-vectors ($k=1$), bivectors ($k=2$), trivectors ($k=3$), and the pseudoscalar ($k=4$). The underlying metric is $\eta = \operatorname{diag}(+1,-1,-1,-1)$, defining Minkowski space-time. The geometric product, $uv = u \cdot v + u \wedge v$, supplies the basic operations: $u \cdot v$ as the Lorentz inner product and $u \wedge v$ as the exterior (bivector) part. Lorentz transformations $\Lambda \in \mathrm{SO}^+(1,3)$ act naturally on grade-1 vectors as $p^\mu \rightarrow \Lambda^\mu_\nu p^\nu$, extended to the full algebra as a homomorphism, so that $\Lambda \cdot (uv) = (\Lambda \cdot u)(\Lambda \cdot v)$. For neural network layers $f$, the strong equivariance constraint $f(\Lambda \cdot x) = \Lambda \cdot f(x)$ holds at every stage. This is enforced mathematically by only permitting “grade-preserving” linear maps: \[ \varphi(x) = \sum_{k=0}^4 v_k \langle x \rangle_k + \sum_{k=0}^4 w_k e_{0123} \langle x \rangle_k, \] where $v_k, w_k\in\mathbb{R}$ are learnable parameters and $e_{0123}$ denotes the unique grade-4 pseudoscalar [2405.14806, 2411.00446, 2502.16630]. Nonlinearities are restricted to functions that commute with Lorentz transformations, including the geometric product $GP(x, y) = xy$, scalar-gated GELU activations $\operatorname{GELU}(\langle x \rangle_0) x$, and grade-wise LayerNorm: \[ \operatorname{LayerNorm}(x) = x / \sqrt{ (1/n) \sum_{c=1}^n \sum_{k=0}^4 |\langle \langle x_c \rangle_k, \langle x_c \rangle_k \rangle| + \epsilon }. \] ## 2. Transformer Architecture and Multivector Attention L-GATr integrates geometric algebra with a multi-head, self-attention Transformer. Each token represents a particle and includes $n$ channels of $\mathrm{Cl}(1,3)$ multivectors (e.g., $n=32$), plus $m$ scalar channels (e.g., for type, time, or other metadata). Four-momenta are encoded into the grade-1 part of one multivector channel; categorical tags are placed in scalars. Attention is carried out by forming queries, keys, and values as multivectors. The attention weights use the Lorentz-invariant inner product, \[ \text{score}_{ij} = \langle q_i, k_j \rangle / \sqrt{16 n_c}, \] which ensures invariance under group action. The architecture supports stacking of $B$ blocks: 1. LayerNorm 2. AttentionBlock: $\operatorname{Linear}(\operatorname{Attention}(\operatorname{Linear}(\bar x), \ldots)) + x$ 3. MLPBlock: includes geometric product and scalar-gated GELU 4. Residual connection and equivariant linear mixing All linear transformations adhere to the grade-wise constraints, and attention scales as $\mathcal{O}(N^2)$ due to FlashAttention backend compatibility [2405.14806, 2411.00446]. ## 3. Lorentz Local Canonicalization and Generalization The Lorentz Local Canonicalization (LLoCa) framework extends L-GATr by equipping each token or node with a learnable local Lorentz frame $L_i \in \mathrm{SO}^+(1,3)$, predicted by a small network (“Frames-Net”). Feature tensors $T_i^{\mu_1\dots\mu_k}$ are canonically mapped to local frames for exact equivariance, and messages are transported as \[ m_{j,L_i}^{a_1\dots a_k} = [\rho_m(R_{ij}) m_{j,L_j}]^{a_1\dots a_k}, \quad R_{ij} = L_i L_j^{-1}, \] ensuring that both local computations and aggregate updates are Lorentz-invariant. This permits mixing arbitrary tensor ranks within attention and feed-forward layers, and enables "drop-in" equivariant upgrades for any Transformer or GNN backbone with minimal computational (<20%) and parameter (<1%) overhead. Mixed representations—splitting total feature dimension into scalar, vector, and tensor channels—enable optimal speed/performance tradeoff [2508.14898]. ## 4. Task-Specific Adaptations: Regression, Classification, and Generation L-GATr supports a unified pipeline for high-energy physics tasks: - **Regression** (e.g., matrix element surrogates for $q \bar q \to Z + n g$): Input tokens are one per incoming/outgoing particle, plus a global summary token. The model predicts standardized log-amplitudes, with the scalar grade of the global token as output, minimizing MSE loss. - **Classification** (e.g., top tagging): Events are tokenized as unordered point clouds, with type tags and optionally additional reference vectors to break to subgroups such as $\mathrm{SO}(3)$. The output head reads out a scalar, using sigmoid/BCE loss. - **Generative modeling**: L-GATr parameterizes continuous normalizing flows via Riemannian flow matching. Latents are mapped via ODE integration over physically valid manifolds (e.g., $(y_m, y_p, \eta, \phi)$ for each particle) and projected back using the network’s computed vector fields, respecting geometric constraints. Evaluations use negative log-likelihood and classifier two-sample tests, providing robust phase space coverage [2405.14806, 2411.00446, 2508.14898]. ## 5. Symmetry Breaking and Flexibility While L-GATr is maximally equivariant by construction, physical tasks at the LHC may require breaking to a subgroup (e.g., by supplying beam-direction or time-like reference vectors as special tokens or channels). This reduces symmetry equivariance to that appropriate for the problem’s residual symmetry. Empirically, providing reference objects as part of the architecture, rather than the data, offers superior performance and flexibility. Symmetry breaking, when performed at the input level (such as explicit beam or time features within LLoCa), allows the network to operate effectively in realistic detector environments [2411.00446, 2508.14898]. ## 6. Experimental Benchmarking and Performance Across regression, classification, and generative benchmarks, L-GATr consistently matches or outperforms strong baselines: - **Regression**: Achieves lowest MSE for high-multiplicity QFT surrogate tasks ($n=4,5$ gluons), with performance robust to dataset size [2405.14806, 2411.00446, 2508.14898]. - **Classification**: Matches or slightly trails top Lorentz-equivariant GNN architectures (LorentzNet, CGENN, PELICAN) in top tagging ($\mathrm{AUC}=0.9870$, background rejection $1/\varepsilon_B \approx 2200$ at $\varepsilon_S=0.3$), and achieves best performance (AUC $0.9885$) on 10-way JetClass tasks when pretrained [2411.00446, 2508.14898]. - **Generative Modeling**: L-GATr parameterized flows yield per-marginal distributions that best match ground truth, especially at distribution tails, with lowest negative log-likelihood and classifier AUCs near $0.5$ (chance-level distinguishability) [2405.14806, 2411.00446, 2508.14898]. - **Computational Scaling**: L-GATr incurs a $\sim$10$\times$ overhead versus vanilla Transformers for small particle multiplicities due to grade-preserving operations, but can handle ${O}(10^3)$ tokens, outscaling equivariant graph-based architectures [2405.14806]. | Task | L-GATr Result | Baseline Comparison | |---------------------|----------------------|----------------------| | QFT Regression | Lowest MSE for $n\geq 3$ | Beats CGENN/Transformer | | Top Tagging | AUC 0.9870, $1/\varepsilon_B\approx 2200$ | Matches SOTA equivariant GNNs | | JetClass Multiway | AUC 0.9885 | Surpasses ParticleNet, ParT | | Event Generation | NLL $\approx$ -32.8 | Superior to MLP, transformer | | Scaling | $N\gtrsim 1000$ tokens | Outscales CGENN, ParT | Ablation studies demonstrate that removing geometric-algebra channels or enforcing symmetry at insufficient granularity significantly degrades performance and data efficiency. ## 7. Extensions and Theoretical Insights Recent work integrates L-GATr with LLoCa, enabling per-particle frame prediction and exact equivariant transport at arbitrary tensorial rank, resulting in architectures that combine the flexibility and speed of standard transformers with guaranteed Lorentz symmetry. In this regime, L-GATr, when enhanced with local frames and explicit rotor transport, can achieve an order of magnitude speedup versus specialized architectures while retaining or surpassing SOTA accuracy, and exhibits optimal scaling with both sample size and multiplicity. A theoretical implication is that building equivariance directly into the architecture obviates the need for networks to "learn" physical symmetries, leading to better generalization and sample efficiency, particularly for high-dimensional, small-sample, or symmetry-bound problems [2508.14898]. ## References - "Lorentz-Equivariant Geometric Algebra Transformers for High-Energy Physics" [2405.14806] - "A Lorentz-Equivariant Transformer for All of the LHC" [2411.00446] - "Probing a Quarkophobic ${\mathbf{W}^\prime$ at the High-Luminosity LHC via Vector Boson Fusion and Lorentz-Equivariant Point Cloud Learning" [2502.16630] - "Lorentz-Equivariance without Limitations" [2508.14898] Source: https://www.emergentmind.com/topics/lorentz-equivariant-geometric-algebra-transformer-l-gatr