EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers

Abstract: As $SE(3)$-equivariant graph neural networks mature as a core tool for 3D atomistic modeling, improving their efficiency, expressivity, and physical consistency has become a central challenge for large-scale applications. In this work, we introduce EquiformerV3, the third generation of the $SE(3)$-equivariant graph attention Transformer, designed to advance all three dimensions: efficiency, expressivity, and generality. Building on EquiformerV2, we have the following three key advances. First, we optimize the software implementation, achieving $1.75\times$ speedup. Second, we introduce simple and effective modifications to EquiformerV2, including equivariant merged layer normalization, improved feedforward network hyper-parameters, and attention with smooth radius cutoff. Third, we propose SwiGLU-$S^2$ activations to incorporate many-body interactions for better theoretical expressivity and to preserve strict equivariance while reducing the complexity of sampling $S^2$ grids. Together, SwiGLU-$S^2$ activations and smooth-cutoff attention enable accurate modeling of smoothly varying potential energy surfaces (PES), generalizing EquiformerV3 to tasks requiring energy-conserving simulations and higher-order derivatives of PES. With these improvements, EquiformerV3 trained with the auxiliary task of denoising non-equilibrium structures (DeNS) achieves state-of-the-art results on OC20, OMat24, and Matbench Discovery.

• The paper presents a refined SE(3)-equivariant Transformer that achieves a 1.75x training speedup and enhanced energy-conserving accuracy in atomistic simulations.
• It introduces innovative techniques such as merged layer normalization, scaled feedforward capacity, and smooth radius cutoff in attention to improve stability and expressivity.
• Empirical results on datasets like OC20, OMat24, and Matbench show significant reductions in error metrics and training time, underscoring its impact on quantum chemistry and material science.

EquiformerV3: Advancing Scalability, Expressivity, and Physical Consistency of SE(3)-Equivariant Graph Attention Transformers

Overview

EquiformerV3 (2604.09130) presents a systematic augmentation of the SE(3)-equivariant graph attention Transformer family for 3D atomistic modeling, specifically targeting improvements in computational efficiency, theoretical expressivity, and task generality. The model addresses challenges that emerge when deploying large-scale neural architectures for quantum-accurate predictions of properties such as potential energy surfaces (PES), forces, and higher-order derivatives. The motivation is grounded in the demands of current quantum chemistry and materials science, where data volume, simulation fidelity, and physical constraints—including strict equivariance and energy conservation—necessitate significant technical advancements. EquiformerV3 consolidates architectural and software-level enhancements, enforcing rigor in symmetry handling, nonlinearity design, and both network and code scalability.

Model Contributions and Technical Innovations

Efficient Implementation

A central software optimization involves extensive operation fusion and compilation support, particularly for eSCN-based convolutions. This reduces memory traffic and the redundancy inherent in applying permutation matrices for SO(2) operations, yielding a 1.75x speedup in training throughput with unaltered accuracy profiles. Tooling refinements include improved pre-computation of constants and explicit dynamic shape handling, allowing effective use of compilation features in contemporary deep learning frameworks such as PyTorch.

Architectural Refinements

EquiformerV3 introduces several notable architectural modifications:

• Equivariant Merged Layer Normalization: Instead of the legacy approach which normalizes irreps features independently by degree (thereby erasing relative magnitude information across degrees), a merged normalization computes a single root-mean-square statistic, preserving the relative scale and thus enhancing the stability and empirical performance in training.
• Scaled Feedforward Capacity: Leveraging the asymmetry in compute cost between edge-wise and node-wise operations, the hidden dimensionality of feedforward networks (FFNs) is increased by 4x, as inspired by recent scaling trends in Transformers, yielding marked gains with minimal added computational burden.
• Smooth Radius Cutoff in Attention: By integrating smoothly decaying envelope functions into the softmax attention rather than only the value transfer, EquiformerV3 guarantees continuity with respect to atomic positions and resolves edge-case instabilities where neighbor sets vary discontinuously—a critical property for learning energy-conserving PES and their derivatives.

Increased Theoretical Expressivity via SwiGLU-S2

SwiGLU-S2 is a newly proposed activation function that synergizes the S2 activation paradigm (projection to and from the sphere) with the bilinear interactions characteristic of self-tensor products, modulated by a SwiGLU gate. This design:

• Introduces many-body interactions into the representational pathway, elevating body-order sensitivity (as evidenced by the body-order counterexample benchmarks), enabling the network to encode and distinguish complex local geometries.
• Maintains strict SE(3) equivariance while drastically reducing the grid sampling cost—as nonlinearities are restricted to scalars and the rest handled through multiplicative interaction—mitigating sampling-induced equivariance errors even for high angular resolution (up to $L_\text{max}=6$).

Empirical Performance

EquiformerV3 delivers robust improvements across canonical datasets and benchmarks:

• OC20 (Open Catalyst 2020): Achieves up to $5.9\times$ speedup in training for the S2EF-2M split, with significant reductions in mean absolute error (MAE) for both energies and forces (e.g., a 1.58 meV/Å reduction in force MAE over previous versions).
• OMat24: Maintains or surpasses force MAE at a fraction of prior model sizes (5–23x reduction compared to SOTA) for crystal structure prediction, with comparable outcomes for stress and energy metrics under both direct and gradient-based predictions.
• Matbench Discovery: Establishes SOTA on the combined performance score (CPS), F1 for stability classification, and thermal conductivity proxy (KSRME), outperforming eSEN and UMA by considerable margins (19–31% boost in thermal conductivity accuracy and 22.6x reduction in training time vs. UMA-M-1.1).

Empirical ablations further confirm that each architectural or implementation enhancement contributes substantial accuracy and efficiency, with SwiGLU-S2 being a necessary condition for high body-order discrimination and equivariance-preserving efficiency.

Implications and Future Perspectives

Practical Impact

EquiformerV3 sets a new standard for physically-consistent modeling in atomistic ML, particularly where energy conservation, higher-order derivative prediction, and simulation stability are non-negotiable. It is thus directly applicable to accelerated materials discovery, simulation of large biomolecular systems, and real-world catalyst development pipelines.

Theoretical Relevance

The architectural advances, such as merged normalization and the SwiGLU-S2 activation, raise the bar for efficient equivariant neural operators, providing insights into the expressivity–efficiency frontier for geometric deep learning. The findings on many-body expressivity clarify theoretical limits and potentials of tensor product-based GNNs for physical modeling.

Limitations and Future Work

The authors note that, beyond $L_{\text{max}}=4$, further scaling does not yield improved results given the current training recipes—implying that future advances may arise from better data curation, more realistic and diverse structure sampling (e.g., via phonon or diatomic augmentations), or improved benchmarking. Additional efficiency may be realized through custom CUDA backends, borrowing from cuEquivariance or OpenEquivariance-style libraries, to unlock even higher-degree or larger-scale models.

Conclusion

EquiformerV3 (2604.09130) synthesizes hardware-aware implementation, symmetry-respecting architecture, and expressive nonlinearity to deliver an SE(3)-equivariant Transformer that outperforms prior art in efficiency, expressivity, and generalization across key atomistic learning benchmarks. The work provides a solid blueprint for the next generation of physically-grounded, scalable, and versatile graph neural models for quantum chemistry and materials simulation.