EquiformerV3: SE(3)-Equivariant Graph Transformer

• EquiformerV3 is a third-generation SE(3)-equivariant graph attention Transformer that leverages novel implementation optimizations, smooth-cutoff attention, and SwiGLU‑S² for efficient 3D atomistic modeling.
• Its architecture enforces strict rotational and translational symmetry using spherical harmonics and merged layer normalization, ensuring smooth potential energy surfaces and energy-conserving dynamics.
• Empirical results show reduced training time with significant improvements in energy and force prediction errors, highlighting enhanced many-body expressivity and simulation reliability.

Searching arXiv for EquiformerV3 and closely related prior work to ground the article with current citations. arxiv_search(query="EquiformerV3", max_results=5, sort_by="submittedDate") arxiv_search(query="EquiformerV2 eSCN eSEN UMA Joshi many-body expressivity atomistic", max_results=10, sort_by="relevance") I found several relevant arXiv entries. I’ll retrieve the most pertinent ones individually to cite accurately. arxiv_search(query="EquiformerV2", max_results=10, sort_by="relevance") arxiv_search(query="eSCN atomistic", max_results=10, sort_by="relevance") arxiv_search(query="eSEN atomistic smooth potential energy surfaces", max_results=10, sort_by="relevance") arxiv_search(query="UMA atomistic materials pretraining equivariant", max_results=10, sort_by="relevance") EquiformerV3 is the third generation of the Equiformer family, an $SE(3)$-equivariant graph attention Transformer for 3D atomistic modeling, introduced to improve efficiency, expressivity, and physical consistency in large-scale applications such as interatomic potential learning and atomistic property prediction (Liao et al., 10 Apr 2026). In the standard formulation used by the model, an atomic structure is represented as a graph with atoms as nodes, nearby pairs as edges, and predictions defined over quantities such as energy, forces, and stress. Its defining architectural commitment is strict treatment of geometric symmetry: translation invariance and equivariance are enforced through relative positions, while rotation equivariance is handled by representing hidden states in bases of irreducible representations of $SO(3)$. The resulting design is motivated not only by pointwise regression accuracy, but also by the requirement that a learned potential energy surface remain smooth enough for energy-conserving molecular dynamics, reliable force prediction as $-\nabla_R E$, and higher-order derivatives relevant to phonons and thermal conductivity (Liao et al., 10 Apr 2026).

1. Lineage and problem setting

EquiformerV3 is presented as a direct successor to EquiformerV2, retaining the overall equivariant graph-attention Transformer paradigm while revising the implementation and several core architectural components (Liao et al., 10 Apr 2026). The paper places it within a broader landscape of $SE(3)$- and $E(3)$-equivariant atomistic models that includes Tensor Field Networks, NequIP, MACE, SE(3)-Transformer, Equiformer (V1), eSCN, eSEN, and UMA (Liao et al., 10 Apr 2026). Relative to EquiformerV2, the stated goals are threefold: faster execution, stronger many-body expressivity, and smoother, more physically reliable potential energy surfaces.

The model inherits from EquiformerV2 the use of eSCN convolutions, whose central role is to reduce the cost of equivariant tensor-product computations and make higher angular degree $L_{\max}$ practical (Liao et al., 10 Apr 2026). EquiformerV2 had already incorporated attention re-normalization, separable $S^2$ activation, and separable layer normalization. EquiformerV3 preserves the high-capacity Transformer-style architecture but identifies several limitations in the earlier generation: implementation inefficiency, normalization that still partially obscured relative scale information across degrees, under-allocation in the node-wise feedforward network, lack of smooth attention cutoff for potential energy surface modeling, and numerical limitations in spherical nonlinearities that could compromise strict equivariance (Liao et al., 10 Apr 2026).

A recurring theme in the paper is that low force mean absolute error on isolated structures is not sufficient for downstream simulation quality. The stated concern is that a model may fit energy and force labels well while still behaving poorly when embedded inside geometry optimization, molecular dynamics, or higher-order response calculations if the learned potential energy surface is not smooth enough or if equivariance is only approximate (Liao et al., 10 Apr 2026). This distinction organizes much of the paper’s design rationale.

2. Equivariant representation and Transformer structure

EquiformerV3 operates on an atomistic graph whose node features encode species $z_i$ and whose edges encode relative displacements $r_{ij}$ within a cutoff radius (Liao et al., 10 Apr 2026). Node features are irreducible-representation-valued tensors indexed by angular degree $L$, order $SO(3)$0, and channel index $SO(3)$1, typically with a common channel count $SO(3)$2 across degrees. A degree-$SO(3)$3 component transforms under rotation by the Wigner matrix $SO(3)$4, so the hidden state is structured as $SO(3)$5 rather than as an unconstrained Euclidean feature vector (Liao et al., 10 Apr 2026).

Interaction between irreps is mediated by tensor products. The paper writes the degree-coupling primitive as

$SO(3)$6

with valid output degrees satisfying $SO(3)$7 (Liao et al., 10 Apr 2026). Relative positions are injected through spherical harmonics $SO(3)$8, coupling geometry into equivariant message passing.

At a block level, the network consists of embedding layers followed by stacked Transformer blocks, each containing equivariant merged layer normalization, equivariant graph attention, a second equivariant merged layer normalization, and a feedforward network using SwiGLU-$SO(3)$9 activation (Liao et al., 10 Apr 2026). The output head produces graph-level targets such as energy and stress, and node-level outputs such as forces or denoising noise when needed.

The graph-attention mechanism retains the standard message form

$-\nabla_R E$0

where $-\nabla_R E$1 is a scalar attention weight and $-\nabla_R E$2 is an equivariant value feature spanning degrees $-\nabla_R E$3 (Liao et al., 10 Apr 2026). The attention block concatenates node embeddings, rotates them with edge-dependent matrices $-\nabla_R E$4, applies an $-\nabla_R E$5 linear layer to obtain scalar attention features and irreps-valued value features, generates logits through layer normalization, leaky ReLU, and linear projection, applies SwiGLU-$-\nabla_R E$6 to value features, rotates back with $-\nabla_R E$7, and aggregates across heads. Because the weights are scalar and the non-scalar operations are equivariant, the architecture preserves equivariance by construction (Liao et al., 10 Apr 2026).

3. Architectural innovations in EquiformerV3

The first innovation is implementation-level. The paper reports that the original EquiformerV2 eSCN implementation performed redundant permutation operations after applying rotated irreps $-\nabla_R E$8. If $-\nabla_R E$9 denotes the required permutation, these permutations can be fused into the rotation itself by defining $SE(3)$0, so that the downstream $SE(3)$1 linear layer acts directly on $SE(3)$2 (Liao et al., 10 Apr 2026). Together with changes needed to enable torch.compile() with dynamic shapes—specifically, precomputing constant tensors and explicitly specifying scatter output shapes—this yielded a reported $SE(3)$3 training speedup on OC20, reducing training time from 270 GPU-hours to 154 GPU-hours while maintaining essentially identical accuracy (Liao et al., 10 Apr 2026).

The second innovation is equivariant merged layer normalization. Earlier normalization schemes normalized each degree independently, which the paper argues destroys relative degree magnitudes. EquiformerV2 partially addressed this with separable normalization between scalar and non-scalar channels; EquiformerV3 replaces that split with a single shared normalization scale across all degrees (Liao et al., 10 Apr 2026). For input irreps $SE(3)$4, the scalar mean is

$SE(3)$5

with scalar RMS

$SE(3)$6

and for $SE(3)$7,

$SE(3)$8

These are merged as

$SE(3)$9

after which all degrees are divided by the same $E(3)$0 (Liao et al., 10 Apr 2026). The paper reports improved energy and force MAE in OC20 ablations.

The third innovation is a reallocation of capacity toward the feedforward network. Because edge-wise tensor-product-like operations dominate compute while node-wise feedforward layers are comparatively cheap, the paper increases FFN hidden size by $E(3)$1 relative to the earlier setting (Liao et al., 10 Apr 2026). In the OC20 ablation, this changed parameter count from 54M to 66M, increased training time by only 8.6%, and improved accuracy.

The fourth innovation is smooth-cutoff attention. The paper argues that smoothing only the value/message term is insufficient in an attention architecture because the softmax denominator still changes discontinuously when neighbors cross the radius threshold (Liao et al., 10 Apr 2026). EquiformerV3 therefore incorporates the envelope directly into both numerator and denominator:

$E(3)$2

and uses

$E(3)$3

The paper describes this change as roughly neutral for direct single-point energy and force regression in OC20 ablations, but important for smoothly varying potential energy surfaces and energy conservation (Liao et al., 10 Apr 2026).

4. SwiGLU-$E(3)$4 and the expressivity–equivariance tradeoff

The paper’s central new nonlinearity is SwiGLU-$E(3)$5, introduced to increase many-body expressivity while preserving strict equivariance and reducing spherical sampling cost (Liao et al., 10 Apr 2026). Classical $E(3)$6 activation projects irreps to spherical grid values, applies a pointwise nonlinearity on sampled values, and projects back. For a single-channel irreps input $E(3)$7,

$E(3)$8

followed by a pointwise nonlinearity $E(3)$9 on the grid and a projection back to irreps through spherical integration (Liao et al., 10 Apr 2026). The paper identifies a limitation of this construction: generic pointwise nonlinearities on spherical grid signals can introduce higher-frequency content, making dense sampling necessary to keep equivariance errors small.

EquiformerV3 instead exploits the observation that tensor-product-like interactions can be implemented efficiently on $L_{\max}$0 by projecting two irreps features to the sphere, multiplying their grid values elementwise, and projecting back. The reported complexity is reduced from $L_{\max}$1 to $L_{\max}$2 because both projection operations and the spherical multiplication have fourth-order complexity (Liao et al., 10 Apr 2026).

The proposed activation is

$L_{\max}$3

where the gate is scalar-only and the non-scalar interaction is bilinear on the sphere (Liao et al., 10 Apr 2026). The stated reason this better preserves strict equivariance is that the nonlinear gating is confined to scalar channels, while multiplication of spherical grid features corresponds to a controlled tensor-product-like interaction rather than an unconstrained pointwise nonlinearity on non-scalar channels.

The paper connects this design to the many-body expressivity arguments of Joshi et al. (2023), stating that self tensor products increase body order and therefore discriminative power in geometric graphs (Liao et al., 10 Apr 2026). In the body-order experiments reproduced in the appendix, gate activation and standard $L_{\max}$4 activation remain at 50% on the tested 2-body, 3-body, and 4-body counterexamples, whereas SwiGLU-$L_{\max}$5 reaches 100% once enough FFNs are stacked (Liao et al., 10 Apr 2026). The paper’s interpretation is that stacked FFNs with SwiGLU-$L_{\max}$6 accumulate higher-order interactions through repeated products.

A practical benefit is reduced sampling cost. For $L_{\max}$7, the paper reports that EquiformerV2 with $L_{\max}$8 activation used $L_{\max}$9, or 324 grid points, whereas EquiformerV3 with SwiGLU-$S^2$0 uses $S^2$1, or 160 grid points, in attention, corresponding to a 50.6% reduction in attention spherical sampling complexity while maintaining strict equivariance (Liao et al., 10 Apr 2026).

5. Physical consistency, training methodology, and implementation regime

The paper’s physical-consistency claim rests chiefly on the combination of smooth-cutoff attention and SwiGLU-$S^2$2 (Liao et al., 10 Apr 2026). The argument is that a learned potential energy surface should be sufficiently smooth that forces, Hessians, and third derivatives are well behaved. Approximate equivariance, cutoff discontinuities, or aliasing artifacts in spherical nonlinearities may have limited effect on benchmark regression metrics while still producing unstable derivatives or non-conservative behavior in simulation. The paper explicitly links its design to prior work, particularly eSEN, that emphasized smooth radius cutoffs and energy conservation, but aims to recover those benefits without discarding attention or higher-degree expressivity (Liao et al., 10 Apr 2026).

Training combines direct prediction and gradient-based methods. On OMat24 and Matbench Discovery, the reported recipe is two-stage: direct pre-training followed by gradient fine-tuning (Liao et al., 10 Apr 2026). The losses are weighted sums over energy, force, stress, and an auxiliary denoising term, with coefficients denoted $S^2$3, $S^2$4, $S^2$5, and $S^2$6.

A major auxiliary task is DeNS, denoising non-equilibrium structures, adopted from prior work by Liao et al. (2024a) (Liao et al., 10 Apr 2026). The procedure corrupts randomly selected atoms with Gaussian noise of standard deviation $S^2$7 in the reported OMat24/MPtrj setups, uses corruption ratio $S^2$8, caps the fraction of corrupted atoms at 0.75, and on OMat24 applies denoising only to structures whose maximum per-atom force norm does not exceed $S^2$9 (Liao et al., 10 Apr 2026). The appendix reports $z_i$0 during direct pretraining and $z_i$1 during fine-tuning. The paper attributes part of its best benchmark performance to this off-equilibrium auxiliary supervision.

Optimization uses AdamW, a cosine learning-rate schedule with linear warmup, EMA with decay 0.999, gradient clipping, Gaussian radial basis functions, and dropout plus stochastic depth during direct pretraining, with dropout and stochastic depth disabled during gradient fine-tuning (Liao et al., 10 Apr 2026). For the OC20 S2EF-2M base setting, the appendix reports a cutoff radius of 12 Å, maximum neighbors 20, 128 radial bases, $z_i$2, $z_i$3, $z_i$4 Transformer blocks, attention heads $z_i$5, batch size 16, 30 epochs, maximum learning rate $z_i$6, weight decay $z_i$7, dropout 0.1, and stochastic depth 0.05 (Liao et al., 10 Apr 2026).

6. Empirical performance, interpretation, and limitations

On OC20 S2EF, the ablation study starts from an EquiformerV2 baseline with energy MAE 21.23 meV and force MAE 19.73 meV/Å at 54M parameters and 270 GPU-hours of training (Liao et al., 10 Apr 2026). Better implementation alone changes the metrics only marginally, to 21.20 and 19.73, while cutting training time to 154 GPU-hours. Adding equivariant merged layer normalization gives 20.79 and 19.28; improved FFN hyperparameters give 20.11 and 18.96 at 66M parameters; smooth attention cutoff gives 20.11 and 18.82; and adding SwiGLU-$z_i$8 gives 19.65 and 18.15 at 91M parameters and 171 GPU-hours (Liao et al., 10 Apr 2026). The paper emphasizes that the full set of improvements yields 41 meV lower energy MAE and 1.58 meV/Å lower force MAE than the V2 baseline, while still reducing total training time because of the software optimization.

On OMat24, the reported validation results distinguish direct prediction from gradient methods, with the paper cautioning that direct-versus-gradient approaches normalize labels differently, so absolute energy and stress comparisons should be interpreted accordingly (Liao et al., 10 Apr 2026). EquiformerV3 with $z_i$9 and gradient fine-tuning reaches force MAE 43.5 meV/Å at 30M parameters, which the paper describes as comparable to EquiformerV2 with five times more parameters and similar to UMA-L with 23 times fewer parameters. EquiformerV3 with $r_{ij}$0 and gradient fine-tuning further improves force MAE to 41.6 meV/Å, the best among the listed models (Liao et al., 10 Apr 2026).

On Matbench Discovery, the results are used as the strongest evidence for the paper’s physical-consistency claims because KSRME depends on higher-order derivatives of the potential energy surface (Liao et al., 10 Apr 2026). In the compliant MPtrj-only setting, EquiformerV3+ with gradient prediction achieves F1 0.863, RMSD 0.070 Å, KSRME 0.275, and CPS 0.830, improving substantially over EquiformerV2+ and reducing KSRME by 19% relative to eSEN-30M-MP+ (Liao et al., 10 Apr 2026). In the larger non-compliant setting, EquiformerV3+ reaches F1 0.931, RMSD 0.059, KSRME 0.118, and CPS 0.902 with 5.7k GPU-hours of training, and the paper identifies it as the first listed model to exceed CPS 0.9 (Liao et al., 10 Apr 2026). It also reports improvements over UMA-M-1.1 with a 22.6$r_{ij}$1 training-time saving, and over PET-OAM-XL with 3.6$r_{ij}$2 lower training time while maintaining higher F1 and comparable KSRME (Liao et al., 10 Apr 2026).

The paper’s own interpretation of these results is tightly partitioned by component: implementation optimization supplies the speedup; merged layer normalization improves optimization and MAE; larger FFNs improve the capacity-to-cost tradeoff; smooth attention cutoff matters primarily for smooth potential energy surfaces and derivative-sensitive tasks; SwiGLU-$r_{ij}$3 improves MAE, many-body expressivity, and equivariance-versus-grid efficiency; DeNS supports the strongest OMat24 and Matbench Discovery pipelines; and gradient fine-tuning is important for the best derivative-sensitive performance (Liao et al., 10 Apr 2026). The explicitly stated limitations are that the architecture remains implementation-heavy, that best performance depends on a multi-stage training recipe, and that larger models brought almost no gain in one Matbench Discovery setting, suggesting that the bottleneck may now lie more in data and training recipe than in architectural capacity. The paper identifies custom CUDA kernels, filtering rattled structures, and adding phonon or diatomic data as concrete future directions (Liao et al., 10 Apr 2026).