Accelerated Deep Atomic Potential Transformer

• The paper introduces a graph-free Transformer model that directly computes all pairwise atomic interactions to predict forces on silicon defects more accurately than GNNs.
• ADAPT tokenizes atoms with raw Cartesian coordinates and elemental descriptors, employing full self-attention to bypass cutoff-restricted graph construction.
• ADAPT achieves significant efficiency gains, reducing training compute by over 200× while lowering force and energy errors compared to state-of-the-art GNN baselines.

Searching arXiv for ADAPT and closely related atomistic-transformer work to ground the article in current papers. {"query":"ADAPT Lightweight Long-Range Machine Learning Force Fields Without Graphs arXiv (Dramko et al., 28 Sep 2025)", "max_results": 5, "sort_by": "submittedDate"} Searching arXiv for related Deep Potential attention and transfer-learning papers. {"query":"DPA-2 a large atomic model as a multi-task learner arXiv (Zhang et al., 2023) DPA-1 (Zhang et al., 2022) DPA4 (Li et al., 1 Jun 2026)", "max_results": 10, "sort_by": "relevance"} Searching arXiv for adjacent work on test-time adaptation and long-range electrostatics in MLIPs. {"query":"Online Test-time Adaptation for Interatomic Potentials arXiv (Cui et al., 2024) opt-DDAP arXiv (H. et al., 13 Apr 2026)", "max_results": 10, "sort_by": "relevance"} Accelerated Deep Atomic Potential Transformer (ADAPT) is a graph-free machine-learning force field (MLFF) for atomistic modeling of crystalline defects, introduced for silicon point defects as a lightweight, long-range alternative to graph neural network (GNN)-based force fields. Its defining design choice is a direct coordinates-in-space formulation in which atoms are treated as tokens and all pairwise atomic interactions are represented by full, unmasked self-attention in a Transformer encoder rather than by cutoff-limited graph construction and message passing. In the reported silicon defect benchmark, ADAPT achieves lower force and energy prediction errors than a retrained state-of-the-art GNN baseline while using only a fraction of the training compute (Dramko et al., 28 Sep 2025).

1. Defect modeling context and design rationale

ADAPT is motivated by the computational cost of first-principles defect calculations, especially density functional theory (DFT), in workflows that require large periodic supercells and repeated force evaluations along structural-relaxation trajectories. In point-defect studies, the defect core is embedded in a bulk crystalline environment, and the relevant response can depend on atoms well beyond an immediate local neighborhood. The ADAPT paper frames this as a mismatch with standard GNN-based MLFFs, which typically impose a cutoff-defined local graph and propagate long-range information only through multiple message-passing layers (Dramko et al., 28 Sep 2025).

The critique is twofold. First, depth-based information propagation is described as susceptible to oversmoothing and oversquashing, so long-range information can degrade as network depth increases. Second, graph construction imposes a discretized topology on a continuous geometric problem. In defect systems, where distortions are localized yet subtle and supercells are large, these limitations are presented as especially consequential. ADAPT therefore abandons graph construction entirely: it does not build edges, does not impose a neighbor cutoff in its default form, and does not use message passing over a sparse graph. Instead, it applies dense attention so that atom $i$ can interact directly with atom $j$ at every layer, regardless of separation in the supercell (Dramko et al., 28 Sep 2025).

This positioning makes point defects a stringent test case. They combine large system size, localized but subtle structural distortions, and physically important nonlocal effects. The paper’s central methodological claim is that a graph-free Transformer can model these characteristics more directly than cutoff-restricted GNNs.

2. Tokenization, attention mechanism, and model structure

In ADAPT, a structure with $n$ atoms is represented as a length-$n$ token sequence. Each atom is encoded as a 12-dimensional token containing raw Cartesian coordinates $(x,y,z)$ together with elemental descriptors: group, period, electronegativity, covalent radius, number of valence electrons, first ionization energy, electron affinity, atomic radius, and molar volume. The authors state explicitly that these descriptors were used because they were naturally available in the raw data, and that the optimal descriptor set remains an open question (Dramko et al., 28 Sep 2025).

For force prediction, each token $\mathbf{x}\in\mathbb{R}^{12}$ is embedded independently by an MLP into a latent dimension $d_{\text{model}}$:

$$MLP(\mathbf{x}) = \mathbf{W}_k \sigma \Bigl(\mathbf{W}_{k-1}\sigma\bigl(\dots\sigma(\mathbf{W}_0 \mathbf{x} + \mathbf{b}_0)\dots\bigr) + \mathbf{b}_{k-1} \Bigr) + \mathbf{b}_k,$$

where $\sigma$ is ReLU. The embedded sequence is then processed by a stack of Transformer encoder blocks with residual attention, feed-forward layers, layer normalization, and dropout during training. Atom–atom interaction occurs only through attention. For head $i$,

$j$0

and

$j$1

All heads are concatenated and projected:

$j$2

Because $j$3, each layer explicitly forms pairwise atom–atom similarity scores for all pairs. This is the mathematical realization of the paper’s claim that ADAPT considers all pairwise atomic interactions (Dramko et al., 28 Sep 2025).

After the final encoder block, per-atom forces are produced by a linear projection,

$j$4

with $j$5, yielding an $j$6 tensor of force components. Forces are therefore predicted directly, not as gradients of a scalar energy model. This is a central departure from conservative MLIPs.

The energy predictor is a separate model. The paper reports evaluation of a Transformer-style decoder, a plain MLP, and an MLP with residual connections, with the MLP+residual architecture performing best. Because the scalar energy model requires fixed-size input, all structures are padded to 220 atoms, corresponding to the $j$7 Si supercell with allowance for dopants (Dramko et al., 28 Sep 2025).

Symmetry treatment is deliberately minimal relative to equivariant MLFFs such as MACE. ADAPT does not build in rotational or translational equivariance. The paper states that the model “lacks explicit geometric equivariances,” while claiming that permutation invariance is inherent to unmasked attention and that translational and rotational invariances can be learned sufficiently well from data. No explicit periodic-image embedding, minimum-image convention, or lattice-vector encoding is documented in the model equations (Dramko et al., 28 Sep 2025).

3. Loss design, optimization setup, and architectural ablations

The force model is trained with an importance-weighted mean-squared error intended to emphasize mechanically relevant atoms near defect centers. For atom $j$8, the importance mask is

$j$9

and the training loss is

$n$0

The paper argues that plain MSE can overreward trivial near-zero predictions on the largely unperturbed bulk atoms, whereas the weighted loss amplifies errors near the defect core. It also notes that this weighting can produce slightly worse unweighted $n$1 error than plain MSE while being more useful in practice for relaxation (Dramko et al., 28 Sep 2025).

Two auxiliary evaluation quantities are used for force vectors. The angular error is

$n$2

and the magnitude discrepancy is

$n$3

These metrics are intended to assess whether the model captures meaningful force directionality rather than collapsing toward near-zero vectors.

The no-graph design is supported by a locality ablation implemented by masking entries in $n$4 with $n$5 before the row-wise softmax. Reported total $n$6 losses are $n$7 at $n$8 allowed interactions, $n$9 at $n$0, $n$1 at $n$2, and $n$3 at $n$4. The paper uses this as direct evidence that full all-to-all interaction is important (Dramko et al., 28 Sep 2025).

For force prediction, the small ADAPT model uses

$n$5

and is trained for 80 epochs in single precision. The large model uses

$n$6

and is trained for 750 epochs. Energy-model candidates were trained for 200 epochs for selection, and the selected MLP+residual model was trained for an additional 200 epochs, for 400 epochs total. The optimizer type, learning rate, weight decay, batch size, scheduler, gradient clipping, and explicit values of $n$7 are not specified in the provided text and are therefore unavailable (Dramko et al., 28 Sep 2025).

An additional architectural observation is that ADAPT Small outperformed ADAPT Large, despite the latter being wider and trained much longer; the paper attributes this to overfitting in the larger model. For the energy predictor, the model-selection table reports $n$8 error $n$9 for the decoder, $(x,y,z)$0 for MLP only, and $(x,y,z)$1 for MLP + residual, indicating that residual connections are a major contributor to scalar energy prediction quality (Dramko et al., 28 Sep 2025).

4. Dataset construction, benchmark protocol, and quantitative performance

The study uses a dataset of silicon defect DFT trajectories drawn from earlier works and restricted to charge-neutral defects. The complex defects are described as being in substitutional-interstitial configuration, and the defect elements span 56 elements across most of the periodic table, excluding noble gases, rare-earth elements, and difficult-to-implant species. From the trajectories, the authors extract 252,240 single-point calculations. The underlying first-principles data were generated with VASP using the PAW method, spin-polarized PBE, a 520 eV plane-wave cutoff, $(x,y,z)$2-point sampling, and a 216-atom Si supercell; structures were optimized at fixed volume until ionic forces fell below $(x,y,z)$3 (Dramko et al., 28 Sep 2025).

For model comparison, ADAPT and a retrained MACE baseline are trained from scratch on 6,082 silicon defect DFT trajectories, while evaluation is performed on 100 held-out structures from trajectories excluded from training. The paper emphasizes that all test cases are complex defects. Reported main metrics are force MAE in $(x,y,z)$4 and energy MAE in eV, with comparisons focused on structures near the beginning of relaxation trajectories because later states have very small forces and can make trivial near-zero predictors appear artificially strong (Dramko et al., 28 Sep 2025).

On the 100-test-structure benchmark, the main reported values are:

• ADAPT Small: force MAE $(x,y,z)$5, energy MAE $(x,y,z)$6
• ADAPT Large: force MAE $(x,y,z)$7
• MACE Retrained: force MAE $(x,y,z)$8, energy MAE $(x,y,z)$9
• MatterSim 5M: force MAE $\mathbf{x}\in\mathbb{R}^{12}$0, energy MAE $\mathbf{x}\in\mathbb{R}^{12}$1

The abstract and text describe a roughly $\mathbf{x}\in\mathbb{R}^{12}$2 reduction in both force and energy errors relative to a state-of-the-art GNN-based model, but the tabulated values imply larger improvements for the specific benchmark table. The provided text notes this discrepancy and suggests that a different aggregation or metric was likely used in the prose. Against MatterSim 5M, the energy MAE corresponds to better than a 30% reduction, matching the text’s stated comparison (Dramko et al., 28 Sep 2025).

The computational-efficiency results are central to the model’s naming. Training the small ADAPT force model took approximately 2.24 minutes per epoch on a single NVIDIA A100 and converged after 80 epochs, for about 3 A100-hours. Retraining MACE took 8.5 minutes per epoch for 300 epochs on 16 NVIDIA A100s, for about 680 A100-hours. The separate ADAPT energy model took 1.93 A100-hours, so force plus energy together required 4.92 A100-hours total. In the paper’s accounting, the ADAPT force model required more than $\mathbf{x}\in\mathbb{R}^{12}$3 less compute than MACE retraining, and force plus energy together were still more than $\mathbf{x}\in\mathbb{R}^{12}$4 faster (Dramko et al., 28 Sep 2025).

The paper attributes this acceleration not to asymptotically better scaling—full attention still implies quadratic dependence on atom count through the $\mathbf{x}\in\mathbb{R}^{12}$5 attention matrix—but to the fact that ADAPT uses dense matrix multiplications that map efficiently to GPUs, whereas GNNs rely heavily on sparse operations and irregular neighbor aggregation.

5. Position within the atomistic-transformer literature

ADAPT belongs to a broader movement toward attention-based atomistic models, but its design is distinct from several adjacent lines of work. “DPA-1: Pretraining of Attention-based Deep Potential Model for Molecular Simulation” introduces local self-attention and learned type embeddings within a Deep Potential framework, while preserving conservative forces through energy differentiation and strict locality within a cutoff (Zhang et al., 2022). “DPA-2: a large atomic model as a multi-task learner” extends that lineage to a multi-task large atomic model with a shared descriptor backbone, task-specific fitting heads, conservative energy-derived forces and virials, and downstream distillation into faster student models (Zhang et al., 2023). Relative to those models, ADAPT is notable for discarding graph locality and local descriptors altogether in favor of dense all-pairs attention over atom tokens (Dramko et al., 28 Sep 2025).

A different graph-free transformer direction appears in “Crystal Transformer Based Universal Atomic Embedding for Accurate and Transferable Prediction of Materials Properties,” which uses a vanilla transformer encoder over atom features and coordinates to learn transferable atomic embeddings for scalar crystal-property prediction rather than simulation-grade interatomic potentials (Jin et al., 2024). This suggests a conceptual overlap at the level of atom-as-token modeling, but the scope is different: CrystalTransformer is centered on universal atomic embeddings and property transfer, whereas ADAPT is a force field for defect energetics and relaxations.

Two additional adjacent developments highlight possible extensions rather than direct architectural relatives. “Online Test-time Adaptation for Interatomic Potentials” proposes a model-agnostic online adaptation framework that updates an MLIP encoder once per test sample using self-supervised objectives and shows improved out-of-distribution accuracy and stable MD trajectories on periodic systems (Cui et al., 2024). “opt-DDAP: Optimisable density-derived atomic point charges via automatic differentiation” provides a differentiable route to optimized atom-centered charges for long-range electrostatics, explicitly positioning those charges as inputs to MLIPs that incorporate electrostatic terms (H. et al., 13 Apr 2026). A plausible implication is that ADAPT’s direct all-pairs formulation could be combined in future with deployment-time adaptation or explicit electrostatic charge models, but neither integration is demonstrated in the ADAPT paper.

At the high-performance equivariant end of the spectrum, “DPA4: Pushing the Accuracy-Cost Frontier of Interatomic Potentials with EMFA SO(2) Convolution” presents an SE(3)-equivariant interatomic-potential architecture with attention-based aggregation and a compiler-friendly conservative training path, emphasizing the accuracy-cost frontier rather than graph removal (Li et al., 1 Jun 2026). In that landscape, ADAPT occupies a different methodological niche: graph-free dense attention, direct force prediction, and defect-focused long-range modeling without explicit equivariance (Dramko et al., 28 Sep 2025).

6. Limitations, caveats, and plausible future directions

The ADAPT paper openly identifies several limitations. The model lacks explicit geometric equivariances, so translation and rotation invariance are not guaranteed analytically and are instead left to be learned from data. The authors therefore caution that ADAPT is likely more data-hungry than equivariant GNNs and may be less robust outside its training distribution. They also state that Transformers generally require substantial amounts of data, making ADAPT unsuitable for tasks with limited training data (Dramko et al., 28 Sep 2025).

A second limitation is the use of separate force and energy models. Because forces are predicted directly rather than obtained as $\mathbf{x}\in\mathbb{R}^{12}$6, the resulting MLFF is non-conservative. The paper treats this as a trade-off that may improve flexibility and efficiency, while also noting that energy–force consistency is important for molecular dynamics. A plausible implication is that the current formulation is better aligned with DFT pre-relaxation or static defect workflows than with long-horizon MD, whereas conservative models such as DPA-2 and DPA4 explicitly preserve the energy-gradient relation (Zhang et al., 2023, Li et al., 1 Jun 2026).

Further caveats concern scope and scaling. Applicability beyond silicon defects remains open; charged defects are left for future work; periodicity is handled only implicitly through supercell coordinates rather than through an explicit periodic architectural mechanism; and full all-pairs attention still scales quadratically with atom number, which may become restrictive for very large supercells (Dramko et al., 28 Sep 2025). The paper’s own phrasing therefore supports a bounded interpretation: ADAPT is a graph-free, accelerator-friendly MLFF designed especially for large periodic defect supercells with important long-range structural effects, not a fully general replacement for conservative, symmetry-aware atomistic potentials.

Potential future extensions are suggested indirectly by related work. opt-DDAP points to one route for adding electrostatically motivated charge information and explicit long-range electrostatic terms (H. et al., 13 Apr 2026). TAIP suggests that online adaptation could improve robustness under distribution shift during deployment (Cui et al., 2024). DPA-2 and DPA4 indicate that pretraining, distillation, and conservative equivariant backbones remain active alternatives for scaling atomistic models to broader chemical domains (Zhang et al., 2023, Li et al., 1 Jun 2026). Within that broader context, ADAPT’s main significance is methodological: it demonstrates that a no-graph Transformer with direct coordinates-in-space inputs and full self-attention can outperform a retrained state-of-the-art GNN baseline on silicon point defects while using dramatically less training compute (Dramko et al., 28 Sep 2025).