DOS Transformer for Adsorption (DOTA)

• DOTA is a deep learning model that combines LDOS feature engineering with a multi-head Transformer encoder, pre-trained on extensive PBE-level DFT data.
• It achieves chemical accuracy by resolving complex adsorption energy challenges like the CO/Pt(111) site-preference puzzle, outperforming traditional descriptors.
• The model enables rapid, high-throughput screening of catalysts through efficient transfer learning and minimal high-fidelity data requirements.

DOS Transformer for Adsorption (DOTA) is a deep learning model introduced by Zhang & Cao for predicting adsorption energies from local density of states (LDOS) information on metallic and intermetallic surfaces. It is designed for the setting in which experimental adsorption energies are scarce, density functional theory (DFT) calculations are computationally expensive for large-scale screening, and machine-learning surrogates must contend with functional dependency, limited high-fidelity labels, and interpretability. DOTA combines LDOS-based feature engineering with a multi-head Transformer encoder, is pretrained on PBE-level DFT data, and can be fine-tuned with minimal experimental or hybrid-functional data to predict adsorption energies with chemical accuracy; the reported study further states that it resolves the CO/Pt(111) site-preference puzzle and outperforms the d-band center and Fermi softness models (Zhang et al., 28 Jul 2025).

1. Scientific context and problem formulation

Adsorption energy is presented as a critical descriptor for high-throughput screening of heterogeneous catalysts and electrode materials. The motivating difficulty is twofold: precise experimental adsorption energies are scarce because the underlying experiments are complex, while direct DFT evaluation remains too costly for large-scale searches. The paper positions DOTA as a response to three longstanding constraints in adsorption modeling: functional dependency, limited labeled high-fidelity data, and lack of interpretability in black-box predictors.

Within this framing, DOTA is not merely a regression model over composition or geometry. It is built around the premise that LDOS contains the orbital-level information needed to encode donation and backbonding behavior across a broad range of adsorption systems. The stated scope of the pretrained model is broad: a single model is reported to cover 11 adsorbates on 1 982 distinct (111) metal/intermetallic surfaces spanning 37 elements. This suggests a deliberate attempt to make the representation transferable across both adsorbate identity and substrate chemistry, rather than tailoring a separate model for each chemisorption problem.

The model is also explicitly situated against traditional electronic descriptors. The paper states that DOTA outperforms the d-band center and Fermi softness models, and it highlights the “CO puzzle” as a benchmark case where conventional approximations and functional choices have difficulty simultaneously describing adsorbate frontier orbitals and metal electronic structure.

2. LDOS input representation and feature engineering

The input is described as LDOS, implemented through spin-polarized projected density of states (PDOS) computed by VASP using PAW/VASP with PBE or other functionals. For each bare surface atom, the PDOS is resolved on a uniform energy grid with $\Delta E = 0.01\ \mathrm{eV}$ centered on the Fermi level, and it is decomposed into angular-momentum channels $\ell \in \{s,p,d,f\}$ (Zhang et al., 28 Jul 2025).

The channel construction is explicit. For the surface, there are 32 PDOS channels per atom, described as eight $\ell,m$ combinations times two spins. For the adsorbate, there are 8 PDOS channels per atom, described as four $\ell$ channels times two spins. The summary gives a concrete example: for a bridge site on a (111) surface, 2 surface atoms contribute $2\times 32 = 64$ surface channels, and the adsorbate contributes 8 channels, for 72 channels in total.

The preprocessing pipeline consists of three stages. First, all DOS curves undergo energy-axis average pooling, with example kernel size 4–8, to reduce noise and normalize grid length. Second, each channel $w_i$ is linearly projected into a $d_{\text{model}}$-dimensional embedding through

$X_i = \phi(W^0 w_i + b^0),$

where $\phi$ is ReLU and $$W^0 \in \mathbb{R}^{d_{\text{model}}\times E_{\text{pool}}}$$ maps the pooled DOS into the model space. Third, the summary states that surface and adsorbate channel embeddings share $\ell \in \{s,p,d,f\}$0 and are concatenated only after the attention encoder.

This representation is physically informed in a narrow and specific sense: it encodes orbital-resolved spectral information directly, rather than relying on a low-dimensional handcrafted descriptor. A plausible implication is that the model is intended to preserve enough spectral structure to distinguish adsorption regimes that appear similar under scalar descriptors such as a single d-band statistic.

3. Transformer encoder and regression head

DOTA uses a stack of $\ell \in \{s,p,d,f\}$1 Encoder blocks. Each block contains four standard components in sequence: Multi-Head Self-Attention with $\ell \in \{s,p,d,f\}$2 heads, Add & LayerNorm, a position-wise feed-forward network (FFN), and a second Add & LayerNorm. The attention formulation is written for concatenated channel embeddings $\ell \in \{s,p,d,f\}$3, where $\ell \in \{s,p,d,f\}$4.

For each head $\ell \in \{s,p,d,f\}$5, the block computes

$\ell \in \{s,p,d,f\}$6

with $\ell \in \{s,p,d,f\}$7 and $\ell \in \{s,p,d,f\}$8. The scaled dot-product attention is then

$\ell \in \{s,p,d,f\}$9

The multi-head output is

$\ell,m$0

with $\ell,m$1.

The model does not treat the channels as a strict sequence. Instead of sinusoidal positional encodings, it uses a learned “channel-type” embedding $\ell,m$2. The summary notes that in a standard sequential case one could use

$\ell,m$3

but the operative choice in DOTA is the learned channel-type embedding.

The FFN is specified as

$\ell,m$4

with $\ell,m$5, $\ell,m$6, and $\ell,m$7. Residual connections and normalization are applied as

$\ell,m$8

$\ell,m$9

The output of the last encoder block, $\ell$0, is flattened and passed through two fully connected layers to yield a scalar adsorption energy $\ell$1. The architectural emphasis is on modeling inter-channel couplings among orbital-resolved DOS channels, which the paper associates with orbital interaction patterns.

4. Pretraining and fine-tuning protocol

The pretraining stage, denoted DOTA-PBE, uses 23 861 adsorption entries covering 11 adsorbates (H, C, N, O, S, CH, CH$\ell$2, CH$\ell$3, NH, OH, H$\ell$4O, SH) on 1 982 distinct (111) metal/intermetallic surfaces from 37 elements. The pretraining loss is the $\ell$5 loss,

$\ell$6

Optimization uses AdamW with initial learning rate $\ell$7, weight decay $\ell$8, batch size 256, and 150 epochs with a 50-epoch warmup; the learning-rate scheduler is exponential cosine annealing. The reported 5-fold cross-validation result is $\ell$9 and $2\times 32 = 64$0 (Zhang et al., 28 Jul 2025).

Fine-tuning is formulated as a few-shot high-fidelity adaptation procedure for chemically accurate CO energies. The augmentation data consist of 988 LDOS–$2\times 32 = 64$1 pairs from PBE, RPBE, and PBEsol on 174 surfaces. The high-fidelity component contains 4 experimental microcalorimetry points for Pd, Cu, Ir, and Ni, together with HSE06-computed gas-phase CO LDOS. The combined input used in fine-tuning is surface LDOS from PBE or another GGA together with adsorbate LDOS from HSE06.

The fine-tuning objective mixes squared and absolute error:

$2\times 32 = 64$2

that is,

$2\times 32 = 64$3

The reported fine-tuning regime uses learning rate $2\times 32 = 64$4, epochs $2\times 32 = 64$5, batch size 32, and minimal regularization. In the paper’s presentation, this stage is what enables transfer from a large PBE-level corpus to chemically accurate predictions under very limited higher-fidelity supervision.

5. Reported accuracy and comparison with traditional descriptors

The reported pretraining performance is $2\times 32 = 64$6 and $2\times 32 = 64$7. Adsorbate-wise, the summary lists $2\times 32 = 64$8 for H and $2\times 32 = 64$9 for NH. On an out-of-sample OH(111) test for Ag and Au, the DOTA error is reported as $w_i$0, whereas d-band model errors are reported as greater than $w_i$1 (Zhang et al., 28 Jul 2025).

The paper gives a direct comparison for OH atop Ag(111) and Au(111). For OH@Ag(111), DOTA-PBE predicts $w_i$2, PBE DFT gives $w_i$3, the d-band center model gives $w_i$4, and the Fermi softness model gives $w_i$5. For OH@Au(111), DOTA-PBE predicts $w_i$6, PBE DFT gives $w_i$7, the d-band center model gives $w_i$8, and the Fermi softness model gives $w_i$9. The Ag–Au adsorption-energy difference is reported as $d_{\text{model}}$0 for DOTA-PBE, $d_{\text{model}}$1 for PBE DFT, $d_{\text{model}}$2 for the d-band center model, and $d_{\text{model}}$3 for the Fermi softness model.

For fine-tuned CO prediction, the best input combination is stated to be HSE06 CO LDOS plus PBE surface LDOS. Under this setting, the model predicts CO-top adsorption on Pt(111) as $d_{\text{model}}$4 versus experiment $d_{\text{model}}$5, and on Rh(111) as $d_{\text{model}}$6 versus experiment $d_{\text{model}}$7; both reported errors are less than $d_{\text{model}}$8. The summary further states that the correct site preference, top versus fcc, is resolved.

A common oversimplification in adsorption modeling is to treat a single electronic descriptor as sufficient. The values reported here indicate that, at least for the cited OH and CO cases, the d-band center and Fermi softness models do not reproduce the same level of quantitative agreement as the LDOS-based Transformer.

6. Interpretability and the CO/Pt(111) puzzle

The model’s interpretability is discussed through integrated gradients and attention-map analysis. For attribution over energy bins $d_{\text{model}}$9, the integrated-gradient score is written as

$X_i = \phi(W^0 w_i + b^0),$0

According to the summary, this reveals which DOS regions dominate a prediction, including CO $X_i = \phi(W^0 w_i + b^0),$1, $X_i = \phi(W^0 w_i + b^0),$2 peaks, and the metal d-band near $X_i = \phi(W^0 w_i + b^0),$3. Attention-map analysis uses the head-wise attention weights $X_i = \phi(W^0 w_i + b^0),$4 to indicate inter-channel coupling, with the explicit example of CO $X_i = \phi(W^0 w_i + b^0),$5 metal d-state interactions (Zhang et al., 28 Jul 2025).

The “CO puzzle” is described as follows. PBE underestimates the CO HOMO–LUMO gap, which leads to overbinding via an artificially low $X_i = \phi(W^0 w_i + b^0),$6 level and consequently the wrong fcc site preference on Pt(111). HSE06 corrects the gap but mis-describes the metal bandwidth. DOTA addresses this by fusing PBE surface LDOS with HSE06 CO LDOS. In the paper’s account, this combination recovers both correct backbonding and donation in the attention layers, yielding chemically accurate energies and the correct top-site preference.

This discussion is significant because it frames interpretability in orbital terms rather than only feature importance in an abstract vector space. A plausible implication is that the model is meant to remain diagnostically useful even when trained as a high-capacity predictor, allowing one to connect individual predictions to familiar chemisorption concepts such as donation, backdonation, and d-state coupling.

7. Screening applications, transferability, and computational role

The application domain emphasized in the paper is high-throughput catalyst and electrode screening. The stated input requirement is only bare surface PDOS together with gas-phase PDOS, so the workflow does not require full adsorbate geometry relaxations. The paper reports a typical speed-up of $X_i = \phi(W^0 w_i + b^0),$7–$X_i = \phi(W^0 w_i + b^0),$8 relative to full adsorption DFT, with the PDOS of a clean slab taking on the order of minutes and Transformer inference taking on the order of milliseconds (Zhang et al., 28 Jul 2025).

Transferability is presented at two levels. First, the pretrained model spans 11 adsorbates and 37 elements in a single network. Second, fine-tuning with as few as 4 experimental points is reported to yield chemical accuracy across new surfaces. The intended computational role is therefore not to replace all first-principles calculations, but to enable rapid down-selection of promising candidates for subsequent full DFT or experimental follow-up.

In this formulation, DOTA occupies an intermediate position between descriptor-based theory and direct high-fidelity simulation. It uses a richer electronic-structure representation than scalar descriptors such as the d-band center, yet it remains substantially cheaper than repeated adsorption calculations. The reported combination of LDOS-based featurization, multi-head self-attention, integrated-gradient attribution, and few-shot fine-tuning is presented as the mechanism by which the model bridges computational and experimental adsorption energetics.