Atomistic Language Models (ALMs) Overview

• Atomistic Language Models (ALMs) are multimodal models that integrate 3D atomic representations with natural language to directly generate and manipulate crystal and biomolecular structures.
• They employ continuous projectors to bridge pretrained atomistic encoders, causal LLMs, and denoising diffusion decoders, ensuring precise geometric and chemical fidelity.
• ALMs offer advanced structural editing and property prediction capabilities while highlighting challenges in numerical precision, tokenization strategies, and domain-specific training.

Searching arXiv for papers on Atomistic LLMs and closely related benchmarks/tokenization methods. Atomistic LLMs (ALMs) are language-model-based systems that operate directly on atomistic structure representations rather than solely on textual surrogates. In the materials setting, ALMs unify a pretrained atomistic encoder, a causal large-language backbone, and a denoising diffusion decoder through purely continuous projectors so that a single model can understand atomistic structures, generate materials from natural language, and optimize crystal structures as instructed by text (Edamadaka et al., 19 Jun 2026). Closely related work extends the same atom-level modeling program to biomolecules and molecular graphs, including atom-by-atom protein generation from SELFIES sequences and atom-level tokenization of local molecular environments for molecular LLMs (Flam-Shepherd et al., 2023, Zhang et al., 28 Nov 2025).

1. Conceptual scope and motivation

The immediate motivation for ALMs in materials science is the mismatch between the centrality of three-dimensional atomic structure and the limitations of conventional language-model pipelines. In the formulation introduced for crystalline materials, atomistic structure and natural language had long been modeled separately, with LLMs either calling atomistic models as tools or being fine-tuned on lossy textual encodings that discard atomistic information (Edamadaka et al., 19 Jun 2026). ALMs are designed to replace that separation with native multimodality.

This problem setting is sharpened by the representational demands of crystallography. Crystallographic Information Files (CIFs) are standard structure representations, but successful manipulation of CIFs requires more than syntax recovery: it requires geometric operations on lattices and atomic positions, consistency of chemical formulae, and tolerance-aware structural equivalence. AtomWorld formalizes this challenge as the “motor skills” problem for LLMs, defining benchmark tasks for reading, writing, and editing three-dimensional atomic structures encoded in CIFs (Lv et al., 6 Oct 2025). The benchmark’s framing suggests that ALMs are not merely larger text models for materials science; they are intended to couple textual competence with spatial and atomistic reasoning.

The term also has a broader domain scope. In biomolecular work, an ALM can denote a model that generates proteins atom by atom from a chemical language representation, including noncanonical residues and protein–drug conjugates, rather than restricting generation to amino-acid vocabularies (Flam-Shepherd et al., 2023). A plausible implication is that “ALM” is best understood as a family of multimodal or atom-level LLMs whose defining property is direct access to atomic structure, whether the substrate is a crystal, a small molecule, or a protein-like macromolecule.

2. Representations and cross-modal interfaces

The materials ALM architecture is defined by three components and two continuous interfaces. A frozen pretrained atomistic encoder $\mathcal{E}$ maps a crystal’s 3D coordinates, lattice, and elements into per-atom embeddings; a causal large-language backbone $\phi$ processes text and “soft-token” atoms; and a denoising diffusion decoder $\mathcal{D}$ reconstructs atomic coordinates and lattice from continuous latent steering signals (Edamadaka et al., 19 Jun 2026). The specific backbone reported is Qwen3-8B, and the diffusion decoder is MatterGen. The input-side projector $P_{\mathrm{in}}$ lifts encoder outputs into the LLM embedding space, while the output-side bridge uses a producer implemented as a Q-Former and a consumer implemented as IP-Adapter cross-attention injected into each GemNet-T block of the diffusion network. The architecture is explicitly described as using purely continuous projectors, with no quantization.

This continuous bridge is important because crystallographic manipulation is fundamentally geometric. For CIF-based tasks, fractional coordinates $(u_i,v_i,w_i)$ are converted to Cartesian positions by

$$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$

and interatomic distances are computed as Euclidean norms in Cartesian space (Lv et al., 6 Oct 2025). Structural editing therefore requires models to invert the fractional–Cartesian mapping when prompts specify Cartesian displacements or distances. AtomWorld’s evaluation further operationalizes structural fidelity through maximum displacement after Hungarian matching and RMSD, with max_dist reported as particularly sensitive because many editing tasks move only a single atom (Lv et al., 6 Oct 2025).

A distinct but related representational strategy appears in AtomDisc, which quantizes local atomic environments into structure-aware discrete tokens embedded directly in an LLM’s token space (Zhang et al., 28 Nov 2025). There, a molecule is represented as a graph, a pre-trained GIN encoder maps each atom and its two-hop neighborhood into a $300$-dimensional embedding, and a learned codebook of size $512$ assigns a symbolic token $\langle\!\text{atom\_}k\rangle$ by nearest-neighbor search. This contrast is structurally important: the materials ALM emphasizes continuous bridging from atomistic embeddings into diffusion steering, whereas AtomDisc emphasizes discrete atom-level vocabulary extension. This suggests two major design regimes for atomistic language modeling: continuous latent interfacing and discrete structure-aware tokenization.

3. Training procedures and controllable generation

The materials ALM is trained in three stages. Stage 1 performs soft-token alignment by training only $P_{\mathrm{in}}$ to describe structures in text under causal language-model cross-entropy. The reported data comprise approximately $\phi$0 million structure-to-description pairs from Robocrystallographer and GPT-Narratives, optimized with AdamW using learning rate $\phi$1 cosine decay, batch size $\phi$2, and $\phi$3k steps (Edamadaka et al., 19 Jun 2026). The result is that $\phi$4 learns to map per-atom embeddings into LLM token space without quantization.

Stage 2, termed ALM Core, adds multimodal instruction tuning. A LoRA adapter is attached to $\phi$5, the atomistic encoder remains frozen, and only LoRA plus $\phi$6 are trained on a five-bucket mixture: describe, property_apps, arXiv abstract generation, CAMEL Q&A, and MaScQA multiple choice (Edamadaka et al., 19 Jun 2026). Hyperparameters include LoRA rank $\phi$7, $\phi$8, learning rates $\phi$9 for LoRA and $\mathcal{D}$0 for $\mathcal{D}$1, batch size $\mathcal{D}$2, and $\mathcal{D}$3k steps. This stage is the point at which the model becomes a genuinely mixed-domain LLM rather than a narrowly aligned structural describer.

Stage 3 specializes the backbone for editing and generation. ALM Edit uses strong conditioning, full $\mathcal{D}$4 finetuning, and MatterGen CSP mode, while ALM Gen uses weak conditioning, LoRA rank $\mathcal{D}$5 on $\mathcal{D}$6, and MatterGen Base (Edamadaka et al., 19 Jun 2026). Both variants optimize a shared diffusion objective with classifier-free guidance dropout $\mathcal{D}$7, and both add auxiliary losses on the producer output: composition-count BCE, contrastive decorrelation, and directional cross-entropy on raise/lower prompts for ALM Edit. The reported diffusion hyperparameters include $\mathcal{D}$8, guidance scale $\mathcal{D}$9 for Edit and $P_{\mathrm{in}}$0 for Gen, producer token count $P_{\mathrm{in}}$1, context window $P_{\mathrm{in}}$2, and atom token count $P_{\mathrm{in}}$3.

Inference-time control is provided by Text-to-Crystal Feynman–Kac (T2C-FK), which replaces a single diffusion chain with an $P_{\mathrm{in}}$4-particle bootstrap SMC that reweights partial trajectories by a reward on the Tweedie $P_{\mathrm{in}}$5 estimate (Edamadaka et al., 19 Jun 2026). The sampler can enforce exact stoichiometry or custom property objectives. Its stoichiometry reward combines Hungarian negative log-likelihood, count $P_{\mathrm{in}}$6 gap, and ratio JS-divergence, and at $P_{\mathrm{in}}$7 atomic numbers are overridden by Hungarian assignment for exact element counts. In the limit $P_{\mathrm{in}}$8, the procedure implements a Feynman–Kac posterior of the form $P_{\mathrm{in}}$9. A plausible implication is that ALMs inherit controllability not only from prompt conditioning but also from explicit inference-time posterior shaping.

4. Benchmarks, metrics, and empirical behavior

ALM Bench is introduced as the first benchmark for text-conditioned crystal generation and optimization, with a seven-task suite containing $(u_i,v_i,w_i)$0 prompt–crystal pairs (Edamadaka et al., 19 Jun 2026). Its tasks include directional editing, crystal structure prediction, application consistency, polymorph generation, doping/substitution, strain, and text-to-structure recovery. Metrics include direction-correct rate using MatterSim-relaxed property changes, Match@1 and Match@K with RMSE@1/20 via pymatgen StructureMatcher, SUN and MSUN for stable/unique/novel yields, de novo metrics $(u_i,v_i,w_i)$1, $(u_i,v_i,w_i)$2, and $(u_i,v_i,w_i)$3, and judge scores for application consistency on a $(u_i,v_i,w_i)$4–$(u_i,v_i,w_i)$5 scale.

The reported performance highlights are substantial. ALM Core breaks the “GNN–LLM wall” on property prediction, with LLM4Mat-Bench MAD/MAE $(u_i,v_i,w_i)$6 on $(u_i,v_i,w_i)$7 slices and MAE $(u_i,v_i,w_i)$8–$(u_i,v_i,w_i)$9 better than prior text LLMs (Edamadaka et al., 19 Jun 2026). ALM Edit sets new state of the art on MP-20 and MPTS-52 crystal structure prediction, achieving best RMSE, including $$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$0 Å on MP-20, and Match@20 of $$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$1. ALM Edit also beats GPT-4o, 4.1, and 5.2 baselines on ALM Bench directional editing and application tasks. ALM Gen achieves top SUN yields on MP-20 at $$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$2 and competitive MSUN on LeMat-GenBench at $$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$3, improving upon unsteered MatterGen by more than $$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$4.

AtomWorld complements ALM Bench by exposing the failure modes of current frontier models on crystallographic operations (Lv et al., 6 Oct 2025). Across ten AtomWorld actions and five CIF-perception subtasks, easy actions such as change, remove, and add achieve approximately $$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$5–$$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$6 success with negligible max_dist, whereas move, move_towards, and insert_between fall to roughly $$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$7–$$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$8 success with mean max_dist of $$\mathbf{r}_i = u_i\,\mathbf{a} + v_i\,\mathbf{b} + w_i\,\mathbf{c},$$9–$300$0 Å. Hard actions including swap, delete_below, and rotate_around fall below $300$1 success, and rotate_around often exceeds $300$2 Å max_dist. CIF-Repair places closed models above $300$3 success and open-source models around $300$4–$300$5; CIF-Gen reaches approximately $300$6–$300$7 for standard prototypes such as NaCl and CaF$300$8, but drops to approximately $300$9 for non-standard stoichiometries in the same prototype. For the Chemical Competence Score, effect sizes are reported as approximately $512$0 for Qwen3-4B, $512$1 for Qwen3-32B, and $512$2 for Llama3-70B. StructProp remains difficult: Gemini 2.5 Pro reaches only approximately $512$3 success on $512$4 manually labeled cases, and most models remain below $512$5 (Lv et al., 6 Oct 2025).

The dominant failure modes are numeric imprecision in coordinate arithmetic, mis-indexing, syntax lapses, context-window limitations for long supercell expansions, and reliance on memorized CIF snippets rather than true geometric understanding (Lv et al., 6 Oct 2025). These observations delimit the present boundary of ALM capability: language-conditioned materials generation is advancing rapidly, but fine-grained crystallographic manipulation remains error-prone.

5. Molecular and biomolecular variants

In biomolecular modeling, the ALM of “Atom-by-atom protein generation and beyond with LLMs” treats every biomolecule as a single sequence of SELFIES tokens, with a vocabulary on the order of thirty tokens covering atoms, bonds, rings, branching, and four special tokens (Flam-Shepherd et al., 2023). SELFIES is used because it captures complete chemical graph information, including atom identity, bond order, ring closures, branching, aromatics, and stereochemistry; no additional learned graph embeddings are required. The model is a GPT-style decoder-only Transformer trained with standard autoregressive cross-entropy, typically using roughly four attention heads, embedding dimension approximately $512$6–$512$7, feed-forward inner dimension approximately $512$8, and on the order of $512$9–$\langle\!\text{atom\_}k\rangle$0 million total parameters. Sequences are up to $\langle\!\text{atom\_}k\rangle$1 tokens, and random SMILES/SELFIES atom-order permutations expand approximately $\langle\!\text{atom\_}k\rangle$2–$\langle\!\text{atom\_}k\rangle$3 raw structures into approximately $\langle\!\text{atom\_}k\rangle$4 training examples for each task.

That model demonstrates that atom-level sequence modeling can recover multiple hierarchical layers of protein organization from atom graphs alone (Flam-Shepherd et al., 2023). When $\langle\!\text{atom\_}k\rangle$5 novel proteins are sampled, approximately $\langle\!\text{atom\_}k\rangle$6 parse successfully into continuous N$\langle\!\text{atom\_}k\rangle$7C backbones with recognizable sidechains and unique primary sequences, AlphaFold2 predictions return per-residue pLDDT mostly in the $\langle\!\text{atom\_}k\rangle$8–$\langle\!\text{atom\_}k\rangle$9 range, and the generated samples exhibit helices, strands, and turns in proportions matching the PDB training set. The same framework is extended beyond canonical amino acids by training on augmented proteins with random small-fragment attachments at every sidechain, after which generated proteins match training distributions in molecular weight, LogP, topological polar surface area, and fragment counts, while also producing novel sidechain chemistries. In the antibody–drug conjugate setting, training on approximately $P_{\mathrm{in}}$0 combined protein–linker–warhead sequences yields about $P_{\mathrm{in}}$1 of samples passing backbone-plus-sidechain parsing, with pLDDT $P_{\mathrm{in}}$2, while generated warheads match LogP, QED, and SA distributions of ZINC.

AtomDisc occupies a neighboring but distinct position in the molecular ALM landscape (Zhang et al., 28 Nov 2025). Its core claim is that quantizing atom-level local environments into structure-aware tokens injects an interpretable inductive bias into molecular LLMs. The codebook has size $P_{\mathrm{in}}$3; embeddings are initialized by $P_{\mathrm{in}}$4-means on a random subset of $P_{\mathrm{in}}$5k atom embeddings; after $P_{\mathrm{in}}$6 epochs on $P_{\mathrm{in}}$7 million atoms from PubChem, nearly all $P_{\mathrm{in}}$8 codes are actively used. The tokens are projected into the $P_{\mathrm{in}}$9-dimensional embedding space of LLaMA-2-7B by a two-layer MLP plus RMSNorm, and training proceeds through codebook training, projector alignment, multi-task instruction pretraining, and supervised fine-tuning. Reported downstream results include an average ROC-AUC of $\phi$00 on seven MoleculeNet classification datasets, QM9 frontier-orbital MAE of $\phi$01 averaged over HOMO, LUMO, and gap, and substantial gains over UniMoT on forward prediction and retrosynthesis generation tasks. Because each atom token corresponds to a tight cluster of local environments, AtomDisc also exposes structure–property associations: tokens $\phi$02 and $\phi$03 distinguish different hydroxyl environments, and swapping them in model input shifts predicted aqueous solubility by approximately $\phi$04 pK units, with Wasserstein distance $\phi$05 and significance $\phi$06 (Zhang et al., 28 Nov 2025). This suggests that atom-level tokenization can serve not only as an input representation, but also as an attribution mechanism.

6. Limitations, ambiguities, and research directions

Current materials ALMs have explicit domain limits. Their encoder and decoder focus on inorganic, crystalline materials; amorphous, biomolecular, or defect-rich systems require new encoders and decoders (Edamadaka et al., 19 Jun 2026). Continuous projectors avoid codebooks, but may be less efficient at extreme scaling than VQ-VAE-like quantization for certain modalities. Strong scaling laws are anticipated, but balancing natural-language retention against strong cross-modal steering already requires careful LoRA-versus-full-finetune trade-offs. Inference-time stoichiometric and symmetry enforcement through T2C-FK adds approximately $\phi$07 compute.

AtomWorld proposes several concrete routes toward more robust ALMs (Lv et al., 6 Oct 2025). These include numeric-aware attention, multimodal chain-of-thought via Visualization-of-Thought with simple 3D sketches or coordinate tables inline, diffusion-based LLMs for valid-structure generation and repair, domain-focused pretraining with millions of CIF files, structure–property pairs, and geometric operation sequences, and task-specific fine-tuning on before-CIF/prompt/after-CIF triples with losses that penalize site mismatches. The same work also reports that a preliminary tool-augmented workflow using code-graph RAG and pymatgen APIs boosts remove success from $\phi$08 to $\phi$09 and insert_between from $\phi$10 to $\phi$11, while rotate_around remains difficult at approximately $\phi$12 success. This suggests that near-term progress may depend as much on symbolic geometry and code execution as on larger language backbones.

A persistent misconception is that atomistic language modeling can be reduced to CIF string completion or retrieval of memorized crystal templates. AtomWorld’s nonstandard-stoichiometry failures and rotation/editing errors argue against that reduction, indicating that syntactic fluency and genuine geometric competence are separable (Lv et al., 6 Oct 2025). Another source of confusion is terminological rather than technical: a separate line of work uses the language of “atoms” for internal sparse representation units in mechanistic interpretability, introducing the atomic inner product and reporting $\phi$13 sparse reconstruction across layers on average and more than $\phi$14 of atoms satisfying a uniqueness condition, compared with $\phi$15 for neurons and $\phi$16 for features (Hu et al., 25 Sep 2025). That usage is conceptually distinct from chemical or materials ALMs.

Taken together, the literature defines ALMs less as a single architecture than as a research program. In materials, the defining features are native multimodality, continuous or structure-aware interfaces to atomistic representations, and evaluation on text-conditioned generation and optimization (Edamadaka et al., 19 Jun 2026). In molecules and proteins, the same program appears as atom-level tokenization, atom-by-atom generation, and direct discovery of structure–property associations (Flam-Shepherd et al., 2023, Zhang et al., 28 Nov 2025). The central open question is therefore no longer whether LLMs can ingest atomistic information, but whether they can do so with the numerical precision, structural faithfulness, and controllability required for autonomous scientific workflows.