---
title: 'CrystalICL: Few-Shot ICL for Crystal Design'
url: https://www.emergentmind.com/topics/crystalicl
type: topic
---

# CrystalICL: Few-Shot ICL for Crystal Design

Searching arXiv for CrystalICL and closely related crystal-generation representation papers to ground the article in current literature.
CrystalICL is a crystal-generation model that explicitly targets few-shot in-context learning (ICL) for periodic materials design. It formulates crystal generation as either conditional generation, \(p_\theta(\cdot \mid s)\), from target conditions such as chemical formula, space group, formation energy, or band gap, or unconditional generation, \(p_\theta(\cdot)\), over crystal distributions. Its central claim is that prior LLM-based crystal generators were effectively restricted to zero-shot use, whereas practical materials design more closely resembles few-shot adaptation from relevant known structures. To address that gap, CrystalICL combines a space-group based crystal serialization, condition- and structure-aware prompt construction, and multi-task instruction tuning on top of Llama2-7b-chat [2508.20143].

## 1. Problem formulation and conceptual scope

CrystalICL represents a crystal unit cell with \(N\) atoms as
\[
\mathcal{M} = (\mathbf{A}, \mathbf{X}, \mathbf{L}),
\]
where \(\mathbf{A} \in \mathbb{R}^{N \times K}\) is one-hot atom-type encoding, \(\mathbf{X} \in \mathbb{R}^{N \times 3}\) are Cartesian coordinates, and \(\mathbf{L} \in \mathbb{R}^{3 \times 3}\) is the lattice matrix. Because periodic crystals are more naturally handled in fractional coordinates, the model adopts
\[
\mathcal{M}=(\mathbf{A},\mathbf{F},\mathbf{L}),
\]
with \(\mathbf{F}\in [0,1)^{N\times3}\). The infinite periodic crystal is written as
\[
\{(\boldsymbol{a}'_i, \boldsymbol{x}'_i) \mid \boldsymbol{a}'_i = \boldsymbol{a}_i, \boldsymbol{x}'_i = \boldsymbol{x}_i + \boldsymbol{k}\mathbf{L}, \forall \boldsymbol{k} \in \mathbb{Z}^{1\times3}\}.
\]
This framing keeps the problem grounded in explicit crystallographic structure rather than composition-only generation [2508.20143].

The model is designed for two regimes. In the conditional regime, given training pairs \(\{(\mathcal{M}_j, s_j)\}_{j=1}^n\), it learns \(p_\theta(\cdot \mid s)\). In the unconditional regime, given \(\{\mathcal{M}_j\}_{j=1}^n\), it learns \(p_\theta(\cdot)\). The intended novelty is not merely LLM-based generation, but train-time and test-time use of demonstrations so that generation quality improves when the prompt includes relevant examples. The paper motivates this by contrasting CrystalICL with CrystalLLM and by noting that, on P5, GPT-3.5 Turbo improves from 0-shot to 3-shot prompting while CrystalLLM degrades, suggesting that prior crystal-specific LLMs do not reliably realize few-shot ICL behavior [2508.20143].

Architecturally, CrystalICL is not a domain-specific backbone pretrained from scratch. It is an instruction-tuned generator built on **Llama2-7b-chat**, fine-tuned with **LoRA**, and specialized through a crystal-specific serialization and retrieval protocol. This places it in a distinct methodological niche relative to encoder-style crystal foundation models such as CLOUD, which rely on masked-language pretraining and downstream fine-tuning rather than prompt-time adaptation [2506.17345].

## 2. Space-group based crystal tokenization

A core contribution is **space-group based crystal tokenization (SGS)**, introduced to reduce the difficulty of modeling crystallographic symmetry in an autoregressive text model. The paper argues that CIF is overly formatted and token-heavy, while XYZ-style coordinate serialization forces the model to infer symmetry entirely from raw atomic coordinates. SGS instead externalizes symmetry through the space group and compresses atomic content through Wyckoff-position representatives [2508.20143].

The key observation is that, for a given space group, atoms of the same element occupying the same Wyckoff position can be represented by a single representative atom. Under SGS, the model therefore generates three components: the **space group symbol**, the **lattice parameters**, and the **element symbols with fractional coordinates for atoms at each Wyckoff position**. In practical serialized form, the output contains the space-group symbol, lattice lengths \(a,b,c\), lattice angles \(\alpha,\beta,\gamma\), and then the representative atomic entries. This reduces both sequence length and the burden of enforcing full symmetry over all atoms [2508.20143].

The representation is conceptually
\[
[\text{space-group symbol}] \;\; [a,b,c] \;\; [\alpha,\beta,\gamma] \;\; \{[\text{element}], [f_x,f_y,f_z]\}_{m=1}^M,
\]
where \(M\) is the number of occupied representative Wyckoff positions rather than the full unit-cell atom count. The paper does not provide a separate formal equation for the SGS string, nor an explicit symbolic mapping from representative coordinates to full Wyckoff orbits, but the intended effect is clear: symmetry constraints are shifted from latent coordinate regularities into an explicit textual scaffold [2508.20143].

This design aligns CrystalICL with a broader trend toward symmetry-aware crystal representations. WyCryst encodes crystals through space groups and Wyckoff occupancy in a VAE framework [2311.17916], and CLOUD uses a coordinate-free sequence representation that begins with symmetry generators and then occupied Wyckoff positions [2506.17345]. CrystalICL differs by using the symmetry-aware representation as a promptable autoregressive language interface rather than as an encoder input or latent variable.

## 3. In-context learning mechanism and instruction tuning

CrystalICL operationalizes few-shot learning through **condition-structure aware hybrid instruction tuning**. Its training set is the union
\[
S_D = S_{D_z} \cup S_{D_f},
\]
where \(S_{D_z}\) contains zero-shot instructions and \(S_{D_f}\) contains few-shot instructions with demonstrations. The model is trained with the autoregressive objective
\[
\mathcal{L}(\theta) = \sum_{(Q_i, R_i)\in S_D} - \log f_\theta (R_i|Q_i),
\]
where \(f_\theta\) is the pretrained LLM, \(Q_i\) is the instruction prompt, and \(R_i\) is the target response [2508.20143].

The zero-shot prompt template provides a condition description and asks for a crystal string. The few-shot template prepends \(K\) demonstrations, each consisting of a condition description and its corresponding crystal string, and then queries the model for a new crystal conditioned on the target description. The demonstrations are drawn from the training set rather than being arbitrary prompt examples, which makes prompt construction part of the method rather than an afterthought [2508.20143].

Demonstration selection follows three strategies. **Condition-based selection** retrieves examples matching the target condition, using anonymized formula patterns for composition, exact match for discrete properties such as space group, and nearest neighbors in absolute value for continuous properties such as band gap. **Structure-based selection** chooses an anchor crystal and retrieves the top \(K-1\) nearest crystals by Euclidean distance in CrystalNN fingerprint space. **Condition-structure based selection** first filters by condition, then retrieves structural neighbors. The method name “condition-structure aware” refers precisely to these retrieval policies [2508.20143].

CrystalICL adds a second training ingredient, **multi-task crystal instruction tuning with property prediction**. The auxiliary task masks a property value in a prompt that includes an SGS crystal string and asks the model to generate the missing value. The paper presents this through prompts of the form “The [Property] is [MASK]” followed by the crystal string. The purpose is to make the model explicitly learn structure–property relationships rather than only property-to-structure mappings. The paper does not give a separate weighted multi-task loss decomposition, but the property-prediction instructions are mixed into the same autoregressive instruction-tuning corpus [2508.20143].

The ablations strongly support this design. Removing the auxiliary task in the **noAux** variant substantially hurts zero-shot performance on MP20 with SGS: pretty formula drops to **0.4167** versus **0.7390** for full CrystalICL, and space group drops to **0.3740** versus **0.9904**. This indicates that the auxiliary property-prediction task is not incidental; it appears to be a major component of the model’s structure–property grounding [2508.20143].

## 4. Training setup, prompting protocol, and evaluation

CrystalICL fine-tunes **Llama2-7b-chat** using **LoRA** with rank **8**, alpha **32**, and dropout **\(5\times10^{-2}\)**. Optimization uses **cosine annealing** with initial learning rate **\(5\times10^{-4}\)**. Training batches and epochs are dataset-specific: **MP20** uses batch size 1 for 10 epochs, **C24** batch size 1 for 10 epochs, **MP30** batch size 1 for 3 epochs, and **P5** batch size 4 for 10 epochs. All training and inference are carried out on **a single Nvidia L40 48G GPU** [2508.20143].

At inference time, decoding uses **top-p = 0.9** and **temperature = 0.9**. For conditional generation, prompts are built from test-set conditions and training-set demonstrations. Inference batch sizes are **6** for cross-domain conditional generation, **8** for conditional generation, and **32** for unconditional generation. In unconditional generation, the model samples 10,000 structures, parses generated strings into CIFs, and rejects invalid parses before resampling [2508.20143].

The evaluation setup covers four conditional-generation benchmarks: **MP20**, **MP30**, **P5**, and **C24**. For unconditional generation, results are reported on **MP20**, **P5**, and **C24**. The paper states that standard splits from prior work are used. Cross-domain conditional generation is also evaluated by training on **MP20** and testing on **P5** and **C24**, which probes whether the prompt-conditioned behavior transfers beyond the training distribution [2508.20143].

Conditional generation is measured by task-specific **success rate**. Formula success requires that the generated atomic composition match the requested composition. Space-group success is evaluated using `SpacegroupAnalyzer` from pymatgen. Formation-energy success requires that the MEGNet-estimated generated formation energy have the **same sign** as the input condition. Band-gap success requires
\[
| \hat{g} - g | < 0.5 \text{ eV},
\]
where \(g\) is the target band gap and \(\hat g\) is the predicted band gap from MEGNet. Unconditional generation is evaluated by **compositional validity**, **structural validity**, **Recall**, **Precision**, and Wasserstein distances for density, formation energy per atom, and number of distinct element types: \(\text{wdist}(\rho)\), \(\text{wdist}(E)\), and \(\text{wdist}(N_{el})\) [2508.20143].

## 5. Empirical performance and ablation results

The central empirical claim is that CrystalICL improves over CrystalLLM in both 0-shot and 3-shot settings and, unlike CrystalLLM, benefits from few-shot prompting. The strongest reported configuration is usually **SGS + few-shot CrystalICL**. On **MP20** with **SGS** and **3-shot**, the success rates are **0.8868** for Pretty Formula, **0.9908** for Space Group, **0.9392** for Formation Energy, and **0.7453** for Band Gap. The corresponding CrystalLLM(SGS) results are **0.4513**, **0.8726**, **0.7984**, and **0.6373**. On **MP30** with the same configuration, CrystalICL reaches **0.9641**, **0.9956**, **0.9789**, and **0.7943**, versus **0.5008**, **0.9006**, **0.8030**, and **0.6687** for CrystalLLM(SGS) [2508.20143].

| Benchmark | CrystalICL (SGS, 3-shot) | CrystalLLM (SGS) |
|---|---:|---:|
| MP20 Pretty Formula | 0.8868 | 0.4513 |
| MP20 Space Group | 0.9908 | 0.8726 |
| MP20 Formation Energy | 0.9392 | 0.7984 |
| MP20 Band Gap | 0.7453 | 0.6373 |
| MP30 Pretty Formula | 0.9641 | 0.5008 |
| MP30 Space Group | 0.9956 | 0.9006 |
| MP30 Formation Energy | 0.9789 | 0.8030 |
| MP30 Band Gap | 0.7943 | 0.6687 |

These numbers indicate that the largest relative gains occur on formula-conditioned generation, while symmetry and property conditioning also improve materially. The paper further reports that, with SGS, CrystalICL exhibits **almost no performance degradation** in cross-domain **space-group conditioned generation** relative to in-domain evaluation, though the full table of cross-domain values is not reproduced in the provided material [2508.20143].

For unconditional generation, the main message is more modest but still substantive: CrystalICL improves substantially over CrystalLLM and is competitive with diffusion-based models on several distributional metrics. On **MP20** with SGS, CrystalICL reports **Valid 0.8555**, **Recall 0.9949**, **Precision 0.9926**, **wdist(\(\rho\)) 0.6039**, **wdist(\(E\)) 0.2568**, and **wdist(\(N_{el}\)) 0.1359**, improving over CrystalLLM(SGS), which reports **0.8144**, **0.9944**, **0.9847**, **0.8356**, **0.3544**, and **0.1743**. On **C24** with SGS, CrystalICL reports **Valid 0.9669**, **Recall 0.9921**, **Precision 0.8928**, **wdist(\(\rho\)) 0.0593**, and **wdist(\(E\)) 1.3061**, versus CrystalLLM(SGS) at **0.9574**, **0.9916**, **0.7965**, **0.0707**, and **3.0564** [2508.20143].

The appendix provides additional realism diagnostics for **MP20** under **SGS 3-shot CrystalICL**: **Atomic Overlap 0.0290**, **Symmetry Adherence 0.9666**, and **Energy-based feasibility 0.9151**. These results are important because the model’s headline success-rate metrics depend partly on surrogate predictors such as MEGNet, whereas overlap and symmetry adherence probe structural plausibility more directly [2508.20143].

The ablation studies clarify which components matter. Under inference-time selection, **condition-based prompt construction** is the strongest strategy. On MP20 with SGS and 3-shot, the best **C** configuration reports **0.9214** for Pretty Formula, **0.9948** for Space Group, **0.9685** for Formation Energy, and **0.7687** for Band Gap. The paper also notes that shuffling example order hurts performance, which suggests that demonstration organization is semantically active rather than inert prompt padding [2508.20143].

The shot-count study is notable because it weakens a simple scaling interpretation of ICL. For condition-based retrieval, **1Shot-C** yields **0.9376 / 0.9946 / 0.9731 / 0.8009**, **2Shot-C** yields **0.9410 / 0.9954 / 0.9683 / 0.7777**, and **3Shot-C** yields **0.9214 / 0.9948 / 0.9685 / 0.7687** across Pretty Formula, Space Group, Formation Energy, and Band Gap. The paper’s conclusion is that the **number of demonstrations has insignificant impact**; retrieval quality matters more than raw shot count [2508.20143].

## 6. Relation to adjacent crystal-generation paradigms, limitations, and significance

CrystalICL occupies a distinct place among recent crystal ML systems. It differs from sequence encoders such as CLOUD, which pretrain a BERT-style model on symmetry-consistent crystal strings and then rely on supervised fine-tuning rather than prompt-time few-shot adaptation [2506.17345]. It also differs from discrete-latent or VQ-based generators such as VQCrystal, where codebook indices behave like tokenized crystal motifs but are manipulated through latent-space search and relaxation rather than in-context demonstrations [2409.06191]. Relative to symmetry-aware generators such as WyCryst, which encode space groups and Wyckoff patterns explicitly in a VAE-style framework, CrystalICL replaces latent-variable decoding with instruction-tuned autoregressive generation and prompt-based adaptation [2311.17916].

This contrast makes CrystalICL significant less as a universal best generator than as a demonstration that LLM-style ICL can be made materially useful for crystal design. The paper’s evidence for genuine ICL is behavioral rather than mechanistic: improvement from 0-shot to 3-shot, sensitivity to retrieval strategy, degradation when demonstrations are shuffled or randomly selected, and stronger cross-domain space-group conditioning than the baseline [2508.20143].

The limitations are equally clear. Cross-domain performance still degrades when moving from MP20 to P5 or C24. Some datasets, particularly P5 and C24, contain many unrealistic or unstable materials, which restricts reliable property-based evaluation. Conditional evaluation on P5 and C24 is correspondingly limited to formula and space group. Generation validity is not perfect, since unconditional sampling requires parsing checks and resampling. SGS can also slightly reduce performance on chemical-formula conditioned tasks relative to XYZ in some settings. Finally, much of the physical assessment still depends on surrogate predictors such as MEGNet rather than direct DFT validation, which suggests that CrystalICL is presently strongest as a prompt-conditioned structure generator rather than as a fully validated inverse-design pipeline [2508.20143].

Taken together, these features define CrystalICL as a symmetry-aware, instruction-tuned LLM system that reorients crystal generation around promptable demonstrations. Its main technical contribution is not only that it generates crystals in text form, but that it makes **few-shot context** operationally useful through SGS serialization, retrieval-based demonstration construction, and auxiliary structure–property supervision. Within the recent crystal-ML literature, it therefore serves as a bridge between symmetry-aware crystal representation work and LLM-style prompt adaptation [2508.20143].

Source: https://www.emergentmind.com/topics/crystalicl