Text to Band Gap: Pre-trained Language Models as Encoders for Semiconductor Band Gap Prediction (2501.03456v2)

Abstract: We investigate the use of transformer-based LLMs, RoBERTa, T5, and LLaMA, for predicting the band gaps of semiconductor materials directly from textual representations that encode key material features such as chemical composition, crystal system, space group, number of atoms per unit cell, valence electron count, and other relevant electronic and structural properties. Quantum chemistry simulations such as DFT provide accurate predictions but are computationally intensive, limiting their feasibility for large-scale materials screening. Shallow ML models offer faster alternatives but typically require extensive data preprocessing to convert non-numerical material features into structured numerical inputs, often at the cost of losing critical descriptive information. In contrast, our approach leverages pretrained LLMs to process textual data directly, eliminating the need for manual feature engineering. We construct material descriptions in two formats: structured strings that combine key features in a consistent template, and natural language narratives generated using the ChatGPT API. For each model, we append a custom regression head and perform task-specific finetuning on a curated dataset of inorganic compounds. Our results show that finetuned LLMs, particularly the decoder-only LLaMA-3 architecture, can outperform conventional approaches in prediction accuracy and flexibility, achieving an MAE of 0.25 eV and R2 of 0.89, compared to the best shallow ML baseline, which achieved an MAE of 0.32 eV and R2 of 0.84. Notably, LLaMA-3 achieves competitive accuracy with minimal finetuning, suggesting its architecture enables more transferable representations for scientific tasks. This work demonstrates the effectiveness of finetuned LLMs for scientific property prediction and provides a scalable, language-native framework for materials informatics.