SemiKong: Curating, Training, and Evaluating A Semiconductor Industry-Specific Large Language Model (2411.13802v2)

Abstract: LLMs have demonstrated the potential to address some issues within the semiconductor industry. However, they are often general-purpose models that lack the specialized knowledge needed to tackle the unique challenges of this sector, such as the intricate physics and chemistry of semiconductor devices and processes. SemiKong, the first industry-specific LLM for the semiconductor domain, provides a foundation that can be used to develop tailored proprietary models. With SemiKong 1.0, we aim to develop a foundational model capable of understanding etching problems at an expert level. Our key contributions include (a) curating a comprehensive corpus of semiconductor-related texts, (b) creating a foundational model with in-depth semiconductor knowledge, and (c) introducing a framework for integrating expert knowledge, thereby advancing the evaluation process of domain-specific AI models. Through fine-tuning a pre-trained LLM using our curated dataset, we have shown that SemiKong outperforms larger, general-purpose LLMs in various semiconductor manufacturing and design tasks. Our extensive experiments underscore the importance of developing domain-specific LLMs as a foundation for company- or tool-specific proprietary models, paving the way for further research and applications in the semiconductor domain. Code and dataset will be available at https://github.com/aitomatic/semikong

• The paper introduces SemiKong, a semiconductor-specific LLM that leverages a curated corpus and adaptive fine-tuning to address industry challenges.
• It details the development process using SemiKong-Corpus, SemiKong-Trainer, and SemiKong-Eval to ensure comprehensive domain understanding and enhanced performance.
• The evaluation framework, incorporating expert feedback, demonstrates superior logical coherence, practicality, and immediate usability compared to general-purpose LLMs.

An Expert Overview of SemiKong: A Domain-Specific LLM for the Semiconductor Industry

In "SEMIKONG: CURATING, TRAINING, AND EVALUATING A SEMICONDUCTOR INDUSTRY-SPECIFIC LLM," the authors address the knowledge gaps in general-purpose LLMs by introducing SemiKong, a domain-specific LLM tailored for the semiconductor industry. This paper outlines the necessity, creation, and evaluation of SemiKong 1.0 to optimize semiconductor manufacturing processes, particularly focusing on etching—a critical process in semiconductor fabrication, where LLMs can significantly impact efficiency and accuracy.

Core Contributions

The primary contributions of this paper can be detailed as follows:

1. SemiKong-Corpus: A meticulously curated corpus of semiconductor-specific texts forms the backbone of this model. The dataset, consisting of more than 20,000 texts, including books and research papers, captures the intricate knowledge necessary for semiconductor manufacturing tasks. This resource is foundational, offering a rich pool of domain-specific terminology and procedural knowledge to train the model.
2. SemiKong-Trainer: The authors applied adaptive pre-training techniques and fine-tuning strategies to create SemiKong. Using Llama3 8B and 70B variants as a starting point, they pre-trained models with domain-specific data before applying Supervised Fine-Tuning (SFT) on instruction datasets. This ensured that the model developed a comprehensive understanding of semiconductor-related queries and problems, particularly optimizing for tasks in the etching process.
3. SemiKong-Eval: A novel evaluation framework was introduced, incorporating expert feedback to produce robust benchmarks that effectively assess AI solutions in the semiconductor domain. This framework emphasizes tailored evaluation criteria focusing on metrics such as clarity, directness, and coherence, ensuring the model's outputs are aligned with the needs of industry specialists.

Performance Evaluation

Through a rigorous evaluation process, SemiKong was compared against both open-source models like Llama3 and commercial counterparts such as GPT-3.5 and Claude-3.5. As indicated in the results, SemiKong 70B outperformed its open-source, generic LLM counterparts across all key performance indicators. Notably, SemiKong demonstrated superior Practicality and Immediate Usability (PIU), logical coherence, and efficiency, aligning with the daily operational demands of semiconductor engineers.

Implications and Future Directions

SemiKong's implications are twofold: On a practical level, it fundamentally enhances the capability to perform tasks related to semiconductor manufacturing with accuracy and efficiency. On a theoretical level, it opens the possibility for further exploration into domain-specific AI models that meet the specific needs of complex industrial domains. The successful implementation and evaluation methodology adopted for SemiKong can be extrapolated to other niche sectors requiring deep expertise.

The future of AI in the semiconductor field will likely see advancements below the etching process, focusing on other specialized operations within the semiconductor workflow. Potential expansions could target comprehensive support for other processes outlined in their developed semiconductor process ontology. Moreover, the methodologies and pipelines developed here can be adapted to support a broader range of industrial applications, potentially revolutionizing process optimization and quality assurance sectors.

In conclusion, SemiKong presents a paper in the applicability and efficacy of domain-specific LLMs. Its successful adaptation and superior performance highlight the importance of specialized training corpora and evaluation metrics in harnessing the full potential of AI, especially in sectors with specific and complex needs such as semiconductor manufacturing.