TinyHelen's First Curriculum: Training and Evaluating Tiny Language Models in a Simpler Language Environment (2501.00522v1)

Abstract: Training LMs and their application agents is increasingly costly due to large datasets and models, making test failures difficult to bear. Simplified language environments serve as primordial training and testing grounds, retaining essential commonsense and communication skills but in a more digestible form, potentially enhancing the learning efficiency of LMs, and thus reducing the required model size and data volume for effective training and evaluation. In these simplified language environments, workable strategies for small models, datasets, and agents may be adaptable to larger models, datasets, and agents in complex language environments. To create such environments, we focus on two aspects: i) minimizing language dataset noise and complexity, and ii) preserving the essential text distribution characteristics. Unlike previous methods, we propose a pipeline to refine text data by eliminating noise, minimizing vocabulary, and maintaining genre-specific patterns (e.g., for books, conversation, code, etc.). Implementing this pipeline with large LMs, we have created a leaner suite of LM training and evaluation datasets: 71M Leaner-Pretrain, 7M Leaner-Instruct, Leaner-Glue for assessing linguistic proficiency, and Leaner-Eval for testing instruction-following ability. Our experiments show that leaner pre-training boosts LM learning efficiency. Tiny LMs trained on these datasets outperform those trained on original datasets in instruction-following across different language granularity levels. Moreover, the Leaner-Pretrain dataset's alignment with conventional large LM training sets enables resource-optimized analysis of how learning objectives, model architectures, and training techniques impact performance on LLMing and downstream tasks. Our code and datasets are available at https://github.com/EmpathYang/TinyHelen.git.

• The paper pioneers a method for curating leaner datasets that enable efficient training of Tiny Language Models.
• It uses a data revision pipeline to eliminate noise, reduce vocabulary, and preserve key linguistic patterns.
• Experimental results show that TLMs trained on simplified datasets outperform larger models in instruction-following tasks.

Training and Evaluating Tiny LLMs in Simplified Environments

The paper "TinyHelen's First Curriculum: Training and Evaluating Tiny LLMs in a Simpler Language Environment" introduces an innovative approach to training LMs in a cost-effective manner, by leveraging simplified language environments. This strategy is motivated by challenges associated with the training of LLMs, particularly the high cost involved in terms of computational resources and data management.

Core Methodology and Dataset

The authors propose a methodology that aims to reduce the size and complexity of datasets without losing essential linguistic characteristics. This allows small models, referred to as Tiny LLMs (TLMs), to be trained more efficiently. The approach focused on minimizing language dataset noise and complexity while preserving key text distribution characteristics, an objective that former methods struggled with.

To create these "Leaner" datasets, a data revision pipeline is introduced. It involves noise elimination, vocabulary reduction, and maintenance of genre-specific linguistic patterns. Implementing this process yielded various leaner datasets such as a 71-million token Leaner-Pretrain, a 7M Leaner-Instruct for instruction tuning, Leaner-Glue for linguistic proficiency assessment, and Leaner-Eval for evaluating instruction-following capabilities.

Experimental Results

Experiments confirmed that TLMs trained on these simplified datasets outperformed those trained on more extensive, conventional datasets, especially in instruction-following tasks. The results also allowed for detailed resource-optimized analysis concerning learning objectives, architectures, and training techniques.

The key findings include:

• Learning Efficiency: Models trained with the simplified Leaner-Pretrain dataset achieved better performance than those trained on larger, noisier datasets. This suggests potential for reduced computational resources and model size without sacrificing effectiveness.
• Instruction-Following: TLMs demonstrated enhanced capability in following instructions when trained on the leaner datasets, even at reduced linguistic input levels.
• Architecture and Curriculum Learning: By analyzing the performance of different architectures like BERT, Llama, XLNet, and Mamba, the paper suggests that simpler datasets are valuable in determining optimal pre-training strategies and assessing model capabilities under streamlined conditions.

Implications and Future Work

The implications of this work are notable both practically and theoretically. Practically, this paper provides a framework for training smaller models with less data, which is crucial for institutions limited by resources. Theoretically, it opens avenues for further exploration into curriculum learning and its potential to emulate human-like learning approaches in machines.

The paper concludes with the acknowledgment of limitations in truly small models showing significant language proficiency and instruction-following abilities. It makes the case for further studies that could explore larger synthetic datasets or advanced approaches in data curation to enhance the development of cost-efficient and effective LLMs.

Future research could deepen insights into curriculum learning and refine methodologies that further close the gap between how humans and machines acquire language. Such exploration may eventually lead to breakthroughs in the generation of self-evolving, text-based autonomous agents, a prospect the paper touches upon as a long-term goal.