On Training Data Influence of GPT Models
Abstract: Amidst the rapid advancements in generative LLMs, the investigation of how training data shapes the performance of GPT models is still emerging. This paper presents GPTfluence, a novel approach that leverages a featurized simulation to assess the impact of training examples on the training dynamics of GPT models. Our approach not only traces the influence of individual training instances on performance trajectories, such as loss and other key metrics, on targeted test points but also enables a comprehensive comparison with existing methods across various training scenarios in GPT models, ranging from 14 million to 2.8 billion parameters, across a range of downstream tasks. Contrary to earlier methods that struggle with generalization to new data, GPTfluence introduces a parameterized simulation of training dynamics, demonstrating robust generalization capabilities to unseen training data. This adaptability is evident across both fine-tuning and instruction-tuning scenarios, spanning tasks in natural language understanding and generation. We make our code and data publicly available at https://github.com/ernie-research/gptfluence.
- Gpt-4 technical report. arXiv preprint arXiv:2303.08774.
- Pythia: A suite for analyzing large language models across training and scaling.
- Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901.
- Ernie-code: Beyond english-centric cross-lingual pretraining for programming languages. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 10628–10650. Association for Computational Linguistics.
- Input similarity from the neural network perspective. Advances in Neural Information Processing Systems, 32.
- BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
- Dsdm: Model-aware dataset selection with datamodels. arXiv preprint arXiv:2401.12926.
- Simfluence: Modeling the influence of individual training examples by simulating training runs. arXiv preprint arXiv:2303.08114.
- Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.
- Datamodels: Predicting predictions from training data. arXiv preprint arXiv:2202.00622.
- Mistral 7b. arXiv preprint arXiv:2310.06825.
- Karthikeyan K and Anders Søgaard. 2021. Revisiting methods for finding influential examples.
- Pang Wei Koh and Percy Liang. 2017. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885–1894. PMLR.
- Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pages 74–81.
- Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173.
- Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 311–318.
- Trak: Attributing model behavior at scale. arXiv preprint arXiv:2303.14186.
- Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33:19920–19930.
- Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
- Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805.
- Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
- Finetuned language models are zero-shot learners.
- Less: Selecting influential data for targeted instruction tuning. arXiv preprint arXiv:2402.04333.
- Representer point selection for explaining deep neural networks. Advances in neural information processing systems, 31.
- Opt: Open pre-trained transformer language models.
Paper Prompts
Sign up for free to create and run prompts on this paper using GPT-5.
Top Community Prompts
Collections
Sign up for free to add this paper to one or more collections.