Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
134 tokens/sec
GPT-4o
10 tokens/sec
Gemini 2.5 Pro Pro
47 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
38 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

SymNoise: Advancing Language Model Fine-tuning with Symmetric Noise (2312.01523v2)

Published 3 Dec 2023 in cs.CL and cs.LG

Abstract: In this paper, we introduce a novel fine-tuning technique for LLMs, which involves incorporating symmetric noise into the embedding process. This method aims to enhance the model's function by more stringently regulating its local curvature, demonstrating superior performance over the current method, NEFTune. When fine-tuning the LLaMA-2-7B model using Alpaca, standard techniques yield a 29.79% score on AlpacaEval. However, our approach, SymNoise, increases this score significantly to 69.04%, using symmetric noisy embeddings. This is a 6.7% improvement over the state-of-the-art method, NEFTune~(64.69%). Furthermore, when tested on various models and stronger baseline instruction datasets, such as Evol-Instruct, ShareGPT, OpenPlatypus, SymNoise consistently outperforms NEFTune. The current literature, including NEFTune, has underscored the importance of more in-depth research into the application of noise-based strategies in the fine-tuning of LLMs. Our approach, SymNoise, is another significant step towards this direction, showing notable improvement over the existing state-of-the-art method.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (36)
  1. Sharegpt. https://sharegpt.com/, 2023.
  2. Akbiyik, M. E. Data augmentation in training {cnn}s: Injecting noise to images, 2020. URL https://openreview.net/forum?id=SkeKtyHYPS.
  3. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020.
  4. Alpagasus: Training a better alpaca with fewer data. arXiv preprint arXiv:2307.08701, 2023.
  5. Vicuna: An open-source chatbot impressing gpt-4 with 90 https://lmsys.org/blog/2023-03-30-vicuna/, Mar 2023.
  6. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022.
  7. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint arXiv:1803.05457, 2018.
  8. Alpacafarm: A simulation framework for methods that learn from human feedback. arXiv preprint arXiv:2305.14387, 2023.
  9. The algorithmic foundations of differential privacy. Foundations and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014.
  10. A framework for few-shot language model evaluation. Version v0. 0.1. Sept, 2021.
  11. Measuring massive multitask language understanding. arXiv preprint arXiv:2009.03300, 2020.
  12. Huh, D. Curvature-corrected learning dynamics in deep neural networks. In International Conference on Machine Learning, pp.  4552–4560. PMLR, 2020.
  13. Neftune: Noisy embeddings improve instruction finetuning, 2023.
  14. On the reliability of watermarks for large language models. arXiv preprint arXiv:2306.04634, 2023.
  15. Robust optimization as data augmentation for large-scale graphs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  60–69, 2022.
  16. Platypus: Quick, cheap, and powerful refinement of llms. arXiv preprint arXiv:2308.07317, 2023.
  17. On explicit curvature regularization in deep generative models. In Topological, Algebraic and Geometric Learning Workshops 2023, pp.  505–518. PMLR, 2023.
  18. Contrastive decoding: Open-ended text generation as optimization. arXiv preprint arXiv:2210.15097, 2022.
  19. Robustness via curvature regularization, and vice versa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.  9078–9086, 2019.
  20. Text-only training for image captioning using noise-injected clip. arXiv preprint arXiv:2211.00575, 2022.
  21. OpenAI. Introducing chatgpt. https://openai.com/blog/chatgpt, 2022.
  22. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems, 35:27730–27744, 2022.
  23. Curvature regularization to prevent distortion in graph embedding. Advances in Neural Information Processing Systems, 33:20779–20790, 2020.
  24. Multitask prompted training enables zero-shot task generalization. arXiv preprint arXiv:2110.08207, 2021.
  25. Spall, J. C. Implementation of the simultaneous perturbation algorithm for stochastic optimization. IEEE Transactions on aerospace and electronic systems, 34(3):817–823, 1998.
  26. Stanford alpaca: An instruction-following llama model, 2023.
  27. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023a.
  28. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288, 2023b.
  29. Self-instruct: Aligning language model with self generated instructions. arXiv preprint arXiv:2212.10560, 2022.
  30. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652, 2021.
  31. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023.
  32. Zeroprompt: scaling prompt-based pretraining to 1,000 tasks improves zero-shot generalization. arXiv preprint arXiv:2201.06910, 2022.
  33. Noise injection-based regularization for point cloud processing. arXiv preprint arXiv:2103.15027, 2021.
  34. Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830, 2019.
  35. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068, 2022.
  36. Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909.11764, 2019.
Citations (1)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now