Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
41 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMs (2402.12030v2)

Published 19 Feb 2024 in cs.CL

Abstract: Deploying LLMs of several billion parameters can be impractical in most industrial use cases due to constraints such as cost, latency limitations, and hardware accessibility. Knowledge distillation (KD) offers a solution by compressing knowledge from resource-intensive large models to smaller ones. Various strategies exist, some relying on the text generated by the teacher model and optionally utilizing his logits to enhance learning. However, these methods based on logits often require both teacher and student models to share the same tokenizer, limiting their applicability across different LLM families. In this paper, we introduce Universal Logit Distillation (ULD) loss, grounded in optimal transport, to address this limitation. Our experimental results demonstrate the effectiveness of ULD loss in enabling distillation across models with different architectures and tokenizers, paving the way to a more widespread use of distillation techniques.

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (64)
  1. The falcon series of open language models. arXiv:2311.16867. arXiv. ArXiv:2311.16867 [cs].
  2. Stochastic wasserstein autoencoder for probabilistic sentence generation.
  3. Knot: Knowledge distillation using optimal transport for solving nlp tasks.
  4. Pythia: A suite for analyzing large language models across training and scaling. arXiv:2304.01373arXiv:2304.01373. arXiv. ArXiv:2304.01373 [cs].
  5. Gpt-neox-20b: An open-source autoregressive language model.
  6. Language models are few-shot learners. volume 33, pages 1877–1901.
  7. Sparks of artificial general intelligence: Early experiments with gpt-4.
  8. Model compression. In Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD ’06, page 535–541, New York, NY, USA. Association for Computing Machinery.
  9. Dialogsum: A real-life scenario dialogue summarization dataset.
  10. Automatic text evaluation through the lens of Wasserstein barycenters. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10450–10466, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  11. Marco Cuturi. 2013. Sinkhorn distances: Lightspeed computation of optimal transport. volume 26.
  12. Cost-effective distillation of large language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 7346–7354.
  13. Qlora: Efficient finetuning of quantized llms.
  14. Training on synthetic data beats real data in multimodal relation extraction.
  15. Revisiting instruction fine-tuned model evaluation to guide industrial applications. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics.
  16. Optimal transport for unsupervised hallucination detection in neural machine translation.
  17. Teacherlm: Teaching to fish rather than giving the fish, language modeling likewise. arXiv:2310.19019. arXiv. ArXiv:2310.19019 [cs].
  18. Generate, Annotate, and Learn: NLP with Synthetic Text. volume 10, pages 826–842.
  19. Distilling the knowledge in a neural network. arXiv:1503.02531arXiv:1503.02531. arXiv. ArXiv:1503.02531 [cs, stat].
  20. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes.
  21. Mistral 7b. arXiv:2310.06825. arXiv. ArXiv:2310.06825 [cs].
  22. Mixtral of experts.
  23. Tinybert: Distilling bert for natural language understanding.
  24. Pubmedqa: A dataset for biomedical research question answering.
  25. A study of bfloat16 for deep learning training.
  26. Yoon Kim and Alexander M. Rush. 2016. Sequence-level knowledge distillation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1317–1327, Austin, Texas. Association for Computational Linguistics.
  27. Alina Kramchaninova and Arne Defauw. 2022. Synthetic data generation for multilingual domain-adaptable question answering systems. In Proceedings of the 23rd Annual Conference of the European Association for Machine Translation, pages 151–160, Ghent, Belgium. European Association for Machine Translation.
  28. Qed: A framework and dataset for explanations in question answering.
  29. Reducing retraining by recycling parameter-efficient prompts.
  30. Fast inference from transformers via speculative decoding. In International Conference on Machine Learning, pages 19274–19286. PMLR.
  31. dis.
  32. Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  33. Llm-pruner: On the structural pruning of large language models.
  34. Small-100: Introducing shallow multilingual machine translation model for low-resource languages. arXiv:2210.11621arXiv:2210.11621. arXiv. ArXiv:2210.11621 [cs].
  35. Crosslingual generalization through multitask finetuning. arXiv:2211.01786. arXiv. ArXiv:2211.01786 [cs].
  36. Training language models to follow instructions with human feedback.
  37. Computational optimal transport: With applications to data science. volume 11, pages 355–607. Now Publishers, Inc.
  38. Squad: 100,000+ questions for machine comprehension of text.
  39. For distillation, tokens are not all you need. In NeurIPS 2023 Workshop on Instruction Tuning and Instruction Following.
  40. A survey of hallucination in large foundation models.
  41. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.
  42. Multitask prompted training enables zero-shot task generalization.
  43. Retrieval augmentation reduces hallucination in conversation.
  44. Beyond accuracy, f-score and roc: A family of discriminant measures for performance evaluation. volume Vol. 4304, pages 1015–1021.
  45. Mobilebert: a compact task-agnostic bert for resource-limited devices.
  46. Inar Timiryasov and Jean-Loup Tastet. 2023. Baby llama: knowledge distillation from an ensemble of teachers trained on a small dataset with no performance penalty.
  47. Llama: Open and efficient foundation language models. arXiv:2302.13971arXiv:2302.13971. arXiv. ArXiv:2302.13971 [cs].
  48. Llama 2: Open foundation and fine-tuned chat models.
  49. An empirical comparison of lm-based question and answer generation methods.
  50. Cédric Villani et al. 2009. Optimal transport: old and new, volume 338. Springer.
  51. Minilmv2: Multi-head self-attention relation distillation for compressing pretrained transformers.
  52. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers.
  53. Generalizing from a few examples: A survey on few-shot learning. volume 53, pages 1–34. ACM New York, NY, USA.
  54. Lamini-lm: A diverse herd of distilled models from large-scale instructions.
  55. Distilled wasserstein learning for word embedding and topic modeling.
  56. Fantastic questions and where to find them: Fairytaleqa – an authentic dataset for narrative comprehension.
  57. Fast discrete distribution clustering using wasserstein barycenter with sparse support.
  58. FiD-ICL: A fusion-in-decoder approach for efficient in-context learning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 8158–8185, Toronto, Canada. Association for Computational Linguistics.
  59. Opt: Open pre-trained transformer language models. arXiv:2205.01068. arXiv. ArXiv:2205.01068 [cs].
  60. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations.
  61. Multistage collaborative knowledge distillation from large language models. arXiv:2311.08640. arXiv. ArXiv:2311.08640 [cs].
  62. Pytorch fsdp: Experiences on scaling fully sharded data parallel.
  63. Tianxun Zhou and Keng-Hwee Chiam. 2023. Synthetic data generation method for data-free knowledge distillation in regression neural networks. volume 227, page 120327. Elsevier.
  64. Wasserstein k𝑘kitalic_k-means for clustering probability distributions.
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Nicolas Boizard (5 papers)
  2. Céline Hudelot (50 papers)
  3. Pierre Colombo (48 papers)
  4. Kevin El Haddad (14 papers)
Citations (6)
View on Semantic Scholar