Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
86 tokens/sec
GPT-4o
11 tokens/sec
Gemini 2.5 Pro Pro
53 tokens/sec
o3 Pro
5 tokens/sec
GPT-4.1 Pro
3 tokens/sec
DeepSeek R1 via Azure Pro
33 tokens/sec
2000 character limit reached

Fusing Models with Complementary Expertise (2310.01542v2)

Published 2 Oct 2023 in cs.LG

Abstract: Training AI models that generalize across tasks and domains has long been among the open problems driving AI research. The emergence of Foundation Models made it easier to obtain expert models for a given task, but the heterogeneity of data that may be encountered at test time often means that any single expert is insufficient. We consider the Fusion of Experts (FoE) problem of fusing outputs of expert models with complementary knowledge of the data distribution and formulate it as an instance of supervised learning. Our method is applicable to both discriminative and generative tasks and leads to significant performance improvements in image and text classification, text summarization, multiple-choice QA, and automatic evaluation of generated text. We also extend our method to the "frugal" setting where it is desired to reduce the number of expert model evaluations at test time. Our implementation is publicly available at https://github.com/hwang595/FoE-ICLR2024.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. The inclusive images competition. In The NeurIPS’18 Competition: From Machine Learning to Intelligent Conversations, pp.  155–186. Springer, 2020.
  2. Open llm leaderboard. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  3. On the Opportunities and Risks of Foundation Models. arXiv:2108.07258 [cs], August 2021.
  4. Leo Breiman. Bagging predictors. Machine learning, 24:123–140, 1996a.
  5. Leo Breiman. Stacked regressions. Machine learning, 24:49–64, 1996b.
  6. Language Models are Few-Shot Learners. arXiv:2005.14165 [cs], June 2020.
  7. Generalized negative correlation learning for deep ensembling. arXiv preprint arXiv:2011.02952, 2020.
  8. Confidence-based model selection: When to take shortcuts for subpopulation shifts. arXiv preprint arXiv:2306.11120, 2023a.
  9. FrugalML: How to use ML prediction APIs more accurately and cheaply. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS’20, pp.  10685–10696, Red Hook, NY, USA, December 2020. Curran Associates Inc. ISBN 978-1-71382-954-6.
  10. Frugalgpt: How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176, 2023b.
  11. Adapting large language models via reading comprehension. arXiv preprint arXiv:2309.09530, 2023.
  12. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
  13. Wikisum: Coherent summarization dataset for efficient human evaluation. 2021.
  14. Thomas G Dietterich. Ensemble methods in machine learning. In International workshop on multiple classifier systems, pp.  1–15. Springer, 2000.
  15. Summeval: Re-evaluating summarization evaluation. Transactions of the Association for Computational Linguistics, 9:391–409, 2021.
  16. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity. The Journal of Machine Learning Research, 23(1):5232–5270, 2022.
  17. Experiments with a new boosting algorithm. In icml, volume 96, pp.  148–156. Citeseer, 1996.
  18. Jerome H. Friedman. Greedy function approximation: A gradient boosting machine. The Annals of Statistics, 29(5):1189–1232, 2001. ISSN 0090-5364, 2168-8966. doi: 10.1214/aos/1013203451.
  19. Specializing Smaller Language Models towards Multi-Step Reasoning. In Proceedings of the 40th International Conference on Machine Learning, pp.  10421–10430. PMLR, July 2023.
  20. Ensemble deep learning: A review. Engineering Applications of Artificial Intelligence, 115:105151, 2022.
  21. Selective Classification for Deep Neural Networks. arXiv:1705.08500 [cs], June 2017.
  22. Newsroom: A dataset of 1.3 million summaries with diverse extractive strategies. arXiv preprint arXiv:1804.11283, 2018.
  23. Textbooks are all you need. arXiv preprint arXiv:2306.11644, 2023.
  24. On calibration of modern neural networks. In International conference on machine learning, pp.  1321–1330. PMLR, 2017.
  25. Neural network ensembles. IEEE transactions on pattern analysis and machine intelligence, 12(10):993–1001, 1990.
  26. The elements of statistical learning: data mining, inference, and prediction, volume 2. Springer, 2009.
  27. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp.  770–778, Las Vegas, NV, USA, June 2016. IEEE. ISBN 978-1-4673-8851-1. doi: 10.1109/CVPR.2016.90.
  28. Measuring massive multitask language understanding. In International Conference on Learning Representations, 2021.
  29. Distilling step-by-step! outperforming larger language models with less training data and smaller model sizes. arXiv preprint arXiv:2305.02301, 2023.
  30. Adaptive mixtures of local experts. Neural computation, 3(1):79–87, 1991.
  31. Exploring the benefits of training expert language models over instruction tuning. arXiv preprint arXiv:2302.03202, 2023.
  32. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.  14165–14178, Toronto, Canada, jul 2023. Association for Computational Linguistics. doi: 10.18653/v1/2023.acl-long.792. URL https://aclanthology.org/2023.acl-long.792.
  33. Hierarchical mixtures of experts and the em algorithm. Neural computation, 6(2):181–214, 1994.
  34. Advances and open problems in federated learning. Foundations and Trends® in Machine Learning, 14(1–2):1–210, 2021.
  35. WILDS: A Benchmark of in-the-Wild Distribution Shifts. arXiv:2012.07421 [cs], December 2020.
  36. Learning multiple layers of features from tiny images. 2009.
  37. Neural network ensembles, cross validation, and active learning. Advances in neural information processing systems, 7, 1994.
  38. Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution. February 2022. doi: 10.48550/arXiv.2202.10054.
  39. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30, 2017.
  40. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019.
  41. Fedmd: Heterogenous federated learning via model distillation. arXiv preprint arXiv:1910.03581, 2019.
  42. Automatic evaluation of summaries using n-gram co-occurrence statistics. In Proceedings of the 2003 human language technology conference of the North American chapter of the association for computational linguistics, pp.  150–157, 2003.
  43. Ensemble distillation for robust model fusion in federated learning. Advances in Neural Information Processing Systems, 33:2351–2363, 2020.
  44. Detecting and Correcting for Label Shift with Black Box Predictors. arXiv:1802.03916 [cs, stat], July 2018.
  45. Ensemble learning via negative correlation. Neural networks, 12(10):1399–1404, 1999.
  46. Inderjeet Mani. Automatic summarization, volume 3. John Benjamins Publishing, 2001.
  47. Mixture of experts: a literature survey. Artificial Intelligence Review, 42:275–293, 2014.
  48. On faithfulness and factuality in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp.  1906–1919, Online, 2020.
  49. Communication-efficient learning of deep networks from decentralized data. In Artificial intelligence and statistics, pp.  1273–1282. PMLR, 2017.
  50. Ibrahim Naji. TSATC: Twitter Sentiment Analysis Training Corpus. In thinknook, 2012.
  51. Understanding softmax confidence and uncertainty. arXiv preprint arXiv:2106.04972, 2021.
  52. Catboost: unbiased boosting with categorical features. Advances in neural information processing systems, 31, 2018.
  53. Summareranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization. arXiv preprint arXiv:2203.06569, 2022.
  54. Out-of-distribution detection and selective generation for conditional language models. In International Conference on Learning Representations, 2023.
  55. Stuart J Russell. Artificial intelligence a modern approach. Pearson Education, Inc., 2010.
  56. BREEDS: Benchmarks for Subpopulation Shift. August 2020. doi: 10.48550/arXiv.2008.04859.
  57. On the evaluation metrics for paraphrase generation. arXiv preprint arXiv:2202.08479, 2022.
  58. Towards out-of-distribution generalization: A survey. arXiv preprint arXiv:2108.13624, 2021.
  59. Investigating societal biases in a poetry composition system. arXiv preprint arXiv:2011.02686, 2020.
  60. Crowd counting with deep negative correlation learning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.  5382–5390, 2018.
  61. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929–1958, 2014.
  62. Asking and answering questions to evaluate the factual consistency of summaries. arXiv preprint arXiv:2004.04228, 2020a.
  63. Federated learning with matched averaging. In International Conference on Learning Representations, 2020b.
  64. A field guide to federated optimization. arXiv preprint arXiv:2107.06917, 2021.
  65. David H Wolpert. Stacked generalization. Neural networks, 5(2):241–259, 1992.
  66. Bartscore: Evaluating generated text as text generation. Advances in Neural Information Processing Systems, 34:27263–27277, 2021.
  67. Mixup: Beyond Empirical Risk Minimization. In International Conference on Learning Representations, February 2018.
  68. Pegasus: Pre-training with extracted gap-sentences for abstractive summarization. In International Conference on Machine Learning, pp.  11328–11339. PMLR, 2020.
  69. Nonlinear regression via deep negative correlation learning. IEEE transactions on pattern analysis and machine intelligence, 43(3):982–998, 2019a.
  70. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675, 2019b.
  71. Towards a unified multi-dimensional evaluator for text generation. arXiv preprint arXiv:2210.07197, 2022.
Citations (19)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets