2000 character limit reached
A Taxonomy of Foundation Model based Systems through the Lens of Software Architecture (2305.05352v6)
Published 9 May 2023 in cs.SE, cs.AI, and cs.CL
Abstract: The recent release of LLM based chatbots, such as ChatGPT, has attracted huge interest in foundation models. It is widely believed that foundation models will serve as the fundamental building blocks for future AI systems. As foundation models are in their early stages, the design of foundation model based systems has not yet been systematically explored. There is limited understanding about the impact of introducing foundation models in software architecture. Therefore, in this paper, we propose a taxonomy of foundation model based systems, which classifies and compares the characteristics of foundation models and design options of foundation model based systems. Our taxonomy comprises three categories: the pretraining and adaptation of foundation models, the architecture design of foundation model based systems, and responsible-AI-by-design. This taxonomy can serve as concrete guidance for making major architectural design decisions when designing foundation model based systems and highlights trade-offs arising from design decisions.
- R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill et al., “On the opportunities and risks of foundation models,” arXiv preprint arXiv:2108.07258, 2021.
- E. A. van Dis, J. Bollen, W. Zuidema, R. van Rooij, and C. L. Bockting, “Chatgpt: five priorities for research,” Nature, vol. 614, no. 7947, pp. 224–226, 2023.
- Q. Lu, L. Zhu, X. Xu, and J. Whittle, “Responsible-ai-by-design: A pattern collection for designing responsible ai systems,” IEEE Software, 2023.
- N. R. Mehta, N. Medvidovic, and S. Phadke, “Towards a taxonomy of software connectors,” in ICSE, 2000, pp. 178–187.
- I. Gorton, J. Klein, and A. Nurgaliev, “Architecture knowledge for evaluating scalable databases,” in 2015 12th Working IEEE/IFIP Conference on Software Architecture. IEEE, 2015, pp. 95–104.
- S. Wu, O. Irsoy, S. Lu, V. Dabravolski, M. Dredze, S. Gehrmann, P. Kambadur, D. Rosenberg, and G. Mann, “Bloomberggpt: A large language model for finance,” arXiv preprint arXiv:2303.17564, 2023.
- M. Moor, O. Banerjee, Z. S. H. Abad, H. M. Krumholz, J. Leskovec, E. J. Topol, and P. Rajpurkar, “Foundation models for generalist medical artificial intelligence,” Nature, vol. 616, no. 7956, pp. 259–265, 2023.
- T. Nguyen, J. Brandstetter, A. Kapoor, J. K. Gupta, and A. Grover, “Climax: A foundation model for weather and climate,” arXiv preprint arXiv:2301.10343, 2023.
- K. M. Jablonka, P. Schwaller, and B. Smit, “Is gpt-3 all you need for machine learning for chemistry?” in AI for Accelerated Materials Design NeurIPS 2022 Workshop.
- S. M. Xie, A. Raghunathan, P. Liang, and T. Ma, “An explanation of in-context learning as implicit bayesian inference,” arXiv preprint arXiv:2111.02080, 2021.
- D. C. Schmidt, J. Spencer-Smith, Q. Fu, and J. White, “Cataloging prompt patterns to enhance the discipline of prompt engineering.”
- Q. Lu, L. Zhu, X. Xu, J. Whittle, D. Zowghi, and A. Jacquet, “Responsible ai pattern catalogue: a multivocal literature review,” arXiv preprint arXiv:2209.04963, 2022.
- V. Garousi, M. Felderer, and M. V. Mäntylä, “Guidelines for including grey literature and conducting multivocal literature reviews in software engineering,” Information and Software Technology, vol. 106, pp. 101–121, 2019. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950584918301939
- B. Kitchenham and S. Charters, “Guidelines for performing systematic literature reviews in software engineering,” Technical report, Ver. 2.3 EBSE Technical Report. EBSE, Tech. Rep., 2007.
- S. M. Anwar, A. Parida, S. Atito, M. Awais, G. Nino, J. Kitler, and M. G. Linguraru, “Spcxr: Self-supervised pretraining using chest x-rays towards a domain specific foundation model,” 2023. [Online]. Available: https://europepmc.org/article/PPR/PPR661659
- B. Lin, Z. Chen, M. Li, H. Lin, H. Xu, Y. Zhu, J. Liu, W. Cai, L. Yang, S. Zhao, C. Wu, L. Chen, X. Chang, Y. Yang, L. Xing, and X. Liang, “Towards medical artificial general intelligence via knowledge-enhanced multimodal pretraining,” 2023.
- E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen, “Lora: Low-rank adaptation of large language models,” arXiv preprint arXiv:2106.09685, 2021.
- H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozière, N. Goyal, E. Hambro, F. Azhar et al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023.
- Y. Shen, K. Song, X. Tan, D. Li, W. Lu, and Y. Zhuang, “Hugginggpt: Solving ai tasks with chatgpt and its friends in huggingface,” arXiv preprint arXiv:2303.17580, 2023.
- D. Driess, F. Xia, M. S. Sajjadi, C. Lynch, A. Chowdhery, B. Ichter, A. Wahid, J. Tompson, Q. Vuong, T. Yu et al., “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378, 2023.
- S. Hong, X. Zheng, J. Chen, Y. Cheng, C. Zhang, Z. Wang, S. K. S. Yau, Z. Lin, L. Zhou, C. Ran et al., “Metagpt: Meta programming for multi-agent collaborative framework,” arXiv preprint arXiv:2308.00352, 2023.
- A. Zeng, A. Wong, S. Welker, K. Choromanski, F. Tombari, A. Purohit, M. Ryoo, V. Sindhwani, J. Lee, V. Vanhoucke et al., “Socratic models: Composing zero-shot multimodal reasoning with language,” arXiv preprint arXiv:2204.00598, 2022.
- OpenAI, “Gpt-4 technical report,” 2023.
- J. Fu, S.-K. Ng, Z. Jiang, and P. Liu, “Gptscore: Evaluate as you desire,” arXiv preprint arXiv:2302.04166, 2023.
- X. Zhou, Y. Jin, H. Zhang, S. Li, and X. Huang, “A map of threats to validity of systematic literature reviews in software engineering,” in 2016 23rd Asia-Pacific Software Engineering Conference (APSEC), 2016, pp. 153–160.
- Qinghua Lu (100 papers)
- Liming Zhu (101 papers)
- Xiwei Xu (87 papers)
- Yue Liu (257 papers)
- Zhenchang Xing (99 papers)
- Jon Whittle (32 papers)