Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Exploring the Capabilities of Large Language Models for Generating Diverse Design Solutions (2405.02345v1)

Published 2 May 2024 in cs.HC and cs.AI

Abstract: Access to large amounts of diverse design solutions can support designers during the early stage of the design process. In this paper, we explore the efficacy of LLMs (LLM) in producing diverse design solutions, investigating the level of impact that parameter tuning and various prompt engineering techniques can have on the diversity of LLM-generated design solutions. Specifically, LLMs are used to generate a total of 4,000 design solutions across five distinct design topics, eight combinations of parameters, and eight different types of prompt engineering techniques, comparing each combination of parameter and prompt engineering method across four different diversity metrics. LLM-generated solutions are compared against 100 human-crowdsourced solutions in each design topic using the same set of diversity metrics. Results indicate that human-generated solutions consistently have greater diversity scores across all design topics. Using a post hoc logistic regression analysis we investigate whether these differences primarily exist at the semantic level. Results show that there is a divide in some design topics between humans and LLM-generated solutions, while others have no clear divide. Taken together, these results contribute to the understanding of LLMs' capabilities in generating a large volume of diverse design solutions and offer insights for future research that leverages LLMs to generate diverse design solutions for a broad range of design tasks (e.g., inspirational stimuli).

PDF HTML Abstract
Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize File Document Download Save Streamline Icon: https://streamlinehq.com PDF File Document Download Save Streamline Icon: https://streamlinehq.com Markdown Bookmark Chat Bubble Oval Streamline Icon: https://streamlinehq.com Chat (Pro)
Definition Search Book Streamline Icon: https://streamlinehq.com
References (50)
  1. Dahl, D. W. and Moreau, P., 2002, “The influence and value of analogical thinking during new product ideation,” Journal of marketing research, 39(1), pp. 47–60.
  2. Kwon, E., Huang, F., and Goucher-Lambert, K., 2022, “Enabling multi-modal search for inspirational design stimuli using deep learning,” AI EDAM, 36.
  3. Fu, K., Murphy, J., Yang, M., Otto, K., Jensen, D., and Wood, K., 2015, “Design-by-analogy: experimental evaluation of a functional analogy search methodology for concept generation improvement,” Research in Engineering Design, 26, pp. 77–95.
  4. Linsey, J. S., Wood, K. L., and Markman, A. B., 2008, “Modality and representation in analogy,” Ai Edam, 22(2), pp. 85–100.
  5. Goucher-Lambert, K. and Cagan, J., 2019, “Crowdsourcing inspiration: Using crowd generated inspirational stimuli to support designer ideation,” Design Studies, 61, pp. 1–29.
  6. Yu, L., Kittur, A., and Kraut, R. E., 2014, “Searching for analogical ideas with crowds,” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, pp. 1225–1234.
  7. Ma, K., Grandi, D., McComb, C., and Goucher-Lambert, K., 2023, “Conceptual Design Generation Using Large Language Models,” International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 116838, American Society of Mechanical Engineers, p. V006T06A021.
  8. Zhu, Q., Zhang, X., and Luo, J., 2023, “Biologically Inspired Design Concept Generation Using Generative Pre-Trained Transformers,” Journal of Mechanical Design, 145(4), p. 041409.
  9. Chan, J., Fu, K., Schunn, C., Cagan, J., Wood, K., and Kotovsky, K., 2011, “On the benefits and pitfalls of analogies for innovative design: Ideation performance based on analogical distance, commonness, and modality of examples,” .
  10. Jiang, S., Hu, J., Wood, K. L., and Luo, J., 2022, “Data-driven design-by-analogy: state-of-the-art and future directions,” Journal of Mechanical Design, 144(2), p. 020801.
  11. Poetz, M. K. and Schreier, M., 2012, “The value of crowdsourcing: can users really compete with professionals in generating new product ideas?” Journal of product innovation management, 29(2), pp. 245–256.
  12. Yu, L., Kittur, A., and Kraut, R. E., 2014, “Distributed analogical idea generation: inventing with crowds,” Proceedings of the SIGCHI conference on Human Factors in Computing Systems, pp. 1245–1254.
  13. Goucher-Lambert, K., Cagan, J., et al., 2017, “Using crowdsourcing to provide analogies for designer ideation in a cognitive study,” DS 87-8 Proceedings of the 21st International Conference on Engineering Design (ICED 17) Vol 8: Human Behaviour in Design, Vancouver, Canada, 21-25.08. 2017, pp. 529–538.
  14. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al., 2020, “Language models are few-shot learners,” Advances in neural information processing systems, 33, pp. 1877–1901.
  15. OpenAI, 2023, “GPT-4 Technical Report,” 2303.08774
  16. Zhu, Q. and Luo, J., 2023, “Generative transformers for design concept generation,” Journal of Computing and Information Science in Engineering, 23(4), p. 041003.
  17. Wang, B., Zuo, H., Cai, Z., Yin, Y., Childs, P., Sun, L., and Chen, L., 2023, “A Task-Decomposed AI-Aided Approach for Generative Conceptual Design,” International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 87349, American Society of Mechanical Engineers, p. V006T06A009.
  18. Goucher-Lambert, K., Gyory, J. T., Kotovsky, K., and Cagan, J., 2020, “Adaptive inspirational design stimuli: using design output to computationally search for stimuli that impact concept generation,” Journal of Mechanical Design, 142(9), p. 091401.
  19. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., and Polosukhin, I., 2017, “Attention is all you need,” Advances in neural information processing systems, 30.
  20. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al., 2022, “Emergent abilities of large language models,” arXiv preprint arXiv:2206.07682.
  21. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al., 2023, “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971.
  22. Driess, D., Xia, F., Sajjadi, M. S., Lynch, C., Chowdhery, A., Ichter, B., Wahid, A., Tompson, J., Vuong, Q., Yu, T., et al., 2023, “Palm-e: An embodied multimodal language model,” arXiv preprint arXiv:2303.03378.
  23. Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., Le, Q. V., Zhou, D., et al., 2022, “Chain-of-thought prompting elicits reasoning in large language models,” Advances in Neural Information Processing Systems, 35, pp. 24824–24837.
  24. Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., and Iwasawa, Y., 2022, “Large language models are zero-shot reasoners,” Advances in neural information processing systems, 35, pp. 22199–22213.
  25. Fu, K., Chan, J., Cagan, J., Kotovsky, K., Schunn, C., and Wood, K., 2013, “The meaning of “near” and “far”: the impact of structuring design databases and the effect of distance of analogy on design output,” Journal of Mechanical Design, 135(2), p. 021007.
  26. Chan, J. and Schunn, C., 2015, “The impact of analogies on creative concept generation: Lessons from an in vivo study in engineering design,” Cognitive science, 39(1), pp. 126–155.
  27. Cagan, J., Chan, J., Fu, K., Schunn, C., Wood, K., Kotovsky, K., et al., 2011, “On the effective use of design-by-analogy: The influences of analogical distance and commonness of analogous designs on ideation performance,” DS 68-7: Proceedings of the 18th International Conference on Engineering Design (ICED 11), Impacting Society through Engineering Design, Vol. 7: Human Behaviour in Design, Lyngby/Copenhagen, Denmark, 15.-19.08. 2011, pp. 85–96.
  28. Regenwetter, L., Srivastava, A., Gutfreund, D., and Ahmed, F., 2023, “Beyond statistical similarity: Rethinking metrics for deep generative models in engineering design,” arXiv preprint arXiv:2302.02913.
  29. Podani, J., 2009, “Convex hulls, habitat filtering, and functional diversity: mathematical elegance versus ecological interpretability,” Community Ecology, 10(2), pp. 244–250.
  30. Kulesza, A., Taskar, B., et al., 2012, “Determinantal point processes for machine learning,” Foundations and Trends® in Machine Learning, 5(2–3), pp. 123–286.
  31. Chen, W. and Ahmed, F., 2021, “Padgan: Learning to generate high-quality novel designs,” Journal of Mechanical Design, 143(3), p. 031703.
  32. Linsey, J. S. and Viswanathan, V. K., 2014, “Overcoming cognitive challenges in bioinspired design and analogy,” Biologically inspired design: computational methods and tools, pp. 221–244.
  33. Tseng, I., Moss, J., Cagan, J., and Kotovsky, K., 2008, “The role of timing and analogical similarity in the stimulation of idea generation in design,” Design studies, 29(3), pp. 203–221.
  34. Toh, C. A. and Miller, S. R., 2014, “The impact of example modality and physical interactions on design creativity,” Journal of Mechanical Design, 136(9), p. 091004.
  35. Linsey, J. S., Markman, A. B., and Wood, K. L., 2012, “Design by Analogy: A Study of the WordTree Method for Problem Re-Representation,” https://doi.org/10.1115/1.4006145Journal of Mechanical Design, 134(4), p. 041009.
  36. Shinn, N., Labash, B., and Gopinath, A., 2023, “Reflexion: an autonomous agent with dynamic memory and self-reflection,” arXiv preprint arXiv:2303.11366.
  37. White, J., Fu, Q., Hays, S., Sandborn, M., Olea, C., Gilbert, H., Elnashar, A., Spencer-Smith, J., and Schmidt, D. C., 2023, “A prompt pattern catalog to enhance prompt engineering with chatgpt,” arXiv preprint arXiv:2302.11382.
  38. Reimers, N. and Gurevych, I., 2019, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv preprint arXiv:1908.10084.
  39. Walsh, H. S. and Andrade, S. R., 2022, “Semantic Search With Sentence-BERT for Design Information Retrieval,” International Design Engineering Technical Conferences and Computers and Information in Engineering Conference, Vol. 86212, American Society of Mechanical Engineers, p. V002T02A066.
  40. Tsumuraya, K., Amano, M., Uehara, M., and Adachi, Y., 2022, “Topic-Based Clustering of Japanese Sentences Using Sentence-BERT,” 2022 Tenth International Symposium on Computing and Networking Workshops (CANDARW), IEEE, pp. 255–260.
  41. Bro, R. and Smilde, A. K., 2014, “Principal component analysis,” Analytical methods, 6(9), pp. 2812–2831.
  42. Team, G., Anil, R., Borgeaud, S., Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al., 2023, “Gemini: a family of highly capable multimodal models,” arXiv preprint arXiv:2312.11805.
  43. Bergstra, J., Bardenet, R., Bengio, Y., and Kégl, B., 2011, “Algorithms for hyper-parameter optimization,” Advances in neural information processing systems, 24.
  44. Barto, A. G., 1997, “Reinforcement learning,” Neural systems for control, Elsevier, pp. 7–30.
  45. Finn, C., Abbeel, P., and Levine, S., 2017, “Model-agnostic meta-learning for fast adaptation of deep networks,” International conference on machine learning, PMLR, pp. 1126–1135.
  46. Reynolds, L. and McDonell, K., 2021, “Prompt programming for large language models: Beyond the few-shot paradigm,” Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pp. 1–7.
  47. Hwang, A. H.-C., 2022, “Too late to be creative? AI-empowered tools in creative processes,” CHI Conference on Human Factors in Computing Systems Extended Abstracts, pp. 1–9.
  48. Inie, N., Falk, J., and Tanimoto, S., 2023, “Designing Participatory AI: Creative Professionals’ Worries and Expectations about Generative AI,” Extended Abstracts of the 2023 CHI Conference on Human Factors in Computing Systems, pp. 1–8.
  49. Kittur, A., Yu, L., Hope, T., Chan, J., Lifshitz-Assaf, H., Gilon, K., Ng, F., Kraut, R. E., and Shahaf, D., 2019, “Scaling up analogical innovation with crowds and AI,” Proceedings of the National Academy of Sciences, 116(6), pp. 1870–1877.
  50. Spearman, C., 1961, “The proof and measurement of association between two things.” .
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (4)
  1. Kevin Ma (6 papers)
  2. Daniele Grandi (13 papers)
  3. Christopher McComb (16 papers)
  4. Kosa Goucher-Lambert (6 papers)
Citations (1)
View on Semantic Scholar
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets