Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
38 tokens/sec
GPT-4o
59 tokens/sec
Gemini 2.5 Pro Pro
41 tokens/sec
o3 Pro
7 tokens/sec
GPT-4.1 Pro
50 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation (2310.12953v3)

Published 19 Oct 2023 in cs.HC and cs.AI

Abstract: Thanks to their generative capabilities, LLMs have become an invaluable tool for creative processes. These models have the capacity to produce hundreds and thousands of visual and textual outputs, offering abundant inspiration for creative endeavors. But are we harnessing their full potential? We argue that current interaction paradigms fall short, guiding users towards rapid convergence on a limited set of ideas, rather than empowering them to explore the vast latent design space in generative models. To address this limitation, we propose a framework that facilitates the structured generation of design space in which users can seamlessly explore, evaluate, and synthesize a multitude of responses. We demonstrate the feasibility and usefulness of this framework through the design and development of an interactive system, Luminate, and a user study with 14 professional writers. Our work advances how we interact with LLMs for creative tasks, introducing a way to harness the creative potential of LLMs.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (71)
  1. ]futurepedia [n. d.]. Futurepedia. https://www.futurepedia.io/ Last accessed 27 August 2023.
  2. ]upwork [n. d.]. Upwork. https://www.upwork.com/. Accessed: September 13, 2023.
  3. Christopher Ahlberg and Ben Shneiderman. 1994. Visual information seeking: Tight coupling of dynamic query filters with starfield displays. In Proceedings of the SIGCHI conference on Human factors in computing systems. 313–317. https://doi.org/10.1145/191666.191775
  4. Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–13. https://doi.org/10.1145/3290605.3300233
  5. Design patterns for data comics. In Proceedings of the 2018 chi conference on human factors in computing systems. 1–12. https://doi.org/10.1145/3173574.3173612
  6. Generative Theories of Interaction. ACM Transactions on Computer-Human Interaction (TOCHI) 28, 6 (2021), 1–54. https://doi.org/10.1145/3468505
  7. Michel Beaudouin-Lafon and Wendy E Mackay. 2007. Prototyping tools and techniques. In The human-computer interaction handbook. CRC Press, 1043–1066. https://www.kth.se/social/upload/52ef5ee4f2765445a466a28a/mackay-lafon-prototypes-52-HCI.pdf
  8. Benjamin B Bederson and James D Hollan. 1994. Pad++ a zooming graphical interface for exploring alternate interface physics. In Proceedings of the 7th annual ACM symposium on User interface software and technology. 17–26. https://doi.org/10.1145/192426.192435
  9. Graphdice: A system for exploring multivariate social networks. In Computer graphics forum, Vol. 29. Wiley Online Library, 863–872.
  10. A constraint-based understanding of design spaces. In Proceedings of the 2014 conference on Designing interactive systems. 453–462. https://doi.org/10.1145/2598510.2598533
  11. Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models. arXiv preprint arXiv:2304.09337 (2023). https://doi.org/10.48550/arXiv.2304.09337
  12. Dennis R Brophy. 2001. Comparing the attributes, activities, and performance of divergent, convergent, and combination thinkers. Creativity research journal 13, 3-4 (2001), 439–455. https://doi.org/10.1207/S15326934CRJ1334_20
  13. Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. https://doi.org/10.1016/B978-0-12-374037-3.X5043-3
  14. Readings in information visualization: using vision to think. Morgan Kaufmann. https://doi.org/10.5555/300679
  15. Creativity factor evaluation: towards a standardized survey metric for creativity support. In Proceedings of the seventh ACM conference on Creativity and cognition. 127–136.
  16. Mapping the design space of human-ai interaction in text summarization. arXiv preprint arXiv:2206.14863 (2022). https://doi.org/10.18653/v1/2022.naacl-main.33
  17. Erin Cherry and Celine Latulipe. 2014. Quantifying the creativity support of digital tools through the creativity support index. ACM Transactions on Computer-Human Interaction (TOCHI) 21, 4 (2014), 1–25. https://doi.org/10.1145/2617588
  18. Jarry HT Claessen and Jarke J Van Wijk. 2011. Flexible linked axes for multivariate data visualization. IEEE Transactions on Visualization and Computer Graphics 17, 12 (2011), 2310–2316. https://doi.org/10.1109/TVCG.2011.201
  19. Nigel Cross. 2004. Expertise in design: an overview. Design studies 25, 5 (2004), 427–441. https://doi.org/10.1016/j.destud.2004.06.002
  20. Beyond text generation: Supporting writers with continuous automatic text summaries. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–13. https://doi.org/10.48550/arXiv.2208.09323
  21. Edward De Bono. 1970. Lateral thinking. New York (1970), 70. https://www.kioulanis.gr/rivips/images/Lateral_thinking.pdf
  22. Zijian Ding and Joel Chan. 2023. Mapping the Design Space of Interactions in Human-AI Text Co-creation Tasks. arXiv e-prints (2023), arXiv–2303. https://doi.org/10.48550/arXiv.2303.06430
  23. An argument for design space reflection. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction. 1–10. https://doi.org/10.1145/2971485.2971528
  24. Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 4 (2010), 1–24. https://doi.org/10.1145/1879831.1879836
  25. Semantic interaction for sensemaking: inferring analytical reasoning for model steering. IEEE Transactions on Visualization and Computer Graphics 18, 12 (2012), 2879–2888. https://doi.org/10.1109/TVCG.2012.260
  26. PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation. arXiv preprint arXiv:2307.09036 (2023). https://doi.org/10.48550/arXiv.2307.09036
  27. Sparks: Inspiration for science writing using language models. In Designing interactive systems conference. 1002–1019. https://doi.org/10.1145/3532106.3533533
  28. Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–20. https://doi.org/10.1145/3544548.3580999
  29. Michael Golembewski and Mark Selby. 2010. Ideation decks: a card-based design ideation tool. In Proceedings of the 8th ACM Conference on Designing Interactive Systems. 89–92. https://doi.org/10.1145/1858171.1858189
  30. Joy Paul Guilford. 1961. Three faces of intellect. (1961). https://doi.org/10.1037/h0046827
  31. Joy Paul Guilford. 1967. The nature of human intelligence. (1967). https://doi.org/10.1017/9781316817049
  32. Kim Halskov and Caroline Lundqvist. 2021. Filtering and informing the design space: Towards design-space thinking. ACM Transactions on Computer-Human Interaction (TOCHI) 28, 1 (2021), 1–28. https://doi.org/10.1145/3434462
  33. CrossCode: Multi-level Visualization of Program Execution. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–13. https://doi.org/10.1145/3544548.3581390
  34. Chris Heape. 2007. The Design Space: the design process as the construction, exploration and expansion of a conceptual space. (2007). https://www.semanticscholar.org/paper/The-Design-Space%3A-the-design-process-as-the-and-of-Heape/40b912badea3b575a8f4bde95df4f83a4427ab78
  35. Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 159–166. https://doi.org/10.1145/302979.303030
  36. Irving Lester Janis. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. (1982). https://www.scirp.org/(S(351jmbntvnsjt1aadkposzje))/reference/ReferencesPapers.aspx?ReferenceID=2122583
  37. David G Jansson and Steven M Smith. 1991. Design fixation. Design studies 12, 1 (1991), 3–11. https://doi.org/10.1017/S0890060414000043
  38. Promptmaker: Prompt-based prototyping with large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–8. https://doi.org/10.1145/3491101.3503564
  39. Martin Jonsson and Jakob Tholander. 2022. Cracking the code: Co-coding with AI in creative programming education. In Proceedings of the 14th Conference on Creativity and Cognition. 5–14. https://doi.org/10.1145/3527927.3532801
  40. Metaphorian: Leveraging Large Language Models to Support Extended Metaphor Creation for Science Writing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. 115–135. https://doi.org/10.1145/3563657.3595996
  41. John Kirwan. 2017. It’s good to have lots of bad ideas. Nature 548, 7668 (2017), 491–491. https://doi.org/10.1038/nj7668-491a
  42. Drawing with Reframer: Emergence and Control in Co-Creative AI. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 264–277. https://doi.org/10.1145/3581641.3584095
  43. Joseph CR Licklider. 1960. Man-computer symbiosis. IRE transactions on human factors in electronics 1 (1960), 4–11. https://doi.org/10.1109/THFE2.1960.4503259
  44. Opal: Multimodal image generation for news illustration. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–17. https://doi.org/10.1145/3526113.3545621
  45. Design space cards: using a card deck to navigate the design space of interactive play. Proceedings of the ACM on Human-Computer Interaction 5, CHI PLAY (2021), 1–21. https://doi.org/10.1145/3474654
  46. Novice-AI music co-creation via AI-steering tools for deep generative models. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–13. https://doi.org/10.1145/3313831.3376739
  47. Bridging the Gap between UX Practitioners’ work practices and AI-enabled design support tools. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–7. https://doi.org/10.1145/3491101.3519809
  48. Exploring high-D spaces with multiform matrices and small multiples. In IEEE Symposium on Information Visualization 2003 (IEEE Cat. No. 03TH8714). IEEE, 31–38. https://doi.org/10.1109/INFVIS.2003.1249006
  49. Design space analysis: Bridging from theory to practice via design rationale. Proceedings of Esprit (1991). https://doi.org/10.1016/0142-694X(94)90026-4
  50. Dimensional reasoning and research design spaces. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition. 367–379. https://doi.org/10.1145/3059454.3059472
  51. Gary Marchionini. 2006. Exploratory search: from finding to understanding. Commun. ACM 49, 4 (2006), 41–46. https://doi.org/10.1145/1121949.1121979
  52. Tiles: a card-based ideation toolkit for the internet of things. In Proceedings of the 2017 conference on designing interactive systems. 587–598. https://doi.org/10.1145/3064663.3064699
  53. Jeyakumar Muthukumarasamy and John T Stasko. 1995. Visualizing program executions on large data sets using semantic zooming. https://doi.org/10.5555/832277.834333
  54. Alex F Osborn. 1953. Applied imagination. (1953). https://archive.org/details/appliedimaginati00osborich
  55. Donald A Schön. 1992. Designing as reflective conversation with the materials of a design situation. Knowledge-based systems 5, 1 (1992), 3–14. https://doi.org/10.1016/0950-7051(92)90020-G
  56. mSpace: improving information access to multimedia domains with multimodal exploratory search. Commun. ACM 49, 4 (2006), 47–49. https://doi.org/10.1145/1121949.1121980
  57. Mary Shaw. 2011. The role of design spaces. IEEE software 29, 1 (2011), 46–50. https://doi.org/10.1109/MS.2011.121
  58. PrivacyToon: Concept-driven Storytelling with Creativity Support for Privacy Concepts. In Designing Interactive Systems Conference. 41–57. https://doi.org/10.1145/3532106.3533557
  59. Coding strip: A pedagogical tool for teaching and learning programming concepts through comics. In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 1–10. https://doi.org/10.1109/VL/HCC50065.2020.9127262
  60. Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. In The 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3586183.3606756
  61. Codetoon: Story ideation, auto comic generation, and structure mapping for code-driven storytelling. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–16.
  62. An aspectual interface for supporting complex search tasks. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 379–386. https://doi.org/10.1145/1571941.1572007
  63. ReelFramer: Co-creating News Reels on Social Media with Generative AI. arXiv preprint arXiv:2304.09653 (2023). https://doi.org/10.48550/arXiv.2304.09653
  64. Bo Westerlund. 2005. Design space conceptual tool–grasping the design process. Nordes 1 (2005). https://doi.org/10.21606/nordes.2005.048
  65. Mikael Wiberg and Erik Stolterman. 2014. What makes a prototype novel? A knowledge contribution concern for interaction design research. In Proceedings of the 8th Nordic conference on human-computer interaction: fun, fast, foundational. 531–540. https://doi.org/10.1145/2639189.2639487
  66. The role of creative thinking in children’s scientific reasoning. Thinking Skills and Creativity 49 (2023), 101375. https://doi.org/10.1016/j.tsc.2023.101375
  67. AI creativity and the human-AI co-creation model. In Human-Computer Interaction. Theory, Methods and Tools: Thematic Area, HCI 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24–29, 2021, Proceedings, Part I 23. Springer, 171–190. https://doi.org/10.1007/978-3-030-78462-1_13
  68. AI as an Active Writer: Interaction strategies with generated text in human-AI collaborative fiction writing. In Joint Proceedings of the ACM IUI Workshops. https://hai-gen.github.io/2022/papers/paper-HAIGEN-YangDaijin.pdf
  69. Wordcraft: story writing with large language models. In 27th International Conference on Intelligent User Interfaces. 841–852. https://doi.org/10.1145/3490099.3511105
  70. Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21. https://doi.org/10.1145/3544548.3581388
  71. VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping. arXiv preprint arXiv:2304.07810 (2023). https://doi.org/10.48550/arXiv.2304.07810
User Edit Pencil Streamline Icon: https://streamlinehq.com
Authors (5)
  1. Sangho Suh (9 papers)
  2. Meng Chen (98 papers)
  3. Bryan Min (5 papers)
  4. Toby Jia-Jun Li (57 papers)
  5. Haijun Xia (24 papers)
Citations (13)
Github Logo Streamline Icon: https://streamlinehq.com

GitHub

X Twitter Logo Streamline Icon: https://streamlinehq.com