Luminate: Structured Generation and Exploration of Design Space with Large Language Models for Human-AI Co-Creation (2310.12953v3)
Abstract: Thanks to their generative capabilities, LLMs have become an invaluable tool for creative processes. These models have the capacity to produce hundreds and thousands of visual and textual outputs, offering abundant inspiration for creative endeavors. But are we harnessing their full potential? We argue that current interaction paradigms fall short, guiding users towards rapid convergence on a limited set of ideas, rather than empowering them to explore the vast latent design space in generative models. To address this limitation, we propose a framework that facilitates the structured generation of design space in which users can seamlessly explore, evaluate, and synthesize a multitude of responses. We demonstrate the feasibility and usefulness of this framework through the design and development of an interactive system, Luminate, and a user study with 14 professional writers. Our work advances how we interact with LLMs for creative tasks, introducing a way to harness the creative potential of LLMs.
- ]futurepedia [n. d.]. Futurepedia. https://www.futurepedia.io/ Last accessed 27 August 2023.
- ]upwork [n. d.]. Upwork. https://www.upwork.com/. Accessed: September 13, 2023.
- Christopher Ahlberg and Ben Shneiderman. 1994. Visual information seeking: Tight coupling of dynamic query filters with starfield displays. In Proceedings of the SIGCHI conference on Human factors in computing systems. 313–317. https://doi.org/10.1145/191666.191775
- Guidelines for human-AI interaction. In Proceedings of the 2019 chi conference on human factors in computing systems. 1–13. https://doi.org/10.1145/3290605.3300233
- Design patterns for data comics. In Proceedings of the 2018 chi conference on human factors in computing systems. 1–12. https://doi.org/10.1145/3173574.3173612
- Generative Theories of Interaction. ACM Transactions on Computer-Human Interaction (TOCHI) 28, 6 (2021), 1–54. https://doi.org/10.1145/3468505
- Michel Beaudouin-Lafon and Wendy E Mackay. 2007. Prototyping tools and techniques. In The human-computer interaction handbook. CRC Press, 1043–1066. https://www.kth.se/social/upload/52ef5ee4f2765445a466a28a/mackay-lafon-prototypes-52-HCI.pdf
- Benjamin B Bederson and James D Hollan. 1994. Pad++ a zooming graphical interface for exploring alternate interface physics. In Proceedings of the 7th annual ACM symposium on User interface software and technology. 17–26. https://doi.org/10.1145/192426.192435
- Graphdice: A system for exploring multivariate social networks. In Computer graphics forum, Vol. 29. Wiley Online Library, 863–872.
- A constraint-based understanding of design spaces. In Proceedings of the 2014 conference on Designing interactive systems. 453–462. https://doi.org/10.1145/2598510.2598533
- Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Models. arXiv preprint arXiv:2304.09337 (2023). https://doi.org/10.48550/arXiv.2304.09337
- Dennis R Brophy. 2001. Comparing the attributes, activities, and performance of divergent, convergent, and combination thinkers. Creativity research journal 13, 3-4 (2001), 439–455. https://doi.org/10.1207/S15326934CRJ1334_20
- Bill Buxton. 2010. Sketching user experiences: getting the design right and the right design. https://doi.org/10.1016/B978-0-12-374037-3.X5043-3
- Readings in information visualization: using vision to think. Morgan Kaufmann. https://doi.org/10.5555/300679
- Creativity factor evaluation: towards a standardized survey metric for creativity support. In Proceedings of the seventh ACM conference on Creativity and cognition. 127–136.
- Mapping the design space of human-ai interaction in text summarization. arXiv preprint arXiv:2206.14863 (2022). https://doi.org/10.18653/v1/2022.naacl-main.33
- Erin Cherry and Celine Latulipe. 2014. Quantifying the creativity support of digital tools through the creativity support index. ACM Transactions on Computer-Human Interaction (TOCHI) 21, 4 (2014), 1–25. https://doi.org/10.1145/2617588
- Jarry HT Claessen and Jarke J Van Wijk. 2011. Flexible linked axes for multivariate data visualization. IEEE Transactions on Visualization and Computer Graphics 17, 12 (2011), 2310–2316. https://doi.org/10.1109/TVCG.2011.201
- Nigel Cross. 2004. Expertise in design: an overview. Design studies 25, 5 (2004), 427–441. https://doi.org/10.1016/j.destud.2004.06.002
- Beyond text generation: Supporting writers with continuous automatic text summaries. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–13. https://doi.org/10.48550/arXiv.2208.09323
- Edward De Bono. 1970. Lateral thinking. New York (1970), 70. https://www.kioulanis.gr/rivips/images/Lateral_thinking.pdf
- Zijian Ding and Joel Chan. 2023. Mapping the Design Space of Interactions in Human-AI Text Co-creation Tasks. arXiv e-prints (2023), arXiv–2303. https://doi.org/10.48550/arXiv.2303.06430
- An argument for design space reflection. In Proceedings of the 9th Nordic Conference on Human-Computer Interaction. 1–10. https://doi.org/10.1145/2971485.2971528
- Parallel prototyping leads to better design results, more divergence, and increased self-efficacy. ACM Transactions on Computer-Human Interaction (TOCHI) 17, 4 (2010), 1–24. https://doi.org/10.1145/1879831.1879836
- Semantic interaction for sensemaking: inferring analytical reasoning for model steering. IEEE Transactions on Visualization and Computer Graphics 18, 12 (2012), 2879–2888. https://doi.org/10.1109/TVCG.2012.260
- PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation. arXiv preprint arXiv:2307.09036 (2023). https://doi.org/10.48550/arXiv.2307.09036
- Sparks: Inspiration for science writing using language models. In Designing interactive systems conference. 1002–1019. https://doi.org/10.1145/3532106.3533533
- Exploring Challenges and Opportunities to Support Designers in Learning to Co-create with AI-based Manufacturing Design Tools. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–20. https://doi.org/10.1145/3544548.3580999
- Michael Golembewski and Mark Selby. 2010. Ideation decks: a card-based design ideation tool. In Proceedings of the 8th ACM Conference on Designing Interactive Systems. 89–92. https://doi.org/10.1145/1858171.1858189
- Joy Paul Guilford. 1961. Three faces of intellect. (1961). https://doi.org/10.1037/h0046827
- Joy Paul Guilford. 1967. The nature of human intelligence. (1967). https://doi.org/10.1017/9781316817049
- Kim Halskov and Caroline Lundqvist. 2021. Filtering and informing the design space: Towards design-space thinking. ACM Transactions on Computer-Human Interaction (TOCHI) 28, 1 (2021), 1–28. https://doi.org/10.1145/3434462
- CrossCode: Multi-level Visualization of Program Execution. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–13. https://doi.org/10.1145/3544548.3581390
- Chris Heape. 2007. The Design Space: the design process as the construction, exploration and expansion of a conceptual space. (2007). https://www.semanticscholar.org/paper/The-Design-Space%3A-the-design-process-as-the-and-of-Heape/40b912badea3b575a8f4bde95df4f83a4427ab78
- Eric Horvitz. 1999. Principles of mixed-initiative user interfaces. In Proceedings of the SIGCHI conference on Human Factors in Computing Systems. 159–166. https://doi.org/10.1145/302979.303030
- Irving Lester Janis. 1982. Groupthink: Psychological studies of policy decisions and fiascoes. (1982). https://www.scirp.org/(S(351jmbntvnsjt1aadkposzje))/reference/ReferencesPapers.aspx?ReferenceID=2122583
- David G Jansson and Steven M Smith. 1991. Design fixation. Design studies 12, 1 (1991), 3–11. https://doi.org/10.1017/S0890060414000043
- Promptmaker: Prompt-based prototyping with large language models. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–8. https://doi.org/10.1145/3491101.3503564
- Martin Jonsson and Jakob Tholander. 2022. Cracking the code: Co-coding with AI in creative programming education. In Proceedings of the 14th Conference on Creativity and Cognition. 5–14. https://doi.org/10.1145/3527927.3532801
- Metaphorian: Leveraging Large Language Models to Support Extended Metaphor Creation for Science Writing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. 115–135. https://doi.org/10.1145/3563657.3595996
- John Kirwan. 2017. It’s good to have lots of bad ideas. Nature 548, 7668 (2017), 491–491. https://doi.org/10.1038/nj7668-491a
- Drawing with Reframer: Emergence and Control in Co-Creative AI. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 264–277. https://doi.org/10.1145/3581641.3584095
- Joseph CR Licklider. 1960. Man-computer symbiosis. IRE transactions on human factors in electronics 1 (1960), 4–11. https://doi.org/10.1109/THFE2.1960.4503259
- Opal: Multimodal image generation for news illustration. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–17. https://doi.org/10.1145/3526113.3545621
- Design space cards: using a card deck to navigate the design space of interactive play. Proceedings of the ACM on Human-Computer Interaction 5, CHI PLAY (2021), 1–21. https://doi.org/10.1145/3474654
- Novice-AI music co-creation via AI-steering tools for deep generative models. In Proceedings of the 2020 CHI conference on human factors in computing systems. 1–13. https://doi.org/10.1145/3313831.3376739
- Bridging the Gap between UX Practitioners’ work practices and AI-enabled design support tools. In CHI Conference on Human Factors in Computing Systems Extended Abstracts. 1–7. https://doi.org/10.1145/3491101.3519809
- Exploring high-D spaces with multiform matrices and small multiples. In IEEE Symposium on Information Visualization 2003 (IEEE Cat. No. 03TH8714). IEEE, 31–38. https://doi.org/10.1109/INFVIS.2003.1249006
- Design space analysis: Bridging from theory to practice via design rationale. Proceedings of Esprit (1991). https://doi.org/10.1016/0142-694X(94)90026-4
- Dimensional reasoning and research design spaces. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition. 367–379. https://doi.org/10.1145/3059454.3059472
- Gary Marchionini. 2006. Exploratory search: from finding to understanding. Commun. ACM 49, 4 (2006), 41–46. https://doi.org/10.1145/1121949.1121979
- Tiles: a card-based ideation toolkit for the internet of things. In Proceedings of the 2017 conference on designing interactive systems. 587–598. https://doi.org/10.1145/3064663.3064699
- Jeyakumar Muthukumarasamy and John T Stasko. 1995. Visualizing program executions on large data sets using semantic zooming. https://doi.org/10.5555/832277.834333
- Alex F Osborn. 1953. Applied imagination. (1953). https://archive.org/details/appliedimaginati00osborich
- Donald A Schön. 1992. Designing as reflective conversation with the materials of a design situation. Knowledge-based systems 5, 1 (1992), 3–14. https://doi.org/10.1016/0950-7051(92)90020-G
- mSpace: improving information access to multimedia domains with multimodal exploratory search. Commun. ACM 49, 4 (2006), 47–49. https://doi.org/10.1145/1121949.1121980
- Mary Shaw. 2011. The role of design spaces. IEEE software 29, 1 (2011), 46–50. https://doi.org/10.1109/MS.2011.121
- PrivacyToon: Concept-driven Storytelling with Creativity Support for Privacy Concepts. In Designing Interactive Systems Conference. 41–57. https://doi.org/10.1145/3532106.3533557
- Coding strip: A pedagogical tool for teaching and learning programming concepts through comics. In 2020 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 1–10. https://doi.org/10.1109/VL/HCC50065.2020.9127262
- Sensecape: Enabling Multilevel Exploration and Sensemaking with Large Language Models. In The 36th Annual ACM Symposium on User Interface Software and Technology (San Francisco, CA, USA) (UIST ’23). Association for Computing Machinery, New York, NY, USA. https://doi.org/10.1145/3586183.3606756
- Codetoon: Story ideation, auto comic generation, and structure mapping for code-driven storytelling. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology. 1–16.
- An aspectual interface for supporting complex search tasks. In Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval. 379–386. https://doi.org/10.1145/1571941.1572007
- ReelFramer: Co-creating News Reels on Social Media with Generative AI. arXiv preprint arXiv:2304.09653 (2023). https://doi.org/10.48550/arXiv.2304.09653
- Bo Westerlund. 2005. Design space conceptual tool–grasping the design process. Nordes 1 (2005). https://doi.org/10.21606/nordes.2005.048
- Mikael Wiberg and Erik Stolterman. 2014. What makes a prototype novel? A knowledge contribution concern for interaction design research. In Proceedings of the 8th Nordic conference on human-computer interaction: fun, fast, foundational. 531–540. https://doi.org/10.1145/2639189.2639487
- The role of creative thinking in children’s scientific reasoning. Thinking Skills and Creativity 49 (2023), 101375. https://doi.org/10.1016/j.tsc.2023.101375
- AI creativity and the human-AI co-creation model. In Human-Computer Interaction. Theory, Methods and Tools: Thematic Area, HCI 2021, Held as Part of the 23rd HCI International Conference, HCII 2021, Virtual Event, July 24–29, 2021, Proceedings, Part I 23. Springer, 171–190. https://doi.org/10.1007/978-3-030-78462-1_13
- AI as an Active Writer: Interaction strategies with generated text in human-AI collaborative fiction writing. In Joint Proceedings of the ACM IUI Workshops. https://hai-gen.github.io/2022/papers/paper-HAIGEN-YangDaijin.pdf
- Wordcraft: story writing with large language models. In 27th International Conference on Intelligent User Interfaces. 841–852. https://doi.org/10.1145/3490099.3511105
- Why Johnny can’t prompt: how non-AI experts try (and fail) to design LLM prompts. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 1–21. https://doi.org/10.1145/3544548.3581388
- VISAR: A Human-AI Argumentative Writing Assistant with Visual Programming and Rapid Draft Prototyping. arXiv preprint arXiv:2304.07810 (2023). https://doi.org/10.48550/arXiv.2304.07810
- Sangho Suh (9 papers)
- Meng Chen (98 papers)
- Bryan Min (5 papers)
- Toby Jia-Jun Li (57 papers)
- Haijun Xia (24 papers)