Papers
Topics
Authors
Recent
2000 character limit reached

DrawTalking: Building Interactive Worlds by Sketching and Speaking (2401.05631v4)

Published 11 Jan 2024 in cs.HC, cs.AI, cs.CL, cs.GR, and cs.ET

Abstract: We introduce DrawTalking, an approach to building and controlling interactive worlds by sketching and speaking while telling stories. It emphasizes user control and flexibility, and gives programming-like capability without requiring code. An early open-ended study with our prototype shows that the mechanics resonate and are applicable to many creative-exploratory use cases, with the potential to inspire and inform research in future natural interfaces for creative exploration and authoring.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (33)
  1. 2022. 50 Years of Fun With Pong. https://computerhistory.org/blog/50-years-of-fun-with-pong/ Section: Curatorial Insights.
  2. Design principles for visual communication. Commun. ACM 54, 4 (April 2011), 60–69. https://doi.org/10.1145/1924421.1924439
  3. Guidelines for Human-AI Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–13. https://doi.org/10.1145/3290605.3300233
  4. AtariAdmin. 2022. New Insight into Breakout’s Origins. https://recharged.atari.com/the-origins-of-breakout/
  5. Richard A. Bolt. 1980. Put-that-there: Voice and gesture at the graphics interface. ACM SIGGRAPH Computer Graphics 14, 3 (July 1980), 262–270. https://doi.org/10.1145/965105.807503
  6. K. Compton and M. Mateas. 2015. Casual Creators. https://www.semanticscholar.org/paper/Casual-Creators-Compton-Mateas/f9add8f5126faab72c9cc591b5fdc7e712936b56
  7. cycling74. 2018. Max. https://cycling74/products/max
  8. Dreams 2020. Dreams. [Playstation 4]. https://www.mediamolecule.com/games/dreams https://indreams.me.
  9. Drawing as a versatile cognitive tool. Nature Reviews Psychology 2, 9 (Sept. 2023), 556–568. https://doi.org/10.1038/s44159-023-00212-w Number: 9 Publisher: Nature Publishing Group.
  10. Scones: towards conversational authoring of sketches. In Proceedings of the 25th International Conference on Intelligent User Interfaces (IUI ’20). Association for Computing Machinery, New York, NY, USA, 313–323. https://doi.org/10.1145/3377325.3377485
  11. Kitty: sketching dynamic and interactive illustrations. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST ’14). Association for Computing Machinery, New York, NY, USA, 395–405. https://doi.org/10.1145/2642918.2647375
  12. James A. Landay. 1996. SILK: sketching interfaces like krazy. In Conference Companion on Human Factors in Computing Systems (CHI ’96). Association for Computing Machinery, New York, NY, USA, 398–399. https://doi.org/10.1145/257089.257396
  13. PixelTone: a multimodal interface for image editing. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI ’13). Association for Computing Machinery, New York, NY, USA, 2185–2194. https://doi.org/10.1145/2470654.2481301
  14. Little Big Planet 2 2011. Little Big Planet 2. [Playstation 3]. https://www.mediamolecule.com/games/littlebigplanet2
  15. Visual Captions: Augmenting Verbal Communication with On-the-fly Visuals. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (CHI).
  16. explosion/spaCy: v3.5.0: New CLI commands, language updates, bug fixes and much more. https://doi.org/10.5281/zenodo.7553910
  17. Dan R. Olsen. 2007. Evaluating user interface systems research. In Proceedings of the 20th annual ACM symposium on User interface software and technology (UIST ’07). Association for Computing Machinery, New York, NY, USA, 251–258. https://doi.org/10.1145/1294211.1294256
  18. Sharon Oviatt. 1999. Ten myths of multimodal interaction. Commun. ACM 42, 11 (Nov. 1999), 74–81. https://doi.org/10.1145/319382.319398
  19. Ken Perlin and Athomas Goldberg. 1996. Improv: a system for scripting interactive actors in virtual worlds. In Proceedings of the 23rd annual conference on Computer graphics and interactive techniques (SIGGRAPH ’96). Association for Computing Machinery, New York, NY, USA, 205–216. https://doi.org/10.1145/237170.237258
  20. Chalktalk : A Visualization and Communication Language – As a Tool in the Domain of Computer Science Education. https://doi.org/10.48550/arXiv.1809.07166 arXiv:1809.07166 [cs].
  21. Scratch: programming for all. Commun. ACM 52, 11 (Nov. 2009), 60–67. https://doi.org/10.1145/1592761.1592779
  22. Constructing Embodied Algebra by Sketching. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (CHI ’21). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3411764.3445460
  23. Andreea Stapleton. 2017. Deixis in Modern Linguistics. Essex Student Journal 9, 1 (Jan. 2017). https://doi.org/10.5526/esj23 Number: 1 Publisher: University of Essex Library Services.
  24. TakeToons: Script-driven Performance Animation. In Proceedings of the 31st Annual ACM Symposium on User Interface Software and Technology (UIST ’18). Association for Computing Machinery, New York, NY, USA, 663–674. https://doi.org/10.1145/3242587.3242618
  25. Ivan E. Sutherland. 1963. Sketchpad: a man-machine graphical communication system. In Proceedings of the May 21-23, 1963, spring joint computer conference (AFIPS ’63 (Spring)). Association for Computing Machinery, New York, NY, USA, 329–346. https://doi.org/10.1145/1461551.1461591
  26. William Robert Sutherland. 1966. The on-line graphical specification of computer procedures. Thesis. Massachusetts Institute of Technology. https://dspace.mit.edu/handle/1721.1/13474 Accepted: 2005-09-21T22:40:43Z.
  27. RealitySketch: Embedding Responsive Graphics and Visualizations in AR through Dynamic Sketching. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST ’20). Association for Computing Machinery, New York, NY, USA, 166–181. https://doi.org/10.1145/3379337.3415892
  28. Phil Turner. 2016. A Make-Believe Narrative for HCI. In Digital Make-Believe, Phil Turner and J. Tuomas Harviainen (Eds.). Springer International Publishing, Cham, 11–26. https://doi.org/10.1007/978-3-319-29553-4_2
  29. Barbara Tversky. 2011. Visualizing Thought. Topics in Cognitive Science 3, 3 (2011), 499–535. https://doi.org/10.1111/j.1756-8765.2010.01113.x _eprint: https://onlinelibrary.wiley.com/doi/pdf/10.1111/j.1756-8765.2010.01113.x.
  30. Bret Victor. 2014. Humane representation of thought: a trail map for the 21st century. In Proceedings of the 27th annual ACM symposium on User interface software and technology (UIST ’14). Association for Computing Machinery, New York, NY, USA, 699. https://doi.org/10.1145/2642918.2642920
  31. Terry Winograd. 1972. Understanding natural language. Cognitive Psychology 3, 1 (Jan. 1972), 1–191. https://doi.org/10.1016/0010-0285(72)90002-3
  32. Haijun Xia. 2020. Crosspower: Bridging Graphics and Linguistics. In Proceedings of the 33rd Annual ACM Symposium on User Interface Software and Technology (UIST ’20). Association for Computing Machinery, New York, NY, USA, 722–734. https://doi.org/10.1145/3379337.3415845
  33. CrossTalk: Intelligent Substrates for Language-Oriented Interaction in Video-Based Communication and Collaboration. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology (UIST ’23). Association for Computing Machinery, New York, NY, USA, 1–16. https://doi.org/10.1145/3586183.3606773

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
Dice Question Streamline Icon: https://streamlinehq.com

Open Problems

We haven't generated a list of open problems mentioned in this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
Lightbulb Streamline Icon: https://streamlinehq.com

Continue Learning

We haven't generated follow-up questions for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Generate Now
List To Do Tasks Checklist Streamline Icon: https://streamlinehq.com

Collections

Sign up for free to add this paper to one or more collections.

User Circle Single Streamline Icon: https://streamlinehq.com Sign Up
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets

This paper has been mentioned in 3 tweets and received 0 likes.

Upgrade to Pro to view all of the tweets about this paper:

Start a free 7-day Pro trial