Papers
Topics
Authors
Recent
Gemini 2.5 Flash
Gemini 2.5 Flash
126 tokens/sec
GPT-4o
47 tokens/sec
Gemini 2.5 Pro Pro
43 tokens/sec
o3 Pro
4 tokens/sec
GPT-4.1 Pro
47 tokens/sec
DeepSeek R1 via Azure Pro
28 tokens/sec
2000 character limit reached

DanceGen: Supporting Choreography Ideation and Prototyping with Generative AI (2405.17827v1)

Published 28 May 2024 in cs.HC

Abstract: Choreography creation requires high proficiency in artistic and technical skills. Choreographers typically go through four stages to create a dance piece: preparation, studio, performance, and reflection. This process is often individualized, complicated, and challenging due to multiple constraints at each stage. To assist choreographers, most prior work has focused on designing digital tools to support the last three stages of the choreography process, with the preparation stage being the least explored. To address this research gap, we introduce an AI-based approach to assist the preparation stage by supporting ideation, creating choreographic prototypes, and documenting creative attempts and outcomes. We address the limitations of existing AI-based motion generation methods for ideation by allowing generated sequences to be edited and modified in an interactive web interface. This capability is motivated by insights from a formative study we conducted with seven choreographers. We evaluated our system's functionality, benefits, and limitations with six expert choreographers. Results highlight the usability of our system, with users reporting increased efficiency, expanded creative possibilities, and an enhanced iterative process. We also identified areas for improvement, such as the relationship between user intent and AI outcome, intuitive and flexible user interaction design, and integration with existing physical choreography prototyping workflows. By reflecting on the evaluation results, we present three insights that aim to inform the development of future AI systems that can empower choreographers.

Definition Search Book Streamline Icon: https://streamlinehq.com
References (88)
  1. Adobe. 2024. Mixamo. Adobe. https://www.mixamo.com/#/
  2. Choreography as mediated through compositional tools for movement: Constructing a historical perspective. In Proceedings of the 2014 International Workshop on Movement and Computing. Association for Computing Machinery, New York, NY, USA, 1–6.
  3. Listen, Denoise, Action! Audio-Driven Motion Synthesis with Diffusion Models. ACM Trans. Graph. 42, 4, Article 44 (jul 2023), 20 pages. https://doi.org/10.1145/3592458
  4. Wearable Choreographer: Designing Soft-Robotics for Dance Practice. In Proceedings of the 2022 ACM Designing Interactive Systems Conference (¡conf-loc¿, ¡city¿Virtual Event¡/city¿, ¡country¿Australia¡/country¿, ¡/conf-loc¿) (DIS ’22). Association for Computing Machinery, New York, NY, USA, 1581–1596. https://doi.org/10.1145/3532106.3533499
  5. Three-dimensional visualization of movement qualities in contemporary dance. In Proceedings of the 5th international conference on movement and computing. Association for Computing Machinery, New York, NY, USA, 1–7.
  6. Jeremy N. Bailenson. 2021. Nonverbal overload: A theoretical argument for the causes of Zoom fatigue. Technology, Mind, and Behavior 2, 1 (feb 23 2021), 1–6. https://tmb.apaopen.org/pub/nonverbal-overload.
  7. Blender. 2024. Blender. Blender. https://www.blender.org/
  8. Virginia Braun and Victoria Clarke. 2006. Using thematic analysis in psychology. Qualitative research in psychology 3, 2 (2006), 77–101.
  9. Virginia Braun and Victoria Clarke. 2012. Thematic analysis. American Psychological Association, Apple Valley, NM, USA.
  10. Virginia Braun and Victoria Clarke. 2019. Reflecting on reflexive thematic analysis. Qualitative research in sport, exercise and health 11, 4 (2019), 589–597.
  11. Composition of multiple figure sequences for dance and animation. In New Advances in Computer Graphics: Proceedings of CG International’89. Springer, Springer, New York, NY, USA, 245–255.
  12. The Dancer in the Eye: Towards a Multi-Layered Computational Framework of Qualities in Movement. In Proceedings of the 3rd International Symposium on Movement and Computing (Thessaloniki, GA, Greece) (MOCO ’16). Association for Computing Machinery, New York, NY, USA, Article 6, 7 pages. https://doi.org/10.1145/2948910.2948927
  13. Shifting Spaces: Using Defamiliarization to Design Choreographic Technologies That Support Co-Creation. In Proceedings of the 6th International Conference on Movement and Computing (Tempe, AZ, USA) (MOCO ’19). Association for Computing Machinery, New York, NY, USA, Article 17, 8 pages. https://doi.org/10.1145/3347122.3347140
  14. Moment by Moment: Creating Movement Sketches with Camera Stillframes. In Proceedings of the 2015 ACM SIGCHI Conference on Creativity and Cognition (Glasgow, United Kingdom) (C&C ’15). Association for Computing Machinery, New York, NY, USA, 131–140. https://doi.org/10.1145/2757226.2757237
  15. Sketching Movement: Designing Creativity Tools for in-Situ, Whole-Body Authorship. In Proceedings of the 2nd International Workshop on Movement and Computing (Vancouver, British Columbia, Canada) (MOCO ’15). Association for Computing Machinery, New York, NY, USA, 68–75. https://doi.org/10.1145/2790994.2791007
  16. Everybody dance now. In Proceedings of the IEEE/CVF international conference on computer vision. IEEE, New York, NY, USA, 5933–5942.
  17. ILVR: Conditioning Method for Denoising Diffusion Probabilistic Models. arXiv:2108.02938 [cs.CV]
  18. How Do Choreographers Craft Dance? Designing for a Choreographer-Technology Partnership. In Proceedings of the 3rd International Symposium on Movement and Computing (Thessaloniki, GA, Greece) (MOCO ’16). Association for Computing Machinery, New York, NY, USA, Article 20, 8 pages. https://doi.org/10.1145/2948910.2948941
  19. Knotation: exploring and documenting choreographic processes. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–12.
  20. Dancing With Drones: Crafting Novel Artistic Expressions Through Intercorporeality. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk) (CHI ’19). Association for Computing Machinery, New York, NY, USA, 1–12. https://doi.org/10.1145/3290605.3300847
  21. XRgonomics: facilitating the creation of ergonomic 3D interfaces. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–11. https://doi.org/10.1145/3411764.3445349
  22. How do experts observe movement?. In Proceedings of the 2nd International Workshop on Movement and Computing. Association for Computing Machinery, New York, NY, USA, 84–91.
  23. Seeing, Sensing and Recognizing Laban Movement Qualities. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 4009–4020. https://doi.org/10.1145/3025453.3025530
  24. Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio. Computers & Graphics 94 (2021), 11–21.
  25. CO/DA: Live-Coding Movement-Sound Interactions for Dance Improvisation. In Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems (¡conf-loc¿, ¡city¿New Orleans¡/city¿, ¡state¿LA¡/state¿, ¡country¿USA¡/country¿, ¡/conf-loc¿) (CHI ’22). Association for Computing Machinery, New York, NY, USA, Article 482, 13 pages. https://doi.org/10.1145/3491102.3501916
  26. Petra Gemeinboeck and Rob Saunders. 2017. Movement Matters: How a Robot Becomes Body. In Proceedings of the 4th International Conference on Movement Computing (London, United Kingdom) (MOCO ’17). Association for Computing Machinery, New York, NY, USA, Article 8, 8 pages. https://doi.org/10.1145/3077981.3078035
  27. Elizabeth Gerber. 2007. Improvisation principles and techniques for design. In Proceedings of the SIGCHI conference on Human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 1069–1072.
  28. Tm2d: Bimodality driven 3d dance generation via music-text integration. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, New York, NY, USA, 9942–9952.
  29. Generating Diverse and Natural 3D Human Motions From Text. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, New York, NY, USA, 5152–5161.
  30. Leva Janauskaitundefined and George Palamas. 2019. Establishing Dialogues Between Movement and Atmospheric Ambiances. In Proceedings of the 6th International Conference on Movement and Computing (Tempe, AZ, USA) (MOCO ’19). Association for Computing Machinery, New York, NY, USA, Article 28, 11 pages. https://doi.org/10.1145/3347122.3359602
  31. MotionGPT: Human Motion as a Foreign Language. arXiv:2306.14795 [cs.CV]
  32. Elizabeth Jochum and Jeroen Derks. 2019. Tonight We Improvise! Real-Time Tracking for Human-Robot Improvisational Dance. In Proceedings of the 6th International Conference on Movement and Computing (Tempe, AZ, USA) (MOCO ’19). Association for Computing Machinery, New York, NY, USA, Article 7, 11 pages. https://doi.org/10.1145/3347122.3347129
  33. Andrew Johnston. 2015. Conceptualising Interaction in Live Performance: Reflections on ’Encoded’. In Proceedings of the 2nd International Workshop on Movement and Computing (Vancouver, British Columbia, Canada) (MOCO ’15). Association for Computing Machinery, New York, NY, USA, 60–67. https://doi.org/10.1145/2790994.2791003
  34. Reinterpreting Schlemmer’s Triadic Ballet: interactive costume for unthinkable movements. In Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–13.
  35. From Skin to Skeleton: Towards Biomechanically Accurate 3D Digital Humans. ACM Trans. Graph. 42, 6, Article 253 (dec 2023), 12 pages. https://doi.org/10.1145/3618381
  36. Choreographic methods for creating novel, high quality dance. In Proceedings, DESFORM 5th international workshop on design & semantics & form. DeSForM, Northumbria University, 188–195.
  37. Vibe: Video inference for human body pose and shape estimation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. IEEE, New York, NY, USA, 5253–5263.
  38. Nathan Kogan. 2002. Careers in the performing arts: A psychological perspective. Communication Research Journal 14, 1 (2002), 1–16.
  39. Rudolf von Laban. 1975. Modern educational dance / by Rudolf Laban. (3rd ed. / revised with additions by lisa ullmann. ed.). Macdonald and Evans, Braintree, MA, USA.
  40. Live dance performance investigating the feminine cyborg metaphor with a motion-activated wearable robot. In Proceedings of the 2020 ACM/IEEE international conference on human-robot interaction. Association for Computing Machinery, New York, NY, USA, 243–251.
  41. When is a Tool a Tool? User Perceptions of System Agency in Human–AI Co-Creative Drawing. In Proceedings of the 2023 ACM Designing Interactive Systems Conference. Association for Computing Machinery, New York, NY, USA, 1978–1996.
  42. Dancing to Music. In Advances in Neural Information Processing Systems, H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (Eds.), Vol. 32. Curran Associates, Inc., New York, NY, USA. https://proceedings.neurips.cc/paper_files/paper/2019/file/7ca57a9f85a19a6e4b9a248c1daca185-Paper.pdf
  43. Learning to Generate Diverse Dance Motions with Transformer. arXiv:2008.08171 [cs.CV]
  44. AI Choreographer: Music Conditioned 3D Dance Generation With AIST++. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, New York, NY, USA, 13401–13412.
  45. FineDance: A Fine-grained Choreography Dataset for 3D Full Body Dance Generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, New York, NY, USA, 10234–10243.
  46. Yimeng Liu. 2024. Interaction Design for Human-AI Choreography Co-creation. arXiv:2405.03999 [cs.HC]
  47. Yimeng Liu and Misha Sra. 2021. Motion Improvisation: 3D Human Motion Synthesis with a Transformer. In Adjunct Proceedings of the 34th Annual ACM Symposium on User Interface Software and Technology (Virtual Event, USA) (UIST ’21 Adjunct). Association for Computing Machinery, New York, NY, USA, 26–28. https://doi.org/10.1145/3474349.3480219
  48. Designing for socially interactive systems. In Proceedings of the 2017 ACM SIGCHI Conference on Creativity and Cognition. Association for Computing Machinery, New York, NY, USA, 39–50.
  49. SMPL: A Skinned Multi-Person Linear Model. ACM Trans. Graphics (Proc. SIGGRAPH Asia) 34, 6 (Oct. 2015), 248:1–248:16.
  50. Takahiro Maeda and Norimichi Ukita. 2022. MotionAug: Augmentation With Physical Correction for Human Motion Prediction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, New York, NY, USA, 6427–6436.
  51. Paul H Mason. 2012. Music, dance and the total art work: choreomusicology in theory and practice. Research in dance education 13, 1 (2012), 5–24.
  52. Sandra Cerny Minton. 2017. Choreography: a basic approach using improvisation. Human Kinetics, Champaign, IL, USA.
  53. The Delay mirror: A technological innovation specific to the dance studio. In Proceedings of the 4th International Conference on Movement Computing. Association for Computing Machinery, New York, NY, USA, 1–6.
  54. Information capacity of full-body movements. In Proceedings of the SIGCHI conference on human factors in computing systems. Association for Computing Machinery, New York, NY, USA, 1289–1298.
  55. Python. 2024. HTTP servers. Python. https://docs.python.org/3/library/http.server.html
  56. DiffDance: Cascaded Human Motion Diffusion Model for Dance Generation. In Proceedings of the 31st ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, USA, 1374–1382.
  57. Single Motion Diffusion. arXiv:2302.05905 [cs.CV]
  58. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning. PMLR, PMLR, Pittsburgh, PA, USA, 8748–8763.
  59. Choreomorphy: A whole-body interaction experience for dance improvisation and visual experimentation. In Proceedings of the 2018 International Conference on Advanced Visual Interfaces. Association for Computing Machinery, New York, NY, USA, 1–9.
  60. Multitask learning for Laban movement analysis. In Proceedings of the 2nd International Workshop on Movement and Computing (Vancouver, British Columbia, Canada) (MOCO ’15). Association for Computing Machinery, New York, NY, USA, 37–44. https://doi.org/10.1145/2790994.2791009
  61. Self-supervised dance video synthesis conditioned on music. In Proceedings of the 28th ACM International Conference on Multimedia. Association for Computing Machinery, New York, NY, USA, 46–54.
  62. Jeba Rezwana and Mary Lou Maher. 2023. Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems. ACM Trans. Comput.-Hum. Interact. 30, 5, Article 67 (sep 2023), 28 pages. https://doi.org/10.1145/3519026
  63. Albert Rothenberg. 2019. The Role of Error in Creativity. https://www.psychologytoday.com/us/blog/creative-explorations/201902/the-role-error-in-creativity.
  64. Affective Movement Generation using Laban Effort and Shape and Hidden Markov Models. arXiv:2006.06071 [cs.HC]
  65. R Keith Sawyer and Stacy DeZutter. 2009. Distributed creativity: How collective creations emerge from collaboration. Psychology of aesthetics, creativity, and the arts 3, 2 (2009), 81.
  66. The choreographer’s notebook: a video annotation system for dancers and choreographers. In Proceedings of the 8th ACM Conference on Creativity and Cognition. Association for Computing Machinery, New York, NY, USA, 197–206.
  67. Bailando: 3D Dance Generation by Actor-Critic GPT With Choreographic Memory. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, New York, NY, USA, 11050–11059.
  68. Experimental Creation of Contemporary Dance Works Using a Body-part Motion Synthesis System. In Proceedings of the 8th International Conference on Movement and Computing. Association for Computing Machinery, New York, NY, USA, 1–5.
  69. Body-part motion synthesis system for contemporary dance creation. In ACM SIGGRAPH 2016 Posters. Association for Computing Machinery, New York, NY, USA, 1–2.
  70. Moving with and without music: scaling and lapsing in time in the performance of contemporary dance. Music Perception 26, 5 (2009), 451–464.
  71. DeepDance: music-to-dance motion choreography with adversarial learning. IEEE Transactions on Multimedia 23 (2020), 497–509.
  72. Unity Technologies. 2024. Unity. Unity. https://unity.com/
  73. Human Motion Diffusion Model. arXiv:2209.14916 [cs.CV]
  74. three.js. 2023. JavaScript 3D Library. https://github.com/mrdoob/three.js.
  75. Edge: Editable dance generation from music. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. IEEE, New York, NY, USA, 448–458.
  76. Barbara Tversky and Juliet Y Chou. 2011. Creativity: depth and breadth. In Design creativity 2010. Springer, Springer, New York, NY, USA, 209–214.
  77. Transflower: Probabilistic Autoregressive Dance Generation with Multimodal Attention. ACM Trans. Graph. 40, 6, Article 195 (dec 2021), 14 pages. https://doi.org/10.1145/3478513.3480570
  78. Tony Veale. 2019. From Conceptual Mash-ups to Badass Blends: A Robust Computational Model of Conceptual Blending. Springer International Publishing, Cham, 71–89. https://doi.org/10.1007/978-3-319-43610-4_4
  79. Motionma: motion modelling and analysis by demonstration. In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1309–1318.
  80. PopBlends: Strategies for conceptual blending with large language models. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–19.
  81. Convolutional sequence generation for skeleton-based action synthesis. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, New York, NY, USA, 4394–4402.
  82. ChoreoNet: Towards Music to Dance Synthesis with Choreographic Action Unit. In Proceedings of the 28th ACM International Conference on Multimedia (Seattle, WA, USA) (MM ’20). Association for Computing Machinery, New York, NY, USA, 744–752. https://doi.org/10.1145/3394171.3414005
  83. Physdiff: Physics-guided human motion diffusion model. In Proceedings of the IEEE/CVF International Conference on Computer Vision. IEEE, New York, NY, USA, 16010–16021.
  84. Music-to-Dance Generation with Multiple Conformer. In Proceedings of the 2022 International Conference on Multimedia Retrieval (Newark, NJ, USA) (ICMR ’22). Association for Computing Machinery, New York, NY, USA, 34–38. https://doi.org/10.1145/3512527.3531430
  85. Dance and choreography in HCI: a two-decade retrospective. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems. Association for Computing Machinery, New York, NY, USA, 1–14.
  86. Here and Now: Creating Improvisational Dance Movements with a Mixed Reality Mirror. In Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems (Hamburg, Germany) (CHI ’23). Association for Computing Machinery, New York, NY, USA, Article 183, 16 pages. https://doi.org/10.1145/3544548.3580666
  87. Zixiang Zhou and Baoyuan Wang. 2023. UDE: A Unified Driving Engine for Human Motion Generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, New York, NY, USA, 5632–5641.
  88. Music2Dance: DanceNet for Music-Driven Dance Generation. ACM Trans. Multimedia Comput. Commun. Appl. 18, 2, Article 65 (feb 2022), 21 pages. https://doi.org/10.1145/3485664
Citations (2)
View on Semantic Scholar

Summary

We haven't generated a summary for this paper yet.

Ai Sparkles Streamline Icon: https://streamlinehq.com Summarize Now
X Twitter Logo Streamline Icon: https://streamlinehq.com

Tweets