CustomDancer: Customized Dance Recommendation by Text-Dance Retrieval

Abstract: Dance serves as both a cultural cornerstone and a medium for personal expression, yet the rapid growth of online dance content has made personalized discovery increasingly difficult. Text-based dance retrieval offers a natural interface for users to search with choreographic intent, but it remains underexplored because dance requires simultaneous reasoning over linguistic semantics, musical rhythm, and full-body motion dynamics. We introduce TD-Data, a large-scale open dataset for text-dance retrieval, containing about 4,000 12-second dance clips, 14.6 hours of motion, 22 genres, and annotations from professional dance experts. On top of this dataset, we propose CustomDancer, a multimodal retrieval framework that aligns text with dance through a CLIP-based text encoder, music and motion encoders, and a music-motion blending module. CustomDancer achieves state-of-the-art performance on TD-Data, reaching 10.23% Recall@1 and improving retrieval quality in both quantitative benchmarks and user preference studies.

• The paper's main contribution is a multimodal framework that integrates explicit music-motion fusion with domain-adapted text encoding to advance choreographic retrieval.
• It introduces TD-Data, a large-scale, expert-annotated 3D dance retrieval dataset with rich metadata ensuring precise textual, rhythmic, and stylistic alignment.
• Empirical results demonstrate improved Recall@1 and high user satisfaction, validating fine-grained synchronization between text, motion, and music.

CustomDancer: Multimodal Text-to-Dance Retrieval with Explicit Motion-Music Fusion

Problem Formulation, Motivation, and Task Overview

CustomDancer addresses the underexplored problem of text-dance retrieval: given a natural-language query, retrieve a dance clip that semantically and rhythmically aligns with the query intent across body motion and music context. Typical search systems are poorly adapted to the choreographic domain due to the need for simultaneous reasoning over linguistic semantics, musical structure, and complex full-body 3D motion. Existing text-to-motion methods often address generic actions and ignore music-content or stylistic details, while video-text retrieval fails to explicitly model the dynamic and synchronized aspects central to dance.

Figure 1: The text-dance retrieval task operationalizes user-choreographic intent, mapping natural-language queries to jointly consistent music-motion clips.

TD-Data: Large-Scale Expert-Annotated 3D Dance Retrieval Dataset

To rigorously evaluate text-dance retrieval, a benchmark dataset is crucial. The TD-Data dataset was constructed by segmenting high-resolution 3D dance motion sequences from FineDance into standardized 12-second clips, then annotating each with genre, tempo, movement signatures, and stylistic attributes by certified experts. These tags were converted to natural-language captions via controlled GPT-4o prompting, producing a corpus of highly diverse and query-faithful textual descriptions spanning 22 genres and 27 professional performers.

Figure 2: The TD-Data pipeline: segmentation, dual-annotator labeling, expert validation, and fluent prompt-driven caption generation.

By separating annotation from performers/genres and validating captions against both motion and music, TD-Data enables controlled, fine-grained studies of retrieval performance and discourages trivial genre-matching shortcuts.

CustomDancer Architecture: Multimodal Encoding and Blending

CustomDancer consists of four primary modules: a CLIP-initialized text encoder (with task-adapted adapters), dedicated temporal music and 3D motion encoders (Transformer-based with staged downsampling for long-range structure), and a music-motion blender that fuses candidate dance features before cross-modal alignment.

Figure 3: The CustomDancer architecture: CLIP-based text encoding and parallel temporal processing of music and motion, with interactive fusion for robust retrieval.

Text queries are mapped into a semantic latent space via CLIP and a lightweight MLP. Music and motion are encoded as temporal feature sequences from Librosa and SMPL input streams, respectively, using stacked Transformer layers for context aggregation. The music-motion blender combines additive and multiplicative interactions to preserve both complementary and agreement information—critical for distinguishing, for example, between visually similar movements performed over musically distinct contexts.

Final text-to-dance alignment is learned using a CLIP-style unidirectional contrastive loss, with cosine similarity driving retrieval. The explicit modeling of both motion and music is shown to be essential for fine-grained retrieval, as ablated variants confirm severe degradation when either channel or sophisticated fusion is omitted.

Quantitative and Qualitative Results

CustomDancer achieves a Recall@1 of 10.23% on the full TD-Data benchmark, outperforming strong video-text retrieval baselines such as XPool and TABLE in both Recall and Mean Rank. This improvement is robust across temporal modeling architectures and fusion strategies: substituting RNN or LSTM for Transformers, or omitting the multiplicative/additive fusion paths, causes marked decreases in retrieval performance.

A user study with choreographers and dance instructors further substantiates the system's advances: CustomDancer obtains the highest ratings on text-motion consistency and text-music relevance short of ground-truth annotator matches, indicating perceptible gains in human-aligned retrieval.

Qualitatively, CustomDancer retrieves dance clips displaying stylistic, rhythmic, and semantic fit to a diverse set of textual queries, from high-energy krump sequences to lyrical ballet phrases, supporting the claim that the model genuinely internalizes choreographic semantics rather than merely exploiting genre-level information.

Figure 4: CustomDancer retrieves stylistically and rhythmically appropriate dance clips for intricate text queries, validating fine-grained multimodal alignment.

Analysis, Implications, and Future Research Directions

CustomDancer empirically demonstrates that explicit music-motion fusion, temporally aware encoding, and domain-adapted text representations are necessary for effective text-dance retrieval. The design choices are motivated by the unique requirements of real-world choreographic recommendation—where matching involves not only the visual shape of movement, but subtle dependencies on rhythm, style, and affect, conditioned on both music and human annotation.

Theoretically, this work shows that general multimodal retrieval architectures are insufficient for choreographic domains unless enhanced with explicit domain priors and synchronized modeling. Practically, CustomDancer and TD-Data provide a testbed for future research into user-facing retrieval and personalized dance recommendation. The authors highlight limitations in handling highly specialized terminology and ambiguous music-motion relationships, underscoring the need for larger-scale, multilingual, and even interactive retrieval benchmarks.

Future research may couple retrieval with controllable generation, enabling post-retrieval adaptation of dance to user-imposed constraints (e.g., style, performer, or new musical context). Advances in music and movement reasoning, user-in-the-loop feedback, and more precise text-conditioned motion representations could further reduce the gap between algorithmic retrieval and expert human judgment.

Conclusion

CustomDancer introduces a rigorous, multimodal framework for natural-language-driven dance retrieval, advancing both the methodological and evaluative state of the art. By integrating expert-labeled, richly annotated 3D motion data with sophisticated cross-modal modeling, the system sets a new standard for text-conditioned dance recommendation and presents a foundation for broader AI-driven choreographic systems. Its empirical gains, robust architecture, and extensible dataset will inform the development of more nuanced, controllable, and user-centric motion retrieval and synthesis systems in the future.