TalkPlayData 2: Multimodal Conversational Dataset

• TalkPlayData 2 is a synthetic, multimodal dataset for conversational music recommendation, featuring structured eight-turn dialogues generated by distinct LLM agents.
• It employs a tailored agentic pipeline that creates explicit listener profiles, goals, and per-turn thought processes, ensuring controlled and realistic interactions.
• The dataset integrates text, audio, and image modalities along with cold-start evaluation splits to support robust benchmarking for both generative and retrieval-based models.

Searching arXiv for the specified paper and closely related conversational music recommendation work. First, I’ll look up "TalkPlayData 2" on arXiv, then find key precursor datasets/papers in multimodal or conversational music recommendation for context. Using the arXiv search tool now. TalkPlayData 2 is a synthetic dataset for multimodal conversational music recommendation generated by an agentic data pipeline in which multiple LLM agents operate under differentiated roles, prompts, and information access. It is designed as a grounded resource for training and evaluating generative recommendation models that must conduct eight-turn dialogues about concrete music tracks using text, audio, and images. The dataset comprises 16,199 conversations and includes explicit user profiles, conversation goals, per-turn thoughts, recommender track selections, and binary goal-progress labels, with public release of both the dataset and generation code announced on the project page (Choi et al., 18 Aug 2025).

1. Definition and scope

TalkPlayData 2 is a publicly released multimodal dataset for conversational music recommendation in which a simulated “Listener” interacts with a simulated “Recsys” over exactly eight turns to discover music tracks from a grounded recommendation pool (Choi et al., 18 Aug 2025). Each conversation is constructed rather than collected from human dialogue, but the generation process is constrained so that the Listener and the recommender do not share the same information. The Listener is conditioned on a profile and a conversation goal, while the recommender sees the profile and the full candidate pool but not the goal. This differential access is a defining property of the dataset’s realism claims.

The dataset is positioned as an advance over earlier conversational recommendation corpora by combining multi-agent generation, multimodal grounding, explicit goals, explicit user profiles, per-turn internal thoughts, and cold-start-aware evaluation splits (Choi et al., 18 Aug 2025). The paper formalizes the creation process as

$$y = f_\theta(x_{\text{profile}}, x_{\text{goal}}, x_{\text{music}})$$

where $f$ denotes the multi-agent creation process, $\theta$ the LLM parameters, $x_{\text{profile}}$ the listener profile, $x_{\text{goal}}$ the conversation goal, and $x_{\text{music}}$ the grounded music pool.

A plausible implication is that TalkPlayData 2 is not merely a dialogue corpus but a structured benchmark for end-to-end conversational recommendation under multimodal, multi-turn, and partially observed conditions. That interpretation follows from the presence of both recommendation targets and explanatory natural language, as well as the explicit warm/cold split design.

2. Agentic generation pipeline

The dataset is created with four LLM agents: Profile LLM, Goal LLM, Listener LLM, and Recsys LLM (Choi et al., 18 Aug 2025). Their separation is central to the pipeline.

The Profile LLM constructs a listener profile from five profiling tracks and demographic variables: age group, country, gender, and preferred language. The Goal LLM selects and customizes a plausible conversation goal from a pool of 44 templates, conditioned on the recommendation pool, the generated profile, and the goal-topic and specificity axes. The Listener LLM role-plays the user; it receives the profile, the goal, and the multimodal content of recommended tracks turn by turn, but it does not see the full pool. The Recsys LLM role-plays the recommender; it sees the profile and the full recommendation pool, including text, audio, image, and track IDs, but it does not see the goal. It must infer intent through the dialogue and recommend only from the available pool.

The paper emphasizes specialized prompting and guardrails. Prompts include constraints such as “recommend ONLY from provided available tracks,” “NO DUPLICATES,” “assess whether this track moves you toward achieving your goal,” and “produce thought then message” (Choi et al., 18 Aug 2025). The logged output contains, per turn, the agent’s thought, the user-facing message, the selected track_id for the recommender, and a binary goal_progress_assessment for the Listener.

The generation algorithm proceeds in two stages. First, the system samples a profiling subset of 5 tracks and a recommendation pool of 16–32 tracks from a session that satisfies the inclusion criteria. Then the Profile and Goal agents derive the final profile and goal. In the conversation stage, Turn 1 begins with the Listener producing a thought and opening message, followed by the recommender producing a thought, a track_id in the pool, and an explanatory message. Turns 2–8 repeat this interaction, with the Listener additionally emitting goal_progress_assessment ∈ {MOVES_TOWARD_GOAL, DOES_NOT_MOVE_TOWARD_GOAL} (Choi et al., 18 Aug 2025).

This architecture is intended to prevent information leakage. The paper states that separate contexts prevent leakage of the Listener’s goal to the Recsys, while pool-only selection and explicit IDs enforce factuality (Choi et al., 18 Aug 2025). This suggests that the dataset is as much an exercise in controlled synthetic data design as in content generation.

3. Multimodal grounding and data schema

TalkPlayData 2 is multimodal at the level of both recommendation and conversation. The dialogue is grounded in track-level text, audio previews, and album art (Choi et al., 18 Aug 2025). The Listener listens to the audio and views the cover art before responding, while the recommender selects tracks using both textual and non-textual cues.

The source data come from LFM-2b sessions from 2005–2020 with user demographics, last.fm tags, and Spotify ID mappings. From Spotify, the pipeline retrieves preview audio snippets, album art, release dates, and popularity as of July 2025. Additional MIR-derived fields include tempo, key, and chords via Madmom, and lyrics via Whisper (Choi et al., 18 Aug 2025). The multimodal assets passed during generation are constrained for cost: a short audio snippet of 3 seconds, represented as 96 multimodal tokens at 32 tokens per second, and album art images at 300×300, represented as 258 tokens per image.

The dataset’s track schema includes Spotify track_id, title, artist, album, release date, popularity, last.fm tags, Whisper-derived lyrics, Madmom-derived tempo/BPM, key, chords, plus the audio preview and cover image (Choi et al., 18 Aug 2025). At the conversation level, the schema includes conversation and user identifiers, split flags, profile fields such as preferred_musical_culture, top_1_artist, and top_1_genre, goal metadata, the recommendation pool, and the full turn-by-turn dialogue records.

The following table summarizes the key structural properties reported for the dataset.

Aspect	Reported value	Notes
Total conversations	16,199	15,199 train + 1,000 test
Turns per conversation	8	Fixed by construction
Approximate total turns	129,592	Derived from fixed turn count
Training users	8,591	Reported in paper
Test users	500	371 warm, 129 cold
Training tracks	43,597	Reported in paper
Test tracks	6,761	3,779 warm, 2,982 cold
Recommendation pool size	16–32 tracks	Per conversation
Profiling subset	5 tracks	Per conversation

The multimodal design distinguishes TalkPlayData 2 from text-only synthetic dialogue generation. A plausible implication is that the dataset is suitable not only for language-model fine-tuning but also for multimodal retrieval, late-fusion, or MLLM-based re-ranking pipelines, as the paper explicitly discusses such downstream architectures (Choi et al., 18 Aug 2025).

4. Conversation goals, profiles, and split design

A major structural innovation of TalkPlayData 2 is its explicit representation of goals and profiles. The Goal LLM selects from 44 templates organized along two axes: topic and specificity (Choi et al., 18 Aug 2025). The topic axis contains 11 codes, A–K: audio-based, lyric-based, visual-musical, contextual/situational, interactive refinement, metadata-rich exploration, mood/emotion-based, artist/discography, cultural/geographic, social/popularity, and temporal/era. The specificity axis consists of the four codes LL, HL, LH, and HH, described as low-vs-high query specificity crossed with low-vs-high target specificity; these correspond to exploratory search, look-up, and fine-grained retrieval behaviors.

The reported specificity distribution is 22%, 34%, 28%, and 16% across the four classes, and topic coverage is complete across the defined codes (Choi et al., 18 Aug 2025). An ablation study reported in the paper shows that removing “goal” or “profile” reduces diversity as measured by KL divergence to uniform and by coverage.

The split strategy is chronological and cold-start-aware. Sessions after 2019 are reserved for testing. The inclusion criterion is sessions with at least 21 tracks and valid Spotify mappings. Of the 1,000 test conversations, 800 are sampled from warm users and 200 from cold users, and the sampling is balanced across country, gender, and age groups (Choi et al., 18 Aug 2025). The paper also reports 500 test users, of whom 371 are warm and 129 cold, and 6,761 test tracks, of which 3,779 are warm and 2,982 cold.

This split design matters methodologically because it enables evaluation under both user and item cold-start conditions. A plausible implication is that the dataset supports more realistic benchmarking than random splits for conversational recommendation, particularly for models expected to leverage profile information, multimodal cues, and turn-wise refinement signals.

5. Evaluation and reported findings

TalkPlayData 2 is evaluated with both LLM-as-a-judge and human assessment (Choi et al., 18 Aug 2025). The LLM judge is Gemini 2.5 Pro, which scores multiple aspects on a 1–4 scale using rubric-specific prompts and, when necessary, access to the relevant textual, audio, and image data for referenced tracks.

The reported average LLM-judge scores are as follows:

Evaluation aspect	Score
Conversation goal plausibility	3.93
Profile appropriateness	3.41
Listener progress-to-goal label accuracy	3.38
Listener thought quality	3.98
Listener message linguistic quality	4.00
Listener message helpfulness toward goal	4.00
Recsys thought quality	3.52
Recsys recommendation quality	3.35
Recsys message linguistic quality	3.69
Recsys message alignment with recommended track	3.83

The paper further states that over 75% of the Listener progress_towards_goal labels scored “Excellent” (4.0), and 73% of the recommender’s recommendation-quality scores were 4.0 (Choi et al., 18 Aug 2025).

In human evaluation on 5-point MOS scales, TalkPlayData 2 reportedly achieves the highest scores among the compared datasets: relevance 4.11 and naturalness 4.15, compared with CPCD at 4.08 and 4.01, LP-MusicDialog at 3.90 and 3.95, and TalkPlayData 1 at 4.04 and 4.01 (Choi et al., 18 Aug 2025). The paper attributes these improvements to multimodal conditioning during generation and to the multi-LLM structure that combines Profile and Goal generation with Listener–Recsys interaction.

The ablations reinforce this interpretation. Removing Goal, Profile, or both degrades diversity, while only the full pipeline attains full topic coverage of 1.00 and the best specificity diversity, measured as the lowest KL divergence (Choi et al., 18 Aug 2025).

6. Recommended uses and methodological significance

The paper presents TalkPlayData 2 as a resource for training generative recommendation systems that jointly predict the next track_id and generate an explanatory message (Choi et al., 18 Aug 2025). The recommended task setup uses as input the profile, goal, conversation history, and either the current candidate pool or a large catalog accessed through retrieval, with multimodal assets supplied either directly to an MLLM or through precomputed embeddings. The primary outputs are the next track_id and a natural-language explanation, with optional generation of a thought field.

For discrete recommendation, the suggested objective is cross-entropy on the gold track_id among candidates, with teacher forcing across turns; ranking variants may use list-wise losses. For language generation, the paper suggests supervised instruction tuning on messages and optionally thoughts. It also proposes using goal_progress_assessment as an auxiliary binary signal for preference modeling or reward modeling (Choi et al., 18 Aug 2025).

For evaluation, the paper lists standard ranking metrics:

$$\mathrm{DCG@k} = \sum_{i=1}^{k}\frac{2^{rel_i}-1}{\log_2(i+1)},$$

$$\mathrm{NDCG@k} = \frac{\mathrm{DCG@k}}{\mathrm{IDCG@k}},$$

$$\mathrm{MRR} = \frac{1}{|Q|}\sum_{q \in Q}\frac{1}{rank_q},$$

$$\mathrm{Recall@k} = \frac{|Rel_k|}{|Rel|}, \qquad \mathrm{Precision@k} = \frac{|Rel_k|}{k}.$$

It also recommends LLM-as-a-judge and human MOS for explanation quality and alignment, and explicitly advises reporting warm-versus-cold slices (Choi et al., 18 Aug 2025).

The paper identifies two broad architectural families for downstream modeling: multimodal LLMs that directly accept text, audio, and image, and two-tower or late-fusion baselines in which one tower encodes conversation history, profile, and goal while another encodes items, after which a smaller LLM verbalizes the recommendation (Choi et al., 18 Aug 2025). This suggests that TalkPlayData 2 is intended as a benchmark spanning both generative and retrieval-augmented paradigms rather than only a single modeling tradition.

7. Relationship to earlier work, limitations, and risks

TalkPlayData 2 is situated relative to prior conversational music recommendation datasets and to earlier text–visual or multimodal systems more generally. The paper contrasts it with human and single-turn corpora such as CPCD and JAMSessions/Text2Tracks, arguing that such datasets lack some combination of multi-turn interaction, profiles, goals, or multimodal alignment (Choi et al., 18 Aug 2025). It also distinguishes TalkPlayData 2 from LP-MusicDialog and TalkPlayData 1, which it characterizes as synthetic multi-turn resources with fewer role constraints and more text-first conditioning. The reported improvements over TalkPlayData 1 include explicit profile and goal generation, structured axes for topics and specificity, multimodal inputs for all agents, per-turn thoughts and progress labels, stricter pool-only and no-duplicate guardrails, and a cold-start-aware test design (Choi et al., 18 Aug 2025).

Several limitations are explicit. First, in-context selection constrains the size of the recommendation pool; extending to larger pools or full catalogs requires retrieval and re-ranking integration. Second, multimodal fidelity is limited by the use of short audio clips of 3 seconds and 300×300 cover images. Third, language and style diversity remain narrower than real-world populations despite the reported naturalness scores. Fourth, biases may be inherited from LFM-2b, Spotify metadata, and the LLMs themselves (Choi et al., 18 Aug 2025).

The paper also addresses ethical and practical risks. Synthetic data may reduce direct privacy risk, but it can amplify source biases. Audio previews and cover art remain subject to platform terms, and downstream users are advised to respect copyright and release only permissible derivatives. Hallucination risk is mitigated through pool-only selection and evaluation checks, but critical findings still require validation (Choi et al., 18 Aug 2025).

A common misconception would be to treat TalkPlayData 2 as a straightforward human dialogue corpus. It is instead a synthetic benchmark built under explicit procedural constraints. Another possible misconception would be to read its high evaluation scores as evidence of unconstrained realism; the paper’s own limitations indicate that realism is bounded by pool size, modality truncation, inherited metadata biases, and the behavior of the underlying multimodal LLMs.

In summary, TalkPlayData 2 defines a specific research object: a grounded, multimodal, eight-turn synthetic dialogue dataset for music recommendation generated by differentiated LLM agents under profile, goal, and pool constraints (Choi et al., 18 Aug 2025). Its significance lies in the combination of multi-agent generation, multimodal conditioning, explicit turn-level supervision, and cold-start-aware benchmarking, which together make it a targeted resource for generative conversational recommender systems.