Role-playing-Video60k: Video-Grounded RPA Corpus

• Role-playing-Video60k is a video-grounded corpus that leverages temporal visual evidence to construct dynamic role profiles with 700k dialogues.
• It employs a two-stage captioning and context synthesis pipeline to generate both dynamic visual tokens and static summaries for enhanced scene continuity.
• The resource balances computational efficiency and model performance, demonstrating improved human-likeness and instruction adherence in role-playing agents.

Searching arXiv for the primary paper and a directly related follow-up paper on video-grounded role-playing. Role-playing-Video60k is a dynamic, video-grounded corpus for role-playing agents (RPAs), introduced together with a multimodal framework for video-guided role-playing in which persona formation is conditioned not only on static textual descriptions but also on temporally evolving visual evidence. The resource comprises 60k videos and 700k corresponding dialogues, and it is designed to supply temporally grounded, persona-relevant visual context and high-quality role-centric dialogues so that RPAs can respond in character with improved human-likeness and instruction adherence (Zhang et al., 17 Sep 2025).

1. Corpus definition, scope, and intended use

Role-playing-Video60k addresses a specific limitation in prior RPA work: existing approaches largely rely on static text or image profiles and therefore miss motions, evolving scenes, and emotional transitions that humans use in situated interaction. The dataset introduces video as a primary modality for constructing dynamic role profiles, with the explicit aim of grounding persona and scene understanding over time (Zhang et al., 17 Sep 2025).

The corpus is sourced from major Chinese social platforms—Xiaohongshu, Douyin, Weibo, and Bilibili—and focuses on everyday-life content, including live clips, lifestyle vlogs, and personal documentaries. The videos are randomly shuffled into 57k training videos and 3k inference videos, while a separate test set contains 328 human-selected questions from social media for evaluation. The release is research-only, and the authors note minimum safety alignment together with the potential for harmful content, recommending careful safety alignment before application. Code and data are available through the project repository (Zhang et al., 17 Sep 2025).

The paper emphasizes content diversity and length-aware temporal policies, but it does not report total hours, average video length, resolution, or FPS. This omission is consequential for benchmarking video efficiency, because the resource is positioned not merely as a dialogue dataset but as a benchmark for temporally grounded multimodal role-playing.

Category	Duration	Characterization
Live	0–5s	Close-up, fine-grained motions around specific moments
Vlog	5s–10min	Day-in-the-life visual storytelling with strong self-expression
Documentary	>10min	Longer narratives with frequent scene transitions and event-level understanding

The paper characterizes the dataset as the first role-playing dataset with video modality at scale. A plausible implication is that its significance lies less in raw volume alone than in the introduction of temporal grounding as a first-class component of role conditioning (Zhang et al., 17 Sep 2025).

2. Construction pipeline and dialogue curation

The annotation pipeline begins with a two-stage video captioning process. First, each video is divided into 64 segments, and one representative frame per segment is sampled. This uniform segmentation is intended to balance coverage and efficiency. Second, an LLM generates a detailed description for each sampled frame, after which a “summary agent” reads the frame descriptions in temporal order and, using Chain-of-Thought (CoT) and In-Context Learning (ICL), produces a comprehensive caption or summary for the entire video (Zhang et al., 17 Sep 2025).

These captions are used differently at training and inference time. During fine-tuning, they serve as the basis for generating Q&A dialogue pairs. During inference, the captions are summarized again to produce a concise role context. The dialogue-generation prompts include four ingredients: the role definition of the video scene, the detailed video description or caption, in-context examples derived from real social media comments, and an explicit instruction for Q&A generation. GPT-4o is instructed to return None if the context is insufficient (Zhang et al., 17 Sep 2025).

Quality control is performed through several filtering stages rather than through conventional human annotation agreement. The pipeline removes answers beginning with “As an AI LLM,” and phrases such as “does not present/show/demonstrate,” as well as other errors. Generation uses temperature=1.0, with max_new_token set by content type: 1024 for live, 2048 for vlog, and 4096 for documentary videos. To increase diversity, multiple strong LLMs are used: Qwen-Max, DeepSeek-R1, GPT-4.1, GPT-4o, Claude 3.7 Sonnet-thinking, and Gemini 2.5 Pro Exp. Postprocessing then applies regular expressions to extract dialogue segments, a second prompt-based selection step to retain scene-aligned dialogues, and final cleaning to remove formatting artifacts such as ** and enumerations (Zhang et al., 17 Sep 2025).

The training format inserts the video at an explicit token position, <video>, and the frames are represented as visual tokens. Each sample typically contains a video ID and category, the <video> insertion token followed by <image> tokens in original temporal order, the role definition and scene description derived from captions, and dialogue Q&A pairs for supervised fine-tuning. At inference, a summary context $S$ is supplied as static persona context. The paper does not report explicit turn counts per sample, inter-annotator agreement, a JSON schema, or a directory layout (Zhang et al., 17 Sep 2025).

Because the dialogues are LLM-synthesized with prompt-based curation, the dataset is best understood as a structured synthetic supervision resource anchored in video-derived descriptions rather than as a manually transcribed conversational corpus. Selective manual review is nevertheless used for duplicate content and static-scene videos, and token limits are adjusted for highly dynamic long videos.

3. Dynamic and static role profiles

The central conceptual contribution of Role-playing-Video60k is the distinction between dynamic role profiles and static role profiles. A dynamic role profile encodes the temporal evolution of persona-relevant visual content by sampling frames from video and feeding them to the LLM in order as a visual prefix. A static role profile has two components: character dialogues from training videos used during fine-tuning, and a summary context from the input video used during inference (Zhang et al., 17 Sep 2025).

The adaptive temporal sampling mechanism is defined for video

$V \in \mathbb{R}^{T \times H \times W \times 3},$

which is transformed into a sampled sequence

$V' \in \mathbb{R}^{t \times H \times W \times 3},$

with $t \le 128$. The policy depends on content type. For live videos, the method uses dense sampling and takes every frame. For vlogs, it uses sparse uniform sampling at one frame per 5 seconds. For documentaries, it applies a three-step keyframe selection algorithm (Zhang et al., 17 Sep 2025).

In the long-video case, candidate frames are obtained by uniformly sampling at 1 fps and computing frame differences

$D(i,j) = \sum_{k=1}^{M} |I_i^k - I_j^k|,$

where $I_i^k$ is the $k$-th pixel of frame $i$, and $M$ is the number of pixels. A frame is added to the candidate set $C$ if $V \in \mathbb{R}^{T \times H \times W \times 3},$0. The candidate set is then partitioned into groups, and intra-group variation is measured by

$V \in \mathbb{R}^{T \times H \times W \times 3},$1

Finally, adjacent-frame deduplication uses CLIP similarity,

$V \in \mathbb{R}^{T \times H \times W \times 3},$2

merging frame $V \in \mathbb{R}^{T \times H \times W \times 3},$3 into frame $V \in \mathbb{R}^{T \times H \times W \times 3},$4 when $V \in \mathbb{R}^{T \times H \times W \times 3},$5, until all adjacent pairs satisfy the threshold criterion. The final keyframe set is denoted $V \in \mathbb{R}^{T \times H \times W \times 3},$6 (Zhang et al., 17 Sep 2025).

The feeding order is integral to the method. Each frame is converted into a special <image> token, stacked in original temporal order, and presented to the LLM as a visual prefix. The paper argues that this preserves scene continuity and motion cues, thereby improving temporal grounding. Static context is added through a separate summary construction process: for an input video of length $V \in \mathbb{R}^{T \times H \times W \times 3},$7, descriptions

$V \in \mathbb{R}^{T \times H \times W \times 3},$8

are produced from uniformly segmented portions, and a summary agent uses a CoT approach to summarize these descriptions into a video summary $V \in \mathbb{R}^{T \times H \times W \times 3},$9, which is then provided as static context (Zhang et al., 17 Sep 2025).

The framework concatenates the dynamic and static profiles in prompts. The <video> and <image> visual prefix grounds temporally evolving content, while the static summary and dialogue-conditioned fine-tuning encode stable persona traits and global scene information. The paper reports that this joint integration yields more consistent, human-like, and instruction-adherent responses. A common misconception is that video-conditioned role-playing can be solved by adding raw frames alone; the ablations instead indicate that concise static summaries remain important for adherence and accuracy (Zhang et al., 17 Sep 2025).

4. Model architecture, prompting, and computational profile

The framework uses InternVL2.5-8B as the base multimodal model. Visual frames are injected as special <image> tokens before text, leveraging InternVL’s vision encoder and alignment layers, and the fine-tuning format explicitly inserts <video> at the intended position in the prompt. Training prompts include role definitions, detailed captions, and Q&A examples, while inference adds both the summary context $V' \in \mathbb{R}^{t \times H \times W \times 3},$0 and the dynamic frame tokens (Zhang et al., 17 Sep 2025).

The training procedure is supervised fine-tuning with LoRA. The optimizer is AdamW with learning rate $V' \in \mathbb{R}^{t \times H \times W \times 3},$1, weight decay $V' \in \mathbb{R}^{t \times H \times W \times 3},$2, and warm-up ratio $V' \in \mathbb{R}^{t \times H \times W \times 3},$3, and training runs for one epoch. To manage long visual prefixes, the maximum number of frames per video is capped at 128. The paper also uses FlashAttention v2.7.4 to reduce attention memory from $V' \in \mathbb{R}^{t \times H \times W \times 3},$4 to $V' \in \mathbb{R}^{t \times H \times W \times 3},$5, accelerating inference for long visual prefixes (Zhang et al., 17 Sep 2025).

The summary context is a context-window management device as much as a semantic one. Rather than concatenating long per-frame descriptions, the method substitutes the concise summary $V' \in \mathbb{R}^{t \times H \times W \times 3},$6, which is intended to keep text context compact and informative. The ablations indicate that this compression is not merely a convenience: it improves both overall performance and video-text relevance relative to longer concatenated descriptions (Zhang et al., 17 Sep 2025).

Inference-time measurements are reported on two NVIDIA RTX A6000 GPUs:

Frames	Approximate inference time	Hardware
0	1.95s	2 × NVIDIA RTX A6000
1	2.72s	2 × NVIDIA RTX A6000
8	5.05s	2 × NVIDIA RTX A6000
16	5.87s	2 × NVIDIA RTX A6000
32	7.58s	2 × NVIDIA RTX A6000
64	17.49s	2 × NVIDIA RTX A6000

Memory usage per GPU increases with frame count. This suggests that the framework’s temporal grounding benefit is coupled to a visible computational cost, which is partly mitigated by FlashAttention and by the 128-frame cap (Zhang et al., 17 Sep 2025).

5. Evaluation protocol and empirical findings

Evaluation is based on eight metrics assessed through LLM-as-a-judge prompting: Character Consistency, Knowledge Hallucination, Utterance Fluency, Tone Consistency, Instruction Adherence, Response Accuracy, Human Likeness, and Video-Text Relevance. Video-Text Relevance is evaluated only in ablations where video is available. To minimize model bias, GPT-4o and GPT-o3-mini are used as judges with temperature=0.0, and three judgments per sample are averaged (Zhang et al., 17 Sep 2025).

The baseline set includes seventeen general LLMs, such as GPT-4.1, Claude 3.7 Sonnet-thinking, Gemini 2.5 Pro Exp, Yi-Large, Qwen-max, and DeepSeek-V3/R1, together with three role-playing expertise models: CharGLM-4, Ernie-char-8k, and Qwen-plus-character. On the main benchmark, InternVL2.5-8B w/ Video SFT scores 72.17 on Consistency, 74.38 on Hallucination, 70.52 on Adherence, 87.93 on Fluency, 69.98 on Human-likeness, 69.26 on Accuracy, 61.75 on Tone, and 72.28 on average. The paper reports that this configuration achieves SOTA in human-likeness among compared models and competitive or better averages despite smaller parameter counts relative to closed-source giants (Zhang et al., 17 Sep 2025).

The ablation studies isolate the effect of video modality and summary context. With Video Inference + Summary Context, 8B w/ Video SFT obtains Avg 66.18 and Video-Text Relevance 23.43, compared with Avg 60.74 and Relevance 14.20 for 8B w/ Text SFT, and Avg 42.29 and Relevance 11.61 for 8B w/o SFT. The paper concludes that video modality during SFT markedly improves most metrics and video-text grounding. A separate ablation shows that 8B w/ Summary Context reaches Avg 66.18 and Relevance 23.43, while 8B w/o Summary Context reaches Avg 64.13 and Relevance 19.37. The reported conclusion is that concise CoT-generated summaries outperform long concatenated descriptions (Zhang et al., 17 Sep 2025).

Human evaluation is reported in a 21-question study in which participants preferred the model over Gemini 2.5 Pro Preview 0325 by 57% to 35%, with 8% “not sure.” Correlations between LLM-judge scores and human scores are also given. For Gemini2.5-Pro-Exp, Pearson reaches 0.5903, Spearman 0.6488, and Kendall 0.4534. For the proposed model, Pearson reaches 0.6655, Spearman 0.6437, and Kendall 0.4989 (Zhang et al., 17 Sep 2025).

The paper also measures an “alignment tax.” After supervised fine-tuning on Role-playing-Video60k, general-benchmark performance declines only slightly on several tasks, for example MMLU from 73.67 to 73.27 and GSM8K from 76.27 to 75.36, while multimodal understanding remains strong. This is relevant to a second common misconception: specialized role-playing fine-tuning need not entail collapse of generalization. In the reported setting, the cost is modest rather than catastrophic (Zhang et al., 17 Sep 2025).

6. Limitations, ethics, and relation to later video-grounded role-playing research

The paper explicitly states several limitations. Resource constraints prevent exploration with larger backbones and denser sampling, only LoRA is used rather than full-parameter fine-tuning, and key corpus statistics such as total hours, resolution, and FPS are not reported. These constraints matter because they leave open whether the gains derive primarily from the dataset itself, the dynamic/static profile design, or the specific efficiency trade-offs imposed by the 128-frame cap (Zhang et al., 17 Sep 2025).

The ethical discussion emphasizes that the release is research-only and that minimum safety alignment may allow harmful or toxic outputs. Because the videos are sourced from social media, the corpus may also reflect demographic, cultural, and platform-specific biases. The summaries and dialogues are LLM-synthesized, so privacy compliance, anonymization where applicable, and filtering of personal identifiers are recommended. The authors further urge careful safety alignment and content moderation before deployment (Zhang et al., 17 Sep 2025).

Role-playing-Video60k also sits within a broader shift toward video-grounded role-playing. A later study on immersive video role-playing introduces EBM-RL, a GRPO-based framework that separates observation, reasoning, and utterance into <perception>, >, and <answer> blocks, and releases another open-source dataset for video-grounded role-playing dialogue built from cinematic material in the Harry Potter and The Lord of the Rings franchises. That work reports improvements in Visual Evidence Grounding, Situational Persona Compatibility, and Conversational Naturalism, as well as zero-shot gains on multiple VideoQA benchmarks (Wang et al., 6 May 2026). This suggests that Role-playing-Video60k is part of a larger methodological transition from static role conditioning toward temporally and atmospherically grounded multimodal role-playing.

The future directions named in the original paper are larger models, denser temporal sampling, and improved integration of video modality for higher video-text relevance. The authors also connect rich roles constructed from dynamic and static perspectives to social applications and digital humans. A plausible implication is that the lasting importance of Role-playing-Video60k lies not only in its scale, but in formalizing the idea that role-playing quality depends on synchronizing persona, scene evolution, and response generation within a single multimodal prompting and fine-tuning regime (Zhang et al., 17 Sep 2025).