---
title: 'AdaRPSet: Adaptive Role-Playing Dataset'
url: https://www.emergentmind.com/topics/adarpset
type: topic
---

# AdaRPSet: Adaptive Role-Playing Dataset

AdaRPSet is a large-scale, structured dataset constructed to train and evaluate large language model (LLM) "Actor Models" specifically for immersive multi-agent role-playing in dynamic narrative environments. It serves as a principal data component in the AdaMARP framework, which addresses limitations in previous LLM-based role-playing systems, such as poor environmental grounding, limited scene/cast adaptability, and insufficient support for orchestrating dynamic character interactions. AdaRPSet is meticulously annotated to reflect complex, multi-dimensional conversational trajectories within both human-authored and LLM-synthesized interactive narratives, using a unified, immersion-oriented message format that encodes internal thought, explicit action, environmental context, and speech [2601.11007].

## 1. Dataset Composition and Statistical Structure

AdaRPSet comprises 22 425 distinct role-playing trajectories, partitioned into two categories:

- **AdaRPSet-Extracted**: 12 525 trajectories derived from 81 public-domain and best-seller books, with plot extraction centered on chapter boundaries (~8 k tokens) and assisted by LLM-based scene segmentation and character profiling.
- **AdaRPSet-Synthesis**: 9 900 trajectories generated by LLM prompt synthesis, distributed evenly over 20 themes (495 per theme; $p_i = 0.05$ per theme), including but not limited to Adventure, Quest, Rescue, Mystery, Betrayal, Magic, and Apocalypse.

These trajectories are further segmented into 14 343 narrative scenes ($N_{\text{scene}}$), spanning a total of 450 235 utterances (agent turns) and yielding an average of 20.08 turns per trajectory.

Control-message injection, used for training orchestration decisions with AdaSMSet, produces a dense annotation of discrete management actions within the AdaRPSet-Synthesis split:

| Manager Action  | Total Count | Mean per Traj. |
|-----------------|------------|----------------|
| init_scene      | 9 900      | 1.00           |
| pick_speaker    | 223 415    | 22.57          |
| switch_scene    | 10 101     | 1.02           |
| add_role        | 9 862      | 1.00           |
| end             | 9 900      | 1.00           |

This distribution encodes continuous scene management, turn-taking, and adaptive character introduction.

## 2. Annotation Schema and Message Encoding

Every utterance in AdaRPSet is formatted using a unified "immersive" schema designed to reflect narrative depth and agent situatedness:

- **[Thought]**: First-person, internal monologue, inaccessible to other characters.
- **(Action)**: Agent actions or behaviors visible to others within the scene.
- **<Environment>**: Exogenous events, ambient phenomena, or environmental cues with no explicit agent.
- **Speech**: Direct character dialogue (unadorned).

Example (in structured LaTeX-style markup):

\[
\underbrace{[\text{I can’t believe she asked that}]}_\text{Thought}
\underbrace{(\text{glances toward the door})}_\text{Action}
\underbrace{<\text{a distant thunder rumbles}>}_\text{Environment}
\text{I’m not sure we should stay here.}
\]

The combined message format enforces explicit, granular modeling of agent cognition, performative acts, environmental context, and dialogue, with consistent wrappers across all source and synthetic splits.

## 3. Training Objectives and Evaluation Metrics

Actor Models trained on AdaRPSet are optimized using a composite loss function:

\[
\mathcal{L} = \mathcal{L}_{CE} + \lambda_{env}\,\mathcal{L}_{env} + \lambda_{cons}\,\mathcal{L}_{cons}
\]

- **Cross-Entropy $\mathcal{L}_{CE}$** is calculated over next-token prediction.
- **Environment Grounding $\mathcal{L}_{env}$** directly penalizes incorrect generation of <Environment> tags:
  \[
  \mathcal{L}_{env} = -\sum_{t\in \mathcal{I}_{env}} \log p_\theta(e_t \mid \text{context}_t)
  \]
  where $\mathcal{I}_{env}$ marks environment-token positions.
- **Consistency Regularizer $\mathcal{L}_{cons}$** (optional): Encourages persona fidelity via squared distance between learned trajectory representations and prescribed profile vectors:
  \[
  \mathcal{L}_{cons} = \sum_{\tau\in \text{batch}} \lVert f_{persona}(\tau) - p^*\rVert^2
  \]

Trajectory-level evaluation is performed via AdaptiveBench, deploying an LLM judge to score simulated trajectories on five dimensions using 0–10 sub-metrics. For example, **Character Consistency (CC)** is scored as:

\[
\mathrm{CC} = \frac{1}{5}\sum_{d=1}^{5} s_d
\]

with $s_d$ assigned to Internal Coherence, Speaking-Style Fidelity, Language Fluency, Profile Fidelity, and Motivation Stability. **Environmental Grounding (EG)** is similarly:

\[
\mathrm{EG} = \frac{1}{2}(s_{\mathrm{Awareness}} + s_{\mathrm{Utilization}})
\]

Other axes include Interpersonal Interaction and Narrative Progression. Aggregated behavioral validity can employ:

\[
\mathrm{ConsistencyScore} = \frac{1}{|\tau|}\sum_{t=1}^{|\tau|} \mathbf{1}[\text{actor\_behavior\_valid}(m_t)]
\]

## 4. Data Collection and Quality Assurance

**AdaRPSet-Extracted** utilizes LLMs (GPT-5-Chat) to assist plot extraction and segmentation from literature, generating 7-dimensional character profiles for each extracted persona. Chapters are chunked at ~8 k tokens, and scenes are cast as immersive-format dialogues.

**AdaRPSet-Synthesis** employs 20 distinct thematic prompts, each generating 495 trajectories formulated to integrate core actions (init_scene, [Thought], (Action), <Environment>, Speech, at least one switch_scene and one add_role per trajectory).

Quality control measures include:

- Automated duplicate detection and culling ($\sim 5$\% duplicate removal) based on main-character names.
- Manual review per synthetic theme to enforce scenario uniqueness and correct control action insertion.
- Spot checks for format and procedural compliance; there is no multi-annotator scoring on the raw data.

## 5. Usage Recommendations, Integration, and Pitfalls

For optimal model performance and generalization:

- Combine AdaRPSet-Extracted and AdaRPSet-Synthesis in equal measure to balance stylistic realism and the modeling of dynamic events.
- Use the unified message-format wrapper for all training samples.
- Representative training hyperparameters (for 7 B/8 B models): micro_batch_size=24, global_batch_size=48, AdamW optimizer, LR=1e–6 with 5% warmup from 1e–7, 8 epochs, max sequence length 16k tokens (left-truncated).

Potential pitfalls include:

- Exclusive training on Extracted data yields strong format compliance but reduces exposure to rapid role/scene adaptation.
- Exclusive use of Synthesis data limits dialogue style diversity and narrative depth.
- Overly constraining instruction prompts may suppress actor flexibility; basic prompts tend to produce superior AdaptiveBench scores.

## 6. Representative Trajectory Example

An exemplary AdaRPSet-Synthesis “Adventure” trajectory (abridged) exhibits the following phenomena:

\[
\begin{array}{l}
\text{Scene Manager: init\_scene}\quad \text{Late afternoon onboard the Orphan Gale, steam hisses under brass fittings, clouds glow below.}\\[4pt]
\text{Isolde Ferrowind: }<\text{The compass trembles}> [Winds shift too fast] \text{Keep sharp, Taron.}\\[4pt]
\text{Taron Corvith: }(\text{sketches wind currents rapidly}) \text{The currents twist like braided rivers...}\\[4pt]
\vdots\\[4pt]
\text{Valdrex: }(\text{emerging from below deck}) \text{Hull’s holding, Captain, but I hear foreign engines.}\\[4pt]
\text{Isolde Ferrowind: }[Competition or ambush] \text{Mark bearing forty-two north by west.}\\[4pt]
\text{Scene Manager: add\_role}\quad \text{reason: A distant signal matches the old legend.}\quad \text{new\_role: Lynath Ocirra (mistrustful sky-ward sentry)}\\[4pt]
\vdots\\[4pt]
\text{Scene Manager: switch\_scene}\quad \text{The Orphan Gale docks at the floating Sky Citadel, basalt shards wrapped in violet energy.}\\[4pt]
\vdots\\[4pt]
\text{Lynath Ocirra: }(\text{hovers by the gangway}) \text{I warned your kind never to breach this corridor.}\\[4pt]
\text{Isolde Ferrowind: }[Her tone recalls old omens] \text{We come to learn, not plunder.}\\[4pt]
\text{Scene Manager: end}\quad \text{reason: Alliance formed and first chapter concludes.}
\end{array}
\]

This illustrates integrated scene control (init_scene, add_role, switch_scene, end), coordinated agent turns with immersive markup, and capacity for dynamic cast/scene adaptations—key phenomena AdaRPSet encodes for adaptive role-playing competency.

## 7. Context and Significance in Adaptive Role-Playing Research

AdaRPSet constitutes a foundational asset for training LLMs capable of narrative-immersive, multi-agent interaction with explicit environmental and interpersonal awareness. Its dual-sourced construct—fusing literature-derived and synthetic, thematically controlled content—directly addresses observed deficits in prior frameworks such as static cast/scene assumptions and inadequate environmental modeling. Modeled jointly with AdaSMSet (for orchestration supervision) and AdaptiveBench (for trajectory-level scoring), AdaRPSet underlies demonstrable improvements in character consistency, environmental grounding, and narrative coherence, enabling models (e.g., 8B actors) to surpass several leading commercial LLMs on core adaptive role-play metrics [2601.11007].

Source: https://www.emergentmind.com/topics/adarpset