AudioRole: Role-Aware Audio Intelligence

• AudioRole is a framework that uses explicit role descriptors to condition voice identity, persona, and conversational dynamics in multi-party interactions.
• It leverages large-scale, annotated audio-text datasets from TV series and synthetic corpora to align semantic content with vocal characteristics.
• Diverse modeling paradigms, including turn-taking control, chain-of-thought reasoning, and role tagging, show marked improvements in acoustic personalization and content fidelity.

Searching arXiv for papers on AudioRole and related speech role-playing benchmarks. AudioRole denotes the use of explicit role information in audio-centric AI systems. In current literature, the term appears in two closely related senses: as an explicitly assigned descriptor that conditions a voice agent’s identity, persona, conversational style, and turn-taking preferences in multi-party interaction, and as the name of a large-scale audio-text dataset for character role-playing in LLMs that aligns semantic content with vocal characteristics (Mitra et al., 11 Jun 2026, Li et al., 27 Sep 2025). Across recent work, AudioRole-related research spans role-aware speech generation, streaming turn-taking, role-conditioned transcription, and multimodal evaluation, with the common objective of modeling not only what is said but also who is speaking, how the speech is delivered, and which social or narrative role it is meant to realize.

1. Conceptual scope

In multi-party spoken interaction, AudioRole is used as an explicit conditioning variable. ModeratorLM defines the AudioRole as encoding not only the assistant’s identity but also persona, conversational style, and turn-taking preferences, such as a proactive moderator or a passive listener; the role descriptor is included as a system prompt, and it modulates both when the agent speaks and how it responds (Mitra et al., 11 Jun 2026). In character-oriented modeling, AudioRole is instead framed as a data problem: the central difficulty is synchronized alignment of semantic content and vocal characteristics, so that a model can reproduce both persona-consistent language and acoustically faithful delivery (Li et al., 27 Sep 2025).

A related but distinct formulation appears in role-aware ASR. In serialized output training for joint ASR and speaker role tagging, the output sequence is augmented with semantically meaningful, role-specific tags such as child/adult or self/other, rather than anonymous speaker identities. The target is serialized as alternating role and lexical tokens, for example

$\mathbf{z} = (s_1, y_1, \ldots, s_T, y_T),$

where $s_t$ are speaker role tokens and $y_t$ are lexical tokens sorted by utterance timestamp (Xu et al., 12 Jun 2025). This formulation treats role as transcript structure rather than persona simulation.

These strands indicate that AudioRole is not limited to voice conversion or to speaker diarization. One line of work emphasizes character embodiment; another emphasizes interaction policy; a third embeds role labels directly into recognition or transcript generation. A plausible implication is that “role” in audio research has become a unifying abstraction linking semantics, paralinguistics, discourse control, and downstream interpretability.

2. Data resources and corpus construction

The AudioRole dataset is curated from 13 TV series spanning 1K+ hours and 1M+ character-grounded dialogues, with synchronized audio-text pairs annotated with speaker identities and contextual metadata; it covers over 115 main characters and includes an AudioRole-Demo subset with six main characters for ARP-Model training and validation (Li et al., 27 Sep 2025). Its construction pipeline concatenates episode audio, applies pyannote-based diarization, extracts context-aware dialogue scenes, generates transcripts with MooER, removes audio under 1 second or over 30 seconds and transcripts with fewer than 2 or more than 512 characters, and applies DeepFilterNet for noise suppression (Li et al., 27 Sep 2025).

Recent role-playing corpora define adjacent design points. RolePlayConv is a synthetic spoken multi-party dataset with 125 assistant roles and 75k+ conversations, each with 3–6 speakers, diverse assistant roles, turn-level reasoning traces, and TTS-assigned voices with gender, age, and accent metadata (Mitra et al., 11 Jun 2026). SpeechRole-Data comprises 98 diverse roles and 112k speech-based single-turn and multi-turn conversations, pairing text, spoken audio, and character metadata, with 78 roles in training and 20 in validation/dev (Jiang et al., 4 Aug 2025). VoxRole contains 13,335 multi-turn dialogues totaling 65.6 hours of speech from 1,228 unique characters across 261 movies, using script alignment, audio denoising, forced alignment, and LLM-assisted persona distillation (Wu et al., 4 Sep 2025). ActorMindBench is hierarchical, with 7,653 utterances, 313 scenes, and 6 roles derived from Season 1 of Friends (Chen et al., 13 Apr 2026).

Resource	Scale	Primary purpose
AudioRole	13 TV series, 1K+ hours, 1M+ dialogues, 115+ characters	Character-grounded audio-text training and ARP evaluation
RolePlayConv	125 roles, 75k+ conversations	Multi-party turn-taking and reasoning supervision
SpeechRole-Data	98 roles, 112k conversations	Training and benchmarking SRPAs
VoxRole	13,335 dialogues, 65.6 hours, 1,228 characters	Speech-based role-playing benchmark
ActorMindBench	7,653 utterances, 313 scenes, 6 roles	Hierarchical speech role-playing benchmark

Taken together, these resources shift role-playing research away from text-only persona simulation toward audio-grounded corpora in which timbre, prosody, timing, and scene context are first-class variables. This suggests that dataset design is now inseparable from evaluation design: corpora increasingly include either contextual metadata, reasoning traces, or human-aligned annotations specifically to supervise role fidelity rather than generic speech synthesis.

3. Modeling paradigms

A baseline AudioRole modeling path fine-tunes GLM-4-Voice on AudioRole-Demo, producing six ARP-Models specialized to different characters (Li et al., 27 Sep 2025). The objective is dual: preserve the target character’s voice and preserve character-consistent content. The same paper explicitly distinguishes this from prior work that could match vocal style on arbitrary text, arguing instead for generation of the right voice and the right words (Li et al., 27 Sep 2025).

ModeratorLM implements a different paradigm: real-time multi-party interaction. It combines a speech encoder with an LLM backbone, processes dynamically sized audio chunks from 0.5s to 3s, sequentially appends chunk embeddings and speaker-annotated transcripts into LLM context, and emits either a turn-taking control token followed by a response or an empty string for no-turn. Turn-taking is an output of the model itself rather than a decision delegated to an external VAD module (Mitra et al., 11 Jun 2026). ModeratorLM-Think adds a dedicated reasoning stage in the form of chain-of-thought traces over conversational context and assigned role (Mitra et al., 11 Jun 2026).

ActorMind instantiates an actor-inspired multi-agent chain-of-thought architecture. The Eye Agent reads role profiles, scene descriptions, and previous text; the Ear Agent applies Speech Emotion Captioning to prior speech signals; the Brain Agent infers the emotional state for the next line; and the Mouth Agent uses retrieval-augmented generation plus TTS to deliver the line with inferred emotion and role-consistent vocal traits (Chen et al., 13 Apr 2026). Its core decomposition is explicitly sequential: perception of script and scene, listening to co-actors, emotion reasoning, and performance.

Role-aware recognition offers yet another mechanism. In joint ASR and speaker role tagging with Whisper, the tokenizer is extended with <spk0> and <spk1>, role annotations are mapped onto these tokens, and the model is fine-tuned to generate a single serialized sequence in one decoding pass using <startoftranscript><en><transcribe><notimestamps> as decoder initialization (Xu et al., 12 Jun 2025). The training objective maximizes $P(\mathbf{z}\mid\mathbf{x})$ for input audio $\mathbf{x}$ and serialized target $\mathbf{z}$, without architectural changes to encoder or decoder layers (Xu et al., 12 Jun 2025).

These architectures differ in immediate task definition—character role-playing, multi-party turn-taking, or role-aware transcription—but converge on explicit role conditioning rather than latent role inference alone. A plausible implication is that AudioRole functions as a control interface as much as a label space.

4. Evaluation frameworks and metrics

AudioRole introduces ARP-Eval, a dual-aspect evaluation framework that measures both response quality and role fidelity (Li et al., 27 Sep 2025). It uses four metrics: Acoustic Quality, Content Quality, Acoustic Personalization, and Content Personalization. Acoustic Personalization is defined through speaker-embedding similarity as

$AP = D_{\cos}(X_b, X_c),$

where $X_b$ is the generated sample and $X_c$ is the reference; lower values indicate better personality match, although experiments report it as a positive score (Li et al., 27 Sep 2025). Content Personalization is scored by GPT-4o multimodal comparison on a 0–2 scale (Li et al., 27 Sep 2025).

For role-conditioned turn-taking, ModeratorLM reports chunk-level precision, recall, and $F1$:

$s_t$0

along with macro-averaged accuracy, false-positive rate, and Reactive Miss Rate (Mitra et al., 11 Jun 2026). This framework evaluates floor-taking decisions directly, not just semantic adequacy of spoken content.

SpeechRole-Eval broadens speech role-playing assessment into three dimensions: fundamental interaction ability, speech expressiveness, and role-playing fidelity. Its metric set includes Instruction Adherence, Speech Fluency, Conversational Coherence, Speech Naturalness, Prosody Consistency, Emotion Appropriateness, Personality Consistency, and Knowledge Consistency, with paired comparison and normalized scoring using Gemini 2.5; human validation reports MSE around 0.13 and Pearson $s_t$1 around 0.72–0.89 across metrics (Jiang et al., 4 Aug 2025). VoxRole combines objective metrics such as Rouge-L, Meteor, BertScore-F1, and UTMOSv2 with a six-dimensional acoustically aware LLM judge covering Human-Likeness, Personality Consistency, Linguistic Fidelity, Relational Coherence, Contextual Coherence, and Paralinguistic Appropriateness; its LLM-based scores show Pearson’s $s_t$2 against human ratings (Wu et al., 4 Sep 2025).

Speech-DRAME argues that zero-shot ALLM judges miss paralinguistic cues and collapse multiple aspects into coarse scores, and therefore separates Archetype Evaluation from Realism Evaluation. Its fine-tuned DRAME-Eval improves Pearson correlation with human ratings from 0.480 to 0.629 in archetypes and from 0.390 to 0.625 in realism (Shi et al., 3 Nov 2025). RoleJudge makes a related intervention by training an audio LLM judge on RoleChat, a 14,032-sample multimodal evaluation dataset with chain-of-thought reasoning annotations, and scores Logical Coherence, Content Relevance, Context Consistency, Emotional Appropriateness, and Style Alignment; RoleJudge reports 86.0% overall accuracy, exceeding Gemini3 Pro at 69.8% and GPT-4o-audio at 53.0% (Fu et al., 15 Apr 2026).

A recurring controversy concerns evaluator bias. AudioRole explicitly notes that models evaluated with the same underlying LLM family may obtain inflated quality scores (Li et al., 27 Sep 2025). Across the benchmark literature, this has made human-aligned evaluation models, contrastive negative samples, and multi-annotator protocols central methodological components rather than auxiliary checks.

5. Empirical findings

On AudioRole-Demo, the ARP-Model attains an average Acoustic Personalization score of 0.31 and a Content Personalization score of 0.36, significantly outperforming the original GLM-4-Voice in acoustic personalization and surpassing the untrained original model by about 38% in content personalization while maintaining the same level as MiniCPM-O-2.6 (Li et al., 27 Sep 2025). The paper also reports a trade-off: Acoustic Quality drops to 6.5 relative to 7.6 for GLM-4-Voice and 7.7 for GPT-4o, which it interprets as reflecting tension between synthetic speech quality and preservation of distinctive character quirks (Li et al., 27 Sep 2025). Data quantity matters: Sheldon, with 12.5 hours of reference audio, reaches AP = 0.42, whereas Tyrion, with 1.34 hours, reaches AP = 0.25 (Li et al., 27 Sep 2025).

In multi-party turn-taking, ModeratorLM improves substantially over non-role-conditioned baselines. ModeratorLM reports precision 0.71, recall 0.57, $s_t$3 0.61, accuracy 0.76, false-positive rate 0.05, and Reactive Miss Rate 0.14, while ModeratorLM-Think reaches precision 0.79, recall 0.82, $s_t$4 0.79, accuracy 0.91, false-positive rate 0.03, and miss rate 0.03 (Mitra et al., 11 Jun 2026). The reported improvements include over 40% gain in precision and more than 70% in recall compared with weaker baselines (Mitra et al., 11 Jun 2026).

ActorMind reports an average RP-MOS of 3.56 ± 0.27, exceeding IndexTTS at 3.05 ± 0.56 and substantially exceeding Qwen_Omni at 1.00 ± 0.00 (Chen et al., 13 Apr 2026). The ablations indicate that removing the Eye Agent or Brain Agent causes the largest drops, underscoring the importance of explicit role profile grounding and emotion reasoning (Chen et al., 13 Apr 2026).

Role-aware ASR also shows measurable gains. On Playlogue, Whisper-large with serialized output training reduces child mtWER from 69.3% to 54.8% and adult mtWER from 37.0% to 23.6% relative to the WavLM-large baseline; on MMCSG, Whisper-small reduces self mtWER from 26.7% to 16.3% and other mtWER from 34.0% to 23.1% (Xu et al., 12 Jun 2025). The paper summarizes this as more than 10% reduction in multi-talker WER and presents the method as feasible for unified speaker-role aware transcription (Xu et al., 12 Jun 2025).

Across these results, coherence and lexical adequacy are often easier targets than acoustic fidelity or long-horizon persona maintenance. VoxRole, for example, shows that coherence is the strongest dimension for all tested models, whereas acoustic quality remains challenging even for the best model (Wu et al., 4 Sep 2025). This suggests that AudioRole research is constrained less by generic language generation than by the coupling of paralinguistics, context, and identity.

6. Related meanings of role in audio research

The term “role” in audio research is broader than speech role-playing. In object-based audio rendering, audio object metadata may include object category or audio role such as Dialog, Background, Music, or Narrator, together with narrative importance, constraints, and preferred renderer. These role labels drive scene adaptation, object refinement, and renderer routing under changing reproduction conditions (Jackson et al., 2017). Here, role is a rendering-control variable rather than a conversational persona.

In operational speech processing, role can denote speaker function. For air traffic control speech recognition, a grammar-based knowledge system uses ICAO phraseology, callsign position rules, and a Bayesian bag-of-words model to identify ATCO versus pilot speech from transcripts, yielding average speaker-role identification accuracy of about 83% under noisy VHF conditions (Prasad et al., 2021). The same work shows that separate or multitask ASR models for ATCO and pilot speech outperform pooled training on noisy data (Prasad et al., 2021).

In music information retrieval, role refers to compositional function. Track-role prediction for single-instrumental sequences classifies Main Melody, Sub Melody, Pad, Riff, Accompaniment, and Bass, reaching 87.1% accuracy in the symbolic domain with MusicBERT-base and 84.3% in the audio domain with PANNs plus Attention Feature Fusion (Han et al., 2024). This usage is conceptually distinct from AudioRole in conversational AI, but it illustrates the same general pattern: “role” serves as a compact semantic label for function within a larger multimodal structure.

Taken together, these adjacent literatures clarify a common misconception. AudioRole is not a single task name but a family of role-centric representations in audio systems. Depending on the problem formulation, role may denote speaker function, dialog policy, character identity, narrative priority, or musical function. The unifying research move is explicit role modeling, whether for generation, recognition, rendering, or evaluation.