DeSRPA: Decoupled Speech Role-Playing Agent via Inference-Time Intervention

Abstract: While LLMs have revolutionized text-based role-playing, creating immersive Speech Role-Playing Agents (SRPAs) requires a seamless bridge between cognitive reasoning and paralinguistic nuances. Current SRPAs primarily rely on end-to-end (E2E) fine-tuning. However, this paradigm suffers from poor generalization to unseen characters due to its reliance on role-specific data, while imposing a "modality alignment tax" that degrades intrinsic LLM reasoning capabilities. We propose DeSRPA, an agentic framework for character role play via inference-time intervention on frozen backbones. DeSRPA employs a dual-level control vector mechanism, Internal Cognitive Steering and External Expressive Rendering, to synchronize "mind" and "voice". Experiments on SpeechRole and OmniCharacter benchmarks demonstrate that DeSRPA significantly outperforms E2E baselines in personality and emotional consistency. It achieves high speech naturalness, narrowing the gap with proprietary models like GPT-4o Audio, while remaining a scalable and training-free paradigm.

• The paper introduces DeSRPA, a novel framework that decouples LLM cognitive reasoning from TTS expressive rendering using inference-time interventions.
• It employs dual-level vector-based control to modulate personality and style separately, achieving robust persona control without parameter updates.
• Experimental results demonstrate enhanced fluency, clarity, and prosodic consistency, outperforming conventional E2E and cascaded speech synthesis approaches.

DeSRPA: Decoupled Speech Role-Playing Agent via Inference-Time Intervention

Introduction and Motivation

Development of Speech-based Role-Playing Agents (SRPAs) has traditionally been constrained by the need for large-scale, role-specific supervision and by the inherent tension between maintaining cognitive depth in language generation and ensuring expressive, consistent speech output. End-to-end (E2E) supervised fine-tuning frameworks, though providing modality alignment, generally struggle with scalability and generalization to unseen characters due to the reliance on narrowly scoped datasets. Further, modality alignment via joint audio-text training often incurs degradation in the intrinsic reasoning abilities of the underlying LLMs. This leads to a trade-off between persona consistency and expressive prosody, a challenge not satisfactorily addressed by conventional cascaded LLM-TTS pipelines, which typically suffer from semantic-acoustic misalignment.

Framework Overview

The Decoupled Speech Role-Playing Agent (DeSRPA) framework introduces an inference-time intervention paradigm that circumvents the limitations of E2E adaptation by leveraging frozen pre-trained LLM and TTS backbones. The architecture decouples reasoning and rendering into dual levels of vector-based control: Internal Cognitive Steering and External Expressive Rendering. The LLM (Qwen3-4B) is modulated by injecting personality and linguistic style vectors directly into its residual stream, achieving persona-contingent text generation. The output is further aligned at the speech synthesis level using StyleTTS 2, which incorporates TTS-side emotion and style control vectors steered by the LLM’s intent outputs.

Figure 1: The DeSRPA framework, decoupling LLM-centered reasoning from expressive TTS rendering through synchronized vector-based interventions, with all modules operating without parameter updates.

Internal Cognitive Steering

DeSRPA’s internal intervention layer uses Sparse Autoencoders to construct disentangled control vectors that target specific layers within the LLM. Personality base vectors and context activation vectors, trained at mid-level transformer layers, govern deep semantic and situational reasoning, while style vectors at higher layers control phrasal and surface-level stylistic expression. Notably, these vectors are not confined to the Big Five but span a 30-facet trait taxonomy, with scaling coefficients for each vector type inferred at runtime from a profile database and refined using human-LLM collaborative annotation (Pearson $r=0.82$ for inter-rater reliability). The result is a highly dynamic, context-adaptive persona embedding, transferable across character identities without any parameter update or end-to-end retraining.

External Expressive Rendering

On the speech generation front, the TTS module (StyleTTS~2) renders voice and prosody by integrating acoustic style vectors obtained via a style subtraction mechanism in the latent space. These acoustic vectors capture emo-stylistic shifts independent of speaker identity through difference vectors computed on parallel emotional and neutral utterances (using ESD and CREMA-D datasets). Speech synthesis is further refined by a Dual-Path Fusion strategy that interpolates between pure reference style and emotion-steered style vectors, ensuring the balance between speaker identity retention (high SIM > 0.85) and emotion execution accuracy (EEA). Fine-grained control over emotional intensity is realized by adjusting a scalar factor $\tau$ in the style steering operation.

Experimental Results

Comprehensive evaluations demonstrate that DeSRPA surpasses major open-source E2E approaches on the SpeechRole and OmniCharacter benchmarks. In objective testing, DeSRPA achieves an EEA of 0.701 and SIM of 0.886 on SpeechRole, outperforming all open-source comparators and rivaling proprietary models like GPT-4o Audio. The framework yields a mean multimodal judge score of 0.8379, exceeding SpeechRole (0.7747) and LLaMA-Omni (0.7452), while showing particular gains in prosodic consistency and emotion appropriateness.

Ablation studies verify the necessity of each control vector subsystem: ablating LLM-side vectors degrades personality and knowledge consistency, while ablating TTS-side vectors dramatically reduces emotion execution. Responsiveness is maintained within practical limits (TTFA: 577 ms), outperforming heavier cascaded pipelines.

In human evaluations on OmniCharacter (stylized out-of-distribution setting), DeSRPA attains the highest scores in fluency (8.70), clarity (9.11), and emotional expressiveness (7.41), confirming the holistic benefits of decoupling and vector-based control, despite modest dips in persona consistency for highly stylized, exaggerated characters.

Theoretical and Practical Implications

DeSRPA demonstrates that character adaptation for speech role-playing can be efficiently and robustly achieved via inference-time injection of cognitive and acoustic control vectors, shifting the paradigm away from compute-heavy end-to-end fine-tuning. This methodology maintains the inherent language understanding and reasoning faculties of frozen LLMs, while its direct, training-free TTS intervention closes the semantic-acoustic alignment gap, enabling scalable, open-domain deployment. The use of disentangled trait and style spaces permits rapid, zero-shot persona adaptation across arbitrary character profiles with substantial gains in both text and speech-level fidelity.

Future Directions

Potential developments include extending the style and persona embedding spaces with unsupervised or cross-lingual attribute vectors, integrating more generalized emotional taxonomies, and exploring real-time, interactive control settings. The framework’s training-free character transfer and its practical scaling properties mechanistically pave a path toward voice-based agents with highly contextualized, persona-consistent behavior, suitable for conversational AI, entertainment, customer service, and assistive technology applications.

Conclusion

DeSRPA establishes a rigorous and generalizable decoupled framework for speech role-playing agents, enabling high-fidelity, scalable persona control by harmonizing LLM reasoning and expressive TTS rendering through inference-time modular interventions. The empirical results underline significant improvement over open-source E2E and cascaded baselines, and the approach narrows the performance gap with proprietary solutions. This work validates style control without end-to-end retraining as a foundation for future advances in speech-centric interactive AI.