PersonaPlex: Adaptive Full-Duplex Speech AI

• PersonaPlex is a full-duplex conversational system that integrates text-based role instructions with audio voice prompts for precise control over persona and speech.
• It employs a hybrid prompting mechanism by conditioning autoregressive joint text–audio generation on structured instructions and short speech samples, trained on a 2250-hour synthetic dialogue corpus.
• The model achieves state-of-the-art role adherence, speaker similarity, and dialog naturalness, making it a robust solution for customizable customer service and other interactive applications.

PersonaPlex enables full-duplex conversational speech modeling with simultaneous, fine-grained control over both voice and role identity. It achieves hybrid prompting by conditioning autoregressive joint text–audio generation on structured textual instructions and short speech samples, thus unifying previously orthogonal advances from instruction-tuned LLMs and zero-shot multispeaker text-to-speech (TTS) systems. PersonaPlex is trained on a large synthetic corpus pairing role-based service and question–answer (QA) dialogues with speaker-cloned renderings. Quantitative and qualitative benchmarks demonstrate state-of-the-art performance in role adherence, speaker similarity, dialog naturalness, and responsive duplex interaction, advancing the development of adaptive speech-driven agents for customer service and beyond (Roy et al., 14 Jan 2026).

1. Motivation and Problem Formulation

Prior full-duplex speech-to-speech (S2S) dialogue models delivered low-latency interactions, but lacked flexibility—supporting only a fixed assistant role and a unique synthesized voice. Real-world applications frequently require agents to assume varied personas (e.g., "bank teller," "technical support representative") and match a user-specified or user-provided voice. While zero-shot voice-cloning TTS can mimic new speakers from minimal samples and LLMs can simulate arbitrary roles via text prompts, these capabilities were not previously integrated in a speech-centric, low-latency, full-duplex system. PersonaPlex reformulates S2S dialogue as dual-conditioned joint text–audio generation, bridging this gap by conditioning autoregressive generation on both a textual role description and a short speech sample (Roy et al., 14 Jan 2026).

2. Hybrid Prompting Mechanism and Model Architecture

PersonaPlex is built on the Moshi duplex backbone, an autoregressive transformer architecture consuming interleaved user audio, agent text, and agent audio streams. The core architectural innovation is the Hybrid System Prompt, comprising two disjoint, boundary-marked prefix segments:

• Textual Role Prompt: A scenario-specific instruction provided via the agent-text channel, describing the persona and task (e.g., "You are Connor Blake, a bank teller at Sunshine Savings. Verify the user’s identity and account balance.").
• Voice Prompt: A 3–5 s target speaker sample inserted into the agent-audio channel, with padding on the agent-text channel.

During inference, a steady 440 Hz tone replaces the user-audio stream to stabilize conditioning. After the prompt, the model generates interleaved reply text and audio, matching the requested persona and voice identity under a unified autoregressive sequence model (Roy et al., 14 Jan 2026).

3. Synthetic Data Generation and Training Pipeline

PersonaPlex is trained using a 2250-hour synthetic corpus of role-prompted, two-speaker dialogue pairs:

• Role Contexts and Transcripts: Service domains (restaurant, bank, insurance) and scenario types (refund, inquiry, unfulfillable request) are sampled and expanded into full two-speaker transcripts using Qwen-3-32B and GPT-OSS-120B. QA scenarios use a fixed "friendly teacher" role with two-turn dialogues.
• Speech Synthesis: 26 k held-out single-speaker files from VoxCeleb, Libriheavy, LibriTTS, CommonAccent, and Fisher corpora seed the multi-speaker Dia-TTS system to synthesize both speakers’ audio, preserving conversational timing and interruptions. For QA, Chatterbox-TTS with random voices and negative silence simulates barge-in effects. A held-out set of 2,630 voices is reserved for speaker-similarity evaluation.

This synthetic data pipeline ensures diverse persona-role conditioning and robust generalization to unseen speakers and conversational scenarios (Roy et al., 14 Jan 2026).

4. Training Objectives and Loss Functions

Training proceeds from a pretrained Moshi checkpoint. Let $T$ denote discrete agent-side text tokens and $A$ acoustic tokens. The objective comprises:

• Role-Conditioning Loss: Cross-entropy (CE) over agent-text tokens, masked to omit the prompt region:

$$L_{\text{role}} = -\sum_{i \in \text{text}} M^{(\text{text})}_i \log P(t_i \mid h_{<i})$$

• Acoustic Generation Loss: Masked CE on agent-audio tokens with per-token weighting:

$$L_{\text{audio}} = -\sum_{j \in \text{audio}} M^{(\text{audio})}_j w(a_j) \log P(a_j \mid h_{<j})$$

where $w(a_j) = 0.02$ for non-semantic units and $1.0$ otherwise.

• Speaker-Similarity Loss: For zero-shot voice cloning, penalize the cosine distance between speaker embeddings of the prompt $v$ and generated speech $g$:

$$L_{\text{spk}} = 1 - \cos( E_{\text{spk}}(v), E_{\text{spk}}(g) )$$

• Total Loss:

$$L_{\text{total}} = L_{\text{role}} + L_{\text{audio}} + \lambda_{\text{spk}} L_{\text{spk}}$$

with $A$0 set small (e.g., $A$1) to avoid disrupting turn-taking fluency (Roy et al., 14 Jan 2026).

5. Benchmarking: Extending Full-Duplex-Bench and New Metrics

To rigorously assess persona adherence and conversational competence, the Service-Duplex-Bench, a 350-question extension to Full-Duplex-Bench, is introduced. This extension covers 50 customer-service contexts, each with a role description and seven role-specific, single-turn questions targeting proper-noun recall, context fidelity, handling of unfulfillable requests, and response to rude customers. This enables direct comparison of generic QA and structured service-role behaviors (Roy et al., 14 Jan 2026).

PersonaPlex is evaluated on:

• Role Adherence: Scored by GPT-4o (1–5 scale) using Service-Duplex-Bench.
• Speaker Similarity: Cosine similarity (WavLM-TDNN SSIM) between the prompt and generated voice.
• Dialog Naturalness: Human DMOS ratings (1–5) via Amazon MTurk.
• Duplex Dynamics: Metrics such as turn-over-rate (TOR), backchannel frequency, and latency from Full-Duplex-Bench.

Table: Principle benchmark results (Full-Duplex-Bench / Service-Duplex-Bench / SSIM):

Model	F-D B	S-D B	SSIM
PersonaPlex	3.90	3.59	0.57
Gemini Live	3.72	3.22	0.00
Qwen-2.5-Omni	3.70	2.37	0.07

On Service-Duplex-Bench (GPT-4o mean score):

• PersonaPlex: 4.48
• Freeze-Omni: 4.02
• Qwen-2.5-Omni: 2.76
• Moshi: 1.75

PersonaPlex matches or exceeds Moshi's responsiveness (TOR = 1.000, latency = 0.400 s) while offering much improved backchannel naturalness (freq = 0.025 vs. Moshi 0.001) (Roy et al., 14 Jan 2026).

6. Qualitative Behaviors and Interaction Analysis

Empirical analyses of PersonaPlex in structured customer-service settings show that it can convincingly "become" distinct personas. For example, in the "Brody Murphy, insurance agent" role, the model calmly declines illogical requests (e.g., same-day insurance enrollment), whereas in the "Wise Teacher" prompt, it adopts an expository tone with clarifying asides. When given a user voice prompt, reply speech exhibits matching accent, pitch, and prosodic variation, even for previously unseen speakers, while still maintaining strict script adherence. During abrupt interruptions, the model produces fluid, context-appropriate backchannels ("Mm-hmm," "I see") rather than clipped truncations, indicating that the hybrid prompting paradigm does not compromise full-duplex temporal dynamics. This demonstrates that both text-based persona instructions and audio-based voice specifications are jointly and robustly supported within an unmodified duplex framework (Roy et al., 14 Jan 2026).

7. Significance, Limitations, and Prospects

PersonaPlex substantiates that joint text–audio conditioning enables simultaneous, high-fidelity control over both role and voice in real-time duplex dialog. State-of-the-art results in role adherence, voice fidelity, dialog naturalness, and responsiveness position PersonaPlex as a foundation for highly flexible, customizable, speech-driven agents in conversational AI. A plausible implication is the broader adoption of hybrid prompting for agent personalization in settings such as customer support, tutoring, and multimodal human–computer interaction. Performance is currently bounded by synthetic-training domain coverage and the limitations of text and audio prompt length, suggesting future research in real-world adaptation and prompt-efficient personalization (Roy et al., 14 Jan 2026).