MimicLM: Zero-Shot Voice Imitation through Autoregressive Modeling of Pseudo-Parallel Speech Corpora

Abstract: Voice imitation aims to transform source speech to match a reference speaker's timbre and speaking style while preserving linguistic content. A straightforward approach is to train on triplets of (source, reference, target), where source and target share the same content but target matches the reference's voice characteristics, yet such data is extremely scarce. Existing approaches either employ carefully designed disentanglement architectures to bypass this data scarcity or leverage external systems to synthesize pseudo-parallel training data. However, the former requires intricate model design, and the latter faces a quality ceiling when synthetic speech is used as training targets. To address these limitations, we propose MimicLM, which takes a novel approach by using synthetic speech as training sources while retaining real recordings as targets. This design enables the model to learn directly from real speech distributions, breaking the synthetic quality ceiling. Building on this data construction approach, we incorporate interleaved text-audio modeling to guide the generation of content-accurate speech and apply post-training with preference alignment to mitigate the inherent distributional mismatch when training on synthetic data. Experiments demonstrate that MimicLM achieves superior voice imitation quality with a simple yet effective architecture, significantly outperforming existing methods in naturalness while maintaining competitive similarity scores across speaker identity, accent, and emotion dimensions.

• The paper presents a novel data construction paradigm that leverages synthetic TTS as source paired with real speech targets to eliminate synthesis quality limits.
• The methodology integrates interleaved text-audio prediction with autoregressive and flow-matching decoders to significantly reduce WER and improve content fidelity.
• Empirical results demonstrate superior performance in speaker and accent similarity, validating the efficiency of role-swapping and preference alignment strategies.

MimicLM: Zero-Shot Voice Imitation through Autoregressive Modeling of Pseudo-Parallel Speech Corpora

Introduction and Motivation

Voice imitation (VI) is a challenging speech generation task that requires accurate transformation of source speech into the timbre, prosody, and expressive style of a reference speaker while maintaining the input’s linguistic content. Constructing appropriate parallel data for learning—triplets of (source, reference, target) with identical content but divergent speaker characteristics—is nontrivial due to the absence of naturally occurring datasets and prohibitive costs of large-scale data collection.

Traditional approaches either attempt explicit disentanglement between content and speaker attributes, deploying complex models and multi-stage pipelines, or generate synthetic training targets using text-to-speech (TTS) or voice conversion (VC) systems. However, the latter strategy creates an inherent quality ceiling; the fidelity of generated speech is constrained by the limitations of the external synthesis system.

MimicLM rethinks the data construction paradigm, inverting the typical roles of synthetic and real speech: TTS-generated speech is used as the source, and real speech serves as the target for model supervision. This innovation provides two pivotal advantages: (1) direct modeling of real speech distributions, thereby removing the synthesis quality ceiling, and (2) improved referential consistency through aligned real recordings of the same speaker.

Architecture and Model Design

The MimicLM framework comprises three principal components: a frozen audio tokenizer, an autoregressive Transformer decoder, and a flow-matching waveform decoder. The system leverages CosyVoice 2.0’s codebooks for discrete audio tokenization and waveform reconstruction and is further distinguished by two training-time augmentations: interleaved text-audio modeling and preference-based post-training alignment.

Figure 1: Overview of the MimicLM architecture. Reference and source audio are encoded, and the decoder generates target speech in a chunked then continuous manner. Frozen components are denoted with snowflake icons.

Four-Stage Data Construction Pipeline

Data construction is articulated in four discrete steps to ensure scalability and quality:

1. Random Speaker Pairing: Sampling two speakers from the Emilia corpus, two adjacent utterances from one, and a single utterance from another.
2. Cross-Speaker Synthesis: Employing TTS to synthesize an utterance in a new speaker's voice, maintaining alignment in textual content.
3. Role-Swapping: Using the synthetic output as the source and real speech as the target, thus enabling the model to learn to generate real speaker audio.
4. ASR-Based Quality Filtering: Using Whisper Large-v3 to enforce a maximum WER of 0.1 for data pair acceptance, thereby removing approximately 33% of potential training samples.

   Figure 2: Four-stage pseudo-parallel data pipeline with role-swapping. Only high-quality pairs with low ASR error are retained for training.

Interleaved Text-Audio Modeling

To address content preservation—especially crucial in VI tasks with style transfer—MimicLM introduces an interleaved prediction structure. During the chunked phase, the model alternates between text and audio tokens, where explicit textual supervision acts as a semantic guide for the subsequent audio token prediction. This structure, implemented with augmented Qwen2 vocabulary, significantly reduces the WER compared to purely audio-supervised learning. The design’s efficacy is empirically supported through an ablation revealing notable content fidelity improvements due to interleaved prediction.

Preference Alignment for Distributional Robustness

Although role-swapping eliminates the synthetic ceiling, the synthetic source/real target training regime can induce domain mismatches at inference when only real utterances are available at both endpoints. MimicLM mitigates this with Direct Preference Optimization (DPO) post-training. By generating multiple candidate outputs for each input, ranking them via WER and acoustic similarity, and optimizing according to these preferences, the model becomes robust to synthetic-to-real distribution shifts, greatly reducing WER when evaluated under real input conditions.

Empirical Results

Evaluation is conducted on SeedTTS test-vc-en and MimicLM-Test, utilizing objective (UTMOS, OVRL, WER, S-SIM, A-SIM, E-SIM) and subjective mean opinion scores.

Numerical Highlights:

• The DPO-aligned MimicLM model reaches WER of 8.25% (full imitation setting), outperforming Vevo (9.10%) and SeedVC v2 (6.32%).
• UTMOS, OVRL, and SIG for MimicLM (DPO) surpass or approach the strongest timbre and full-voice imitation baselines.
• Speaker similarity (S-SIM) and accent similarity (A-SIM) for MimicLM (DPO) are competitive with or exceed most existing systems.
• Subjective N-MOS and S-MOS for the DPO model achieve 4.71 and 4.62, distinctly higher than SeedVC and Vevo scores.
• On MimicLM-Test, DPO reduces WER on real/real inputs from 15.80% to 13.81%, outperforming Vevo’s 17.99%.

Ablations and Data Scaling

Ablations confirm the necessity of each key component:

• Role-swapping is critical for high-fidelity and high similarity outputs.
• Interleaved text-audio modeling is essential for minimizing WER, especially under style transfer.
• Preference alignment substantially reduces inference-time error rates on real audio.

Scaling analysis reveals substantial and consistent improvements for both naturalness and similarity metrics as data volume increases, indicating effective learning in the low-resource and large-data regime alike.

Figure 3: Training data scaling curves for WER and S-SIM. Gains are non-saturating as data grows.

Implications and Theoretical Impact

MimicLM demonstrates that scalable, high-quality voice imitation is achievable without elaborate disentanglement architectures or multi-stage pipelines, provided the data construction strategy inverts the traditional use of synthetic speech. This paradigm strips away the prevalent dependency on synthesis system quality for generation and establishes a model that is directly optimized for naturalistic real speech. The findings indicate that end-to-end autoregressive modeling, fuelled by well-curated pseudo-parallel corpora and robust post-training preference alignment, can match or surpass the VI capabilities of much more intricate systems.

Practically, this approach enables better generalization and higher quality in real-world applications like assistive technologies, content creation, and expressive speech synthesis. Theoretically, it challenges the necessity for complex factorization modules and emphasizes the importance of data regime design and preference modeling, suggesting a broader role for such strategies in other speech and non-speech generative models.

Limitations and Future Directions

Despite its effectiveness, MimicLM’s reliance on high-quality TTS for data generation introduces an upstream dependency and potential biases. Additionally, the computational burden of mass-scale TTS synthesis and DPO alignment is non-negligible. Although English is the primary evaluation domain, initial multilingual experiments suggest generalizability, but real-world robustness to extreme variation in speaker style, environmental noise, and accent diversity warrants comprehensive exploration.

Future research should investigate further closing the synthetic-to-real gap, enhanced cross-lingual transfer, and more efficient preference data generation mechanisms. Integration of advanced controllability and safety measures (e.g., watermarking, detection) is also essential to address potential misuse.

Conclusion

MimicLM redefines the foundation for voice imitation by learning directly from real speech targets, leveraging synthetic sources only for scalable data construction. Interleaved text-audio modeling and preference-aligned post-training enable the system to outperform or match current state-of-the-art methods, minimizing complexity while maximizing fidelity, intelligibility, and similarity. The role-swapping strategy resolves longstanding quality constraints inherent to synthetic target training, empowering future research and application in expressive speech generation.