Autoregressive Diffusion DPO for Speech Models

• ARDM-DPO is a post-training preference-alignment method that fine-tunes pretrained speech autoregressive diffusion models using a DPO-style objective.
• It addresses challenges like monotone expressiveness and poor long-text robustness by leveraging automatic, metric-driven pairwise preference data.
• Experiments with the DiTAR model show improved F0 variance, reduced word error rate, and maintained speaker similarity for zero-shot TTS outputs.

Searching arXiv for the specified papers and closely related work to ground the article. Autoregressive Diffusion-Direct Preference Optimization (ARDM-DPO) is a post-training preference-alignment method for speech autoregressive diffusion models (ARDMs) that fine-tunes a pretrained ARDM toward preferred speech outputs using a DPO-style objective rather than explicit reward modeling or reinforcement learning. In the formulation introduced for zero-shot text-to-speech (TTS), the method is applied to DiTAR and is designed to improve two weak points of zero-shot speech generation—expressiveness and robustness on hard or long texts—while preserving speaker similarity and naturalness reasonably well (Liu et al., 23 Sep 2025).

1. Definition and problem setting

ARDM-DPO is defined for the setting in which a pretrained speech ARDM already generates intelligible speech, but its outputs remain suboptimal with respect to preference criteria such as prosodic expressiveness or robustness on difficult text. The intended adjustment is not full retraining but post-training alignment: the generation distribution is biased toward preferred outputs while remaining close to the base model distribution (Liu et al., 23 Sep 2025).

The model class of interest is an autoregressive diffusion model for speech. Unlike discrete-token TTS systems, ARDMs generate continuous latent speech tokens. Generation is autoregressive over token positions, but each token is denoised through a diffusion process rather than produced by direct next-token prediction. This is presented as attractive for speech because it preserves fine acoustic detail and avoids the complexity of discrete speech tokenization (Liu et al., 23 Sep 2025).

The preference-alignment problem is motivated by concrete failure modes of zero-shot speech generation. Emotion prompts may still produce monotone speech; hard long texts may trigger omissions or insertions; and naive attempts to increase expressiveness may degrade speaker similarity and naturalness. ARDM-DPO addresses this by adapting the DPO paradigm to the ARDM setting, using preference pairs rather than an explicit reward model (Liu et al., 23 Sep 2025).

2. ARDM generative structure and mathematical formulation

The sampling process is modeled as a Markov chain over states indexed by token index $n$ and diffusion time $t$. Each state contains the already denoised history $x_{<n}^0$ and the current noisy token $x_n^t$. For a fixed number of tokens $N$ and diffusion steps $T$, the ARDM denoises token $n$ from $t=T$ down to $0$, then advances to token $n+1$ by reinitializing noise (Liu et al., 23 Sep 2025).

The forward perturbation is specified as

$t$0

with decreasing $t$1 and increasing $t$2. A DDPM-like sampling step is written as

$t$3

For ARDMs, the conditional score is

$t$4

so denoising proceeds one token at a time. The trajectory notation $t$5 denotes the full ARDM sampling trajectory including all intermediate states (Liu et al., 23 Sep 2025).

Preferences are assumed over output pairs $t$6, where $t$7 is preferred over $t$8, and are modeled with a Bradley–Terry likelihood: $t$9 The paper then formulates KL-regularized policy optimization over ARDM trajectories: $x_{<n}^0$0 where $x_{<n}^0$1 is the fine-tuned ARDM policy, $x_{<n}^0$2 is the reference model, and $x_{<n}^0$3 controls closeness to the reference. The reward-optimal policy takes the standard DPO form

$x_{<n}^0$4

with

$x_{<n}^0$5

Marginalizing over intermediate ARDM states yields a log-ratio expression over output samples: $x_{<n}^0$6 This provides the theoretical basis for adapting DPO to autoregressive diffusion speech generation (Liu et al., 23 Sep 2025).

3. Preference objective and denoising-loss interpretation

At trajectory level, ARDM-DPO defines

$x_{<n}^0$7

For a preference pair $x_{<n}^0$8, the DPO preference likelihood becomes

$x_{<n}^0$9

where the expectation is over diffusion time $x_n^t$0, token index $x_n^t$1, and the corresponding ARDM transition distributions (Liu et al., 23 Sep 2025).

Because direct optimization is intractable, the expectations are moved outside the sigmoid using Jensen’s inequality, producing a lower bound $x_n^t$2. The inner expectation of $x_n^t$3 can then be expressed as a difference of KL divergences: $x_n^t$4 Under the assumption that the ARDM is trained with a denoising objective, this becomes a difference of squared denoising errors: $x_n^t$5 The resulting ARDM-DPO objective is

$x_n^t$6

Its stated interpretation is that, relative to the reference model, the preferred sample is trained to receive a better normalized denoising score than the dispreferred sample (Liu et al., 23 Sep 2025).

The empirical instantiation uses DiTAR, a diffusion transformer autoregressive model using $x_n^t$7-prediction and continuous diffusion time $x_n^t$8, where $x_n^t$9 is pure noise. Noisy tokens are generated as

$N$0

with $N$1, and

$N$2

The specialized preference objective becomes

$N$3

where $N$4 is the token dimension and $N$5 in the reported experiments. The time-dependent weight $N$6 is omitted following prior Diffusion-DPO practice (Liu et al., 23 Sep 2025).

4. Specialization to DiTAR and training pipeline

The reported experiments fine-tune a 0.4B-parameter DiTAR model pretrained on around 280,000 hours of Chinese and English speech. The architecture uses 24 Transformer blocks for the language-model component, 4 blocks for the diffusion head, hidden size 1024, and 16 attention heads (Liu et al., 23 Sep 2025).

The same sampler is used for data generation and evaluation: a 16-step DDPM sampler with a linear time schedule and LM Guidance with weight $N$7, described as similar to classifier-free guidance. DPO training is conducted on 32 A100 GPUs with local batch size 1 pair, gradient accumulation 32, and effective batch size 1024 pairs. The optimizer is AdamW with learning rate $N$8, weight decay 0.01, and $N$9 (Liu et al., 23 Sep 2025).

A notable implementation feature is that the preference pairs are not derived from human preference labels directly. Instead, the method constructs preference data from automatic reward signals that differ by task. This places ARDM-DPO within a class of preference-optimization methods that preserve the pairwise-comparison interface of DPO while replacing direct human annotation with metric-defined winners and losers (Liu et al., 23 Sep 2025).

This suggests that, in the reported formulation, ARDM-DPO functions as a bridge between preference learning and metric-guided post-training for speech diffusion models. The paper does not describe this as explicit reward-model training; rather, the automatic metrics are used only to construct pairwise preference data (Liu et al., 23 Sep 2025).

5. Preference data construction and evaluation methodology

Two task-specific preference datasets are reported. For expressiveness, the data source is LibriTTS. For each prompt-text pair, the base model generates 32 candidates, scores them by F0 variance (F0V), and selects the highest-F0V sample as winner and the lowest-F0V sample as loser. This yields 256k preference pairs, approximately 1,000 hours. The purpose is to increase prosodic expressiveness and reduce monotonicity (Liu et al., 23 Sep 2025).

For long-text robustness, prompts come from DidiSpeech-2, described as 227h and 1500 speakers. The texts are long Chinese sentences with artificially introduced repetitive phrases or clauses. For each prompt-text pair, the base model generates 16 candidates and scores them using a phoneme-based CTC model. The reward proxy is negative log-likelihood per phoneme, with lower NLL indicating better text alignment and intelligibility. The winner is the sample with the lowest CTC NLL and the loser is the sample with the highest CTC NLL. This produces 430k preference pairs, about 3,500 hours. The CTC model itself is a small transformer with 6 transformer blocks, hidden size 1024, and 16 heads (Liu et al., 23 Sep 2025).

Evaluation combines objective and subjective protocols. Objective metrics are F0 variance for expressiveness; WER for English from Whisper-large-v3; CER for Chinese from Paraformer-zh; speaker similarity (SIM) using WavLM-TDCNN speaker embeddings; and token average KL divergence as a proxy for divergence from the reference model: $T$0 This quantity is reported as a measure of how far the fine-tuned model moves from the base model (Liu et al., 23 Sep 2025).

Subjective evaluation uses pairwise listening tests with 20 listeners comparing base and DPO outputs on naturalness, speaker similarity, and expressiveness. For the long-text task, subjective comparison is reported on 40 test cases (Liu et al., 23 Sep 2025).

6. Empirical findings, baselines, and limitations

On the expressiveness task, the base DiTAR model has F0V 14.2, SIM 0.770, and WER 5.17. ARDM-DPO with $T$1 and 200 steps achieves F0V 29.2, SIM 0.765, WER 3.73, and KL 0.010. The paper describes this as the strongest result in the table and notes that it nearly doubles F0 variance with only a small speaker-similarity drop (Liu et al., 23 Sep 2025).

The expressiveness baselines are Best-of-16, Best-of-64, and RAFT. Best-of-16 obtains F0V 22.5; Best-of-64 obtains F0V 26.6; and RAFT iterations improve F0V only to about 20.1 while degrading SIM and WER more. The reported conclusion is that DPO outperforms rejection-sampling fine-tuning and best-of-$T$2 selection on this task. A grid search over $T$3 indicates that larger $T$4 strengthens the KL constraint, leading to better SIM preservation but smaller F0V gains. Early stopping is recommended because training can become unstable and quality may degrade over longer runs. The paper also notes an observation that, although the DPO objective should reduce loss on winners and increase it on losers, in practice both losses can increase during training, similar to phenomena reported in LLM DPO (Liu et al., 23 Sep 2025).

Subjectively, listeners report that DPO slightly reduces naturalness and speaker similarity but significantly improves expressiveness on the expressiveness task. On the long-text task, the base model has NLL 0.55, SIM 0.711, and CER 8.37. ARDM-DPO with $T$5 and 9000 steps achieves NLL 0.32, SIM 0.712, CER 6.32, and KL 0.009. The paper characterizes this as roughly a 25% CER reduction with essentially unchanged SIM, indicating strong prior preservation (Liu et al., 23 Sep 2025).

For long-text robustness, Best-of-8 based on CER gives CER 4.99 but worse NLL; Best-of-8 based on NLL gives NLL 0.27 but CER 6.79. The paper presents DPO as providing a more balanced improvement and as being better aligned with the training objective than ad hoc selection. In subjective evaluation on 40 test cases, the DPO model performs almost identically to the base model on naturalness and speaker similarity: naturalness 4.4% lose / 88.7% tie / 6.9% win, and speaker similarity 2.1% lose / 94.3% tie / 3.6% win. The main gain is therefore described as robustness and intelligibility rather than an obvious shift in perceived quality (Liu et al., 23 Sep 2025).

The principal contribution is stated as showing that DPO can be adapted cleanly to autoregressive diffusion speech models by expressing preference alignment in terms of diffusion denoising losses and reference-model ratios. The authors further emphasize that this is, to their knowledge, the first preference-alignment method tailored to ARDMs for TTS. The main limitations identified are training instability on the expressiveness task, strong dependence on the quality of metric-driven preference data construction, and open questions about why both preferred and dispreferred diffusion losses can increase during DPO (Liu et al., 23 Sep 2025).

7. Relation to autoregressive DPO theory

ARDM-DPO is specific to autoregressive diffusion speech models, whereas Autoregressive DPO (ADPO) revisits DPO for autoregressive LLMs by making the autoregressive assumption explicit before applying the Bradley–Terry model (Oi et al., 10 Feb 2026). The two methods address different model classes and derive different training objectives, but both are motivated by aligning preference optimization more closely with autoregressive generation.

In ADPO, the central mathematical distinction from standard DPO is that the summation over token-level log-ratios is moved outside the log-sigmoid, yielding a sum of prefix-wise preference terms rather than a single response-level term. The paper presents this as a theoretically consistent refinement of standard DPO rather than a departure from it, and introduces the distinction between token length $T$6 and feedback length $T$7 to characterize preference granularity (Oi et al., 10 Feb 2026).

A plausible implication is that ARDM-DPO and ADPO occupy analogous positions in different autoregressive generative settings. ARDM-DPO adapts preference optimization to a trajectory over token indices and diffusion times in continuous latent speech generation, whereas ADPO adapts it to prefix-wise discrete autoregressive sequence modeling. The commonality is the attempt to reconcile DPO-style preference alignment with the internal factorization structure of the generator, but the concrete losses, state spaces, and theoretical constructions remain distinct (Liu et al., 23 Sep 2025, Oi et al., 10 Feb 2026).