Dual-Channel Naturalness Estimator

Updated 4 July 2026

The paper demonstrates that encoding user and system audio channels separately and fusing them at the segment level yields improved in-domain conversation naturalness, as measured by higher PCC and lower MSE.
It compares explicit dual-channel separation to single-channel fusion, showing that while dual-channel models achieve better in-domain performance, single-channel methods offer greater out-of-domain robustness.
The estimator leverages pretrained encoders like Whisper, AES, and WavLM with synthetic augmentation to optimize predictions on extensive conversational datasets, emphasizing structured paired-signal analysis.

Searching arXiv for the primary paper and closely related dual-channel naturalness work. A dual-channel naturalness estimator is an automatic model that predicts perceived naturalness from paired, time-aligned inputs rather than from a single stream. In the most explicit speech formulation, it estimates the human-likeness of a two-speaker, multi-turn conversation by encoding the user and system audio channels separately, fusing them at the segment level, and regressing to human Mean Opinion Score (MOS) judgments (Xu et al., 2 Mar 2026). Closely related formulations appear in stereoscopic image quality assessment, where left and right views are modeled jointly (Bourbia et al., 2021); in neural machine translation, where naturalness and content preservation are treated as two reward channels (Lai et al., 2024); in visual assessment of physical-world adversarial attacks, where rating priors and gaze priors form two aligned channels (Li et al., 2023); and in turn-taking evaluation, where future two-speaker voice-activity likelihood is scored over two channels (Zhang et al., 1 Jul 2026). Taken together, these works define the topic as a family of estimators in which naturalness is treated as a property of coordinated paired signals rather than of isolated outputs.

1. Conceptual basis and problem definition

In conversational speech, the relevant distinction is between utterance-level naturalness and conversation-level naturalness. The conversational formulation evaluates two related targets on two-channel recordings: conversation naturalness, defined as the perceived human-likeness of the entire two-speaker conversation, and system naturalness, defined as the perceived human-likeness of the system’s audio channel while listeners are exposed to both channels so that turn-taking, fillers, and overlaps influence judgment (Xu et al., 2 Mar 2026). Each recording is rated by at least 5 raters on a 1–5 MOS scale, where 1 = not natural at all and 5 = indistinguishable from humans, and the mean opinion score is used as the ground-truth label; inter-rater reliability is not reported.

The central motivation is that existing speech naturalness predictors were designed for single-speaker utterances and primarily respond to quality or fluency, not to interactional dynamics. The conversational study states that such predictors do not capture smooth turn-taking, fillers or backchannels, overlap management, or the appropriateness of expressions given context (Xu et al., 2 Mar 2026). Empirically, on the in-domain ConvTTS dataset, NISQA and UTMOSv2 correlate poorly, and sometimes negatively, with human conversation and system naturalness. For conversation naturalness, NISQA PCC is reported as Mean $-0.18$ , Min $-0.13$ , Max $-0.18$ , Median $-0.18$ , while UTMOSv2 gives Mean $-0.19$ , Min $-0.16$ , Max $-0.18$ , Median $-0.18$ (Xu et al., 2 Mar 2026).

A closely related conceptual move appears in other fields. In stereoscopic image quality assessment, naturalness is defined not only by the statistics of each view but also by the statistical dependence across the two views, and the auxiliary task predicts naturalness-analysis features adapted to stereo images (Bourbia et al., 2021). In neural machine translation, naturalness is treated separately from content preservation, and the two are combined into a single reward (Lai et al., 2024). In physical-world attack assessment, naturalness is modeled through both human rating behavior and human attentional behavior (Li et al., 2023). This suggests that “dual-channel” is not tied to one modality; it denotes an estimator whose target depends on structured interaction between paired signals or paired priors.

2. Conversational speech architecture

The conversational speech estimator in "Conversational Speech Naturalness Predictor" processes natively captured two-channel user–system recordings, so no diarization is required (Xu et al., 2 Mar 2026). For the proposed model, both channels are divided into fixed-length 30-second segments, and variable-length conversations therefore produce a variable number $T$ of segments. For each segment and each channel, the model computes a weighted sum of all hidden layers of a pretrained encoder and then mean-pools over time to obtain a fixed-size vector. The weighted sum is learned end-to-end with the predictor.

The channel-specific segment embeddings are written as

$h_t^U = E_U(x_t^U), \qquad h_t^S = E_S(x_t^S),$

where $-0.13$ 0 and $-0.13$ 1 denote the user and system audio segments, and $-0.13$ 2 and $-0.13$ 3 are the encoders (Xu et al., 2 Mar 2026). Fusion is performed by concatenation followed by a 3-layer MLP with hidden size 768, GeLU activation, and dropout 0.1:

$-0.13$ 4

A final linear layer maps the fused representation to a segment-level MOS prediction,

$-0.13$ 5

and the conversation-level score is the mean over segments:

$-0.13$ 6

The evaluated encoders are WavLM-large, Audiobox-Aesthetics (AES), and Whisper large-v3 (Xu et al., 2 Mar 2026). A single-channel system-only variant uses only $-0.13$ 7, while a combined-channel ablation concatenates user and system waveforms into one input and encodes them jointly. The paper reports that explicit dual-channel separation is superior to combining both channels into a single input: on ConvTTS conversation naturalness, the combined-channel ablation reaches 0.458 PCC versus 0.482 for explicit dual-channel separation; on ConvTTS system naturalness, 0.557 versus 0.570; and on out-of-domain FDX-Conv system naturalness, 0.110 versus 0.290 (Xu et al., 2 Mar 2026).

No explicit RNN, LSTM, or Transformer temporal module is applied over the 30-second segments. Aggregation is by segment-wise averaging rather than attention pooling. The paper notes that attention pooling could be formulated, but mean pooling is the method actually used (Xu et al., 2 Mar 2026).

3. Training protocol, datasets, and augmentation

The loss is mean squared error to human MOS,

$-0.13$ 8

with no explicit correlation-maximization term in the objective (Xu et al., 2 Mar 2026). Optimization uses Adam. AES and Whisper predictors are trained with batch size 32 and learning rate 0.002, while WavLM predictors use batch size 16 and learning rate 0.001. Training runs for 30 epochs, and the checkpoint with the lowest development loss is selected each epoch.

The in-domain dataset, ConvTTS, contains 6,579 synthetic two-channel user–system conversations generated with an internal offline TTS model, split into 4,579 train, 1,000 development, and 1,000 evaluation conversations (Xu et al., 2 Mar 2026). Mean duration is 49.4 s with standard deviation 19.3 s. All conversations have conversation naturalness labels, and subsets have system naturalness labels. The out-of-domain dataset, FDX-Conv, contains 490 conversations from a real full-duplex model interacting with internal workers and includes overlaps and interruptions typical of simultaneous listening and speaking; it has system naturalness labels only (Xu et al., 2 Mar 2026).

A large-scale synthetic augmentation stage pretrains the estimator on 5000 hours of augmented conversations with conversation naturalness ratings (Xu et al., 2 Mar 2026). The augmented dialogues are generated from transcripts produced by Llama-3.1-405B, while the training-set conversations are used as reference signals for both user and system channels to mimic speaker characteristics. The original conversation’s MOS label is assigned to the augmented sample. Pretraining runs for 5 epochs with batch size 32 and learning rate 0.001, with evaluation every 1000 steps and checkpoint selection by highest development correlation, after which the model is fine-tuned on real ConvTTS.

In-domain gains from augmentation are negligible or negative, but out-of-domain robustness improves. For Whisper dual-channel on FDX-Conv system naturalness, PCC rises from 0.290 without pretraining to 0.358 with pretraining, a reported +23.45% relative gain (Xu et al., 2 Mar 2026). This suggests that the augmentation is functioning primarily as a robustness prior rather than as a direct in-domain performance booster.

4. Empirical behavior and design trade-offs

On ConvTTS conversation naturalness, the best single-channel result is Whisper with PCC 0.433, SRC 0.406, and MSE 0.221, while the best dual-channel result is Whisper with PCC 0.482, SRC 0.451, and MSE 0.208 (Xu et al., 2 Mar 2026). On ConvTTS system naturalness, Whisper single-channel reaches 0.550/0.538/0.226 and Whisper dual-channel reaches 0.570/0.560/0.218, which is the best reported in-domain system-naturalness result (Xu et al., 2 Mar 2026). These numbers reverse the negative correlations observed for NISQA and UTMOSv2 on the same data.

The out-of-domain result is more nuanced. On FDX-Conv system naturalness, the best model is not dual-channel but single-channel Whisper, with PCC 0.362, SRC 0.358, and MSE 0.438 (Xu et al., 2 Mar 2026). The corresponding dual-channel Whisper model drops to PCC 0.290, SRC 0.285, and MSE 0.633. The paper attributes this degradation to user-channel mismatch: ConvTTS uses TTS on the user side, whereas FDX-Conv has real human user speech, which makes dual-channel features noisier out of domain.

The encoder comparison is consistent: Whisper outperforms AES and WavLM across tasks and datasets, and the authors attribute this to its 680k-hour multilingual pretraining (Xu et al., 2 Mar 2026). At the same time, the fusion mechanism remains simple. Concatenation followed by an MLP is sufficient to obtain gains; no cross-attention or co-attention mechanism is required. Statistical significance tests are not reported.

A recurrent misconception is that dual-channel modeling is automatically superior once two channels are available. The conversational evidence does not support that general claim. Dual-channel modeling is consistently better in-domain, but single-channel system-only inference is lighter and more robust out of domain (Xu et al., 2 Mar 2026). Another misconception is that any naturalness predictor trained on single utterances can be aggregated to conversation level. The negative PCC values for NISQA and UTMOSv2 on ConvTTS show that this extrapolation can fail badly (Xu et al., 2 Mar 2026).

The same label is used in several adjacent literatures, although the “channels” differ. In stereoscopic image quality assessment, the dual-channel estimator is the auxiliary branch of a multi-task CNN that predicts a 108-D feature vector derived from dual-tree complex wavelet magnitudes, Gamma marginals, and a Gaussian copula over left and right views (Bourbia et al., 2021). The stereo model uses sub-networks for the left and right images, additional fusion modules, a 2048-D binocular feature vector, and a joint loss in which the naturalness branch improves the quality-prediction branch. The full model improves from 0.953/0.935 to 0.957/0.942 on LIVE Phase I and from 0.902/0.897 to 0.921/0.915 on LIVE Phase II when compared with the version without the auxiliary task (Bourbia et al., 2021).

In neural machine translation, the term is operationalized as a two-channel reward: one channel scores naturalness with a translationese classifier, and the other scores content preservation with COMET (Lai et al., 2024). The naturalness reward is thresholded at $-0.13$ 9, the content reward at $-0.18$ 0, and the overall reward is the harmonic mean when both channels are nonzero. In the English-to-Dutch literary setting, the best dual-channel configuration, COMET + MT-HT classifier, yields MTLD 93.3 versus 90.4 for the base MT system while maintaining COMET 82.2 versus 82.3, KIWI 80.6 versus 80.4, and MetricX 2.63 versus 2.66 (Lai et al., 2024).

In visual naturalness assessment for physical-world adversarial attacks, Dual Prior Alignment (DPA) implements two channels: rating prior alignment and attentive prior alignment (Li et al., 2023). The first aligns a prototype-based rating distribution with the empirical human rating distribution via KL divergence, and the second aligns a modified Grad-CAM attention map with human gaze saliency via a Frobenius-norm loss. On PAN, DPA achieves SROCC 0.7501, PLCC 0.7727, and gaze-alignment cosine similarity 0.7178 (Li et al., 2023).

In turn-taking evaluation, TurnNat uses two speaker channels and models the joint future voice-activity state over a 2 s horizon partitioned into four bins per speaker, resulting in $-0.18$ 1 future-activity states (Zhang et al., 1 Jul 2026). Frame-level negative log-likelihood is pooled over turn-taking boundary units and aggregated via mean and tail statistics into a dialogue-level score. The best reported configuration reaches pairwise accuracy 88.0% and C-index 0.676 on a controlled perturbation benchmark (Zhang et al., 1 Jul 2026).

These formulations are not identical. A plausible common pattern is that the estimator preserves channel separation long enough to represent cross-channel dependence explicitly, and only then fuses or aggregates the resulting signals.

6. Boundary cases, limitations, and future directions

The topic has important boundary conditions. "Deep Learning Based Assessment of Synthetic Speech Naturalness" does not present a dual-channel naturalness estimator; it presents a single-ended CNN–biLSTM predictor, although its transfer learning labels are derived from POLQA, which is a dual-channel full-reference speech-quality measure (Mittag et al., 2021). This is relevant because “dual-channel” can refer either to model input structure or to the provenance of supervisory signals, and the paper explicitly distinguishes those cases.

Within conversational speech, several limitations are stated directly. The estimator uses audio alone and does not incorporate text-based discourse coherence or alignment (Xu et al., 2 Mar 2026). It has no explicit turn-structure or overlap model beyond segment pooling, even though dual-channel input allows some interactional cues to be learned implicitly. More-than-two-speaker scenarios are not addressed. Code and data release are not indicated, and both ConvTTS and FDX-Conv are internal datasets.

Several future directions are named rather than demonstrated. The conversational paper proposes cross-attention between channels, integration of ASR text for multimodal naturalness, correlation-aware training objectives, adaptation to multi-speaker and multilingual conditions, and online streaming via segment-wise scoring (Xu et al., 2 Mar 2026). TurnNat identifies sensitivity to VAD errors and to channel misalignment, and notes that its evaluation focuses on timing naturalness rather than semantic appropriateness or prosodic style (Zhang et al., 1 Jul 2026). In neural machine translation, domain transfer beyond literary English→Dutch is said to require retraining evaluators and possibly COMET calibration (Lai et al., 2024). In visual attack assessment, the PAN-to-PAN-phys drop indicates a notable domain gap (Li et al., 2023).

A final misconception is that naturalness is a monolithic scalar independent of the evaluation context. Across these works, naturalness is operationalized as dialog-level human-likeness, binocular statistical regularity, translationese reduction balanced against adequacy, or alignment with human ratings and gaze. This suggests that a dual-channel naturalness estimator is best understood not as a single architecture, but as a modeling principle: naturalness is estimated from paired evidence whose dependency structure is itself perceptually meaningful.