Dual-Path Speech Encoder (DPSE)

• DPSE is an architecture that routes input signals through parallel local and global streams to capture fine-grained and long-range speech dependencies.
• For speech separation, DPSE segments recordings into chunks and processes them with intra- and inter-chunk layers, reducing error rates and computational cost.
• In audio-driven facial animation (KSDiff), DPSE disentangles head-pose and expression features using separate pooling windows and prosody-conditioned FiLM modulation.

Searching arXiv for the cited papers to ground the article in the relevant literature. tool call: arxiv_search({"query":"id:(Li et al., 2021) OR id:(Raj et al., 2021) OR id:(Lyu et al., 24 Sep 2025)","max_results":10,"sort_by":"relevance"}) Dual-Path Speech Encoder (DPSE) denotes a class of speech representation architectures that route an input signal through two complementary processing paths after a shared front end. In the literature provided here, the term appears explicitly in KSDiff, where a DPSE maps raw audio to head-pose features $\mathbf{f}_h^{1:T}$ and expression features $\mathbf{f}_e^{1:T}$ for talking-face generation (Lyu et al., 24 Sep 2025). Closely related work in continuous speech separation and streaming multi-talker ASR uses the same dual-path principle under the name “dual-path model,” alternating local-window and cross-window processing over long recordings (Li et al., 2021), and extending the idea to Streaming Unmixing and Recognition Transducer (SURT) for multi-turn meetings (Raj et al., 2021). This suggests that DPSE is best understood not as a single fixed blueprint but as a reusable architectural pattern for factorizing speech structure across complementary temporal or functional pathways.

1. Historical lineage and problem setting

Dual-path modeling entered long-recording speech processing as a response to the limitations of fixed-window continuous speech separation (CSS). In CSS, a long, partially overlapped recording with a varying number of speakers is difficult to process with utterance-level separation methods because segmenting the recording into independent windows fails to model long dependency in speech and leads to sub-optimum performance (Li et al., 2021). The dual-path formulation addresses this by combining local-window modeling with cross-window dependency modeling in a stacked architecture.

The same motivation reappears in streaming multi-talker ASR. “Continuous Streaming Multi-Talker ASR with Dual-path Transducers” investigates multi-turn meetings containing multiple speakers using SURT and reports that naively extending the single-turn model to this harder setting incurs a performance penalty; the proposed remedy is the dual-path modeling strategy first used for time-domain speech separation (Raj et al., 2021). The paper further states that both LSTM- and Transformer-based dual-path models improve word error rate (WER) performance while yielding faster convergence, and that chunk width randomization and curriculum learning are important training strategies in this setting (Raj et al., 2021).

A different but conceptually related use appears in KSDiff. There, the DPSE is not primarily a separator of overlapping speakers, but a speech encoder that disentangles expression-related and head-pose-related features for audio-driven facial animation (Lyu et al., 24 Sep 2025). The dual-path idea therefore spans at least two technical interpretations: temporal factorization for long-context speech separation and semantic factorization for motion-conditioned speech representation.

2. Local–global dual-path modeling for long recordings

In “Dual-Path Modeling for Long Recording Speech Separation in Meetings,” the encoder begins by segmenting a long discrete-time mixture $x[n]$ of length $T$ into $B$ overlapping chunks of length $L$ with hop size $H$: $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$ An STFT-domain formulation is also given. If

$W \in \mathbb{R}^{T_f\times F}$

is the magnitude spectrum of $x[n]$, then the segmented windows are

$\mathbf{f}_e^{1:T}$0

so that

$\mathbf{f}_e^{1:T}$1

The paper uses $\mathbf{f}_e^{1:T}$2 frames and hop $\mathbf{f}_e^{1:T}$3 (Li et al., 2021).

After a bottleneck linear layer or $\mathbf{f}_e^{1:T}$4 convolution maps each chunk from $\mathbf{f}_e^{1:T}$5 to $\mathbf{f}_e^{1:T}$6 with $\mathbf{f}_e^{1:T}$7, the network stacks $\mathbf{f}_e^{1:T}$8 identical dual-path blocks. Each block has an intra-chunk layer and an inter-chunk layer. For chunk $\mathbf{f}_e^{1:T}$9, the intra-chunk model $x[n]$0 produces

$x[n]$1

followed by bottleneck projection, layer normalization, and a residual connection: $x[n]$2 The inter-chunk layer then aggregates across chunks at each within-chunk time index $x[n]$3 by constructing

$x[n]$4

applying a chunk-axis model $x[n]$5,

$x[n]$6

and again using projection, layer normalization, and residual addition: $x[n]$7 The reassembled tensor $x[n]$8 is then passed to the next dual-path block (Li et al., 2021).

This formulation makes the encoder explicitly bi-scale. The intra-chunk layer captures local structure within a window, while the inter-chunk layer models long-range dependencies across the meeting timeline. That division is the defining technical property of the dual-path model in the separation literature.

3. Speech-feature disentanglement in KSDiff

KSDiff’s DPSE starts from raw waveform $x[n]$9 and produces two time-aligned sequences of speech embeddings: head-pose features $T$0 and expression features $T$1 (Lyu et al., 24 Sep 2025). Its shared front end applies a frozen Wav2Vec speech encoder, followed by projection and a Multi-Scale Dilated Convolution (MSDC) block: $T$2

$T$3

The MSDC block is defined as

$T$4

with branch outputs

$T$5

The concatenation $T$6 is projected back to dimension $T$7 and added residually to $T$8 (Lyu et al., 24 Sep 2025).

After this shared stage, the architecture splits into two parallel paths. The head-pose path emphasizes low-frequency structure by applying two temporal pooling windows of different sizes: $T$9 Each pooled representation is followed by a learned linear projection that preserves dimensionality.

The expression path emphasizes high-frequency and prosodic information. The waveform is used to extract per-frame fundamental frequency $B$0 and energy, yielding $B$1, which is passed through two parallel linear layers to produce FiLM parameters $B$2. The shared representation is then modulated as

$B$3

and short-window pooling followed by linear projection produces

$B$4

The paper characterizes this as an expression path driven by prosody-conditioned FiLM modulation (Lyu et al., 24 Sep 2025).

4. Cross-attention refinement and downstream coupling

KSDiff refines the three pooled token streams $B$5 using multi-head cross-attention. Let $B$6 denote these token streams, and let the corresponding key/value features be $B$7, where $B$8 and $B$9 (Lyu et al., 24 Sep 2025). For each branch,

$L$0

$L$1

For the coarse and fine head-pose branches, $L$2 is concatenated and projected to yield $L$3; for the expression branch, $L$4 (Lyu et al., 24 Sep 2025).

These outputs are then integrated into the remainder of KSDiff. Together with transcript embeddings $L$5, they feed two autoregressive Transformer predictors in Keyframe Establishment Learning (KEL), which produce binary keyframe sequences $L$6 and $L$7. In the Dual-Path Motion Generator, described as a conditional diffusion model based on DiffSpeaker, each branch is conditioned on either $L$8 or $L$9 to generate head-pose coefficients $H$0 and expression coefficients $H$1 (Lyu et al., 24 Sep 2025).

A common misconception is that the DPSE in KSDiff introduces an explicit disentanglement penalty. The paper states the opposite: it does not introduce an explicit adversarial or orthogonality loss on $H$2; instead, disentanglement is enforced by the architecture through separate pooling windows and FiLM modulation, together with downstream supervision from weighted BCE in KEL and diffusion, reconstruction, velocity, and multi-resolution spectral losses in the motion generator (Lyu et al., 24 Sep 2025). In that formulation, disentanglement is an inductive bias rather than a direct constraint.

5. Design choices and hyperparameters

The long-recording separation variant and the KSDiff variant instantiate the dual-path principle with markedly different operating points. In CSS, the speech encoder front end uses STFT with $H$3, hop $H$4, and magnitude $H$5 with $H$6, followed by segmentation into $H$7 and a bottleneck mapper to $H$8 (Li et al., 2021). The RNN configuration uses 2 DP-blocks, each with two 1-layer BiLSTMs, for approximately 13.9 M parameters. The Transformer configuration uses 5 DP-blocks, each local/global module being one Transformer-encoder with $H$9, $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$0, and 4 heads, for approximately 8.2 M parameters. The boosted DP-Transformer with convolutional sampling reaches approximately 10.1 M parameters (Li et al., 2021).

The convolutional-sampling variant inserts a 1D convolution between DP-blocks to downsample along the within-chunk time axis by $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$1, and a transposed 1D convolution before the last block to upsample back. The paper states that exact kernel size, stride, and padding are not specified there, while noting that one would typically choose kernel size $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$2, stride $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$3, and padding $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$4 in a re-implementation (Li et al., 2021). This caveat is important because it marks a boundary between the published architecture and re-implementation heuristics.

KSDiff’s DPSE uses a hidden dimension $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$5 for all embeddings; MSDC uses kernel size $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$6, $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$7 branches with dilations $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$8, and dropout 0.1. The front end uses stride $$B \;=\;\Big\lfloor \frac{T - L}{H}\Big\rfloor + 1, \quad x^{(b)}[\,\ell\,] \;=\; x\bigl[b\,H + \ell\bigr], \quad \ell=0,\dots,L-1,\; b=0,\dots,B-1.$$9 to obtain $W \in \mathbb{R}^{T_f\times F}$0, the fused projection dimension is $W \in \mathbb{R}^{T_f\times F}$1, and the pooling windows are $W \in \mathbb{R}^{T_f\times F}$2 s for coarse head motion, $W \in \mathbb{R}^{T_f\times F}$3 s for fine head motion, and $W \in \mathbb{R}^{T_f\times F}$4 s for expression. FiLM conditioning is implemented with two linear layers producing $W \in \mathbb{R}^{T_f\times F}$5. Multi-head cross-attention uses $W \in \mathbb{R}^{T_f\times F}$6-dimensional queries, keys, and values, and the number of heads is 8 in the KEL transformer. The optimizer is AdamW with learning rate $W \in \mathbb{R}^{T_f\times F}$7, batch size $W \in \mathbb{R}^{T_f\times F}$8, and 100 k iterations with 5 k warmup (Lyu et al., 24 Sep 2025).

These choices indicate that “dual-path” is not tied to one particular module family. In the separation work it is realized as local/global sequence modeling over chunked spectrograms; in KSDiff it is realized as pathway-specific pooling, prosody-conditioned FiLM, and cross-attention refinement over Wav2Vec-derived features.

6. Empirical behavior, effectiveness, and interpretation

The empirical evidence supplied in the cited works supports the practical value of dual-path encoders, but the measured gains depend on the task. In LibriCSS separation, the paper reports WER after continuous separation, stitching, and off-the-shelf ASR. For a 2.4 s window, the reported WERs are as follows: Mixture gives 15.4, 11.5, 21.7, 27.0, 34.3, and 40.5 for overlap conditions 0S, 0L, 10, 20, 30, and 40; Trans gives 16.0, 14.4, 19.0, 22.6, 29.5, and 33.5; DP-Trans gives 15.6, 14.7, 18.8, 22.8, 29.1, and 32.3; DP-Trans+ gives 14.2, 12.3, 17.4, 22.4, 29.1, and 32.5 (Li et al., 2021). The same study reports multiply-accumulate counts for a 100 s meeting of 31.5 G MACs for Trans, 31.5 G MACs for DP-Trans, and 21.4 G MACs for DP-Trans+, corresponding to approximately 30% relative reduction in computation for the convolutionally sampled dual-path Transformer (Li et al., 2021).

The SURT study extends the evaluation from 2-speaker single-turn sessions to multi-turn meetings containing multiple speakers and reports that LSTM- and Transformer-based DP models improve WER while yielding faster convergence. It also reports that chunk width randomization and curriculum learning are important through ablation studies, and that the models perform competitively with offline separation-based methods on LibriCSS meeting data (Raj et al., 2021). Because the detailed architectural sections are not included in the provided material, the evidence that can be stated here is limited to those high-level findings.

In KSDiff, Table 4 on HDTF reports a sharp degradation when the DPSE speech split is removed. The complete model gives LSE-C $W \in \mathbb{R}^{T_f\times F}$9, LSE-D $x[n]$0, LVE $x[n]$1, Diversity $x[n]$2, and BeatAlign $x[n]$3, while “w/o speech split” gives LSE-C $x[n]$4, LSE-D $x[n]$5, LVE $x[n]$6, Diversity $x[n]$7, and BeatAlign $x[n]$8 (Lyu et al., 24 Sep 2025). The paper states that this confirms that the two-path architectural split in DPSE is critical for both lip-sync accuracy and natural head-pose diversity, and that the complete KSDiff pipeline with DPSE yields state-of-the-art LVE, LSE-C, Diversity, and Beat-Align scores (Lyu et al., 24 Sep 2025).

Taken together, these results support a narrow but robust conclusion: dual-path speech encoders are effective when the target problem has separable structure that benefits from two complementary views of the signal. In long-recording separation and streaming ASR, that structure is local versus global temporal dependency. In KSDiff, it is head-pose-related versus expression-related speech information. A plausible implication is that the success of DPSEs depends less on the existence of exactly two branches than on whether the chosen split aligns with the supervisory and temporal structure of the downstream task.