Residual Weighted-Sum Encoding in Unified Speech Models

• Residual Weighted-Sum Encoding (RWSE) is a mechanism that fuses encoder layers using task-specific softmax-normalized weights and a residual connection to enhance speech processing.
• It leverages a two-stage fusion—combining weighted sum of all layers with direct addition of the final layer—to support joint optimization for diarization, separation, and ASR.
• Empirical evidence shows RWSE improves speaker diarization error rates and signal quality metrics by aligning low-, mid-, and high-level acoustic cues across tasks.

Residual Weighted-Sum Encoding (RWSE) is the layer-fusion mechanism introduced within the unified multi-speaker encoder (UME) architecture for joint speaker diarization (SD), speech separation (SS), and multi-speaker automatic speech recognition (ASR). In that formulation, the hidden representations from all layers of a shared speech foundation model—the OWSMv3.1 encoder—are combined by task-dependent softmax-normalized learnable weights, and the resulting weighted sum is added residually to the final encoder layer to produce the encoder output used by downstream task branches. In the source paper, RWSE is defined compactly in the speech-encoder section rather than as a standalone module, but it functions as the central representation interface for the multitask system (Shakeel et al., 28 Aug 2025).

1. Formal definition

RWSE is defined after the input speech mixture $X$ is passed through a stack of $L$ encoder layers. The layerwise hidden states are given by

$H_{(l)} = \mathrm{SpeechEnc}_{(l)}(X). \tag{2}$

The paper also specifies the input as a single-channel mixture waveform

$X = \{x_t \in \mathbb{R} \mid t=1,\dots,T\},$

and the layer outputs as

$$H_{(l)} = \left\{\mathbf{h}_{(t,l)} \in \mathbb{R}^{D^{\text{enc}}} \mid t = 1,\dots,T^{\text{enc}}\right\},$$

with $l \in \{1,\dots,L\}$. All encoder layers are then fused through a task-specific weighted sum,

$$H^{\text{ws}} = \sum_{l=1}^{L} \omega_{(l)}^{\text{task}} H_{(l)}. \tag{3}$$

The weights $\omega_{(l)}^{\text{task}}$ are described as softmax-normalized learnable weights. RWSE is obtained by adding this weighted sum to the final encoder layer,

$H^{\text{enc}} = H^{\text{ws}} + H_{(L)}. \tag{4}$

The paper states that this residual addition is introduced “to amplify the influence of the final layer” across tasks during training (Shakeel et al., 28 Aug 2025).

The mechanism is therefore a two-stage fusion rule: first, a task-specific weighted combination of all encoder layers; second, direct residual reinforcement of the top layer. The natural reading of the notation is that the weighting is layerwise, scalar, and task dependent. The paper does not specify weighting by time frame, channel, attention head, speaker stream, or input utterance, and it does not mention layerwise feature normalization before summation, post-sum normalization, residual scaling, or a learned gate on the residual branch. A faithful interpretation is thus a softmax over per-layer scalar parameters, followed by linear fusion and direct addition of $H_{(L)}$.

2. Motivation and semantic role

The motivation for RWSE follows from the paper’s broader claim that SD, SS, and multi-speaker ASR are usually trained independently and therefore fail to exploit their interdependencies. In a unified model, a single encoder representation must simultaneously support speaker attribution, source disentanglement, and linguistic recognition. The paper argues that relying on only one encoder layer—especially the top layer—may be suboptimal because different tasks require information at different semantic levels (Shakeel et al., 28 Aug 2025).

The immediate rationale for multi-layer fusion is drawn from prior findings, as summarized in the UME paper, that different encoder layers encode different types of information in SD and ASR tasks. The paper further notes preliminary observations that intermediate layers extract a rich hierarchy of information, and that in WavLM large, initial layers and last layers are more critical for SD and ASR tasks. This suggests a division of labor across the encoder hierarchy: lower and intermediate layers plausibly retain more local acoustic or speaker-related cues, whereas later layers plausibly encode more semantic or linguistic structure. RWSE is designed to preserve this hierarchy rather than collapsing the representation to a top-layer-only abstraction.

The residual component has a narrower purpose. The weighted sum grants access to all layers, while the residual addition preserves and amplifies the strongest final representation. The paper frames this as a way “to effectively use information from different semantic levels” and hypothesizes that RWSE “introduces information exchange and better bottom-up alignment to all the tasks from different semantic levels.” The paper does not formalize the term “bottom-up alignment.” A plausible interpretation is that all task branches consume a common representation that mixes low-, mid-, and high-level cues, so task supervision can act on the encoder hierarchy in a more coordinated way than if each branch read only the final layer.

3. Position within the UME architecture

RWSE is applied at the output of the shared OWSMv3.1 encoder and acts as the bridge between the shared speech foundation model and the three downstream task branches. The implementation section states that UME employs the pre-trained supervised SFM encoder OWSMv3.1 (medium) as a shared feature extractor for all tasks, and that learnable weights with RWSE are used “to optimize all OWSMv3.1 layers jointly” (Shakeel et al., 28 Aug 2025).

In the diarization branch, the paper states that the hidden state representations $L$0 from the speech encoder are used to estimate speaker activity, and the implementation specifies that RWSE features with frame length 400 and frameshift 640 samples are input to a 1-layer RNN attractor with hidden size 1024. In the separation branch, RWSE does not replace the separation frontend; instead, upsampled RWSE features are concatenated with Conv-TasNet features:

$L$1

The implementation further specifies that 1024-dimensional OWSM hidden representations are concatenated with 256-dimensional Conv-TasNet features, and that the RWSE features are upsampled because OWSM has 40 ms frame shift. In the ASR branch, RWSE is the direct shared acoustic input to the speaker-differentiating encoder blocks:

$L$2

The implementation states that OWSM features are projected to 128 dimensions and passed through four speaker-differentiating transformer blocks plus convolutional subsampling.

Architecturally, RWSE is therefore neither an auxiliary regularizer nor a branch-local enhancement. It is the common representational interface through which the shared encoder is exposed to SD, SS, and ASR. The figure description in the paper states that RWSE of intermediate layers acts as a bridge between these tasks, which is consistent with its role as the point at which shared hierarchical representations are made task-usable.

4. Empirical behavior and ablation evidence

The UME paper reports that the unified system substantially improves over single-task baselines dedicated to SD, SS, and multi-speaker ASR on LibriMix evaluation sets, and that for SD it outperforms previous studies, achieving diarization error rates of 1.37% on Libri2Mix and 2.29% on Libri3Mix in clean settings. The ablation evidence isolating RWSE compares three variants: A1, without weighted sum; A2, with weighted sum; and A3, with RWSE. These comparisons provide the main direct evidence for the residualized formulation (Shakeel et al., 28 Aug 2025).

For diarization, the clearest isolated gains appear when comparing weighted sum alone with weighted sum plus residual top-layer addition. In the two-speaker noisy mixboth setting with ASR initialization, DER improves from 2.45 for weighted sum to 2.14 for RWSE. In the corresponding three-speaker noisy mixboth setting, DER improves from 3.15 to 3.01. Against the reproduced EEND baseline, the RWSE-based UME reaches 2.14 versus 4.62 in the two-speaker noisy condition. These results support the conclusion that residualizing the layer fusion is particularly beneficial for diarization.

For separation, the gains over non-residual weighted sum are smaller but consistent. In the two-speaker noisy mixboth setting with ASR initialization, STOI improves from 89.82 to 89.95, SDR from 12.68 to 12.76, and SI-SNR from 12.12 to 12.22. In the three-speaker noisy mixboth setting, STOI improves from 86.48 to 86.71, SDR from 10.69 to 10.79, and SI-SNR from 10.07 to 10.18. Relative to the Conv-TasNet baseline, the two-speaker noisy SDR rises from 11.48 to 12.76 in the RWSE-based UME.

For ASR, the evidence is more mixed. In the noisy mixboth setting, the comparison between A1 without weighted sum and A3 with RWSE shows an improvement on Libri2Mix WER from 21.1 to 19.6, but RWSE is worse on some of the other reported sets than the best A1 setting. In clean mixclean conditions, the RWSE-based model reports 6.4 on Libri2Mix, 15.9 on Libri3Mix, 5.5 on LS2Mix, and 12.9 on LS3Mix, but the paper does not provide a clean matched ablation against weighted sum only or no weighted sum. Accordingly, the ablations establish a stronger RWSE-specific case for diarization than for ASR.

An important limitation of the ablation design is that the paper does not compare RWSE against simple layer averaging, concatenation across all encoder layers, learned input-conditioned gating, or an explicitly labeled last-layer-only baseline. The nearest reported baseline to last-layer-only is “w/o weighted sum,” and the nearest direct comparator to RWSE is the non-residual weighted sum.

5. Optimization properties, efficiency, and limitations

The paper gives little direct RWSE-specific cost analysis, but the mechanism itself is structurally lightweight. It requires retaining the hidden states from all encoder layers, learning one scalar weight per layer per task, computing a weighted sum, and adding the last layer. This suggests a very small parameter overhead relative to the shared encoder and downstream branches, while shifting most of the additional memory burden to storage of intermediate layer states rather than to the fusion parameters themselves (Shakeel et al., 28 Aug 2025).

The implementation statement that learnable weights with RWSE are used “to optimize all OWSMv3.1 layers jointly” suggests an optimization role beyond simple feature aggregation. Because all hidden states contribute to the downstream losses through the RWSE fusion, supervision can in principle shape the full encoder hierarchy rather than only the top layer. The paper associates this with “information exchange,” “bottom-up alignment,” and improved learnability, although it does not provide a formal optimization analysis.

Several reproducibility-relevant details remain unspecified. The paper does not fully resolve whether separate RWSE modules are instantiated explicitly for SD, SS, and ASR or whether some practical sharing is used despite the task-specific notation. It does not state whether the layer outputs are taken before or after layer normalization inside OWSM. It says that the encoder is fine-tuned during UME training, but it does not specify whether any initial freeze period is used. It also does not mention dropout or regularization on the RWSE weights, residual scaling, or gating of the residual branch.

The empirical scope also leaves open questions about task-specific layer preference. Although the notation $L$3 indicates task-dependent weights, the paper does not report the learned weight distributions, whether SD prefers lower layers and ASR higher layers, or whether the layer preferences shift during training. This omission is notable because the paper’s own motivation is explicitly semantic-level specialization across layers.

Training stability remains a separate concern. The paper states that training without ASR initialization often diverged in the three-speaker case and that ASR initialization is essential for stability. RWSE appears in the final stable and best-performing configuration, but the paper does not claim that RWSE itself resolves optimization instability.

6. Terminology, acronym ambiguity, and related concepts

The acronym “RWSE” is not unique across arXiv literature. In the graph-transformer paper “Simple Path Structural Encoding for Graph Transformers,” RWSE denotes Random Walk Structural Encoding, not Residual Weighted-Sum Encoding. There, RWSE is a pairwise structural encoding derived from multi-hop random-walk transition probabilities,

$L$4

and injected into self-attention as an edge- or pair-conditioned feature. That usage is unrelated to the speech multitask layer-fusion mechanism described above, and the graph paper explicitly treats RWSE as an existing graph encoding later replaced by SPSE (Airale et al., 13 Feb 2025).

A second related but distinct line appears in “WriteSAE: Sparse Autoencoders for Recurrent State.” That paper does not mention Residual Weighted-Sum Encoding explicitly, but it does formulate a sparse weighted decomposition of recurrent matrix state as

$L$5

A plausible conceptual relation is that both RWSE and WriteSAE rely on weighted combinations of internal representations, but the substrates differ fundamentally: RWSE fuses encoder layers in a shared speech representation, whereas WriteSAE decomposes native matrix-cache writes in recurrent LLMs. The latter therefore belongs to a broader family of weighted-sum internal encodings without being an instance of Residual Weighted-Sum Encoding in the UME sense (Young, 12 May 2026).

Within speech multitask modeling, RWSE specifically denotes the residualized layer-fusion rule

$L$6

used to expose multi-level encoder information to SD, SS, and multi-speaker ASR simultaneously. Its distinctive feature is not merely layer averaging, but the combination of task-specific softmax-normalized weighting over all layers with explicit residual reinforcement of the final encoder state.