Unified Multi-Speaker Encoder (UME)

• UME is a unified speech modeling architecture that combines diarization, separation, and ASR through a shared encoder and dedicated task branches.
• It utilizes techniques such as residual weighted-sum encoding and layer-specific projections to extract both semantic and speaker representations effectively.
• Empirical results demonstrate enhanced performance in diarization and ASR, highlighting the importance of methods like ASR initialization and branch supervision.

Searching arXiv for the cited papers to ground the article and confirm bibliographic details. Unified Multi-Speaker Encoder (UME) denotes a class of unified speech modeling architectures in which a single shared speech encoder is trained to support multiple speaker-related or speaker-conditioned objectives through task-specific branches rather than through fully separate encoders. In the cited literature, the term is used explicitly for an end-to-end architecture that jointly addresses speaker diarization, speech separation, and multi-speaker automatic speech recognition with a shared speech foundational encoder (Shakeel et al., 28 Aug 2025). Closely related work extends the same organizing principle to arbitrary utterance-level attributes, showing that a single speech foundation model can jointly learn semantic representations and speaker representations from one shared self-supervised backbone (Bouziane et al., 9 Mar 2026). An earlier antecedent is a unified recurrent model for speech recognition and speaker recognition in which the output of one task is fed to the input of the other across time, emphasizing cross-task coupling for negatively correlated tasks rather than naive parameter sharing (Tang et al., 2016).

1. Conceptual scope and defining properties

The unifying premise behind UME is that speech tasks such as speaker diarization, speech separation, multi-speaker ASR, semantic retrieval, and speaker recognition are interdependent and may benefit from a common encoder that exposes reusable intermediate structure. In this view, the encoder is “unified” not because all information is collapsed into a single undifferentiated vector, but because one shared backbone supports multiple simultaneous representation targets or downstream heads.

A central distinction in the literature is between ordinary shared-feature multitask learning and UME-style designs. In the 2026 utterance-level framework, a single SSL speech encoder is reused by several attribute-specific branches, each supervised by a different frozen teacher embedding space, so that the shared backbone can preserve both semantic and speaker information (Bouziane et al., 9 Mar 2026). In the 2025 overlapping-speech model, the shared speech foundational encoder supports three tasks at once—speaker diarization, speech separation, and multi-speaker ASR—through dedicated task modules and joint optimization (Shakeel et al., 28 Aug 2025). In the 2016 recurrent model, speech recognition and speaker recognition are treated as negatively correlated tasks, so the architecture couples them by recurrent feedback rather than by only sharing hidden layers (Tang et al., 2016).

A common misconception is that “unified” implies one universal embedding for all attributes. The 2026 formulation explicitly rejects that interpretation: the same shared encoder is used for all attributes, but each task has its own projection, layer-weighting, pooling, and teacher-alignment branch (Bouziane et al., 9 Mar 2026). This suggests that UME is better understood as a shared representational substrate with controlled task specialization.

2. Architectural motifs across UME formulations

Across the cited work, several architectural motifs recur. The first is a shared encoder that exposes multi-layer hidden states. In the utterance-level formulation, the SSL backbone produces hidden states

$H^{(\ell)} \in \mathbb{R}^{T \times D},$

where $T$ is the number of time frames, $D$ the hidden dimension, and $\ell$ indexes the layer (Bouziane et al., 9 Mar 2026). In the overlapping-speech formulation, OWSM v3.1 serves as the shared speech foundation model encoder, implemented as a stack of $L$ E-Branchformer encoder layers, with

$H_{(l)} = \mathrm{SpeechEnc}_{(l)}(X)$

for mixture input $X$ (Shakeel et al., 28 Aug 2025).

The second motif is task-specific extraction from shared hidden states. In the 2026 framework, each attribute $\tau$ has a branch that first applies a task-specific linear map at each layer,

$$\tilde{H}^{(\ell)}_{\tau} = H^{(\ell)} W^{(\ell)\top}_{\tau} + b^{(\ell)}_{\tau},$$

then combines layers with learned interpolation weights

$$\lambda_{\tau,\ell} = \frac{\exp(s_{\tau,\ell})}{\sum_{j=1}^{n} \exp(s_{\tau,j})}, \qquad \sum_{\ell=1}^{n} \lambda_{\tau,\ell}=1,$$

forming

$T$0

followed by layer normalization and attribute-specific attentive pooling to obtain an utterance-level vector $T$1 (Bouziane et al., 9 Mar 2026). The paper states that this projection is important because it reduces the amount of task interference.

The third motif is explicit use of information from multiple encoder depths. The overlapping-speech UME introduces residual weighted-sum encoding (RWSE). For a task-specific weighted sum,

$T$2

with softmax-normalized $T$3, the encoded representation is

$T$4

RWSE is presented as a mechanism for information exchange across layers and bottom-up alignment between different tasks (Shakeel et al., 28 Aug 2025).

The fourth motif is branch-specific supervision. In the utterance-level model, the semantic branch aligns speech to multilingual text embeddings from BGE-M3 and the speaker branch aligns speech to speaker embeddings from a pretrained ECAPA-TDNN model trained on VoxCeleb; the speech encoder is initialized with w2v-BERT 2.0 and trained by cosine-similarity alignment in a multi-task teacher–student distillation setup (Bouziane et al., 9 Mar 2026). In the overlapping-speech UME, each branch instead optimizes a task-native criterion: binary cross entropy under PIT for diarization, SI-SDR loss under PIT for separation, and a CTC/attention hybrid objective for ASR (Shakeel et al., 28 Aug 2025).

Work	Shared mechanism	Task-specific specialization
"Unifying Diarization, Separation, and ASR with Multi-Speaker Encoder" (Shakeel et al., 28 Aug 2025)	Shared speech foundational encoder with RWSE	Diarization, Conv-TasNet separation, CTC/attention ASR heads
"Learning Multiple Utterance-Level Attribute Representations with a Unified Speech Encoder" (Bouziane et al., 9 Mar 2026)	Shared SSL backbone with layer interpolation	Semantic and speaker branches aligned to frozen teachers
"Multi-task Recurrent Model for Speech and Speaker Recognition" (Tang et al., 2016)	Two recurrent components with cross-task recurrent feedback	Separate ASR and SRE branches coupled across time

3. UME for overlapping speech: diarization, separation, and ASR

The 2025 UME architecture addresses overlapping speech by treating speaker diarization, speech separation, and multi-speaker ASR as different views of the same input scene (Shakeel et al., 28 Aug 2025). The input mixture is modeled as

$T$5

where $T$6 is clean speech of speaker $T$7, $T$8 is the binary activity sequence for speaker $T$9, and $D$0 is additive noise. The ground-truth activity sequences supervise diarization.

The diarization head adopts an EEND-style module with permutation invariant training. A linear layer maps encoder features to speaker activity outputs, and the loss is binary cross entropy under PIT. The separation branch uses Conv-TasNet. The mixture is first encoded by a 1-D convolution, and the key UME mechanism is to concatenate upsampled RWSE features with separation features: $D$1 A TCN then predicts masks for each speaker, after which a transposed convolution decoder reconstructs separated waveforms (Shakeel et al., 28 Aug 2025). The ASR branch uses a CTC/attention hybrid multi-speaker architecture: the shared encoder output is passed to speaker-differentiating encoder blocks, and each speaker’s token sequence is produced by an attention decoder.

Joint training is performed with

$D$2

with the main equal-weight setting $D$3 (Shakeel et al., 28 Aug 2025). The paper also reports ASR-only and ASR-heavy settings.

The experimental setup uses LibriMix (460 hours), including Libri2Mix and Libri3Mix at 16 kHz with max mode, and evaluates on Libri2Mix and Libri3Mix with 100% overlap and LibriSpeech2Mix and LibriSpeech3Mix with partial random overlap of at least 0.5 s (Shakeel et al., 28 Aug 2025). The reported best diarization results are 1.37% DER on Libri2Mix and 2.29% DER on Libri3Mix. For multi-speaker ASR on mixclean, the reported WERs are 6.4% on Libri2Mix and 15.9% on Libri3Mix. For separation on mixclean with RWSE and ASR initialization, the best reported results are STOI 95.64, SDR 17.41 dB, and SI-SNR 17.06 dB on Libri2Mix, and STOI 91.25, SDR 13.07 dB, and SI-SNR 12.58 dB on Libri3Mix (Shakeel et al., 28 Aug 2025).

The paper identifies ASR initialization as a practical requirement for stable optimization, especially for three-speaker mixtures, reporting that training can diverge without ASR pretraining in several settings (Shakeel et al., 28 Aug 2025). A plausible implication is that the linguistic structure induced by the ASR branch regularizes the shared encoder strongly enough to stabilize the broader multitask problem.

4. UME-style post-training for semantic and speaker representations

The 2026 work reformulates the unification problem at the utterance level rather than in overlapping-speech decoding. Its central contribution is to show that a single shared speech encoder can be adapted to produce multiple utterance-level embeddings for different attributes instead of only one semantic embedding (Bouziane et al., 9 Mar 2026). The motivating concern is that speech foundation models such as wav2vec 2.0, HuBERT, or w2v-BERT mostly output frame-level acoustic representations, and semantic-only post-training may suppress other informative paralinguistic attributes such as speaker identity.

Training follows a multi-task teacher–student distillation paradigm. For each attribute $D$4, there is a frozen teacher defining the target embedding space. The semantic branch uses multilingual text embeddings from BGE-M3; the speaker branch uses speaker embeddings from a pretrained ECAPA-TDNN model trained on VoxCeleb. After task-specific projection, layer weighting, normalization, and attentive pooling, the final utterance embedding is $D$5-normalized and aligned to the teacher representation by cosine similarity (Bouziane et al., 9 Mar 2026). The loss is described as cosine-similarity alignment, and the global objective is the sum over tasks.

Empirically, the semantic branch is evaluated on speech $D$6 speech retrieval on VoxPopuli, speech $D$7 text retrieval on MTEDx, and generalization to low-resource and unseen languages on FLEURS (Bouziane et al., 9 Mar 2026). The key result is that adding speaker supervision causes only small degradation, if any, in semantic retrieval quality: the joint model Att(sem+spk) stays very close to the semantic-only baseline Att(sem) and generally outperforms SONAR on the reported retrieval benchmarks. On FLEURS, the joint model remains near the semantic baseline, clearly exceeds SONAR, and shows a slight gain on the my-en pair.

For speaker verification on VoxCeleb1-O, the speaker branch approaches the ECAPA-TDNN teacher. The reported numbers are: ECAPA-TDNN with EER 0.90% and MinDCF 0.1104; Att(spk) with EER 0.93% and MinDCF 0.1285; and Att(sem+spk) with EER 0.91% and MinDCF 0.1253 (Bouziane et al., 9 Mar 2026). The paper interprets this as evidence that joint semantic+speaker training does not significantly hurt speaker discrimination and may slightly improve over the single-task speaker branch.

A noteworthy analysis concerns layer usage. The semantic branch concentrates on middle layers, with strong peaks around layers 13–14, while the speaker branch spreads weight more broadly and shifts toward higher layers, peaking around layers 23–24 (Bouziane et al., 9 Mar 2026). This suggests that a shared encoder can preserve different kinds of information at different depths and that branch-specific layer selection is a mechanism for mitigating interference.

5. Relation to earlier unified ASR–SRE recurrent models

The 2016 paper provides an earlier unified architecture for speech recognition and speaker recognition, predating foundation-model-based UME formulations but addressing a closely related problem (Tang et al., 2016). Its central claim is that although ASR and SRE are highly correlated at the task level, they are negatively correlated at the representation level: ASR seeks invariance to speaker variation, whereas SRE seeks invariance to linguistic variation. For that reason, the proposed solution is not ordinary shared-feature multitask learning.

The model contains two LSTM-based components, one for ASR and one for SRE, each with its own parameters and recurrent state, but both receiving the same acoustic input $D$8. Cross-task recurrent feedback is introduced so that ASR receives information from SRE’s previous hidden outputs and SRE receives information from ASR’s previous hidden outputs. In the representative formulation, feedback from recurrent and nonrecurrent projections is injected into the nonlinear activation path $D$9 of the other task (Tang et al., 2016).

Joint training conceptually minimizes the sum of the two supervised losses, with frame-level classification for phone or pdf targets and speaker labels, implemented in the Kaldi nnet/LSTM setup with natural stochastic gradient descent (Tang et al., 2016). On WSJ, the baseline ASR system reports a total WER of 7.41%, while the baseline r-vector SRE system reports EER 1.84% with cosine scoring. Multi-task variants reduce WER to about 7.05–7.28% and reduce EER to 0.55–0.71% in the best configurations; one highlighted setting, using feedback from $\ell$0 only and injecting it into $\ell$1 together, yields WER 7.05% and EER 0.55% (Tang et al., 2016).

This earlier model differs structurally from later UME systems. It does not use a shared foundation encoder plus task heads, nor does it learn utterance-level attribute embeddings or RWSE over transformer-like layers. Its significance lies instead in demonstrating that negatively correlated speech and speaker objectives can be trained in one unified framework if the coupling mechanism is designed to preserve task specialization (Tang et al., 2016). A plausible historical interpretation is that this paper foreshadows later UME designs by establishing that joint content–speaker modeling need not collapse into negative transfer.

6. Empirical behavior, misconceptions, and limitations

Across the cited work, the principal empirical claim is that joint modeling can preserve or improve performance across tasks that are often trained separately. In the overlapping-speech setting, UME improves over single-task baselines dedicated to speaker diarization, speech separation, and multi-speaker ASR on LibriMix evaluation sets, with the strongest reported diarization numbers of 1.37% and 2.29% DER on Libri2Mix and Libri3Mix, respectively (Shakeel et al., 28 Aug 2025). In the utterance-level setting, joint semantic and speaker supervision maintains strong multilingual retrieval and near-teacher speaker verification performance (Bouziane et al., 9 Mar 2026). In the earlier recurrent setting, carefully designed inter-task feedback improves both ASR and SRE on WSJ (Tang et al., 2016).

Several misconceptions are explicitly addressed by these results. One is that speaker supervision necessarily damages semantic structure. The 2026 paper reports only small degradation, if any, in retrieval quality under joint semantic+speaker training, and the speaker branch remains nearly on par with the ECAPA-TDNN teacher (Bouziane et al., 9 Mar 2026). Another is that unification is equivalent to simple parameter sharing. The 2016 paper argues the opposite for negatively correlated tasks, and the later systems likewise rely on branch-specific projections, pooling, PIT, or recurrent feedback rather than on undifferentiated sharing alone (Tang et al., 2016).

The limitations are also concrete. The overlapping-speech UME notes that training can be unstable without ASR initialization, especially for three-speaker cases; separation is based on Conv-TasNet, which the paper itself notes is not state-of-the-art; evaluations are primarily on simulated LibriMix-style mixtures; and the demonstrations assume fixed known speaker counts in the benchmark settings (Shakeel et al., 28 Aug 2025). The utterance-level model is framed around semantic and speaker attributes as the main example, so broader generalization to other arbitrary utterance-level attributes is a research direction implied by the framework rather than exhaustively established in the reported experiments (Bouziane et al., 9 Mar 2026). The 2016 recurrent model is limited by a small WSJ-based setup, frame-level supervision, and single-domain evaluation (Tang et al., 2016).

Taken together, the literature presents UME not as one immutable architecture but as a design principle for unified speech encoding: one common speech model, explicit task- or attribute-specific extraction mechanisms, and joint optimization across objectives that would otherwise be handled by isolated systems. This suggests that the central technical problem is not whether content, speaker identity, diarization structure, or separation cues can coexist in one encoder, but how branch design, layer selection, recurrent coupling, and initialization determine whether they do.