whisperM2M: Multi-Talker ASR & Speech Translation

• whisperM2M is a framework that extends Whisper for both multi-talker automatic speech recognition and many-to-many speech translation by using modular separation and target talker identification.
• It employs a frozen Whisper encoder with a sidecar separator and a dedicated target talker identifier to efficiently isolate and transcribe multiple speakers.
• The system also integrates a compressed multilingual translation architecture with KVPSN speculative decoding, achieving significant speedups while maintaining state-of-the-art accuracy.

whisperM2M refers to a family of recent techniques and model enhancements designed to extend the Whisper architecture for multi-talker and many-to-many speech translation tasks. These systems address both multi-talker automatic speech recognition (ASR) and multilingual speech-to-text (ST) with a focus on parameter efficiency, deployment practicality, and strong empirical performance. Distinct instantiations of whisperM2M include (1) joint multi-talker and target-talker ASR systems with modular separation and identification, and (2) multilingual, compressed, and speculative translation architectures yielding state-of-the-art speed/accuracy tradeoffs in many-to-many speech translation (Meng et al., 2024, Le et al., 15 Aug 2025).

1. Joint Multi-Talker and Target-Talker Speech Recognition

The whisperM2M ASR framework implements a modular pipeline that repurposes the Whisper foundation model and incorporates external separation and identification modules, allowing joint transcription of multiple speakers or extraction of a specific target talker’s utterances from a mixture (Meng et al., 2024).

• Base Encoder: The Whisper encoder is used as a frozen feature extractor. Mixed audio waveforms $x$ are processed via log-Mel frontend and the first two Transformer encoder layers. The layer-2 outputs $h_t \in \mathbb{R}^C$ (for $t=1...T$) serve as predominantly acoustic representations and are not updated during training.
• Sidecar Separator: The Sidecar module, built as a Conv-TasNet–style separator with $K=8$ dilated 1-D convolution blocks and $R=3$ repeats, operates on the time-stacked embedding $$E_{\mathrm{mix}} = [h_1; h_2; \ldots; h_T] \in \mathbb{R}^{T \times C}$$ to produce $S$ talker-wise binary masks $M^i$. Separation is performed by elementwise multiplication: $\tilde{E}^i = M^i \odot E_{\mathrm{mix}}$ for $i = 1\ldots S$.
• Embedding Re-injection: Each separated embedding stream is passed through the remaining Whisper encoder layers ($h_t \in \mathbb{R}^C$0), yielding $h_t \in \mathbb{R}^C$1 branch-specific representations for decoding.
• Parameter Freezing: All original Whisper parameters (encoder and decoder) are frozen. Only the Sidecar (8–18M parameters, model size dependent), Target Talker Identifier (TTI), and soft prompts are trained.

2. Target Talker Identifier and Soft Prompt Tuning

To enable target-talker extraction, whisperM2M incorporates an identification mechanism and decoder adaptation method (Meng et al., 2024).

• Target Talker Identifier (TTI): Given a 3-second enrollment signal $h_t \in \mathbb{R}^C$2 from the targeted speaker, the frozen Whisper frontend and encoder produce embeddings, averaged to yield $h_t \in \mathbb{R}^C$3. For the first 150 frames of each branch output, a small MLP computes logits $h_t \in \mathbb{R}^C$4, softmaxed to obtain the branch probabilities $h_t \in \mathbb{R}^C$5. The target talker is assigned as $h_t \in \mathbb{R}^C$6. Only the selected branch is passed to the decoder for target-aware transcription.
• Soft Prompt Tuning: The decoder is augmented with $h_t \in \mathbb{R}^C$7 trainable prefix vectors $h_t \in \mathbb{R}^C$8 inserted in the input sequence between special tokens. The decoder output is formalized as $h_t \in \mathbb{R}^C$9, with $t=1...T$0 as the encoder output of the chosen branch.
• Training Objective:

$t=1...T$1

where $t=1...T$2 is computed via permutation invariant training (PIT) and $t=1...T$3 is the cross-entropy between $t=1...T$4 and the correct branch.

3. Many-to-Many Speech Translation with Model Compression and Speculation

In the multilingual domain, whisperM2M refers to a compressed and efficient adaptation of Whisper Medium for ST with several architectural innovations (Le et al., 15 Aug 2025):

• Prompt Reformulation: The “language-ID” token is repurposed to denote the target language. Non-essential prompt fields are removed for minimalistic prompting.
• Multi-task MT/ST Distillation: A 6-layer text encoder (hidden size 1024) is grafted onto the model to support text-to-text MT. Speech→text (ST) and MT tasks share decoder parameters. A KL-based distribution matching loss is activated once MT loss falls below ST loss.
• Decoder Pruning: The 24-layer original Whisper decoder is pruned to 12 layers (first and last retained), halving the autoregressive cost; active parameters reduce from 769M to 561M (27% reduction).
• Low-Rank Adaptation: First 20 encoder layers frozen; LoRA adapters inserted (rank $t=1...T$5).
• Base Loss Function:

$t=1...T$6

with $t=1...T$7

4. KVPSN Speculative Decoding and Dual-Scale Acceleration

whisperM2M introduces the Key-Value Parasitic Speculative Network (KVPSN) to accelerate inference (Le et al., 15 Aug 2025):

• KVPSN Block: Each block receives the embedding of the most recently generated output token and performs cross-model attention over “flattened” key–value tensors from specific decoder-layer groups ($t=1...T$8, $t=1...T$9 groups, each spanning 4 layers).
  • Key equations:
  • $K=8$0
  • $K=8$1
  • $K=8$2
• Speculation Algorithm: At each decoding step, the base model outputs $K=8$3. If the token was speculated by KVPSN, it is validated; accepted if in top-k of base’s probability. If rejected, decoding rolls back and continues with the base output. The next token embedding is then used for KVPSN speculation. In ideal settings, compute reduction approaches $K=8$4; practically, with top-1 validation, $K=8$5 speedup is observed.

5. Experimental Results and Comparison

Empirical results demonstrate the efficacy of both ASR and ST whisperM2M variants with strong SOTA performance:

ASR Results (Meng et al., 2024):

Model	2-spk LibriMix WER (%)	3-spk LibriMix WER (%)	2-spk Target WER (%)	2-spk AishellMix CER (%)
D2V-Sidecar-DB (prev SOTA)	9.69	33.91	—	—
Whisper-small-SS-TTI	9.39	—	—	—
Whisper-medium-SS-TTI	6.56	—	—	—
Whisper-large-SS-TTI	4.66	16.79	7.97	28.94 → 17.81*

ST Results (Le et al., 15 Aug 2025):

Model	Params (M)	CoVoST2 BLEU X→EN	CoVoST2 BLEU EN→X	Decoder Latency (ms)	Relative Speed (%)
Whisper Medium (ft)	769	36.1	37.6	16.5	100
whisperM2M (12L)	561	37.0	38.9	8.7	189
+ KVPSN (top-1)	605	37.0	38.8	6.4	259

*One-batch tuning on AishellMix yields 17.81% CER.

whisperM2M yields up to 2.6× decoder speedup (16.5 ms → 6.4 ms/token) with negligible BLEU loss (≤0.05 points at $K=8$6).

6. Limitations and Future Directions

• For multi-talker ASR, a fixed number of talkers $K=8$7 must be specified; PIT-based assignment restricts explicit variable-$K=8$8 handling. Future extensions may incorporate SOT or HEAT architectures to support variable-speaker scenarios (Meng et al., 2024).
• In target-talker ASR, the TTI operates exclusively on the initial 3-second cue, which may ungeneralize when the target’s speech occurs elsewhere. Aggregating identification over the entire utterance may mitigate this limitation.
• For whisperM2M speech translation, KVPSN’s efficiency degrades for sequences longer than 500 tokens due to increased cross-attention cost. Multi-token speculation and cross-family model adaptation remain open areas (Le et al., 15 Aug 2025).
• Sidecar/TTI modules introduce modest run-time overhead (1–3% trainable parameters); pruning and quantization are expected to reduce this cost further.

7. Practical Considerations and State-of-the-Art Comparison

whisperM2M models balance accuracy, speed, and memory efficiency, enabling local deployment on resource-constrained hardware.

• Memory requirements: 605M parameters occupy $K=8$92.4GB (fp32), $R=3$01.2GB (fp16) (Le et al., 15 Aug 2025).
• Throughput: $R=3$1150 tokens/sec on a single A100/H800-class GPU (ALTP ≈ 6.4 ms/token).
• Competitive baselines: whisperM2M achieves lower latency and higher or comparable BLEU to SeamlessM4T Medium (10 ms, lower BLEU) and is significantly faster than SeamlessM4T Large v2 (31 ms, higher BLEU but 4× slower).
• LoRA/quantization are compatible for further compression.
• A plausible implication is that whisperM2M strategies become the default for research and deployment settings prioritizing speed, parameter efficiency, and robust adaptation without loss of accuracy.

In summary, whisperM2M techniques advance the state of the art in multi-talker speech recognition and many-to-many speech translation through modular separation, efficient parameterization, speculative acceleration, and combined training schemes, setting new benchmarks for both empirical performance and practical deployment (Meng et al., 2024, Le et al., 15 Aug 2025).