M2MeT Challenge: Multi-Channel Transcription

Updated 8 January 2026

The paper demonstrates how overlap-aware diarization and advanced ASR methods address the challenges of multi-channel, multi-party meeting transcription.
It details both modular and end-to-end approaches, including TS-VAD and multi-frame cross-channel attention, achieving significant improvements (e.g., DER reduction to as low as 2.26%).
The challenge establishes rigorous evaluation protocols using large-scale Mandarin meeting corpora with metrics like DER, CER, and cp-CER, advancing reproducible research in realistic meeting scenarios.

The Multi-channel Multi-party Meeting Transcription (M2MeT) Challenge is a benchmark and community testbed for robust speaker diarization and multi-speaker automatic speech recognition (ASR) in realistic meeting scenarios. It addresses the unique signal processing and modeling problems of far-field, multi-party, overlapped, and noisy meeting recordings using large annotated corpus resources and a well-defined evaluation protocol. With the advent of high-quality Mandarin meeting corpora such as AliMeeting (118.75 h, 8-channel microphone arrays), M2MeT provides reproducible infrastructure across speaker diarization, overlap-aware ASR, and modular or end-to-end pipelines.

1. Corpus, Tracks, and Evaluation Protocols

M2MeT is anchored on the AliMeeting dataset (Yu et al., 2021), which comprises 118.75 hours of Mandarin meetings, recorded via 8-channel circular microphone arrays and synchronous near-field headset mics. Sessions last 15–30 minutes and feature 2–4 speakers (Train: 212 sessions; Eval: 8; Test: 20), with an average overlap ratio of 42%, reaching up to 59% in 4-speaker sessions. Annotation includes character-level transcripts, utterance boundaries, speaker IDs, and precise overlap marking.

Two tracks are delineated:

Track 1: Speaker Diarization. Input is multi-channel far-field audio. Output is the time-stamped assignment of speaker labels, permitting overlaps and unknown speaker counts.
Track 2: Multi-speaker ASR. Input is identical; output is session-level transcription with explicit speaker-change tokens, scored via character error rate (CER).

Evaluation is conducted chiefly via Diarization Error Rate (DER), which aggregates missed-speaker, false-alarm, and speaker-confusion time, with a 0.25 s collar exclusion. For ASR, CER (or WER for English) is computed with support for permutation-invariant scoring schemes in overlapped segments, notably FIFO utterance-based and speaker-based best-permutation concatenation (Yu et al., 2021). M2MeT 2.0 introduces speaker-attributed ASR (SA-ASR), requiring joint transcript and speaker label output scored by concatenated minimum permutation CER (cp-CER) (Liang et al., 2023).

2. Modeling Paradigms

2.1 Classical and Modular Pipelines

M2MeT builds on successful modular architectures:

Diarization: Typically x-vector or d-vector embedding networks (ResNet-34, ECAPA-TDNN), followed by spectral clustering, agglomerative hierarchical clustering (AHC), or Bayesian HMM (VBx), often with PLDA scoring.
Overlap Resolution: Dedicated Overlapped Speech Detectors (OSD), e.g. SincNet-BiLSTM (Wang et al., 2022), multi-channel U-Net (Tian et al., 2022), or discriminative multi-stream neural nets with spatial and temporal attention (DMSNet) (Wang et al., 2022).
Fusion: Output label-level combination using DOVER-Lap (Wang et al., 2022), which exploits system diversity and improves confusion errors.

2.2 Target-Speaker Voice Activity Detection (TS-VAD)

TS-VAD is central for high-overlap diarization.

The DKU_DukeECE system employs a ResNet-34 front-end to produce frame-level embeddings, clustered to extract target speaker embeddings. These embeddings are concatenated with local acoustic representations and classified by Transformer encoders and BiLSTM back-ends, using binary cross-entropy loss (Wang et al., 2022).
Performance: Single-channel TS-VAD reduces baseline clustering DER by 75% (12.68% → 3.14%); multi-channel TS-VAD with cross-channel self-attention further reduces DER to 2.26% (Wang et al., 2022).

2.3 Overlap-aware Feature Fusion

Feature-fusion strategies integrate acoustic, spatial (DOA), and speaker i-vector features. The FFM-TS-VAD system fuses all cues at the input to transformer encoder layers, achieving robust performance in closely spaced speaker scenarios (Zheng et al., 2022).

Data augmentation simulates small-angle spatial confusables, forcing reliance on speaker embeddings when spatial cues are weak.

2.4 End-to-End and Joint Approaches

M2MeT systems are increasingly end-to-end:

Cross-channel attention: Multi-channel self-attention (e.g., channel-wise Transformer heads) enables nonlinear spatial fusion within the diarization model, outperforming late fusion of independently trained single-channel networks (Wang et al., 2022, Yu et al., 2022).
Multi-frame cross-channel attention: MFCCA extends attention over time and channel, fusing adjacent-frame and cross-channel knowledge before convolutional merging and downstream ASR decoding. With channel masking during training, MFCCA achieves up to 37% CER reduction versus single-channel models and surpasses previous state-of-the-art (Yu et al., 2022).
Speaker-attributed ASR (SA-ASR): M2MeT 2.0 and subsequent systems require simultaneous transcription and speaker attribution for each token. Pipelines typically employ modular diarization (TS-VAD or clustering+OSD), followed by speaker-conditioned Conformer/Paraformer/U2++ ASR with permutations resolved using cp-CER (Liang et al., 2023, Lyu et al., 2023). Direct E2E architectures with serialized output training or neural fusion remain less effective given limited in-domain data.

3. Signal Processing and Data Augmentation

Robustness to noise, reverberation, and variable array geometry is handled via:

Beamforming: Delay-and-sum (beamformIt, DAS), MVDR (often DNN-assisted), or fully neural beamformers (FaSNet-TAC), combined with dereverberation (WPE, Kalman filter) (Cui et al., 2024, Shen et al., 2022).
Data simulation: RIR synthesis (image method), spatial mixing, additive MUSAN/openRIR noise, multi-channel overlapped waveform construction mimic real meeting acoustics and overlap patterns (Ye et al., 2022, Liang et al., 2023).
Unsupervised/semi-supervised training: Three-stage CSS training incorporates simulated mixtures, teacher-student adaptation on unlabeled meeting audio, and downstream ASR loss-based fine-tuning (Wang et al., 2022).

4. Experimental Benchmarks

A selection of representative DER and CER/cp-CER benchmarks from the challenge:

System	Task	DER (%)	CER (%)	cp-CER (%)
Baseline (Kaldi-VBx)	Diarization	15.24	—	—
DKU_DukeECE MC-TS-VAD	Diarization	2.98	—	—
FFM-TS-VAD	Diarization (fusion)	3.28	—	—
RoyalFlush (SA-ASR)	ASR	—	18.79	—
MFCCA (8-ch)	ASR	—	19.4–21.3	—
Official Modular (2.0)	SA-ASR (fixed)	—	—	8.84
PP-MeT (open)	SA-ASR (2.0)	—	—	11.27
Baseline SA-Transformer	SA-ASR (2.0)	—	—	41.55

Notably, modular pipelines with overlap-aware diarization and strong ASR back-ends (Paraformer, U2++, Conformer) consistently outperform direct end-to-end architectures (Liang et al., 2023, Lyu et al., 2023, Yu et al., 2022). Fully neural spatial fusion yields further incremental gains if jointly optimized with ASR objectives (Cui et al., 2024).

5. Lessons, Controversies, and Future Directions

Overlap Modeling: Overlap-aware approaches (TS-VAD, OSD+DOA, separation) are the decisive factor for lowering DER and cp-CER as overlap ratio increases (Wang et al., 2022, Liang et al., 2023).

Fusion Strategies: DOVER-Lap, ROVER, and system-level model averaging confer 2–15% relative improvements when computational cost is unconstrained (Yu et al., 2022). Late model fusion remains standard but direct neural fusion methods are emerging.

Signal Processing Front-ends: Fixed beamformers remain strong baselines; neural front-ends only surpass classical approaches when co-optimized with downstream ASR (Cui et al., 2024). There remains debate whether neural channel fusion alone provides significant gain.

End-to-End SA-ASR: Despite strong motivation, direct E2E architectures underperform modular pipelines without vast in-domain annotated training. Hybrid or self-supervised pre-training paradigms may close this gap.

Data Scarcity and Augmentation: Data simulation—especially realistic room/acoustic/overlap modeling—remains vital for generalization (Liang et al., 2023, Ye et al., 2022). Semi-supervised or self-supervised adaptation promises further improvements (Wang et al., 2022).

Metrics and Generalization: Concatenated minimum-permutation CER or cpWER is now the standard for scoring speaker-attributed transcription. Extensions to handle insertion/deletion of speaker streams and latency constraints are emerging topics (Liang et al., 2023).

A plausible implication is that future M2MeT systems will tightly integrate multi-channel spatial modeling, overlap-aware temporal labeling, and speaker-conditioned recognition within joint optimization frameworks, potentially incorporating large-scale cross-lingual and SSL pre-trained backbones.

6. Conclusion

The M2MeT Challenge establishes rigorous, reproducible benchmarks for multi-channel, multi-party diarization and speaker-attributed ASR in realistic meetings. Major progress is driven by overlap-aware architectures (TS-VAD, MFCCA, OSD/DMSNet), multi-channel spatial fusion (self-attention, beamforming), and large pre-trained or simulated data pipelines. Modular systems leveraging robust front-ends, advanced signal fusion, and speaker conditioning outperform direct end-to-end approaches under current data regimes, with DER under 3% and cp-CER below 9% now achievable for Mandarin far-field corpora. Open questions remain on E2E model scaling, optimal fusion strategies, array-agnostic spatial learning, and extension to low-resource or cross-lingual domains. Continued innovation will likely leverage massive self-supervised spatial modeling and unified sequence labeling to address the persistent challenges in multi-party meeting transcription (Wang et al., 2022, Yu et al., 2021, Liang et al., 2023, Yu et al., 2022).