DI-cpWER: Diarization-Invariant ASR Metric
- The paper introduces DI-cpWER, a metric that estimates the cpWER if hypothesis speaker labels were optimally corrected, highlighting the impact of speaker confusion.
- DI-cpWER is computed by swapping the reference and hypothesis roles in the cpWER framework, and a greedy approximation is used to reduce its otherwise exponential computational complexity.
- This metric aids in system analysis by distinguishing recognition errors from speaker-attribution errors, providing an oracle-like lower bound on transcription quality.
Searching arXiv for the specified papers and closely related work on DI-cpWER and tcpWER. Diarization-invariant cpWER (DI-cpWER) is a variant of meeting-level word error rate for long-form multi-talker speech recognition that is designed to answer a specific diagnostic question: how well a speaker-attributed transcript would score if the system’s speaker labels were corrected perfectly. It was introduced to estimate the impact of speaker confusion or speaker-attribution errors on concatenated minimum-permutation WER (cpWER), while preserving the cpWER-style Levenshtein framework and normalization by the number of reference words (Neumann et al., 4 Aug 2025). In the broader metric family, DI-cpWER sits alongside speaker-attributed measures such as cpWER and tcpWER, and speaker-agnostic measures such as ORC-WER and MIMO-WER (Neumann et al., 2023).
1. Position within long-form multi-talker WER definitions
The classical WER is not directly applicable to long transcripts containing multiple speakers and complex speaking patterns, so a family of extended WER definitions has been developed for long-form multi-talker speech recognition. In this family, cpWER and tcpWER are speaker-attributed approaches that count speaker confusion errors, whereas ORC-WER and MIMO-WER are speaker-agnostic approaches that aim to ignore speaker confusion errors (Neumann et al., 4 Aug 2025).
The standard cpWER is defined by concatenating utterances by speaker and then choosing the best one-to-one matching between reference speakers and hypothesis streams. In the notation of the later formulation, with reference segments $\rv$, hypothesis segments $\hv$, reference speaker labels $\rlbl$, hypothesis labels $\hlbl$, and ordinary Levenshtein distance $\lev(\cdot,\cdot)$, cpWER is
$\distance^\text{cp} = \min_{\perm} \sum_{\ispk=1}^{\nspk} \lev\!\Big( \select(\ind_{\{i:\rlble_i = \ispk\})\rv,\; \select(\ind_{\{i:\hlble_i=\pi_\ispk\})\hv \Big).$
Intuitively, cpWER penalizes recognition errors, speaker assignment errors, and indirectly segmentation issues that create mismatched concatenations. In meeting transcription, it is treated as the standard metric for diarization-style systems whose outputs are grouped into speaker streams but whose speaker identities may be permuted relative to the reference (Neumann et al., 4 Aug 2025).
MeetEval places cpWER in direct relation to two other core variants. ORC-WER is described as a special case of MIMO-WER where reference speaker labels are ignored but the temporal order of utterances is preserved. MIMO-WER is the more general multi-output metric for systems with multiple output streams that may not be speaker-aligned, including SOT-style systems that may output speakers in arbitrary order (Neumann et al., 2023).
2. Formal definition of DI-cpWER
DI-cpWER reverses the side on which the cpWER-style assignment problem is solved. Instead of permuting reference speakers to best match the hypothesis streams, it corrects the hypothesis speaker labels to best fit the reference. Its formal definition is
$\distance^\text{DI-cp} = \min_{v_1,\dots} \sum_{\istream=1}^{\nstream} \lev\!\Big( \select(\ind_{\{i:\rlble_i=\istream\})\rv,\; \select(v_\istream)\hv \Big).$
The paper characterizes this as the same optimization structure as ORC-WER, but with reference and hypothesis swapped, and states explicitly: “The Diarization-Invariant cpWER and diarization-invariant tcpWER can be computed by swapping reference and hypothesis in the ORC-WER ... while keeping the denominator” (Neumann et al., 4 Aug 2025).
Operationally, DI-cpWER keeps the reference speaker grouping fixed, reassigns the hypothesis segments to speakers or streams in the way that minimizes total Levenshtein error, and then computes the resulting WER. This makes the metric invariant to speaker label switches on the hypothesis side. In that sense, DI-cpWER is a speaker-attribution-invariant version of cpWER rather than a speaker-agnostic metric in the ORC-WER sense (Neumann et al., 4 Aug 2025).
3. Error semantics and relation to speaker confusion
The central analytical use of DI-cpWER is to estimate how much of cpWER is attributable to speaker confusion. The paper’s summary claim is that the difference between DI-cpWER and cpWER reflects the impact of speaker confusions on WER. Since cpWER includes penalties caused by wrong speaker labels and DI-cpWER removes those penalties by optimally relabeling the hypothesis side, the gap
is used as a proxy for speaker-attribution error (Neumann et al., 4 Aug 2025).
This quantity is not presented as an exact decomposition. The paper explicitly cautions that it is only a rough estimate, because a speaker confusion can have arbitrary effects on the Levenshtein computation. Accordingly, the gap should be interpreted as a metric-level estimate of the diarization impact on transcription quality rather than as a precisely separable count of “speaker confusion words” (Neumann et al., 4 Aug 2025).
DI-cpWER is also distinguished from ORC-WER. ORC-WER ignores speaker attribution errors by assigning reference utterances to output streams while preserving global order and utterance consistency. DI-cpWER instead preserves the cpWER framing but makes the hypothesis speaker labels invariant. The paper notes two asymmetries: utterance merges affect DI-cpWER negatively and do not affect ORC-WER, whereas utterance splits affect DI-cpWER positively and ORC-WER negatively. This is why DI-cpWER is tailored to the question “what is the cpWER if hypothesis speaker labels were corrected?” rather than collapsing to ORC-WER (Neumann et al., 4 Aug 2025).
4. Temporal plausibility and the connection to tcpWER
The motivation for adding temporal structure to cpWER comes from a limitation already identified in meeting-transcription evaluation: a purely text-based alignment can match common words such as “the” or “and” to occurrences that are much earlier or later in the meeting, yielding an artificially good WER despite a poor actual transcript alignment. MeetEval therefore introduced tcpWER, a time-constrained cpWER that replaces ordinary Levenshtein distance with a temporally constrained Levenshtein distance, so that a reference word and a hypothesis word may only be matched as correct or substitution if their time intervals overlap or are separated by less than a collar $\collar$ (Neumann et al., 2023).
For words that are not allowed to match, the edit costs are set so that insertion plus deletion is preferred over an illegal substitution or correct match:
$\costsub = \costcorr \ge \costins + \costdel.$
A key algorithmic consequence is pruning of the dynamic-programming search: cells in the Levenshtein matrix where no time-legal matches exist can be skipped, leaving a diagonal band and reducing computation. MeetEval presents this temporal constraint as a way to obtain a matching that more closely resembles the actual transcription quality and to penalize systems that provide poor time annotations (Neumann et al., 2023).
Within the later metric framework, DI-cpWER is explicitly linked to this temporal line of work through the statement that diarization-invariant tcpWER can be computed by swapping reference and hypothesis in ORC-WER while keeping the denominator (Neumann et al., 4 Aug 2025). This indicates that diarization invariance and temporal plausibility are treated as composable properties of the metric family.
MeetEval also addresses the practical absence of exact word-level timestamps by approximating word timing from segment-level annotations. It discusses full-segment annotation, equally sized intervals, character-based intervals, and character-based timepoints. The recommended approximation is character-based intervals for references and character-based timepoints for hypotheses. The collar is used to absorb annotation imprecision, pseudo-word timing approximation errors, dataset-specific offsets, and slight timestamp discrepancies; a collar of about 5 s is recommended as a reasonable default in the reported Libri-CSS experiments (Neumann et al., 2023).
5. Exact computation and greedy approximation
Exact DI-cpWER computation is described as combinatorially expensive. The paper states that it is analogous to ORC-WER and can be exponential in the number of reference segments in the naive formulation (Neumann et al., 4 Aug 2025).
To make the metric practical, a greedy approximation is proposed. Its high-level procedure is: start from the cpWER assignment, iterate over hypothesis utterances, try moving each utterance to each possible stream, keep the move if it improves total Levenshtein distance, and repeat until no improvement is possible. The paper describes this as a greedy label-correction algorithm (Neumann et al., 4 Aug 2025).
A two-stage refinement is used to mitigate local minima caused by label swaps between utterances. The recommendation is first to run greedy updates with substitution cost $\hv$0, so that a substitution can be traded for an insertion plus deletion, and then to rerun with the standard $\hv$1. This is intended to help the algorithm escape some local minima (Neumann et al., 4 Aug 2025).
The complexity claim is that the greedy method has polynomial complexity rather than exponential, with runtime depending on the product of the number of output streams and reference utterances. Appendix optimizations include reuse of previously computed Levenshtein results, updating only the affected stream distances, avoiding recomputation of full Levenshtein matrices, and exploiting symmetry between forward and reverse dynamic-programming computations. Experimentally, the approximation is reported to find the exact matching 86% of the time, with an average deviation of less than 0.02 percentage points, while the abstract reports <0.1% deviation in experiments (Neumann et al., 4 Aug 2025).
6. Applications, visualization, and limitations
DI-cpWER is presented as a metric for system analysis, not for leaderboard ranking. Its stated role is to estimate what cpWER would be if diarization or speaker attribution were perfect, and to help determine whether the main problem in a long-form multi-talker ASR system lies in recognition, segmentation, or speaker attribution. The paper emphasizes that higher-level errors such as speaker confusions cannot be reliably inferred automatically from the transcript alone, so DI-cpWER provides a useful oracle-like lower bound on cpWER (Neumann et al., 4 Aug 2025).
The metric is particularly relevant in settings where outputs contain multiple streams, diarized segments, or serialized outputs with ambiguous speaker attribution. The paper identifies meeting transcription, continuous speech separation pipelines, CHiME-style multi-speaker ASR, and systems where transcription and diarization are jointly estimated as representative contexts in which the distinction between “what was said” and “who said it” is analytically important (Neumann et al., 4 Aug 2025).
To expose those distinctions, the paper advocates trace visualizations of alignments. A numerical gap between cpWER and DI-cpWER is sufficient to infer that there is at least one speaker-attribution error, but an alignment trace can reveal exactly which words were assigned to the wrong speaker. In one illustrative example, the same sample yields cpWER = 87.5%, ORC-WER = 50.0%, MIMO-WER = 37.5%, and DI-cpWER = 25.0%, showing that many cpWER errors can disappear once hypothesis speaker labels are corrected (Neumann et al., 4 Aug 2025).
The principal caveat is that DI-cpWER is not a fair ranking metric. The paper states that it can be gamed by splitting segments into smaller units and allowing the assignment mechanism to exploit that flexibility. Relatedly, word-level DI-cpWER and word-level ORC-WER, where each word is treated as its own segment, are described as more aggressive variants that generally under-estimate errors and are not recommended for serious evaluation, although they are included for illustration (Neumann et al., 4 Aug 2025).