---
title: 'DI-cpWER: Diarization-Invariant ASR Metric'
url: https://www.emergentmind.com/topics/diarization-invariant-cpwer-di-cpwer
type: topic
---

# DI-cpWER: Diarization-Invariant ASR Metric

Searching arXiv for the specified papers and closely related work on DI-cpWER and tcpWER.
Diarization-invariant cpWER (DI-cpWER) is a variant of meeting-level word error rate for long-form multi-talker speech recognition that is designed to answer a specific diagnostic question: **how well a speaker-attributed transcript would score if the system’s speaker labels were corrected perfectly**. It was introduced to estimate the impact of speaker confusion or speaker-attribution errors on concatenated minimum-permutation WER (cpWER), while preserving the cpWER-style Levenshtein framework and normalization by the number of reference words [2508.02112]. In the broader metric family, DI-cpWER sits alongside speaker-attributed measures such as cpWER and tcpWER, and speaker-agnostic measures such as ORC-WER and MIMO-WER [2307.11394].

## 1. Position within long-form multi-talker WER definitions

The classical WER is not directly applicable to long transcripts containing multiple speakers and complex speaking patterns, so a family of extended WER definitions has been developed for long-form multi-talker speech recognition. In this family, **cpWER** and **tcpWER** are speaker-attributed approaches that count speaker confusion errors, whereas **ORC-WER** and **MIMO-WER** are speaker-agnostic approaches that aim to ignore speaker confusion errors [2508.02112].

The standard cpWER is defined by concatenating utterances by speaker and then choosing the best one-to-one matching between reference speakers and hypothesis streams. In the notation of the later formulation, with reference segments $\rv$, hypothesis segments $\hv$, reference speaker labels $\rlbl$, hypothesis labels $\hlbl$, and ordinary Levenshtein distance $\lev(\cdot,\cdot)$, cpWER is

$$
\distance^\text{cp} = \min_{\perm} \sum_{\ispk=1}^{\nspk} \lev\!\Big( \select(\ind_{\{i:\rlble_i = \ispk\})\rv,\; \select(\ind_{\{i:\hlble_i=\pi_\ispk\})\hv \Big).
$$

Intuitively, cpWER penalizes recognition errors, speaker assignment errors, and indirectly segmentation issues that create mismatched concatenations. In meeting transcription, it is treated as the standard metric for diarization-style systems whose outputs are grouped into speaker streams but whose speaker identities may be permuted relative to the reference [2508.02112].

MeetEval places cpWER in direct relation to two other core variants. ORC-WER is described as a special case of MIMO-WER where reference speaker labels are ignored but the temporal order of utterances is preserved. MIMO-WER is the more general multi-output metric for systems with multiple output streams that may not be speaker-aligned, including SOT-style systems that may output speakers in arbitrary order [2307.11394].

## 2. Formal definition of DI-cpWER

DI-cpWER reverses the side on which the cpWER-style assignment problem is solved. Instead of permuting reference speakers to best match the hypothesis streams, it **corrects the hypothesis speaker labels** to best fit the reference. Its formal definition is

$$
\distance^\text{DI-cp} = \min_{v_1,\dots} \sum_{\istream=1}^{\nstream} \lev\!\Big( \select(\ind_{\{i:\rlble_i=\istream\})\rv,\; \select(v_\istream)\hv \Big).
$$

The paper characterizes this as the **same optimization structure as ORC-WER, but with reference and hypothesis swapped**, and states explicitly: “The Diarization-Invariant cpWER and diarization-invariant tcpWER can be computed by swapping reference and hypothesis in the ORC-WER ... while keeping the denominator” [2508.02112].

Operationally, DI-cpWER keeps the reference speaker grouping fixed, reassigns the hypothesis segments to speakers or streams in the way that minimizes total Levenshtein error, and then computes the resulting WER. This makes the metric invariant to speaker label switches on the hypothesis side. In that sense, DI-cpWER is a speaker-attribution-invariant version of cpWER rather than a speaker-agnostic metric in the ORC-WER sense [2508.02112].

## 3. Error semantics and relation to speaker confusion

The central analytical use of DI-cpWER is to estimate how much of cpWER is attributable to speaker confusion. The paper’s summary claim is that **the difference between DI-cpWER and cpWER reflects the impact of speaker confusions on WER**. Since cpWER includes penalties caused by wrong speaker labels and DI-cpWER removes those penalties by optimally relabeling the hypothesis side, the gap

$$
\text{cpWER} - \text{DI-cpWER}
$$

is used as a proxy for speaker-attribution error [2508.02112].

This quantity is not presented as an exact decomposition. The paper explicitly cautions that it is only a **rough estimate**, because a speaker confusion can have arbitrary effects on the Levenshtein computation. Accordingly, the gap should be interpreted as a metric-level estimate of the diarization impact on transcription quality rather than as a precisely separable count of “speaker confusion words” [2508.02112].

DI-cpWER is also distinguished from ORC-WER. ORC-WER ignores speaker attribution errors by assigning reference utterances to output streams while preserving global order and utterance consistency. DI-cpWER instead preserves the cpWER framing but makes the hypothesis speaker labels invariant. The paper notes two asymmetries: utterance **merges** affect DI-cpWER negatively and do not affect ORC-WER, whereas utterance **splits** affect DI-cpWER positively and ORC-WER negatively. This is why DI-cpWER is tailored to the question “what is the cpWER if hypothesis speaker labels were corrected?” rather than collapsing to ORC-WER [2508.02112].

## 4. Temporal plausibility and the connection to tcpWER

The motivation for adding temporal structure to cpWER comes from a limitation already identified in meeting-transcription evaluation: a purely text-based alignment can match common words such as “the” or “and” to occurrences that are much earlier or later in the meeting, yielding an artificially good WER despite a poor actual transcript alignment. MeetEval therefore introduced **tcpWER**, a time-constrained cpWER that replaces ordinary Levenshtein distance with a temporally constrained Levenshtein distance, so that a reference word and a hypothesis word may only be matched as correct or substitution if their time intervals overlap or are separated by less than a collar $\collar$ [2307.11394].

For words that are not allowed to match, the edit costs are set so that insertion plus deletion is preferred over an illegal substitution or correct match:

$$
\costsub = \costcorr \ge \costins + \costdel.
$$

A key algorithmic consequence is pruning of the dynamic-programming search: cells in the Levenshtein matrix where no time-legal matches exist can be skipped, leaving a diagonal band and reducing computation. MeetEval presents this temporal constraint as a way to obtain a matching that more closely resembles the actual transcription quality and to penalize systems that provide poor time annotations [2307.11394].

Within the later metric framework, DI-cpWER is explicitly linked to this temporal line of work through the statement that **diarization-invariant tcpWER** can be computed by swapping reference and hypothesis in ORC-WER while keeping the denominator [2508.02112]. This indicates that diarization invariance and temporal plausibility are treated as composable properties of the metric family.

MeetEval also addresses the practical absence of exact word-level timestamps by approximating word timing from segment-level annotations. It discusses full-segment annotation, equally sized intervals, character-based intervals, and character-based timepoints. The recommended approximation is **character-based intervals** for references and **character-based timepoints** for hypotheses. The collar is used to absorb annotation imprecision, pseudo-word timing approximation errors, dataset-specific offsets, and slight timestamp discrepancies; a collar of about **5 s** is recommended as a reasonable default in the reported Libri-CSS experiments [2307.11394].

## 5. Exact computation and greedy approximation

Exact DI-cpWER computation is described as combinatorially expensive. The paper states that it is analogous to ORC-WER and can be **exponential** in the number of reference segments in the naive formulation [2508.02112].

To make the metric practical, a **greedy approximation** is proposed. Its high-level procedure is: start from the cpWER assignment, iterate over hypothesis utterances, try moving each utterance to each possible stream, keep the move if it improves total Levenshtein distance, and repeat until no improvement is possible. The paper describes this as a greedy label-correction algorithm [2508.02112].

A two-stage refinement is used to mitigate local minima caused by label swaps between utterances. The recommendation is first to run greedy updates with substitution cost $C_S = 2$, so that a substitution can be traded for an insertion plus deletion, and then to rerun with the standard $C_S = 1$. This is intended to help the algorithm escape some local minima [2508.02112].

The complexity claim is that the greedy method has **polynomial complexity rather than exponential**, with runtime depending on the product of the number of output streams and reference utterances. Appendix optimizations include reuse of previously computed Levenshtein results, updating only the affected stream distances, avoiding recomputation of full Levenshtein matrices, and exploiting symmetry between forward and reverse dynamic-programming computations. Experimentally, the approximation is reported to find the exact matching **86% of the time**, with an average deviation of **less than 0.02 percentage points**, while the abstract reports **<0.1% deviation** in experiments [2508.02112].

## 6. Applications, visualization, and limitations

DI-cpWER is presented as a metric for **system analysis**, not for leaderboard ranking. Its stated role is to estimate what cpWER would be if diarization or speaker attribution were perfect, and to help determine whether the main problem in a long-form multi-talker ASR system lies in recognition, segmentation, or speaker attribution. The paper emphasizes that higher-level errors such as speaker confusions cannot be reliably inferred automatically from the transcript alone, so DI-cpWER provides a useful **oracle-like lower bound** on cpWER [2508.02112].

The metric is particularly relevant in settings where outputs contain multiple streams, diarized segments, or serialized outputs with ambiguous speaker attribution. The paper identifies meeting transcription, continuous speech separation pipelines, CHiME-style multi-speaker ASR, and systems where transcription and diarization are jointly estimated as representative contexts in which the distinction between “what was said” and “who said it” is analytically important [2508.02112].

To expose those distinctions, the paper advocates **trace visualizations** of alignments. A numerical gap between cpWER and DI-cpWER is sufficient to infer that there is at least one speaker-attribution error, but an alignment trace can reveal exactly which words were assigned to the wrong speaker. In one illustrative example, the same sample yields **cpWER = 87.5%**, **ORC-WER = 50.0%**, **MIMO-WER = 37.5%**, and **DI-cpWER = 25.0%**, showing that many cpWER errors can disappear once hypothesis speaker labels are corrected [2508.02112].

The principal caveat is that DI-cpWER is **not** a fair ranking metric. The paper states that it can be gamed by splitting segments into smaller units and allowing the assignment mechanism to exploit that flexibility. Relatedly, word-level DI-cpWER and word-level ORC-WER, where each word is treated as its own segment, are described as more aggressive variants that generally under-estimate errors and are not recommended for serious evaluation, although they are included for illustration [2508.02112].

Source: https://www.emergentmind.com/topics/diarization-invariant-cpwer-di-cpwer