Optimal Reference Combination WER (ORC-WER)

• ORC-WER is a speaker-agnostic word error rate metric that evaluates multi-speaker ASR systems by merging references into a single temporal sequence without splitting utterances.
• It employs a dynamic-programming strategy to compute the minimal edit distance efficiently, reducing complexity compared to earlier exponential approaches.
• ORC-WER bridges the gap between cpWER and MIMO-WER by ignoring speaker labeling errors while strictly preserving global utterance order.

Searching arXiv for recent and foundational papers on ORC-WER and related multi-speaker WER definitions. Optimal Reference Combination Word Error Rate (ORC-WER) is a speaker-agnostic Word Error Rate (WER) definition for evaluating automatic speech recognition (ASR) systems that process recordings containing multiple speakers and produce multiple output word sequences. It is intended for settings such as meeting transcription, Continuous Speech Separation (CSS), and serialized output training (SOT)-style recognition, where output streams may not have consistent speaker identities and where classical single-speaker WER is not directly applicable. In ORC-WER, all reference utterances are merged into a single reference sequence ordered by their original temporal occurrence, and the score is the minimal possible WER obtained by optimally assigning each utterance in that globally sorted reference to a single hypothesis channel, under the constraint that utterances cannot be split across multiple channels (Neumann et al., 2022). This definition evaluates content recovery while ignoring speaker label errors, but it preserves utterance integrity and global order, which distinguishes it from more permissive speaker-agnostic measures and from speaker-attributed measures such as cpWER (Neumann et al., 2022).

1. Concept and scope

ORC-WER addresses the assignment ambiguity that arises in long-form multi-speaker recognition. In such scenarios, hypotheses and references exist as collections of segments, and modern systems such as CSS-style systems may output streams without clear or consistent speaker labels. Classical WER cannot handle this ambiguity because it assumes a deterministic comparison between a single reference transcript and a single hypothesis transcript (Neumann et al., 4 Aug 2025).

The defining property of ORC-WER is that it is speaker-agnostic but not unconstrained. It ignores speaker attribution errors, yet it requires that every reference utterance be assigned intact to one output stream. This prevents a hypothesis from receiving artificially favorable credit when parts of a reference utterance are distributed across multiple output streams. The 2022 formulation explicitly motivates this behavior as a way to avoid counting as correct those hypotheses where parts of an utterance are spread over several outputs (Neumann et al., 2022).

This places ORC-WER between two other families of metrics. Relative to cpWER, it is less strict because it does not penalize speaker labeling errors. Relative to MIMO-WER, it is more restrictive because it preserves a global utterance order. The long-form comparison paper states that speaker-attributed approaches such as cpWER and tcpWER count speaker confusion errors, whereas speaker-agnostic approaches such as ORC-WER and MIMO-WER ignore speaker confusion errors but differ in how they handle temporal misalignment and ordering constraints (Neumann et al., 4 Aug 2025).

2. Formal definition

In the 2022 general framework, ORC-WER is obtained by reducing the multi-input multi-output setting to the case $\nspk = 1$ after merging the references. If the merged reference sequence is denoted by $\treforc$ and the hypothesis channels by $\{\thyp_1, \ldots, \thyp_{\nchn}\}$, ORC-WER seeks the minimal WER achievable by assigning each utterance in the merged reference to one hypothesis channel, without allowing utterance splits (Neumann et al., 2022).

The long-form 2025 paper gives the ORC-WER distance as

$\distance^\text{ORC} = \min_{(v_1, \dots, v_\nstream) \in \mathcal{V}} \sum_{\istream=1}^\nstream \lev(\select(v_\istream)\rv,\; \select(\ind_{\{i:\hlble_i=\istream\})}\hv),$

where $\lev(\cdot,\cdot)$ is the Levenshtein distance, each $v_\istream$ selects the reference utterances mapped to stream $\istream$, and the set $\mathcal{V}$ contains assignments in which each reference segment is assigned exactly once and the original sequence order is preserved (Neumann et al., 4 Aug 2025).

MeetEval presents the same idea in toolkit form as

$\mathrm{lev}^{\text{(ORC)}} = \min_{\rho_1, \dots, \rho_J} \sum_{\ichn=1}^{\nchn} \mathrm{lev}\left(\bigoplus_{\iutt\in\rho_\ichn}\tref_{\iutt}, \bigoplus_{\iutt'=1}^{\nutt_{\thyp,\ichn}}\thyp_{\ichn,\iutt'}\right),$

where the reference utterances are sorted in time while speaker labels are ignored, and the optimization ranges over partitions of reference utterances across hypothesis streams that preserve temporal order (Neumann et al., 2023).

In all of these formulations, the normalized ORC-WER is the total edit distance divided by the total number of reference words (Neumann et al., 2022).

3. Dynamic-programming computation

A central contribution of the 2022 work is an efficient implementation of ORC-WER based on dynamic programming in a multi-dimensional Levenshtein distance tensor. The general MIMO recursion extends the Wagner-Fischer algorithm to multiple reference and multiple hypothesis sequences, and ORC-WER appears as the special case $\nspk = 1$ after reference merging (Neumann et al., 2022).

The computation uses a multi-dimensional tensor $\treforc$0 whose entries represent the minimum edit distance for every combination of progress through the reference and the hypothesis channels. To prevent utterance splits, the algorithm introduces a channel change token at utterance boundaries and permits channel assignment changes only at those boundaries. Equation (4) in the 2022 paper specializes the recurrence to a channel-conditioned tensor $\treforc$1 and restricts channel switching to change-token positions (Neumann et al., 2022).

This formulation addresses a previously noted computational obstacle. Earlier brute-force approaches were exponential in the number of utterances, which made exact ORC-WER intractable for longer recordings. The dynamic-programming formulation reduces the computation to polynomial complexity in the number of utterances, and the paper reports that this enables efficient computation for much longer recordings, with benchmarks extending beyond 70 utterances compared with previous limits around 23 (Neumann et al., 2022). MeetEval adopts this polynomial-time dynamic-programming approach and notes that the complexity does not depend on the number of reference speakers and is polynomial in the number of utterances (Neumann et al., 2023).

A plausible implication is that ORC-WER became practical as a routine evaluation metric only after this reformulation. The earlier definition existed, but its direct computation suffered from exponential scaling (Neumann et al., 2022).

4. Temporal constraints and timing approximations

Standard ORC-WER is defined on word sequences and does not require speaker labels or detailed word timings. However, several later works emphasize that unconstrained Levenshtein alignment can produce implausible matches over long temporal gaps. To address this, MeetEval discusses a time-constrained extension, tcORC-WER, in which substitutions or correct matches are only allowed when word timings overlap or the gap lies within a specified collar (Neumann et al., 2023).

The long-form 2025 paper gives a corresponding time-constrained edit cost:

$\treforc$2

The paper argues that the time constraint makes the metric more human plausible and also reduces computational cost because only alignments within the collar need to be considered (Neumann et al., 4 Aug 2025).

MeetEval further addresses the common case in which exact word-level timings are unavailable. It presents several approximations derived from segment-level timings: full-segment assignment, equally-sized intervals, character-based approximation, and character-based time points. The reported recommendation is to use the character-based approximation for references and character-based time points for hypotheses (Neumann et al., 2023). The paper states that the character-based approximation yields median differences of less than 100 ms between pseudo word boundaries and ground-truth alignments, and that with a collar of 4–5 seconds, time-constrained WER using pseudo word-level timings closely tracks scoring with true word-level annotations (Neumann et al., 2023).

This suggests that timing-aware ORC-WER can be deployed even when only segment-level timestamps are available, provided the collar is chosen conservatively.

5. Relation to other WER definitions

ORC-WER is best understood within the broader landscape of long-form multi-talker WER definitions. The 2022 framework explicitly states that commonly used WER definitions for multi-speaker scenarios can be viewed as specializations of a general MIMO-WER framework (Neumann et al., 2022).

The principal contrasts are as follows.

Metric	Speaker treatment	Key constraint
cpWER	Speaker-attributed	Best speaker permutation
ORC-WER	Speaker-agnostic	Global order preserved; no utterance splits
MIMO-WER	Speaker-agnostic	More permissive with respect to ordering
asclite	Speaker-agnostic with temporal info	Relies on word time-stamps

cpWER is speaker-attributed and penalizes diarization or speaker assignment errors. It finds a permutation between speaker-reference and hypothesis outputs and computes a single-input single-output WER over the best assignment. The 2022 paper describes cpWER as an upper bound to ORC-WER and MIMO-WER and recommends it when speaker labeling is important (Neumann et al., 2022).

ORC-WER is speaker-agnostic and does not judge speaker identity, but it does require each utterance to appear intact on one hypothesis stream. The 2022 discussion characterizes it as a lower bound on cpWER, because speaker errors are not counted, and an upper bound on MIMO-WER, because it imposes a global utterance order (Neumann et al., 2022).

MIMO-WER is more general. The 2022 paper presents it as a fully generalized error-rate scheme for arbitrary numbers of speakers and channels, including settings where utterances can be reordered, and the 2025 comparison states that it is more permissive than ORC-WER (Neumann et al., 2022); (Neumann et al., 4 Aug 2025).

asclite uses a multi-dimensional Levenshtein distance and relies on time-stamps. The 2022 paper notes that it does not penalize utterance splits, so hypotheses that spread utterances over channels can score unrealistically well (Neumann et al., 2022).

The 2025 comparison adds DI-cpWER, a diarization-invariant cpWER whose difference from cpWER reflects the impact of speaker confusions on WER (Neumann et al., 4 Aug 2025). This does not alter ORC-WER itself, but it clarifies the boundary between speaker-agnostic content evaluation and speaker-attributed evaluation.

6. Applications and empirical use

ORC-WER is used primarily to evaluate multi-output ASR systems whose streams are not canonically mapped to speakers. The 2022 paper recommends it for continuous, speaker-agnostic evaluation of multi-speaker ASR systems that output multiple channels but do not output speaker labels, especially when detailed speaker or time information is unavailable (Neumann et al., 2022).

In meeting transcription, ORC-WER is often paired with cpWER. A modular meeting recognition pipeline using TF-GridNet CSS, a speaker-agnostic recognizer, and d-vector-based diarization evaluates recognition quality with ORC-WER and full-pipeline quality with cpWER (Neumann et al., 2023). That paper states that ORC-WER assigns each reference utterance to one system output stream such that the WER is minimized and ignores speaker label errors, whereas cpWER takes the assigned speaker labels into account (Neumann et al., 2023). Reported Libri-CSS results include ORC-WER values of 26.5 for no separation with Whisper, 6.8 for TF-GridNet with Whisper, 6.4 for TF-GridNet with ESPnet, and 3.5 for clean signals with Whisper (Neumann et al., 2023).

MeetEval operationalizes ORC-WER in an open-source toolkit that provides a unified interface for cpWER, ORC-WER, and MIMO-WER, among other WER definitions (Neumann et al., 2023). This is significant because it makes exact or timing-aware ORC-WER computation reproducible and standardizable across evaluation settings.

ORC-WER also appears outside multi-speaker meeting transcription in a different, segment-level oracle sense. In work on ASR system combination and selection, “ORC-WER” is used for the minimum WER obtainable by selecting the best system for each segment from a set of hypotheses (Jalalvand et al., 2017); (Gündüz et al., 2024). In the 2017 quality-estimation-for-ROVER paper, this segment-level oracle is written as

$\treforc$3

and serves as a local oracle for ranking or fusion (Jalalvand et al., 2017). AutoMode-ASR uses the same oracle concept as a lower bound for segment-level system selection and reports an oracle WER of 6.5 compared with 11.1 for its learned selector and 13.4 for a single-best baseline (Gündüz et al., 2024). The terminology is shared, but the task differs from the multi-speaker stream-assignment problem defined in (Neumann et al., 2022).

7. Limitations, recommendations, and interpretive cautions

ORC-WER is not a universal measure of transcript quality. The 2022 paper states that it is not applicable to systems that allow utterance reordering, such as non-FIFO serialization, because its definition preserves global utterance order (Neumann et al., 2022). The 2025 comparison similarly positions MIMO-WER as the more permissive choice when ordering flexibility must be tolerated (Neumann et al., 4 Aug 2025).

The metric can also be misleading when utterance boundaries in the references are highly inaccurate or ambiguous, although the 2022 paper notes that such cases are rare in practice (Neumann et al., 2022). This caveat matters because ORC-WER enforces integrity at the utterance level; if the reference segmentation is itself unstable, the split-penalty can reflect annotation artifacts as much as recognition behavior.

A further limitation is evaluative rather than algorithmic: ORC-WER ignores speaker attribution. The 2023 meeting-recognition paper therefore recommends interpreting ORC-WER together with cpWER, since ORC-WER reflects the quality of separation and recognition, while cpWER captures the overall capability of the pipeline including diarization (Neumann et al., 2023).

The recommendations across the cited literature are consistent. ORC-WER is appropriate when the goal is speaker-agnostic content evaluation for multi-speaker ASR systems whose outputs are arranged in multiple streams without reliable speaker labels, and when preserving utterance integrity is important (Neumann et al., 2022). Time-constrained ORC-WER is preferable when temporal plausibility and computational tractability are both concerns (Neumann et al., 4 Aug 2025); (Neumann et al., 2023). When speaker attribution is itself part of the task, cpWER or related speaker-attributed metrics are more suitable (Neumann et al., 2022).