Lexical Speaker Error Correction (LSEC)

• Lexical Speaker Error Correction (LSEC) is a suite of post-processing techniques that use contextual lexical and acoustic cues to correct word-level speaker labeling errors in ASR pipelines.
• It employs transformer-based models with both lexical-only and lexical-acoustic fusion methods to effectively address errors at speaker turns and overlapping speech regions.
• LSEC achieves significant performance gains, reducing Word Diarization Error Rate and Speaker Attributed WER on conversational benchmarks.

Lexical Speaker Error Correction (LSEC) is a family of post-processing algorithms designed to improve word-level speaker labeling in automatic speech recognition (ASR) pipelines with separate diarization modules. Unlike traditional speaker diarization, which relies primarily on acoustic clustering and segmentation, LSEC leverages lexical and contextual information—often with large pre-trained LLMs—to detect and fix word-level speaker assignment errors, especially those occurring at speaker turns and in regions of overlap. More recent LSEC frameworks further incorporate word-level or frame-level acoustic speaker probabilities, yielding multimodal models with robust gains on multiple conversational benchmarks.

1. Motivation and Problem Definition

Standard conversational ASR pipelines produce an output transcript and a hypothesized mapping of words to speakers by aligning recognized words to time-stamped diarization segments. Several challenges arise in this setting:

• Acoustic diarization systems (SD) are prone to cluster boundary errors, especially around rapid speaker-turns, overlapping speech, and short utterances.
• The reconciliation of SD and ASR outputs (i.e., mapping word timestamps to speaker segments) is inherently noisy, introducing systematic errors at word-level speaker assignments.
• Lexical regularities in conversation (e.g., turn-taking cues, syntactic and semantic continuities) are inaccessible to purely acoustic models.

Lexical Speaker Error Correction addresses these weaknesses by post-processing the SD-ASR output with a separate module that utilizes lexical context—often large-scale pre-trained LMs—optionally fused with acoustic cues, to directly reduce word-level speaker labeling errors. The canonical task is: given a word sequence $w_1,\ldots,w_N$ and an initial sequence of hypothesized speaker tags $s_1,\ldots,s_N$, predict a corrected sequence $s^*_1,\ldots,s^*_N$ such that $s^*_i$ better matches the ground-truth assignment for each word $w_i$ (Kirakosyan et al., 2024, Paturi et al., 2023, Paturi et al., 2024, Kumar et al., 14 Jan 2025).

2. Core LSEC Methodologies

2.1. Lexical-only Correction with Transformer Models

The foundational LSEC models are text-only, operating exclusively on the ASR hypothesis and initial speaker labeling. The core architectural paradigm is:

• Input: ASR-decoded wordpiece sequence $w_1,\ldots,w_N$ and initial speaker tags $s_1,\ldots,s_N$.
• Word tokens are embedded via a frozen/fine-tuned pre-trained LM such as RoBERTa-Base or ALBERT-base.
• Speaker tags are embedded (e.g., via a learned vector for each speaker label) and summed or concatenated with word embeddings at each position (Kirakosyan et al., 2024, Paturi et al., 2023).
• A shallow Transformer encoder processes the combined embeddings in parallel (non-autoregressive).
• A per-token softmax layer outputs $P(\text{speaker}=k \mid w_{1:N}, s_{1:N})$ for $k=1,\ldots,K$ speakers.
• Inference applies the model within a window centered at putative speaker-change points or in sliding windows (Kirakosyan et al., 2024, Paturi et al., 2023).
• For two-speaker scenarios, permutation-invariant cross-entropy loss is standard to accommodate speaker-label ambiguity (Kirakosyan et al., 2024).

2.2. Lexical–Acoustic Fusion: AG-LSEC and SEAL Frameworks

Recent advances integrate acoustic speaker scores, derived from SD systems such as EEND, into the LSEC pipeline:

• Acoustic speaker scores: EEND computes frame-level speaker posteriors $p_{s,t}$, which are filtered and mean/median-pooled over each word’s time span to yield word-level soft speaker scores $s_1,\ldots,s_N$0 for each word $s_1,\ldots,s_N$1 (Paturi et al., 2024, Kumar et al., 14 Jan 2025).
• Early fusion: Word embeddings are concatenated at the first subword position of each word with the corresponding $s_1,\ldots,s_N$2, providing the model with both lexical and acoustic cues during encoding (Paturi et al., 2024).
• Late fusion: Lexical posterior $s_1,\ldots,s_N$3 from the LSEC frontend is combined with acoustic posteriors $s_1,\ldots,s_N$4 via an auxiliary feed-forward layer (Paturi et al., 2024).
• LLM-based correction with acoustic conditioning: Fine-tuned LLMs (e.g., Mistral-7B Instruct) receive prompts that interleave the transcript with inline, discretized speaker confidence tokens (“low/medium/high”), constraining output to only re-label speakers and not alter the transcript words (Kumar et al., 14 Jan 2025).
• Constrained decoding: Output is forced to match the input word sequence exactly, ensuring changes only in speaker attribution, not word recognition (Kumar et al., 14 Jan 2025).

2.3. Beam Search and Contextual Inference

An alternative methodology frames LSEC as a joint probabilistic decoding problem—optimizing over both word and speaker assignments:

• The joint objective factors as $s_1,\ldots,s_N$5, where $s_1,\ldots,s_N$6 are acoustic SD outputs, $s_1,\ldots,s_N$7 is the speaker sequence, and $s_1,\ldots,s_N$8 is the word sequence (Park et al., 2023).
• Beam search maintains parallel hypotheses over word and speaker sequences, scoring extensions by a weighted sum of acoustic model, LLM-based lexical speaker probability, and lexical word probability.
• General-purpose LLMs (e.g., Megatron-GPT) are prompted at each step to predict $s_1,\ldots,s_N$9 and $s^*_1,\ldots,s^*_N$0 (Park et al., 2023).

3. Training, Evaluation, and Datasets

3.1. Training Protocols

• Data is typically constructed from conversational corpora (Fisher, CALLHOME, RT03-CTS), using ASR transcripts and SD outputs (Paturi et al., 2023, Paturi et al., 2024).
• LSEC models are initially trained on simulated errors: random speaker-tag flips and/or word-level corruptions; curriculum schedules decrease corruption over training epochs (Paturi et al., 2023).
• Models are fine-tuned on real SD–ASR reconciled data, using ground-truth annotated speaker labels for supervision (Paturi et al., 2023, Paturi et al., 2024).

3.2. Metrics

• Word Diarization Error Rate (WDER): the fraction of output words whose assigned speaker differs from reference, including ASR insertions, deletions, and substitutions (Paturi et al., 2024, Paturi et al., 2023, Kirakosyan et al., 2024).
• Speaker Attributed WER (SA-WER): WER computed per speaker and averaged; improvement is measured by reduction in $s^*_1,\ldots,s^*_N$1SA-WER relative to baseline (Park et al., 2023).
• Capital-pair WER (cpWER): error rate that counts any word with a mis-assigned speaker as a WER event (Kirakosyan et al., 2024, Kumar et al., 14 Jan 2025).

4. Performance and Empirical Gains

4.1. Lexical-only LSEC

• LSEC reduces WDER by 15–30% on telephony data (e.g., Fisher: 2.26% → 1.53%; RT03-CTS: 2.18% → 1.59%) (Paturi et al., 2023).
• Largest absolute improvements are observed near speaker turns and in segments with rapid alternations.
• Accuracy saturates quickly with moderate transcript data size; simulated-error pretraining is highly effective (Paturi et al., 2023).

4.2. Lexical–Acoustic Fusion Methods

• AG-LSEC early-fusion architecture yields up to 40% relative WDER reduction over diarization-only baselines (e.g., Fisher: 2.56% → 1.56%; RT03-CTS: 2.64% → 1.56%) (Paturi et al., 2024).
• The extension over LSEC is significant: 23–26% additional WDER reduction (Fisher: 2.03% → 1.56%) (Paturi et al., 2024).
• SEAL, implementing acoustic conditioning plus constrained decoding, achieves 24–43% relative reduction in speaker error rates across Fisher, Callhome, and RT03-CTS, with 10–15% extra gain from decoding constraints (Kumar et al., 14 Jan 2025).
• Contextual beam search methods leveraging LLM lexical priors attain up to 39.8% relative decrease in $s^*_1,\ldots,s^*_N$2SA-WER (Park et al., 2023).

Example Table: WDER reductions on Fisher (selected approaches)

Method	WDER (%)	Relative Reduction vs. Baseline
SD+ASR Baseline	2.56	–
LSEC (lexical only)	2.03	21%
AG-LSEC Early Fusion	1.56	39%
SEAL (LLM+acoustic)	1.46*	~43%*

*Approximated from reported cpWER reductions.

5. Architectural and Algorithmic Considerations

• Sliding window strategies: LSEC is typically run locally (e.g., 30-word windows, centered at speaker changes or in a sliding manner) to accommodate varying speaker counts and maintain context coherence (Paturi et al., 2023, Kirakosyan et al., 2024).
• Permutation invariance: supervised losses explicitly accommodate arbitrary labelings (K! permutations for $s^*_1,\ldots,s^*_N$3 speakers), especially when ground-truth speaker identities are anonymous (Kirakosyan et al., 2024).
• Speaker-turn probability modeling: Enhanced diarization can compute word-level speaker-turn probabilities via bi-directional GRUs, which are then fused with acoustic adjacency matrices to bias spectral clustering toward likely turn boundaries (Park et al., 2020).
• Error correction constraint: Fine-tuned LLMs with constrained decoding are restricted from altering the text output, focusing solely on speaker label correction (Kumar et al., 14 Jan 2025).

6. Limitations, Extensions, and Research Challenges

• Speaker count scalability: Most implementations focus on 2-speaker local windows; multi-party global modeling demands sophisticated permutation-invariant objectives or clustering (Kirakosyan et al., 2024, Paturi et al., 2024).
• Language dependence: All published evaluations are in English; extensions to multilingual or code-switched contexts remain open (Paturi et al., 2023, Kumar et al., 14 Jan 2025).
• Acoustic–lexical integration: Effective integration hinges on quality word-level acoustic speaker scores and robust lexical embeddings; early fusion exhibits strongest empirical gains (Paturi et al., 2024).
• Computational trade-offs: Full LLM scoring is computationally intensive; hybrid approaches (e.g., LLM for speaker probability, n-gram for word probability) lower cost while retaining ∼90% of the gain (Park et al., 2023).
• Windowing and overcorrection: Applying LSEC too densely (excessive overlap or global relabeling) degrades performance, as spurious corrections accumulate (Kirakosyan et al., 2024).
• Error analysis: LSEC corrects most errors at speaker turns and overlaps but remains challenged by long-span speaker swaps or ambiguous context (Paturi et al., 2023, Paturi et al., 2024).
• Evaluation boundary: Modern constraints guarantee zero word-level errors (via constrained decoding), decoupling speaker error correction from ASR error propagation (Kumar et al., 14 Jan 2025).
• Potential future directions include local modeling of more than two speakers, integration of frame-level acoustic embeddings, joint end-to-end training, adaptation to unseen dialects or domains, and generalization to related sequence-labeling tasks such as role identification (Paturi et al., 2024, Kumar et al., 14 Jan 2025).

7. Comparative Analysis and Practical Implications

• LSEC and its descendants demonstrate that word-level speaker error correction—especially with explicit context modeling and acoustic-lexical fusion—yields substantial improvements over classic diarization and reconciliation pipelines (Paturi et al., 2024, Kumar et al., 14 Jan 2025).
• Large-scale pre-trained LMs function as robust contextual priors, able to correct labeling errors that are inaccessible to acoustic clustering.
• Acoustic grounding further reduces overcorrections and hallucinations, as evidenced by early-fusion AG-LSEC and SEAL's performance.
• As LLM infrastructures and diarization backends evolve, LSEC architectures are expected to continue delivering state-of-the-art speaker-attribution accuracy across increasingly varied and challenging conversational data (Kumar et al., 14 Jan 2025, Paturi et al., 2024, Park et al., 2023).

References:

(Paturi et al., 2023, Paturi et al., 2024, Kumar et al., 14 Jan 2025, Kirakosyan et al., 2024, Park et al., 2023, Park et al., 2020)