--- title: Knowledge Graph-Aided ASR Error Correction url: https://www.emergentmind.com/topics/knowledge-graph-aided-asr-error-correction type: topic --- # Knowledge Graph-Aided ASR Error Correction Knowledge graph-aided ASR error correction refers to a class of approaches that leverage structured, graph-based representations of domain or world knowledge to improve the accuracy of automatic speech recognition (ASR) systems, particularly in domains where named entities and terminological precision are crucial. By integrating semantic, relational, and sometimes phonetic information from knowledge graphs (KGs), such systems can disambiguate entity mentions, correct misleading outputs, and reduce error rates in downstream tasks such as spoken question answering (SQA), virtual assistance, and general transcription. ## 1. Motivation and Problem Formulation Conventional ASR systems frequently struggle with the recognition of rare or ambiguous entities, technical terms, and homophones, resulting in high word error rates (WER) and sentence error rates (SER), especially in domain-specific applications such as medical QA and virtual assistants. These errors often stem from acoustic confusability, limited vocabulary coverage, or insufficient linguistic context in n-gram or neural language models. Knowledge graph-aided frameworks address these challenges by injecting structured factual and relational knowledge—entity types, aliases, relationships, and (in some cases) phonetic confusables—into the error correction pipeline, usually at the post-ASR or lattice rescoring stage. The objective is to align ASR hypotheses more closely with real-world entity relationships or domain constraints and to exploit external world knowledge unattainable to base ASR models [2106.11292, 1705.08018, 2602.00981]. ## 2. Construction and Utilization of Knowledge Graphs Effective application of KGs to ASR error correction depends on the precise construction and real-time exploitation of suitable graphs: - **Entity and Relation Extraction**: In domain-agnostic settings, entities are extracted from ASR hypotheses using mention detection and linking systems (e.g., DBpedia Spotlight for TED-Lium) [1705.08018]. In medical domains, entity recognition leverages lexicons or NER models to identify domain concepts. - **Graph Construction**: - *Semantic edges*: In MedSpeak, nodes represent UMLS medical concepts with standardized CUIs, linked by relations such as *classifies*, *due_to*, or *plays_role* [2602.00981]. - *Phonetic edges*: Unique to MedSpeak, undirected edges connect node pairs with similar Double Metaphone encodings and low Levenshtein distance in CMU-dictionary pronunciations, explicitly modeling phonetic confusability prevalent in medical speech [2602.00981]. - *General domains*: Graphs are constructed by fetching triples from resources such as DBpedia, using entity URIs and relations extracted with SPARQL queries and filtering high-degree hubs [1705.08018]. - **Representation for Inference**: The retrieved KG subgraphs for all detected entities are serialized as plain text (e.g., for LLM prompting) or embedded numerically (e.g., with TransE embeddings), depending on the downstream error correction method. ## 3. Integration Mechanisms: Semantic and Phonetic Context in Scoring and Correction Three principal integration methodologies are apparent across recent systems: - **Log-linear Feature Integration**: In discriminative n-gram language modeling (DEAL), KG-based features (e.g., entity–entity co-occurrence, type-specific n-grams) are activated during candidate scoring, with weights learned discriminatively [2106.11292]. The overall score for a candidate hypothesis $h$ in utterance $u$ is $$ s(u,h) = \sum_{f=0}^{F} w_f x_{u,h,f} $$ where $f=0$ is the base system score, and $f \ge 1$ enumerates KG-induced features. - **Embedding-based Relatedness**: KG triples retrieved for a hypothesis are embedded via TransE; the semantic relatedness cost between consecutive entities is computed as $$ \beta(H) = \sum_{t=1}^{T-1} \delta_t = \sum_{t=1}^{T-1} \min_n \|M_{t+1}^n - M_t^n\| $$ where $M_t^n$ is the $n$-th triple embedding for entity $e_t$ [1705.08018]. - **Prompt-based Semantic/Phonetic Context for LLMs**: In MedSpeak, the union of a term's semantic and phonetic neighbors is supplied to the LLM via prompt engineering, enabling the model to resolve both context and potential confusables during correction. No explicit embedding learning is performed; instead, raw relational and phonetic context is input to the language model [2602.00981]. In an *Editor's term* for this approach, one can describe it as "contextual candidate augmentation": candidate entity corrections are presented as explicitly contextualized options for the LLM's attention mechanism. ## 4. Correction Algorithms and Model Architectures The concrete error correction workflow varies by system: - **N-best/Lattice Rescoring**: Both [1705.08018] and [2106.11292] rerank the N-best or lattice hypotheses output by a decoder using KG-derived features or cost terms. Hypotheses are rescored in a log-linear framework, with KG features (embedding distances, n-gram triggers) interpolated with acoustic and language model scores. Feature weights are tuned by grid search or discriminative loss minimization (e.g., minimum WER objective). - **LLM-based Correction**: MedSpeak employs a fine-tuned Llama-3.1-8B-Instruct LLM. The model is trained to take as input the noisy ASR transcript, multiple-choice options, and truncated KG context—outputting both a corrected transcript and the predicted answer. Formally: $$ L(\theta) = -\sum_{i=1}^T \log P_\theta(y_i \mid x, y_{