--- title: 'MedSpeak: Medical ASR Correction' url: https://www.emergentmind.com/topics/medspeak type: topic --- # MedSpeak: Medical ASR Correction MedSpeak is a knowledge graph–aided error correction and question-answering (QA) framework designed to address the persistent failure modes of automatic speech recognition (ASR) in the medical domain. By leveraging a structured medical knowledge graph (KG) with explicit semantic and phonetic relationships, MedSpeak refines noisy ASR transcripts and improves downstream medical spoken QA by incorporating large language model (LLM)–driven reasoning. The approach systematically resolves domain-specific misrecognitions, particularly for specialized terminology and phonetic confusions, and delivers state-of-the-art accuracy for both ASR correction and clinical multiple-choice QA scenarios [2602.00981]. ## 1. Background and Motivation Spoken medical QA systems, including LLM-based clinical dialogue agents, rely heavily on ASR outputs as their textual substrate. However, general-purpose ASR models exhibit substantially higher error rates on medical terms compared to non-domain speech, leading to: - Substitution errors between clinically distinct entities (e.g., "hypoplasia" misrecognized as "hyperplasia"). - Phonetic confusion between near-homophones with different semantics ("hypertension" vs. "hypotension"). - Propagation of recognition errors into downstream QA, resulting in wrong diagnoses or decision support. Prior attempts at fine-tuning ASR on limited medical audio have yielded only modest improvements, remaining data hungry and incapable of systematically resolving phonetic near-miss errors. Retrieval-augmented pipelines and KG-based techniques typically neglect phonetic ambiguities, instead relying on textual snippets or semantic edges alone, which is insufficient for high-stakes domains with dense, confusable terminology. MedSpeak's innovation is the integration of both semantic and phonetic relationships within a unified KG–LLM correction pipeline, targeting domain-specific ASR error modes [2602.00981]. ## 2. System Architecture MedSpeak is realized as a modular five-stage pipeline: 1. **Noisy ASR Input:** Initial transcript ($\hat t$) produced by a state-of-the-art ASR model (e.g., Whisper) from raw medical audio. 2. **Knowledge Graph Integration:** Identification of candidate medical terms in $\hat t$; retrieval of semantic ($\mathcal K_{\mathrm{sem}}$) and phonetic ($\mathcal K_{\mathrm{phon}}$) subgraphs for correction. 3. **Phonetic Encoder:** Embedding of both original and candidate tokens via a phonetic encoder, parameterized to capture spelling–pronunciation correspondences (e.g., Double Metaphone+CMUdict lexicon). 4. **LLM-Based Reasoning:** Concatenation of system instructions, user block (noisy transcript, options, retrieved KG context) as prompt; input to a fine-tuned LLaMA-derived large language model. 5. **Final Prediction:** Joint output of the corrected transcript ($\tilde t$) and QA answer ($o^*$) by the LLM. Pipeline summary: Audio → ASR → Noisy Transcript → KG Retrieval → Encoder + Scoring → Candidate Reranking → Corrected Transcript → LLM (with options + KG) → Answer & Final Transcript [2602.00981]. ## 3. Medical Knowledge Graph Representation The MedSpeak KG is formally defined as $G = (V, E)$: - $V$: medical concepts (e.g., "hypoplasia," "cerebral atrophy"). - $E$: edges typed by relation ("classifies", "due_to", "phonetic"). Each node $v \in V$ is assigned an embedding $h_v \in \mathbb{R}^d$ (pretrained via translational KG objectives such as TransE or GAT over UMLS). Edges are likewise embedded—either as learned matrices $W_r \in \mathbb{R}^{d\times d}$ by relation $r$, or via vector representations. For semantic proximity of terms $w_i, w_j$ within KG, MedSpeak defines: \[ s_{sem}(w_i, w_j) = \sigma\bigl(h_{w_i}^T W_r h_{w_j}\bigr) \] where $\sigma(\cdot)$ is the sigmoid and $W_r$ is relation-specific [2602.00981]. ## 4. Phonetic Error Correction via KG A distinguishing component of MedSpeak is the explicit modeling of phonetic confusability. Each word $w$ and candidate term $t$ are mapped to phonetic embeddings by $f_{phon}$ (e.g., Double Metaphone features, encoded as vectors): \[ d_{phon}(w, t) = \| f_{phon}(w) - f_{phon}(t) \|_2 \] Candidates are ranked for possible correction in terms of a joint score: \[ s(w, t) = \lambda s_{sem}(w, t) + (1 - \lambda)(1 - d_{phon}(w, t)) \] with $\lambda \in [0, 1]$ controlling the semantic vs. phonetic tradeoff. This formulation ensures prioritization of candidates that are both semantically proximate and phonetically plausible, outperforming prior approaches that neglect either axis [2602.00981]. ## 5. LLM Integration and Reasoning for Spoken QA The LLM module is explicitly prompted to generate corrected transcripts alongside multiple-choice answers, using the following protocol: - **System instruction:** "You must output exactly two lines: Corrected Text: … Correct Option: A/B/C/D." - **User context:** comprises the noisy ASR output, choice set $\{A, B, C, D\}$, and the truncated KG context (containing candidate nodes and edges derived as above). The LLM (LLaMA-derived, with in-domain fine-tuning) receives this structured prompt, enabling joint editing of the ASR transcript and downstream QA by leveraging both in-context semantic/phonetic cues and the candidate answer set [2602.00981]. Training is end-to-end, with a causal language modeling objective: \[ \mathcal{L}(\theta) = -\sum_{i=1}^{|y|} \log P_\theta(y_i | x, y_{