VeriLLMed: Interactive Visual Debugging of Medical Large Language Models with Knowledge Graphs

Abstract: LLMs show promise in medical diagnosis, but real-world deployment remains challenging due to high-stakes clinical decisions and imperfect reasoning reliability. As a result, careful inspection of model behavior is essential for assessing whether diagnostic reasoning is reliable and clinically grounded. However, debugging medical LLMs remains difficult. First, developers often lack sufficient medical domain expertise to interpret model errors in clinically meaningful terms. Second, models can fail across a large and diverse set of instances involving different input types, tasks, and reasoning steps, making it challenging for developers to prioritize which errors deserve focused inspection. Third, developers struggle to identify recurring error patterns across cases, as existing debugging practices are largely instance-centric and rely on manual inspection of isolated failures. To address these challenges, we present VeriLLMed, a visual analytics system that integrates external biomedical knowledge to audit and debug medical LLM diagnostic reasoning. VeriLLMed transforms model outputs into comparable reasoning paths, constructs knowledge graph-grounded reference paths, and identifies three recurring classes of diagnosis errors: relation errors, branch errors, and missing errors. Case studies and expert evaluation demonstrate that VeriLLMed helps developers identify clinically implausible reasoning and generate actionable insights that can inform the improvement of medical LLMs.

• The paper demonstrates a novel visual analytics system that aligns LLM outputs with Bio-KGs to systematically identify structural reasoning errors.
• It employs multi-level visualization interfaces, including projection and Sankey views, to detect relation, branch, and missing errors in diagnostic reasoning.
• Case studies show that addressing these structural errors improved diagnostic accuracy, enhancing benchmark performance by 2.8%.

Visual Debugging of Medical LLM Reasoning via Knowledge Graphs: An Expert Analysis of VeriLLMed (2604.23356)

Problem Setting and Motivation

Medical diagnostic LLMs are increasingly evaluated on medical QA benchmarks but their deployment is hampered by unreliable reasoning and high-stakes consequences. Existing debugging methods are limited by the lack of accessible medical grounding, the difficulty of prioritizing errors, and the inability to identify recurring error patterns. VeriLLMed addresses these bottlenecks through a visual analytics system grounded in biomedical knowledge graphs (Bio-KGs), systematically mapping model outputs and reasoning traces to structured entities and relations for interpretability and debugging.

System Architecture and Computational Pipeline

VeriLLMed processes medical QA cases by extracting structured reasoning paths from LLM predictions and aligns both model-generated entities and ground truth entities to the Bio-KG. This alignment enables comparison of reasoning trajectories in a shared concept space and systematic identification of structural errors: relation errors (medically invalid transitions), branch errors (reasoning diverges into irrelevant concept regions), and missing errors (omission of diagnostically relevant entities).

Figure 1: The data processing pipeline aligns model reasoning with Bio-KG reference paths, identifies error types, and exposes recurrent error patterns for visual analytics.

The alignment method employs exact matching, domain-specific embeddings (MedEmbed), and LLM-based disambiguation, mitigating common biomedical entity ambiguity. Reference reasoning paths are generated by searching for shortest directed paths in the Bio-KG between question entities and correct answers, pruned via LLMs for semantic validity. Structural errors are quantified via graph reachability, and error patterns are expanded using downstream node neighborhoods for aggregated analysis.

Interactive Visual Debugging Interface

VeriLLMed provides a multi-level interface for coordinated reasoning inspection:

• Dataset Overview summarizes error distributions by type and supports initial triage.
• Projection View leverages node2vec and t-SNE for spatial localization of error-intense regions in the KG embedding space.
• Path View visualizes reasoning neighborhoods via glyphs encoding reference participation, error frequency, and connection structure.

  Figure 2: (A) Path View glyphs encode entity reasoning participation and error types in the KG embedding; (B) matrix alternative is considered, but dropped for scalability and interpretability.

• Error View utilizes a comparative Sankey layout to contrast aggregated erroneous and reference reasoning steps under semantic grouping, supporting pattern-level expansion and high-level explanation.

  Figure 3: (A) Comparative Sankey layout for error pattern summarization; (B) symmetric alternative is less effective for correspondence between erroneous and correct transitions.

• Detail View and Instance View link aggregate evidence back to case-level reasoning for granular validation.

Design Study and Workflow Evaluation

Formative studies with LLM developers highlight requirements for accessible medical knowledge integration, error typology, prioritized investigation, recurrent pattern discovery, and case-level validation. Expert interviews post-implementation reveal that VeriLLMed matches practical debugging logic and reduces manual error tracing by making structural reasoning errors visually explicit and easily explorable. Knowledge graph grounding provides valuable external reference, especially for non-expert developers.

Case Studies: Recurrent Diagnostic Error Identification

The authors illustrate the system in two analytic walkthroughs:

• Case 1: Missing errors concentrated around nonspecific symptoms (e.g., fever). Model omits integrating systemic symptoms, leading to lower accuracy and recurrent infectious disease misclassification. Adding representative examples to prompts corrected previously misdiagnosed cases, boosting benchmark accuracy by 2.8%.
• Case 2: Relation errors linked to dermatologic symptoms (e.g., pruritus). Model's reasoning erroneously shifts toward distant dermatologic associations instead of systemic autoimmune and chronic non-dermatologic conditions, confirmed by expanded error patterns and Sankey comparison.

  Figure 4: Diagnostic case analysis reveals relation error bias around dermatologic symptoms, with model reasoning diverging from clinically grounded KG-based alternatives.

Implications, Limitations, and Future Directions

VeriLLMed demonstrates structural reasoning-level debugging is feasible and effective for medical LLMs using Bio-KG grounding. The approach generalizes to other reasoning-intensive domains where structured reference knowledge is available. Limitations include reliance on entity extraction alignment accuracy, coverage constraints of the KG, and focus on unimodal QA. Extension to multimodal diagnostic reasoning and integration of uncertainty-aware alignment and prompt revision support are suggested.

The taxonomy of error types (relation, branch, missing) is tightly coupled to the structural units of medical KGs, allowing operational debugging workflows. However, expressive and contextual medical errors may require augmentation of KG representations or alternative knowledge sources.

Conclusion

VeriLLMed provides a comprehensive visual analytics workflow for debugging medical LLM diagnostic reasoning at a structural level by leveraging Bio-KGs. The system efficiently exposes recurrent structural reasoning errors and facilitates actionable refinement hypotheses. Expert user studies demonstrate improved debugging efficiency and hypothesis generation. The framework is extensible to other domains with structured reference knowledge, highlighting its broader applicability to post-hoc LLM reasoning audits in safety-critical settings.