MEDLAYXPLAIN: Benchmarking the Expert-Lay Gap in Medical Vision-Language Models

Abstract: Medical Vision-LLMs (Med-VLMs) achieve strong expert-level performance, yet their ability to generate patient-accessible descriptions remains underexplored. With the 21st Century Cures Act now mandating immediate patient access to diagnostic imaging results, evaluating whether Med-VLMs can bridge this Expert-Lay Gap is both urgent and clinically consequential for patient education and shared decision-making. To this end, we introduce MedLayXPlain, the first large-scale multimodal benchmark and evaluation framework for Medical Lay Language Generation (MLLG). MedLayXPlain-122K provides 122,789 region-grounded samples across 8 imaging modalities from 12 publicly available source datasets, each comprising a medical image with paired expert and lay captions anchored in a three-level Unified Medical Language System (UMLS) ontology hierarchy spanning 7 semantic groups, 43 semantic types, and 2,411 medical concepts. Lay captions are constructed via Hierarchical Ontology-Verified Refinement (HOVER), a three-step pipeline combining patient-centric vocabulary mapping, LLM-based constrained rewriting, and cross-model visual verification to enforce semantic equivalence while preventing hallucination. We further introduce MedLayEval, a lightweight 3B evaluator distilled from a 27B verifier that scores expert-lay alignment across five clinically grounded attributes, addressing the poor correlation between standard NLG metrics and clinical judgment. Benchmarking 33 VLMs on MedLayXPlain-122K reveals a systematic Expert-Lay Gap: medical VLMs achieve strong expert captioning but suffer significant lay-register degradation, while general-purpose VLMs produce more accessible language yet lack clinical precision, confirming that neither current paradigm adequately serves patient-facing communication.

• The paper introduces MedLayXPlain, the first multimodal benchmark to evaluate medical VLMs in translating expert clinical language to patient-friendly text.
• It employs the HOVER pipeline, integrating ontology-based mapping and constrained LLM rewriting, to maintain semantic equivalence and clinical precision.
• Benchmarking 33 models reveals a significant trade-off between clinical fidelity and readability, highlighting the need for dual objective training in Med-VLMs.

MedLayXPlain: Benchmarking the Expert-Lay Gap in Medical Vision-LLMs

Motivation and Problem Formulation

Medical vision-LLMs (Med-VLMs) have demonstrated strong performance at expert clinical tasks, primarily due to training on technical corpora designed for clinician communication. Recent regulatory mandates, such as the US 21st Century Cures Act, now require direct patient access to diagnostic results, creating an urgent need to translate imaging findings into patient-accessible language. Despite this regulatory shift, Med-VLMs’ actual ability to generate comprehensible, clinically faithful lay descriptions remains unquantified and unaddressed. MedLayXPlain introduces the first large-scale multimodal benchmark and evaluation framework specifically for medical lay language generation (MLLG), bridging this unexamined expert-lay gap and addressing both the scalability and ontological grounding absent from previous efforts.

Figure 1: MedLayXPlain positions itself as the only benchmark providing multimodal region-grounded expert-lay pairs, meeting clinical needs for patient-accessible report rewriting and bridging existing dataset limitations.

Benchmark Construction and Dataset Architecture

MedLayXPlain comprises 122,789 region-grounded samples across 8 modalities—pathology, MRI, CT, X-ray, PET, dermoscopy, endoscopy, and ultrasound—drawn from 12 public datasets. Each sample consists of a medical image, region-of-interest (ROI) annotation, expert caption, and lay caption, with all captions grounded in the three-level UMLS ontology hierarchy (semantic group, type, concept). Benchmark construction utilizes the Hierarchical Ontology-VErified Refinement (HOVER) pipeline:

• Entity extraction and mapping of expert captions to UMLS concepts,
• Patient-friendly synonym retrieval primarily from Consumer Health Vocabulary (CHV) with robust fallback,
• Constrained rewriting using LLMs, enforcing semantic equivalence and parenthetical term retention,
• Cross-model visual verification by independent VLMs, scoring five clinical and accessibility attributes and triggering revision or rejection as necessary.

This process guarantees that lay captions preserve clinical fidelity while avoiding hallucination and optimizing readability.

Figure 2: The HOVER pipeline enforces three steps: ontology-driven mapping, constrained LLM rewriting, and external cross-model visual verification to ensure semantic/clinical equivalence.

Figure 3: MedLayXPlain dataset statistics highlight modality distribution and concept-level vocabulary flow, supporting diverse clinical tasks and providing a comprehensive semantic coverage.

MedLayEval: Multimodal Reference-Free Evaluator

Existing evaluation metrics penalize successful register transfer due to vocabulary shifts, are modality-limited, or incur prohibitive inference costs. MedLayEval is a distilled 3B multimodal evaluator trained via supervised regression from a 27B “teacher” verifier. It regresses scores for five attributes: modality consistency, anatomical accuracy, finding completeness, factual correctness, and lay readability. The architecture combines LoRA adaptation, attention-masked mean pooling, and a two-layer MLP regression head, enabling attribute-level interpretability and efficient inference.

MedLayEval is agnostic to reference captions and can reliably discriminate between high-fidelity and low-fidelity expert-to-lay translations. Pearson $r = 0.880$ vs. the teacher verifier demonstrates strong attribute score tracking.

Benchmarking VLMs: Systematic Expert-Lay Gap

A systematic evaluation of 33 VLMs—including proprietary, open-source generalist, and medical-specialized models—was performed using MedLayEval on a modality-stratified 5,000-sample test set. The analysis revealed:

• Strong numerical results for certain modalities and architectures: Gemini 2.5 Flash and MedGemma-27B achieve $S=70.6$ and $67.2$, respectively, while LLaVA-Med-7B scores only $23.9$, confirming a 46.7-point spread and a persisting open challenge.
• Contradictory claims regarding model specialization: Generalist VLMs outperform most specialized medical VLMs in patient-accessibility, but often suffer in clinical precision, indicating no paradigm offers both optimal fidelity and accessibility.
• Trade-off between clinical fidelity and readability: Medical VLMs retain factual accuracy but exhibit substantial readability deficits; generalists produce accessible but less precise outputs.
• Metric disagreement: Reference-based metrics (e.g., GREEN, BLEU-4) fail to rank lay performance, penalizing register transfer in favor of expert jargon, underscoring the necessity of evaluators like MedLayEval.

  Figure 4: VLM leaderboard reveals the large spread in MedLayEval scores, identifying expert-to-lay captioning as an unresolved challenge.

  

  Figure 5: Score polygons show that supplying expert captions significantly improves lay output across all top models; removing the image modestly degrades anatomical dependence only for certain architectures.

Qualitative Analysis and Model Behavior

Qualitative examples across modalities demonstrate successful register transfer, from brain MRI and abdominal CT—where swelling and cancer terms are correctly rephrased—to pathology and ultrasound. The pipeline consistently preserves all medical entities and delivers patient-friendly explanations, with frequent parenthetical term retention enabling seamless cross-referencing.

Figure 6: MRI example shows entity mapping and register-optimized captions, delivering semantically equivalent patient explanations.

Figure 7: CT example demonstrates rewrite of expert region-of-interest findings into accessible, actionable description.

Practical and Theoretical Implications

MedLayXPlain conclusively establishes the expert-lay gap as a distinct axis in Med-VLM development. The HOVER pipeline and MedLayEval evaluator are scalable, modality-agnostic, and can be extended to new datasets or clinical settings. Benchmark results advocate for joint training on clinical fidelity and patient accessibility and highlight the insufficiency of domain adaptation alone. This has direct practical implications for patient engagement, educational workflows, and regulatory compliance. Theoretically, MedLayXPlain exposes the need for ontology-grounded evaluation and region-level grounding, emphasizing the complexity of multimodal register transfer.

Future Directions

The benchmark framework supports expansion to multilingual lay captioning, integration of additional vocabularies, and refinement of visual verification protocols. As Med-VLMs mature, training on region-grounded, lay-annotated data is likely to become standard, and MedLayEval-like evaluators may become ubiquitous for validating patient-facing outputs.

Conclusion

MedLayXPlain delivers the first comprehensive multimodal benchmark for medical lay language generation, with rigorous ontological grounding and region annotation. Empirical analysis exposes persistent trade-offs across VLM architectures and scales, demanding explicit focus on both clinical precision and patient accessibility. Future Med-VLMs must incorporate lay communication as a primary training objective to meet real-world regulatory and patient needs.