MedConceptEval: Medical Concept Evaluation

• MedConceptEval is a framework for evaluating clinical concept fidelity and semantic alignment in various medical AI tasks.
• It aggregates methods such as exact span detection, descriptor bank similarity, and graded relevance scoring via UMLS hierarchies.
• The approach shifts evaluation from mere lexical overlap to verifying clinically meaningful concept extraction and reasoning.

Searching arXiv for papers referring to “MedConceptEval” and the cited identifiers. Calling arXiv search tool. arXiv search results indicate that “MedConceptEval” is used in multiple, non-identical senses across recent medical-AI papers, including note-generation evaluation, interpretability evaluation for medical VLMs, and broader concept-recognition benchmarks (Kamal et al., 7 Aug 2025, Haque et al., 13 Apr 2026, Osborne et al., 2014). MedConceptEval denotes a set of concept-centered evaluation formulations in medical artificial intelligence rather than a single standardized metric. In the cited literature, the name has been used for clinical concept recognition and CUI mapping in text, section-level semantic alignment for generated SOAP notes, semantic verification of latent concepts in medical vision-LLMs, and semantically aware relevance scoring for medical image retrieval (Osborne et al., 2014, Kamal et al., 12 Jun 2025, Haque et al., 13 Apr 2026, Wei et al., 16 Jun 2025). This suggests that the term functions as an umbrella for evaluations in which medically meaningful concepts, rather than only lexical overlap or coarse end-task scores, are treated as the primary unit of assessment.

1. Scope, naming, and recurrent structure

The uses of MedConceptEval share a concern with whether model outputs correspond to clinically meaningful entities, descriptors, or ontology-linked concepts. They differ, however, in the object being scored, the reference signal, and the mathematical form of the metric. In some papers the target is explicit concept extraction from text; in others it is semantic alignment of generated note sections; in others it is interpretability or retrieval relevance grounded in concepts (Osborne et al., 2014, Kamal et al., 7 Aug 2025, Haque et al., 13 Apr 2026, Wei et al., 16 Jun 2025).

Formulation	Object evaluated	Representative paper
Boundary detection and CUI mapping	Clinical text spans and UMLS concepts	(Osborne et al., 2014)
Descriptor-bank similarity	SOAP note sections	(Kamal et al., 7 Aug 2025)
Aligned/Unaligned/Uncertain semantic verification	Latent concepts in medical VLMs	(Haque et al., 13 Apr 2026)
Approximate matching over UMLS distances	Medical image retrieval rankings	(Wei et al., 16 Jun 2025)

A recurring structural pattern is evident. First, a system produces candidate concepts, whether by NER, ontology mapping, sparse latent units, or descriptor-conditioned text generation. Second, those concepts are compared to a medically grounded reference: gold spans and CUIs, expert descriptor banks, radiology reports, or UMLS-based concept sets. Third, evaluation is aggregated with a task-specific score such as $F_1$, cosine similarity, image-level concept proportions, or NDCG-style ranking measures. This suggests that MedConceptEval is best understood as a concept-centric evaluation design pattern rather than as a single canonical benchmark.

2. Clinical concept recognition and normalization

An early concept-recognition use of this evaluation style appears in work comparing YTEX and MetaMap for ShARe/CLEF eHealth Task 1, where Task 1a is boundary detection and Task 1b is CUI mapping (Osborne et al., 2014). The setup distinguishes strict and relaxed scoring. Under strict scoring, span boundaries must match exactly and the CUI must be correct; under relaxed scoring, any overlap of predicted and gold span counts as a hit and the CUI match is less stringent. Gold annotations include both exact spans and CUIs, with discontinuous spans adjudicated but not scored here.

The pipeline applied identical post-processing to MetaMap and YTEX outputs. Spurious or overly broad concepts were removed through three filters: a hand-curated stop list of 20 high-level UMLS CUIs such as “Disease” C0012634 and “Injury” C0175677; elimination of any concept whose preferred name contains “mouse” or “mice”; and retention only of concepts whose semantic type is in the set required by ShARe/CLEF. Boundary precision on the strict task was further increased by leftward expansion when a recognized span was immediately preceded by one of the tokens LV, MCA, LA, abd, PEA, LE, LGI, ICA, C2, B12, RCA, RUQ, GI, VF, "lower", "chronic" (Osborne et al., 2014).

The reported results show a clear split between exact-span and overlap-tolerant behavior. On strict boundary detection, YTEX achieved $P=0.584$, $R=0.446$, and $F_1=0.505$, whereas MetaMap achieved $P=0.796$, $R=0.487$, and $F_1=0.604$; the detailed report attributes this primarily to a precision gain of approximately $0.212$ for MetaMap. On relaxed boundary detection, YTEX achieved $P=0.942$, $R=0.601$, and $P=0.584$0, whereas MetaMap achieved $P=0.584$1, $P=0.584$2, and $P=0.584$3, giving YTEX a relaxed-task $P=0.584$4 advantage of $P=0.584$5. For relaxed CUI mapping accuracy, the report gives YTEX at approximately $P=0.584$6 and MetaMap at approximately $P=0.584$7, and notes that this is approximately $P=0.584$8 higher although the original paper reported approximately $P=0.584$9 (Osborne et al., 2014).

A related text-mining formulation appears in adverse drug reaction detection from medical forums, where the task is explicitly decomposed into concept identification and concept normalisation (Metke-Jimenez et al., 2015). The corpus consists of 1,250 AskaPatient posts with 9,115 annotated spans, split into 875 training posts and 375 test posts. Identification is evaluated under both strict and relaxed matching using precision, recall, $R=0.446$0, and accuracy, while normalisation is evaluated with strict and relaxed effectiveness measures. In that study, a linear-chain CRF for identification combined with Ontoserver for normalisation achieved the strongest ADR results, with strict ADR identification $R=0.446$1, relaxed ADR identification $R=0.446$2, and strict ADR normalisation effectiveness $R=0.446$3; the study also notes that relaxed normalisation can be misleading when span identification is poor (Metke-Jimenez et al., 2015).

Taken together, these text-oriented variants establish a foundational meaning of MedConceptEval as exacting evaluation over span detection, ontology mapping, and post-processing choices. They also show that “concept evaluation” in clinical NLP is highly sensitive to boundary criteria, vocabulary restrictions, and whether one scores exact or approximate matches.

3. Descriptor-bank evaluation for SOAP note generation

In multimodal SOAP-note generation for dermatology, MedConceptEval is defined as a section-level, concept-based semantic similarity metric for the Subjective, Objective, Assessment, and Plan sections (Kamal et al., 7 Aug 2025, Kamal et al., 12 Jun 2025). Its purpose is to measure how well each generated section aligns with a curated set of expert medical concepts, or “descriptor bank,” for the target disease class. The descriptor banks are condition-specific and were curated from authoritative clinical resources including Mayo Clinic, the National Cancer Institute, the American Cancer Society, and the NHS, with optional synonym expansion by an LLM and manual filtering for clinical validity.

Let $R=0.446$4 be the set of target dermatologic conditions, and for each condition $R=0.446$5 let $R=0.446$6 denote the descriptor bank. Using ClinicalBERT as the embedding function $R=0.446$7, the text of section $R=0.446$8 is embedded as $R=0.446$9, and each descriptor is embedded as $F_1=0.505$0. Pairwise cosine similarities are then computed: $F_1=0.505$1 Two section-level scores are defined: $F_1=0.505$2

$F_1=0.505$3

Across $F_1=0.505$4 representative cases, grand averages are reported as

$F_1=0.505$5

All quantities lie in $F_1=0.505$6, although positive values near $F_1=0.505$7 indicate strong semantic alignment (Kamal et al., 7 Aug 2025).

The practical workflow is modular. Descriptor embeddings are precomputed. For each generated SOAP section, the section text is embedded with the same ClinicalBERT pipeline, cosine similarities to all descriptors in the condition-specific bank are computed, and both the average and maximum similarity are recorded. Reporting can be done per case, per section, or as averages over multiple cases. The metric is thus interpretable at section level: average similarity indicates overall concept coverage, while maximum similarity highlights the single best-matched concept (Kamal et al., 7 Aug 2025).

The detailed SOAP-note study reports MedConceptEval scores for six dermatological conditions—BCC, MEL, SCC, SEK, ACK, and NEV—across four sections, averaged over five cases each (Kamal et al., 12 Jun 2025). For example, melanoma Plan sections achieved $F_1=0.505$8 and $F_1=0.505$9, while NEV Plan sections achieved $P=0.796$0 and $P=0.796$1. Statistical analysis found a significant effect of SOAP section on AvgSim, with $P=0.796$2, but no significant effect of lesion type, with $P=0.796$3 (Kamal et al., 12 Jun 2025).

The strengths and limitations of this formulation are stated explicitly. Its strengths are clinical grounding, interpretability, and section-level resolution. Its limitations are descriptor-bank dependence, embedding sensitivity, false positives due to embedding proximity, and dilution of strong matches by low-relevance concepts in $P=0.796$4. The report also notes that paraphrasing without precise medical terms can reduce similarity even when the text is semantically correct, and that high cosine similarity does not verify factual correctness or syntactic structure (Kamal et al., 7 Aug 2025, Kamal et al., 12 Jun 2025).

4. Semantic verification for latent concepts in medical vision-LLMs

A substantially different MedConceptEval formulation appears in unsupervised concept discovery for medical VLMs (Haque et al., 13 Apr 2026). Here the task is not note generation or span detection but post hoc verification of named latent concepts extracted from a pretrained VLM. The pipeline begins with a held-out medical volume $P=0.796$5, from which a fixed feature vector $P=0.796$6 is extracted using a frozen 3D VLM encoder, Merlin. An over-complete sparse autoencoder is then trained on these features: $P=0.796$7 with $P=0.796$8 and objective

$P=0.796$9

The framework then builds a textual concept vocabulary by querying UMLS for abdomen-related anatomical and pathological terms and expanding synonyms with ChatGPT to obtain a dictionary $R=0.487$0. Each term is encoded with the frozen VLM text encoder, $R=0.487$1. For the $R=0.487$2-th sparse autoencoder unit, the corresponding decoder column $R=0.487$3 is compared to all text embeddings by cosine similarity,

$R=0.487$4

and the concept label is assigned as

$R=0.487$5

For a new test volume, activations are thresholded to obtain $R=0.487$6, mapped to predicted concept names $R=0.487$7, and each predicted concept is judged against the paired radiology report $R=0.487$8 by a frozen medical LLM evaluator, MedGemma (Haque et al., 13 Apr 2026).

The evaluator returns, for each predicted concept $R=0.487$9, a probability distribution

$F_1=0.604$0

followed by a discrete verdict

$F_1=0.604$1

The image-level scores are then

$F_1=0.604$2

with

$F_1=0.604$3

The prompt template instructs the evaluator to classify the report–concept relation as Aligned, Unaligned, or Uncertain and to output probabilities for all three classes (Haque et al., 13 Apr 2026).

Published results show dataset-dependent behavior. On the MerlinPlus dataset, using the top-25 activated concepts per volume yields a median $F_1=0.604$4–$F_1=0.604$5, $F_1=0.604$6, and $F_1=0.604$7 covering the remainder. Expanding to $F_1=0.604$8 lowers alignment modestly and raises uncertainty. On AbdomenAtlas3.0, median $F_1=0.604$9 and $0.212$0 for $0.212$1, with only a small rise in uncertainty at $0.212$2. The three scores are reported to be nearly invariant to MedGemma decoding temperature $0.212$3 (Haque et al., 13 Apr 2026).

The limitations are integral to interpretation. Any concept present in the image but omitted from the report is penalized as Unaligned; the fixed vocabulary based on UMLS plus ChatGPT expansions cannot cover every clinical term; and semantic naming via cosine similarity inherits biases and vision-text misalignments from the pretrained VLM. Accordingly, this MedConceptEval variant measures report-grounded semantic support for discovered concepts, not ground-truth presence in the image independent of reporting practice (Haque et al., 13 Apr 2026).

5. Semantically aware relevance for medical image retrieval

In content-based medical image retrieval, MedConceptEval is instantiated as a semantically aware relevance framework called nn-CUI@K (Wei et al., 16 Jun 2025). Each image $0.212$4 is associated with text $0.212$5, from which a set of UMLS concept identifiers is extracted using an off-the-shelf NER tool such as MedCAT or QuickUMLS. The image is then represented by

$0.212$6

Given two images with concept sets $0.212$7 and $0.212$8, the goal is to define a graded relevance score $0.212$9 that rewards both exact overlap and semantically close non-identical concepts (Wei et al., 16 Jun 2025).

The underlying distance is derived from the UMLS “is_a” hierarchy. If $P=0.942$0 is the directed acyclic graph of CUIs and parent-child relations, then the distance between two concepts is defined as the shortest-path length: $P=0.942$1 Examples in the report include C0042449 (“Veins”) and C0005847 (“Blood vessels”) with $P=0.942$2, and Brain stem (C0006121) versus Head (C0018670) with $P=0.942$3 (Wei et al., 16 Jun 2025).

The relevance score extends Intersection-over-Union by adding approximate matches. Let $P=0.942$4 be the set of CUIs in $P=0.942$5 that have at least one partner in the other set within graph distance $P=0.942$6, excluding exact overlaps already in $P=0.942$7. Then

$P=0.942$8

When $P=0.942$9, this reduces to plain IoU; for $R=0.601$0, it becomes nn-IoU. The paper uses a simple threshold-based greedy matching, while noting that bipartite matching such as the Hungarian algorithm could be substituted to enforce one-to-one matching (Wei et al., 16 Jun 2025).

For ranked retrieval evaluation, $R=0.601$1 is inserted into an NDCG@K protocol. For each query image, relevance values to the retrieved images are used to compute DCG, an ideal ranking sorted by true relevance yields IDCG, and the final score is the dataset average of $R=0.601$2. On ROCOv2, where the dataset contains 60,163 train, 9,948 validation, and 9,928 test images with 3.4 CUIs per image on average, nn-IoU with $R=0.601$3 and $R=0.601$4 consistently outperformed plain IoU. For example, on the Modality$R=0.601$5Organ task, Precision@30 improved from $R=0.601$6 for IoU to $R=0.601$7 for nn-IoU; on the Organ task, Precision@30 improved from $R=0.601$8 to $R=0.601$9 (Wei et al., 16 Jun 2025).

This formulation treats concept similarity as a graded graph property rather than a binary overlap. Its stated advantages are that no manual relevance labels are required, graded similarity is captured, the method is grounded in a comprehensive public knowledge graph, and it is plug-and-play with any CBIR system via NDCG@K. Its limitations are equally explicit: only “is_a” distances are used; shortest-path distance ignores concept depth and specificity; greedy matching may double-count CUIs; and precomputing pairwise distances in UMLS can be expensive (Wei et al., 16 Jun 2025).

6. Related concept-centric benchmarks and antecedents

The broader literature contains several adjacent frameworks that clarify what MedConceptEval is, and is not. MedicalBench formulates medical concept extraction as a verification task over note–concept pairs with sentence-level evidence retrieval, emphasizing implicit concepts rather than only explicit mentions (Yang et al., 5 Apr 2026). Built from MIMIC-IV discharge summaries and human-verified ICD-10 codes, it contains 823 examples, of which 352 are positives and 471 negatives; approximately $P=0.584$00 of positives are explicit and $P=0.584$01 are implicit. The benchmark measures micro-averaged precision, recall, and $P=0.584$02 for concept extraction and macro-averaged sentence recall for evidence retrieval. Benchmarking results remain modest, with Gemini-3-pro-preview reaching $P=0.584$03, Claude-opus-4.5 $P=0.584$04, GPT-5 $P=0.584$05, and evidence retrieval recall around $P=0.584$06 for the best models (Yang et al., 5 Apr 2026). Although this benchmark is not named MedConceptEval in the paper title, it addresses the same core problem of medically faithful concept verification with evidence grounding.

MedConceptsQA is an open-source multiple-choice benchmark for medical concepts across ICD9-CM, ICD10-CM, ICD9-PROC, ICD10-PROC, and ATC (Shoham et al., 2024). It contains 819,832 four-way multiple-choice questions, with difficulty calibrated by graph distance in the underlying code hierarchy. Accuracy is the primary metric, evaluated in zero-shot and 4-shot settings with repeated trials and $P=0.584$07 confidence intervals. The reported findings show that several pre-trained clinical LLMs cluster near random guessing at $P=0.584$08–$P=0.584$09 accuracy, while GPT-4 reaches approximately $P=0.584$10 zero-shot and $P=0.584$11 few-shot (Shoham et al., 2024). The benchmark’s design principles—hierarchy-aware distractors, controlled-vocabulary descriptions, clear random baselines, and explicit error analysis—provide methodological guidance for concept-centric evaluation, even though its task is question answering rather than concept extraction or note scoring.

A further antecedent is the “Concept-Centric Visual Turing Test” framework, which evaluates image classifiers through binary clinically meaningful concepts with Gaussian-process modelling and active querying (Fountoukidou et al., 2019). The framework defines a finite set of binary concepts, exhaustively annotates a held-out validation set, models per-concept performance as a function of model confidence, and chooses queries by maximizing remaining posterior uncertainty. The paper reports that uncertainty-based strategies reduce the number of queries needed for confident evaluation and expose both dataset bias and method bias (Fountoukidou et al., 2019). This is not a MedConceptEval metric in name, but it is concept-centric in the same sense: the object of evaluation is not a single accuracy number but a structured profile over clinically interpretable concepts.

These neighboring benchmarks show that concept-centered evaluation in medicine spans at least four regimes: extraction, verification, generation assessment, and retrieval. A plausible implication is that the diversity of MedConceptEval formulations reflects the breadth of medical AI tasks for which ontology-level or descriptor-level fidelity is considered more informative than lexical or aggregate task metrics alone.

7. Limitations, misconceptions, and interpretive cautions

A common misconception is that MedConceptEval refers to one standardized metric with a fixed formula. The literature does not support that reading. Depending on the paper, MedConceptEval may mean exact and relaxed span scoring with CUI mapping, cosine similarity to descriptor banks, Aligned/Unaligned/Uncertain proportions from an LLM evaluator, or UMLS-graph-aware approximate matching in retrieval (Osborne et al., 2014, Kamal et al., 7 Aug 2025, Haque et al., 13 Apr 2026, Wei et al., 16 Jun 2025). Any comparison across papers therefore requires attention to the scored object, the reference concept set, and the aggregation rule.

A second misconception is that concept-level agreement guarantees factual correctness. The SOAP-note formulation explicitly states that cosine similarity may be inflated by generic medical wording and does not verify factual correctness or syntactic structure (Kamal et al., 12 Jun 2025). The VLM interpretability formulation penalizes concepts omitted from the report as Unaligned even when they may be present in the image, so report sparsity can be confounded with model error (Haque et al., 13 Apr 2026). In ADR normalisation, relaxed effectiveness can appear artificially high when identification quality is poor, making end-to-end interpretation dependent on the interaction between extraction and mapping (Metke-Jimenez et al., 2015).

A third interpretive issue concerns ontology and vocabulary dependence. Descriptor-bank methods depend on the completeness and quality of curated concepts; UMLS-distance methods depend on the choice of relation type and threshold $P=0.584$12; latent concept naming by cosine similarity inherits biases from pretrained vision-text encoders; and traditional concept extraction pipelines depend heavily on semantic-type filters, stop lists, and abbreviation handling (Osborne et al., 2014, Haque et al., 13 Apr 2026, Wei et al., 16 Jun 2025). This suggests that MedConceptEval results are inseparable from the representational substrate—UMLS, SNOMED CT, ATC, expert descriptor banks, or ICD hierarchies—used to define what counts as a concept.

Despite this heterogeneity, the term marks an important shift in evaluation philosophy. Across the cited work, evaluation is pushed away from purely surface-level overlap and toward clinically structured comparison: exact spans versus relaxed spans, code mappings versus free text, descriptor coverage versus generic fluency, report-grounded concept support versus uninterpretable latent activations, and ontology-aware approximate relevance versus binary overlap. In that sense, MedConceptEval names a family of methods for asking whether a medical AI system is getting the concepts right, not merely whether it is producing plausible outputs.