Clinical Coherence Score (CCS)

• Clinical Coherence Score (CCS) is a metric that quantifies the semantic alignment between a case-specific input-derived caption and the generated SOAP note sections.
• It employs ClinicalBERT embeddings and cosine similarity to evaluate coherence across Subjective, Objective, Assessment, and Plan sections, ensuring the note reflects provided clinical data.
• Designed for weakly supervised settings, CCS assesses note fidelity without relying on gold references, highlighting potential content inconsistencies.

Searching arXiv for the cited paper and the ClinicalBERT reference to ground the article in current records. Clinical Coherence Score (CCS) is a metric introduced to evaluate whether a generated SOAP note is consistent with the clinical inputs provided to a model, particularly in weakly supervised multimodal dermatology documentation settings. In "Towards Scalable SOAP Note Generation: A Weakly Supervised Multimodal Framework" (Kamal et al., 12 Jun 2025), CCS is defined as a measure of semantic consistency between a case-specific clinical caption, synthesized from lesion images and structured clinical features, and each section of a generated SOAP note. Unlike reference-based metrics, CCS is designed to assess faithfulness to the input features themselves, thereby providing a grounding-oriented evaluation when gold reference notes are unavailable or limited.

1. Definition and motivation

CCS quantifies the semantic alignment between the clinical caption derived from the input features for a case and the text of the Subjective, Objective, Assessment, and Plan sections of the generated SOAP note. The metric addresses the question: “Does the note reflect what was actually provided about the patient and lesion?” (Kamal et al., 12 Jun 2025)

The underlying intuition is that if a model faithfully grounds its output in the given inputs, each SOAP section should semantically align with the input-derived caption. A high CCS therefore indicates that the generated text is coherent with the facts, symptoms, and descriptors present in the inputs, whereas a lower CCS suggests drift or hallucination.

The metric is motivated by the specific setting of dermatology SOAP generation from limited inputs, namely an image plus sparse structured features. In that setting, CCS complements concept-based and reference-based evaluations by checking coherence with the inputs directly, and it does so without requiring a gold reference note. This suggests that CCS is particularly suited to data-scarce or weakly supervised regimes in which conventional note-to-note comparison is insufficient.

2. Formal specification

CCS is computed using ClinicalBERT-based contextual embeddings and cosine similarity (Kamal et al., 12 Jun 2025). Let $c$ denote the clinical caption derived from the input features for a case, and let $n_S, n_O, n_A, n_P$ denote the text content for the Subjective, Objective, Assessment, and Plan sections of the generated SOAP note.

The embedding function is defined as follows:

$f:\text{text} \rightarrow \mathbb{R}^d$

where $f(\cdot)$ is the ClinicalBERT encoder mapping text to a $d$-dimensional embedding.

Cosine similarity between two vectors $u, v \in \mathbb{R}^d$ is given by:

$\mathrm{sim}(u,v)=\frac{u^\top v}{\|u\|_2\|v\|_2}$

The section-level embeddings are:

$e_c=f(c)$

$$e_S=f(n_S), \quad e_O=f(n_O), \quad e_A=f(n_A), \quad e_P=f(n_P)$$

The corresponding section-level coherence scores are:

$s_S=\mathrm{sim}(e_c,e_S)$

$n_S, n_O, n_A, n_P$0

$n_S, n_O, n_A, n_P$1

$n_S, n_O, n_A, n_P$2

The per-note CCS is then the unweighted arithmetic mean of the four section-level cosine similarities:

$n_S, n_O, n_A, n_P$3

For dataset-level summaries, the metric is averaged over cases:

$n_S, n_O, n_A, n_P$4

No weighting scheme, normalization beyond cosine, or thresholding is used in the reported formulation. The paper states explicitly that the four section-level cosine similarities are simply averaged.

3. Computational procedure

The CCS computation pipeline begins from three inputs: a lesion image, structured clinical features, and a generated SOAP note with Subjective, Objective, Assessment, and Plan sections (Kamal et al., 12 Jun 2025). The procedure consists of four stages.

First, structured features and image-derived descriptors available to the model are converted into a clinical caption $n_S, n_O, n_A, n_P$5 that succinctly summarizes the input case. This caption serves as the anchor for CCS and is also used throughout the broader framework during pre-training and inference.

Second, ClinicalBERT is used to encode both the caption and each SOAP section into contextual embeddings in a shared clinical semantic space.

Third, cosine similarity is computed between the caption embedding and the embedding of each SOAP section. This yields section-level alignment scores for Subjective, Objective, Assessment, and Plan.

Fourth, the final CCS is obtained by averaging the four section-level similarities.

The paper provides concise pseudocode for this procedure:

$n_S, n_O, n_A, n_P$9

The implementation burden is correspondingly low. For each case, five texts are embedded and four cosine similarities are computed, then averaged. The paper characterizes CCS as lightweight and notes that it requires only ClinicalBERT for scoring, even though the generation model in the overall framework is Vision-LLaMA 3.2 fine-tuned via QLoRA (PEFT) and SFT for the SOAP task.

4. Relation to other evaluation metrics

CCS is positioned as complementary to both concept-based and reference-based evaluation methods (Kamal et al., 12 Jun 2025). Its distinctive role is to test whether a generated note is faithful to the actual case inputs rather than whether it merely resembles a reference note or uses condition-appropriate terminology.

Metric family	Primary comparison target	Core evaluation question
CCS	Case-specific caption derived from actual inputs	Does the note reflect the given case?
MedConceptEval	Curated disease-specific descriptor banks	Does the note use clinically appropriate concepts for the presumed condition?
BLEU, ROUGE, CHRF++, BERTScore, ClinicalBERT F1	Human-authored reference note	How much lexical or semantic overlap exists with a reference?

MedConceptEval assesses alignment to curated disease-specific descriptor banks by encoding each section and comparing it to those expert concept sets. CCS instead evaluates alignment to the case-specific caption derived from the actual inputs, namely image and structured features. The distinction is substantive: CCS tests case faithfulness, whereas MedConceptEval tests concept appropriateness for the presumed condition.

Standard metrics such as BLEU, ROUGE, CHRF++, BERTScore, and ClinicalBERT F1 compare a candidate note to a human-authored reference and thereby assess lexical or semantic overlap. CCS does not require a reference at all. In the weakly supervised setting described in the paper, this makes it a practical grounding check when annotated target notes are limited.

The section-wise design of CCS also highlights where unsupported content may enter the note. The paper explicitly notes that this can expose sections that add unsupported diagnoses or plans not inferable from the provided inputs.

5. Empirical characterization

The empirical validation reported for CCS is limited but concrete. The paper states that CCS was computed on three lesion-image cases with generated SOAP notes, using ClinicalBERT embeddings and cosine similarity per section, followed by averaging across sections (Kamal et al., 12 Jun 2025).

The reported results show that Case 3 achieved the highest overall coherence. Subjective and Assessment consistently showed strong alignment, while Plan was slightly lower. The interpretation offered in the paper is that plan-specific details are usually not present in the caption and are instead generated by the model.

For Case 3, the reported section-level values are:

Section	Score
Subjective	0.9318
Objective	0.9334
Assessment	0.9208
Plan	0.8981
Average	0.9210

The corresponding computation is:

$n_S, n_O, n_A, n_P$6

This example illustrates that Subjective and Objective are tightly grounded in the caption, with Assessment close behind, while Plan trails slightly because plans typically introduce management actions not necessarily present in the caption.

The paper does not report Pearson or Spearman correlation between CCS and human expert ratings, nor does it report inter-rater reliability for CCS-related judgments. It further states that although qualitative LLM-as-a-Judge evaluation using Flow-Judge-v0.1 was conducted, CCS-specific correlation analyses were not performed. Formal ablations, such as removing image inputs or perturbing captions, were also not reported. The only sensitivity-related observation described is that Plan was lower, which is consistent with its tendency to contain content extending beyond the input features.

6. Limitations, implementation constraints, and broader applicability

The principal limitation identified for CCS is its dependence on the fidelity of the synthesized caption (Kamal et al., 12 Jun 2025). If the caption omits crucial features or includes errors, the resulting score may misrepresent the true coherence between inputs and note. This dependence is structural, because the caption is the sole semantic anchor against which all four SOAP sections are compared.

A second limitation is encoder dependence. CCS uses ClinicalBERT, and its embedding space and vocabulary coverage affect sensitivity. Domain mismatch or out-of-vocabulary terms can reduce alignment scores. The metric is therefore not independent of representation quality.

A third limitation is section imbalance. Some SOAP sections, particularly Plan, may legitimately contain content not derivable from raw inputs, such as patient education or follow-up. Lower Plan scores should therefore be interpreted cautiously rather than treated as automatic evidence of hallucination.

A fourth limitation arises from image–text alignment errors upstream. If the pipeline that produces caption content overemphasizes or underemphasizes image-derived features, CCS may penalize otherwise appropriate reasoning. This suggests that CCS measures coherence with the captioned representation of the inputs, not directly with the raw multimodal inputs themselves.

The implementation details are intentionally minimal. CCS uses default ClinicalBERT encoding and cosine similarity, with no special hyperparameters beyond standard text pre-processing. No weighting or normalization beyond cosine is applied, and the paper does not release CCS code. The provided pseudocode is described as sufficient for a straightforward implementation with common transformer libraries.

The paper also states that CCS is conceptually modality- and specialty-agnostic. For cardiology, radiology, or primary care SOAP notes, a case-specific caption can be derived from EHR fields, labs, or imaging summaries and then compared to generated sections via a domain-appropriate clinical encoder such as BioClinicalBERT or PubMedBERT. If the documentation schema includes additional sections, CCS can be generalized accordingly:

$n_S, n_O, n_A, n_P$7

where $n_S, n_O, n_A, n_P$8 is the number of sections in the documentation schema.

A plausible implication is that the metric’s generalization depends less on dermatology-specific content than on the availability of a reliable case summary and an encoder suitable for the target domain. Within the source paper, however, this broader applicability is presented conceptually rather than through direct experimental validation.