Skin-SOAP: Weakly Supervised SOAP Note Generation

• Skin-SOAP is a multimodal framework that integrates dermoscopic images with minimal clinical text to generate structured SOAP notes.
• It employs retrieval-augmented synthesis, structured prompting, and parameter-efficient fine-tuning of Vision-LLaMA to reduce manual annotation needs.
• Evaluations show improved semantic alignment and clinical coherence, highlighting its potential to reduce clinician burnout and documentation errors.

Skin-SOAP is a weakly supervised multimodal framework for generating clinically structured SOAP notes from limited inputs, specifically lesion images and sparse clinical text. It is designed for dermatologic documentation, where physicians record visits using the four canonical SOAP sections—Subjective, Objective, Assessment, and Plan—and where manual note generation is labor-intensive and contributes to clinician burnout. The framework combines dermoscopic imagery, structured metadata, retrieval-augmented medical context, and a vision-language transformer to produce clinically grounded notes while reducing reliance on manual annotations and large annotated corpora (Kamal et al., 7 Aug 2025).

1. Clinical setting and problem formulation

Skin carcinoma is described as the most prevalent form of cancer globally and as accounting for over $8 billion in annual healthcare expenditures. Within this setting, early diagnosis, accurate and timely treatment, and accurate documentation of skin lesions are all presented as operationally important. SOAP notes standardize clinical communication, but producing the four sections manually is reported as labor-intensive, to reduce face-to-face time with patients, and to contribute to clinician burnout (Kamal et al., 7 Aug 2025).

Skin-SOAP is positioned against two resource constraints that are particularly acute in dermatology. First, existing automated-note systems are said either to rely heavily on text dialogues such as doctor–patient transcripts or on large annotated corpora. Second, dermatology requires visual reasoning over dermoscopic images, whereas general-purpose LLMs are described as lacking built-in visual reasoning and as often drifting from clinical facts. The framework therefore integrates lesion images with sparse structured metadata, uses weak supervision in place of full SOAP-note annotation, and grounds generations by retrieving authoritative medical passages (Kamal et al., 7 Aug 2025).

A precursor preprint, "Towards Scalable SOAP Note Generation: A Weakly Supervised Multimodal Framework" (Kamal et al., 12 Jun 2025), presents the same core problem formulation, the same multimodal inputs, and the same evaluation emphasis. This suggests that the August 2025 paper formalized the framework under the explicit name Skin-SOAP after an earlier presentation of the same methodological program (Kamal et al., 12 Jun 2025).

2. Data model, inputs, and preprocessing pipeline

The framework is built on the PAD-UFES-20 dataset. The later description specifies 2,298 dermoscopic images corresponding to 1,641 distinct skin lesions from 1,373 patients. It includes six lesion classes: Melanoma (MEL), Basal Cell Carcinoma (BCC), Squamous Cell Carcinoma (SCC), Actinic Keratosis (ACK), Seborrheic Keratosis (SEK), and Nevus (NEV). Each case includes a CSV file of 26 structured clinical features, including lesion diameter, anatomical location, biopsy status, symptoms such as itching and bleeding, patient demographics, and family history; about 58% are biopsy-confirmed and the remainder rely on expert consensus (Kamal et al., 7 Aug 2025).

The preprocessing pipeline converts these inputs into a form suitable for multimodal generation. Structured metadata are first transformed by GPT-3.5 into a short free-text clinical caption that narrativizes the attributes. The earlier description gives an example of such a caption: “59-year-old male, 8 mm pigmented lesion on the forearm, bleeding intermittently…”. That caption is then used as a retrieval query over a vector-indexed medical knowledge base or ChromaDB vector index constructed from dermatology sources such as NCI, ACS, NHS, and, in the earlier description, STX Skin Cancer Institute. The retrieved passages are concatenated with the original caption to form a context-augmented prompt (Kamal et al., 7 Aug 2025).

This pipeline is central to the framework’s weak-supervision strategy. The retrieved context is intended to reduce hallucinations and to inject lesion-specific clinical knowledge into the synthetic training signal. No human-annotated full SOAP corpus in dermatology is assumed. Instead, pseudo-labels are produced from image-plus-metadata inputs and retrieved text, which allows the framework to scale in the absence of manually authored target notes (Kamal et al., 12 Jun 2025).

3. Architecture and weakly supervised training

Skin-SOAP fine-tunes Vision-LLaMA 3.2 as a multimodal generator. The image encoder is the model’s vision backbone, which ingests the dermoscopic image and produces a dense visual feature map $V \in \mathbb{R}^{h\times w\times d}$, or, in the earlier description, a sequence of visual embeddings. The text encoder is the LLaMA 3.2 LLM, which tokenizes and embeds the clinical caption together with any retrieved passages into $T \in \mathbb{R}^{m\times d}$. Cross-attention layers then fuse visual and textual representations into a unified multimodal representation $F$ (Kamal et al., 7 Aug 2025).

The decoder is an auto-regressive LLM head that generates the four SOAP sections in order. The model uses structured prompting and special section tokens so that output is delimited as “Subjective: …”, “Objective: …”, “Assessment: …”, and “Plan: …”, or, in the more explicit tokenized form, “<S>Subjective: …” through “<P>Plan: …” (Kamal et al., 7 Aug 2025).

Weak supervision is implemented through retrieval-augmented synthesis. GPT-3.5 first summarizes the structured metadata into a preliminary clinical caption. Retrieval-augmented generation then supplies clinically grounded context passages. The combined text and image are fed into pretrained Vision-LLaMA with a template prompting it to output a weakly supervised SOAP note; these synthesized notes become pseudo-labels for downstream training (Kamal et al., 12 Jun 2025).

The training regime has two named components. Parameter-Efficient Fine-Tuning (PEFT) is performed via QLoRA, with low-rank adapters inserted into transformer projections. The earlier description specifies rank $r=8$ and scaling $\alpha=16$, with adapters injected into the query, key, value, and output projections while all other weights are frozen. The low-rank update is written as

$\Delta W = A B,\quad W' = W + \Delta W,$

where $A\in\mathbb R^{d\times r}$, $B\in\mathbb R^{r\times d}$, and $r\ll d$ (Kamal et al., 7 Aug 2025).

Supervised Fine-Tuning (SFT) is then performed on the weakly supervised image-text pairs using token-level cross-entropy: $$\mathcal{L}_{\mathrm{SFT}} = -\sum_{i=1}^{n} y_i \log \hat y_i,$$ where $T \in \mathbb{R}^{m\times d}$0 is the reference token and $T \in \mathbb{R}^{m\times d}$1 is the model probability. A broader multimodal loss decomposition is also discussed,

$T \in \mathbb{R}^{m\times d}$2

although the setup is described as collapsing these into the end-to-end cross-entropy objective $T \in \mathbb{R}^{m\times d}$3 (Kamal et al., 7 Aug 2025).

A notable design detail is that no standard image augmentation such as flips or crops is reported. Instead, augmentation is said to arise from varying retrieved context passages. This places retrieval not only in the role of factual grounding but also in the role of data diversification (Kamal et al., 12 Jun 2025).

4. Inference and note structure

At inference time, the incoming lesion’s structured features are again converted into a clinical caption. Domain passages may optionally be retrieved with that caption. The dermoscopic image, caption, and any retrieved context are then passed to the fine-tuned Vision-LLaMA model, which decodes the SOAP note auto-regressively under section-specific prompting (Kamal et al., 12 Jun 2025).

The output is explicitly structured around the four conventional SOAP segments. In this framework, the structured prompt does not merely format a free-form answer after generation; it acts as a control mechanism that encourages the decoder to emit section-bounded content in the order Subjective, Objective, Assessment, and Plan. Post-processing is minimal: the generated text is split on section headings to form the final structured note (Kamal et al., 12 Jun 2025).

This organization has direct clinical significance because one of the identified failure modes is structural misplacement. The papers report examples in which diagnostic labels such as “BCC” appear in Subjective rather than Assessment. That error mode indicates that section control is both a formatting constraint and a semantic one: the model must learn not just which facts to state, but where in the SOAP ontology they belong (Kamal et al., 7 Aug 2025).

5. Evaluation framework and reported results

Two evaluation metrics are introduced specifically for clinical relevance. MedConceptEval measures semantic alignment between a generated SOAP section and a curated bank of disease-specific descriptors. In the later formulation, descriptor banks $T \in \mathbb{R}^{m\times d}$4 are built for lesion class $T \in \mathbb{R}^{m\times d}$5 and SOAP section $T \in \mathbb{R}^{m\times d}$6 from authoritative sources such as Mayo Clinic, ACS, and NHS. The generated section and each descriptor are embedded with ClinicalBERT, and average and maximum cosine similarity are computed: $T \in \mathbb{R}^{m\times d}$7 The earlier paper reports average similarity across six conditions of 0.76–0.78 for Subjective, 0.77–0.80 for Objective, 0.78–0.82 for Assessment, and 0.77–0.86 for Plan, with Assessment the highest-alignment section overall and Plan the highest for melanoma and nevus (Kamal et al., 12 Jun 2025).

Clinical Coherence Score (CCS) measures semantic consistency between the generated note and the original clinical caption. The section-level definition is

$T \in \mathbb{R}^{m\times d}$8

where $T \in \mathbb{R}^{m\times d}$9 and $F$0 are ClinicalBERT embeddings of the caption and section. The earlier formulation reports note-level CCS as the mean over the four SOAP sections: $F$1 Across three cases, the earlier paper reports averages of 0.9112, 0.8993, and 0.9210. The later paper summarizes this as generated notes averaging approximately 0.91 across three cases, versus approximately 0.44–0.65 for dermatologist-written ground truth, and reports a two-way ANOVA with a highly significant effect of note type $F$2, indicating closer reproduction of caption terminology by Skin-SOAP (Kamal et al., 7 Aug 2025).

The framework is also compared with GPT-4o, DeepSeek Janus Pro, and Claude using ROUGE-1/2/L, METEOR, CHRF++, BERT Score or BERT-F1, and ClinicalBERT-F1. In Case 1, Skin-SOAP reports METEOR 0.2221 versus 0.1745 for GPT-4o, 0.1804 for Janus Pro, and 0.1952 for Claude; CHRF++ 44.91 versus 44.90, 37.25, and 38.92; BERT-F1 0.1223 versus 0.0619, 0.1144, and 0.0117; and ClinicalBERT-F1 0.7750 versus 0.7409, 0.7348, and 0.6974. In Case 2, ROUGE-2 is 0.1250 versus 0.0966, 0.1122, and 0.0671. In Case 3, ClinicalBERT-F1 is 0.7890 versus 0.8098, 0.7573, and 0.7846. The qualitative LLM-as-judge evaluation with Flow-Judge-v0.1 assigns Skin-SOAP a perfect 20/20 across Structure, Readability, Completeness, and Medical Relevance, while GPT-4o, Claude, and Janus Pro score 18–19/20 (Kamal et al., 12 Jun 2025).

Ablation studies further characterize the system. Omitting retrieval-augmented context produces a 4–6 point drop in MedConceptEval and introduces occasional hallucinations. Removing PEFT in favor of full fine-tuning yields comparable final accuracy but requires more than 4 times GPU memory and 3 times training time. Removing image input and using caption-only input reduces CCS by approximately 0.08 and clinical BERT-F1 by 5–7 points, which the paper interprets as evidence for the value of visual features (Kamal et al., 7 Aug 2025).

6. Limitations, error modes, and research trajectory

The principal limitations are tied to data scope, retrieval dependence, and evaluation scale. The framework depends on the quality and coverage of the retrieval corpus, so biased, outdated, incomplete, or otherwise deficient knowledge passages can propagate errors. Evaluation is restricted to a single-source dataset, PAD-UFES-20, and to a small human-annotated test set of only three expert SOAP notes, which constrains large-scale validation and broad generalizability. The papers also note that metadata schemas vary across institutions, which further hinders generalization (Kamal et al., 12 Jun 2025).

Hallucination risk is explicitly acknowledged under ambiguous inputs. Retrieval-augmented generation is reported to mitigate that risk but not eliminate it. Additional error modes include structural misplacement of content across SOAP sections and under-generation of subtle symptom detail in SCC and BCC cases, which is reflected in somewhat lower MedConceptEval scores for those classes (Kamal et al., 7 Aug 2025).

The future work agenda is correspondingly pragmatic. Proposed directions include expansion to multiple, clinically heterogeneous datasets and harmonization of metadata formats; incorporation of human-in-the-loop correction or interactive refinement for critical cases; development of longitudinal benchmarks that capture SOAP note evolution over multiple visits; and integration of uncertainty estimation or retrieval validation to flag potentially unsafe outputs in real-world deployment. The later paper also proposes stronger multimodal pretraining, including contrastive image–text objectives, as a possible route to improved grounding (Kamal et al., 12 Jun 2025).

A common misunderstanding would be to read the higher CCS of generated notes relative to dermatologist-written notes as a blanket claim of superior clinical quality. The paper’s stated interpretation is narrower: Skin-SOAP reproduces caption terminology more closely. That distinction matters because CCS measures semantic alignment to the provided input caption, whereas clinical adequacy also depends on correct synthesis, section placement, and safe handling of ambiguity (Kamal et al., 7 Aug 2025).