---
title: 'SkinGPT-4: Dermatology Multimodal System'
url: https://www.emergentmind.com/topics/skingpt-4
type: topic
---

# SkinGPT-4: Dermatology Multimodal System

SkinGPT-4 is a dermatology-specific multimodal diagnostic system built by fine-tuning MiniGPT-4 on skin disease images, clinical concepts, and doctors’ notes so that uploaded skin photographs can be analyzed in natural language, assigned likely disease categories, and paired with interactive explanations and treatment recommendations [2304.10691]. In subsequent literature, SkinGPT-4 functions both as a concrete system and as a reference point for a broader class of “SkinGPT-style” dermatology vision-language systems, especially in debates over multimodal reasoning quality, clinical trustworthiness, fairness across skin tones, and the relative merits of end-to-end generation versus modular diagnostic pipelines [2510.00055][2603.26122].

## 1. Origin, clinical motivation, and intended use

The original SkinGPT-4 paper presents the system as a response to three stated obstacles in dermatology: shortage of dermatologists, difficulty of accurately interpreting skin disease images, and the burden of generating understandable reports for patients [2304.10691]. The authors frame the system for teledermatology-like use, especially where patients upload ordinary skin photographs and where specialist access is limited, including rural settings. They also emphasize local deployment as a privacy-preserving alternative to remote proprietary APIs, and later studies continue to describe SkinGPT-4 as a workflow-augmenting tool intended for underserved communities and preliminary image interpretation rather than autonomous care [2304.10691][2510.00055].

SkinGPT-4 is not described as a narrow classifier. The original paper positions it as an interactive assistant that can describe lesion appearance, identify disease categories, explain likely causes, recommend treatment and medication, and continue a follow-up dialogue after the initial image interpretation [2304.10691]. That combination differentiates it, in the authors’ framing, from earlier dermatology AI systems that mainly returned class labels and from text-only large language models that could converse but not directly interpret skin images [2304.10691].

The system’s intended role is explicitly assistive. The original paper states that SkinGPT-4 is not a substitute for doctors, noting that real dermatologic consultation depends on follow-up questioning, additional clinical information, and human empathy [2304.10691]. Later evaluation work on bias and hallucination preserves the same positioning: SkinGPT-4 is clinically promising, but its safest use is as support for clinicians and patients rather than as a fully autonomous dermatology decision engine [2510.00055].

## 2. Model architecture and training design

SkinGPT-4 is built by fine-tuning MiniGPT-4 rather than by introducing a new backbone from scratch [2304.10691]. In the original system description, MiniGPT-4 consists of Vicuna as the language decoder, itself built on LLaMA, together with the same vision encoder as BLIP-2: a ViT backbone, a pre-trained Q-Former, and a linear projection layer that aligns visual features to the language model [2304.10691]. The inference path described in the paper is image upload, ViT feature extraction, Q-Transformer embedding generation, alignment to language space, and Vicuna-based generation of diagnosis, explanation, and recommendations [2304.10691].

The paper’s main methodological contribution is a two-step dermatology-specific fine-tuning procedure [2304.10691]. In the first step, the model is trained on skin images paired with clinical concept descriptions so that it learns to express dermatologic morphology in natural language. In the second step, it is further fine-tuned on skin images paired with doctors’ descriptions and disease categories so that it learns disease-type diagnosis and richer medical explanation [2304.10691]. The authors argue qualitatively that the first stage improves morphology recognition, the second stage improves diagnosis, and the combination is necessary for both visual description and disease identification [2304.10691].

The first-stage dataset is based on SKINCON and contains 3,886 images: 3,230 from Fitzpatrick17k with dense annotations for 48 clinical concepts and 656 from the Diverse Dermatology Images dataset [2304.10691]. The concept vocabulary includes morphology and appearance terms such as vesicle, papule, macule, plaque, nodule, ulcer, crust, scale, telangiectasia, erythema, lichenification, umbilicated, pigmented, and cyst [2304.10691]. The second-stage dataset contains 49,043 image-text pairs, combining 18,856 Dermnet images and 30,187 private in-house image-description pairs [2304.10691]. Across both steps, the paper reports a total training corpus of 52,929 skin disease images [2304.10691].

The reported training settings are concrete but incomplete. The original paper gives max epochs 20, iterations per epoch 5000, warmup steps 5000, batch size 2, learning rate \(1e{-4}\), max text length 160, training time approximately 9 hours, training on 2 \(\times\) NVIDIA V100 (32GB), and inference on 1 \(\times\) NVIDIA V100 (32GB) [2304.10691]. It does not provide explicit mathematical loss definitions, optimizer type, weight decay, image resolution, exact trainable parameter count, or a detailed split strategy [2304.10691]. A later bias-audit paper describes SkinGPT-4 in MiniGPT-4 terms as a vision-language model with a ViT image encoder, Q-Former, alignment layer, and Llama-2-13B text decoder, reinforcing the characterization of SkinGPT-4 as a MiniGPT-4-style dermatology VLM [2510.00055].

## 3. Tasks, outputs, and empirical evaluation

SkinGPT-4 is evaluated in the original paper through a clinical-review protocol rather than standard benchmark classification metrics [2304.10691]. The evaluation set contains 150 real-life cases, with 10 cases per disease group across 15 groups, and certified dermatologists review the outputs generated in response to four prompts: description of the skin disease, additional visual features, cause explanation, and treatment or medication recommendation [2304.10691]. The assessed items include diagnostic correctness or relevance, informativeness, usefulness, physician utility, patient utility, privacy implications of local deployment, and willingness to use the system [2304.10691].

The reported numbers reflect expert agreement rather than exact-label accuracy. The paper states that 78.76% of SkinGPT-4’s diagnoses were evaluated as correct or relevant by certified dermatologists, with 73.13% “strongly agree” and 5.63% “agree” [2304.10691]. Causes and potential treatments were judged 80.63% informative and 83.13% useful, while dermatologists felt the system could help doctors in diagnosis at 85% and help patients understand their disease better at 81.25% [2304.10691]. Agreement on the value of local deployment for privacy protection was 91.88%, and willingness to use SkinGPT-4 was 75% [2304.10691]. The paper also notes response times of seconds, contrasted with online dermatologist consultation waiting times of minutes, although no exact latency values are reported [2304.10691].

The task surface exposed to users is broader than diagnosis alone. The system is designed to support image-based dermatology diagnosis, lesion-feature description, disease-category prediction, explanation of likely cause, treatment and medication recommendation, and interactive follow-up dialogue [2304.10691]. At the same time, the original paper does not describe a formal differential-diagnosis list, calibrated uncertainty estimate, or abstention mechanism [2304.10691].

Later comparative work measures SkinGPT-4 in more conventional classification settings, but these comparisons are reimplementations rather than the original evaluation protocol. DermETAS-SNA, for example, reports an author-implemented SkinGPT-4 baseline with overall accuracy 52.92%, precision 54.57%, recall 46.83%, F1-score 48.51%, and MCC 0.50 on a 23-class DermNet-based task [2512.08998]. That study argues the comparison is meaningful but not fully standardized, since exact split details and exhaustive replication settings are not fully documented [2512.08998]. This indicates that SkinGPT-4’s published evidence base is strongest for clinician-reviewed interactive output quality, whereas later papers often treat it as a benchmark baseline under different task definitions [2304.10691][2512.08998].

## 4. Reliability, bias, and safety concerns

The most important limitation of SkinGPT-4 as a clinical concept is the separation between fluent dermatology-like output and fully trustworthy multimodal reasoning. The original paper itself does not provide formal safety filters, dangerous-treatment suppression rules, escalation policies, adverse-event monitoring, or calibration analysis [2304.10691]. It also does not report subgroup fairness, external validation across institutions, or prospective clinical deployment [2304.10691].

A later mixed-methods audit focuses directly on skin-tone bias in SkinGPT-4 and finds substantial disparities across Fitzpatrick types [2510.00055]. Using SCIN and dermatologist evaluation over 300 cases from six diseases—eczema, allergic-contact dermatitis, psoriasis, tinea, urticaria, and others—the study reports an average demographic parity of 0.10 across Fitzpatrick types and lightest-versus-darkest disparities of 0.10 for diagnostic accuracy, 0.10 for informativity, 0.11 for physician utility, and 0.15 for patient utility [2510.00055]. The paper attributes weaker performance on darker skin to underrepresentation in training data and broader imbalance in dermatology datasets, and it reports hallucinations in artifacts and anatomy at a rate of 17.8% [2510.00055].

The same study also shows that SkinGPT-4’s pretrained backbone can be repurposed into more controlled classifiers with better fairness properties [2510.00055]. By freezing the vision transformer layers and appending an MLP head, with oversampling and fairness-aware analysis, the customized models achieve average F1, precision, and AUROC of 0.75, 0.78, and 0.78 across visually similar disease pairs; one fine-tuned setting reaches accuracy 0.8095, AUPRC 0.8613, AUROC 0.9082, F1 0.8000, precision 0.8083, and recall 0.8095 [2510.00055]. The paper’s practical conclusion is that targeted supervised adaptation of the SkinGPT-4 backbone may be safer and fairer than unrestricted generative diagnosis [2510.00055].

A broader caution comes from work on multimodal GPT-4 vision in medicine. In NEJM Image Challenges, GPT-4V reaches dermatology accuracy of 94.1% versus physician accuracy 79.4%, but hidden flaws remain in 18.8% of dermatology cases where the final answer is correct [2401.08396]. Because SkinGPT-4 belongs to the same class of dermatology-facing image-language systems, this is not direct evidence about SkinGPT-4 itself; however, it strongly indicates that benchmark success can coexist with flawed image comprehension, brittle rationale quality, and “correct answer for the wrong reasons” in skin-image reasoning [2401.08396]. This suggests that high apparent performance for SkinGPT-4-style systems should not be equated with faithful lesion understanding unless morphology description, knowledge recall, and reasoning quality are audited separately.

## 5. Related systems and methodological descendants

Later literature often treats SkinGPT-4 as a baseline, a predecessor, or a reference point for alternative dermatology AI architectures.

| System | Main focus | Relation to SkinGPT-4 |
|---|---|---|
| DermPrompt [2404.17749] | GPT-4V retrieval, reranking, and alignment pipeline | SkinGPT-4-like teledermatology workflow |
| Skin-SOAP [2508.05019] | Weakly supervised SOAP-note generation | Explicitly narrower than SkinGPT-4 |
| SkinGPT-X [2603.26122] | Multi-agent diagnosis with self-evolving memory | Related SkinGPT-style extension |
| DermETAS-SNA [2512.08998] | ETAS ViT + StackNet + RAG explanations | Direct comparator against SkinGPT-4 |
| SkinGPT-R1 [2511.15242] | Adapter-only dermatology reasoning CoT | SkinGPT-style reasoning-centric model |

DermPrompt decomposes dermatology answer generation into retrieval, reranking, and aligner modules, with GPT-4V acting as a candidate proposer rather than a retriever over an external index [2404.17749]. In validation, context-dependent Naive CoT retrieval using image plus brief history achieves retrieval accuracy 0.851063, while the best standard reranker—Medical Guidelines with image and context—reaches Top-2 0.617021 and Top-1 0.531915 [2404.17749]. The paper’s Multi-Agent Conversation framework then improves top-1 accuracy from 0.53333 for MG-GR to 0.73333 on a 15-example subset with 3–5 candidates [2404.17749]. Relative to SkinGPT-4, this shifts emphasis from end-to-end fine-tuning toward staged prompt engineering and critique-based differential diagnosis.

Skin-SOAP moves in a different direction. It is a weakly supervised multimodal framework for generating structured SOAP notes from lesion images and sparse clinical text, using PAD-UFES-20, clinical-caption generation from 26 structured features, retrieval over authoritative skin-cancer sources, and QLoRA fine-tuning of Vision-LLaMA 3.2 [2508.05019]. The paper explicitly contrasts its workflow-oriented documentation scope with SkinGPT-4’s broader dermatological diagnostic reasoning, treating SkinGPT-4 as a multimodal LLM for diagnosis while presenting Skin-SOAP as a note-generation system [2508.05019].

SkinGPT-X is presented not as SkinGPT-4 itself but as a related, more structured system for fine-grained and rare-disease diagnosis [2603.26122]. It decomposes diagnosis into a Vision Agent, a PanDerm-based pre-diagnosis agent, textbook retrieval from an Oxford Handbook–derived Skin Handbook, a self-evolving memory called EvoDerma-Mem, and a Case-Review Agent [2603.26122]. The paper reports a +9.6% accuracy improvement on DDI31 and a +13% weighted F1 gain on Dermnet over the state-of-the-art model, as well as gains on a 498-category Dermnet benchmark and an eight-class rare skin disease dataset of 564 samples [2603.26122]. The paper cites prior SkinGPT-4 work and treats SkinGPT-X as part of the same conceptual lineage, but it does not define a formal version lineage [2603.26122].

DermETAS-SNA reinterprets the design problem as one of specialized visual classification plus grounded explanation [2512.08998]. It uses Dermatology-focused Evolutionary Transformer Architecture Search on SKINCON, 23 one-vs-all disease classifiers on DermNet, a StackNet meta-classifier, and a DERM-RAG explanation layer backed by Gemini 2.5 Pro over roughly 5,000 pages of dermatology textbooks [2512.08998]. On the authors’ 23-class benchmark, StackNet reports F1-score 56.30% versus 48.51% for SkinGPT-4, and the response-generation layer receives 92% agreement from eight licensed medical doctors versus 48.2% for SkinGPT-4 [2512.08998]. The paper interprets the difference as evidence for modular specialization, one-vs-all training, and grounded retrieval [2512.08998].

SkinGPT-R1 shifts the emphasis further toward explicit reasoning traces [2511.15242]. Built on Vision-R1-7B with frozen-backbone adapter-only dual distillation, it uses DermCoT, DermEval, and DermBench to make dermatologic chain of thought explicit, step by step, and verifiable [2511.15242]. On DermBench, across 14 models, SkinGPT-R1 achieves an average score of 4.031 out of 5 and improves the average score over Vision-R1 by about 41% [2511.15242]. The paper cites SkinGPT-4 as prior related work but does not present SkinGPT-R1 as a formally defined version upgrade [2511.15242].

## 6. Historical significance and research trajectory

SkinGPT-4 occupies an early and influential position in dermatology-focused multimodal language modeling because it reframed skin-image AI from a label-only prediction problem into an interactive visual-language task that joins morphology description, disease suggestion, explanation, and treatment-oriented text [2304.10691]. Its most distinctive contribution is not a novel backbone but a domain-adaptation recipe: first align image features to dermatologic clinical concepts, then fine-tune on doctors’ notes so that the model can speak in medically meaningful natural language about skin disease [2304.10691].

Subsequent work suggests that the original SkinGPT-4 formulation opened several diverging research directions. One direction keeps the interactive dermatology-assistant objective but replaces monolithic generation with modular retrieval, reranking, critique, and memory mechanisms [2404.17749][2603.26122]. A second direction constrains the output format toward workflow-specific clinical documentation, as in SOAP-note generation [2508.05019]. A third direction treats SkinGPT-4 mainly as a reusable visual backbone that can be adapted into fairer or more accurate classifiers under stronger task constraints [2510.00055]. A fourth direction makes chain-of-thought reasoning itself the primary object of optimization and evaluation [2511.15242].

This suggests that SkinGPT-4’s long-term importance lies less in any single reported metric than in the design space it helped define. The central questions that later papers repeatedly revisit are whether dermatology systems should be end-to-end or staged, whether explanation should be free-form or grounded, whether accuracy claims are trustworthy without rationale auditing, and whether broad multimodal generation should yield to narrower but safer specialist modules [2404.17749][2401.08396][2510.00055][2512.08998]. In that sense, SkinGPT-4 marks an early transition from conventional dermatology classifiers toward clinically styled multimodal assistants, while later work increasingly treats transparency, fairness, and workflow modularity as first-class requirements rather than secondary refinements.

Source: https://www.emergentmind.com/topics/skingpt-4