SkinGPT-4: Dermatology Multimodal System
- SkinGPT-4 is a dermatology-specific multimodal diagnostic system that fine-tunes MiniGPT-4 to interpret skin images and clinical data.
- It integrates a two-step training process using clinical concept descriptions and doctors’ notes to generate interactive disease explanations and treatment recommendations.
- Clinical evaluations show strong potential for teledermatology support, while highlighting challenges in fairness, safety, and bias across diverse skin tones.
SkinGPT-4 is a dermatology-specific multimodal diagnostic system built by fine-tuning MiniGPT-4 on skin disease images, clinical concepts, and doctors’ notes so that uploaded skin photographs can be analyzed in natural language, assigned likely disease categories, and paired with interactive explanations and treatment recommendations (Zhou et al., 2023). In subsequent literature, SkinGPT-4 functions both as a concrete system and as a reference point for a broader class of “SkinGPT-style” dermatology vision-language systems, especially in debates over multimodal reasoning quality, clinical trustworthiness, fairness across skin tones, and the relative merits of end-to-end generation versus modular diagnostic pipelines (Nijjer et al., 28 Sep 2025, Chen et al., 27 Mar 2026).
1. Origin, clinical motivation, and intended use
The original SkinGPT-4 paper presents the system as a response to three stated obstacles in dermatology: shortage of dermatologists, difficulty of accurately interpreting skin disease images, and the burden of generating understandable reports for patients (Zhou et al., 2023). The authors frame the system for teledermatology-like use, especially where patients upload ordinary skin photographs and where specialist access is limited, including rural settings. They also emphasize local deployment as a privacy-preserving alternative to remote proprietary APIs, and later studies continue to describe SkinGPT-4 as a workflow-augmenting tool intended for underserved communities and preliminary image interpretation rather than autonomous care (Zhou et al., 2023, Nijjer et al., 28 Sep 2025).
SkinGPT-4 is not described as a narrow classifier. The original paper positions it as an interactive assistant that can describe lesion appearance, identify disease categories, explain likely causes, recommend treatment and medication, and continue a follow-up dialogue after the initial image interpretation (Zhou et al., 2023). That combination differentiates it, in the authors’ framing, from earlier dermatology AI systems that mainly returned class labels and from text-only LLMs that could converse but not directly interpret skin images (Zhou et al., 2023).
The system’s intended role is explicitly assistive. The original paper states that SkinGPT-4 is not a substitute for doctors, noting that real dermatologic consultation depends on follow-up questioning, additional clinical information, and human empathy (Zhou et al., 2023). Later evaluation work on bias and hallucination preserves the same positioning: SkinGPT-4 is clinically promising, but its safest use is as support for clinicians and patients rather than as a fully autonomous dermatology decision engine (Nijjer et al., 28 Sep 2025).
2. Model architecture and training design
SkinGPT-4 is built by fine-tuning MiniGPT-4 rather than by introducing a new backbone from scratch (Zhou et al., 2023). In the original system description, MiniGPT-4 consists of Vicuna as the language decoder, itself built on LLaMA, together with the same vision encoder as BLIP-2: a ViT backbone, a pre-trained Q-Former, and a linear projection layer that aligns visual features to the LLM (Zhou et al., 2023). The inference path described in the paper is image upload, ViT feature extraction, Q-Transformer embedding generation, alignment to language space, and Vicuna-based generation of diagnosis, explanation, and recommendations (Zhou et al., 2023).
The paper’s main methodological contribution is a two-step dermatology-specific fine-tuning procedure (Zhou et al., 2023). In the first step, the model is trained on skin images paired with clinical concept descriptions so that it learns to express dermatologic morphology in natural language. In the second step, it is further fine-tuned on skin images paired with doctors’ descriptions and disease categories so that it learns disease-type diagnosis and richer medical explanation (Zhou et al., 2023). The authors argue qualitatively that the first stage improves morphology recognition, the second stage improves diagnosis, and the combination is necessary for both visual description and disease identification (Zhou et al., 2023).
The first-stage dataset is based on SKINCON and contains 3,886 images: 3,230 from Fitzpatrick17k with dense annotations for 48 clinical concepts and 656 from the Diverse Dermatology Images dataset (Zhou et al., 2023). The concept vocabulary includes morphology and appearance terms such as vesicle, papule, macule, plaque, nodule, ulcer, crust, scale, telangiectasia, erythema, lichenification, umbilicated, pigmented, and cyst (Zhou et al., 2023). The second-stage dataset contains 49,043 image-text pairs, combining 18,856 Dermnet images and 30,187 private in-house image-description pairs (Zhou et al., 2023). Across both steps, the paper reports a total training corpus of 52,929 skin disease images (Zhou et al., 2023).
The reported training settings are concrete but incomplete. The original paper gives max epochs 20, iterations per epoch 5000, warmup steps 5000, batch size 2, learning rate , max text length 160, training time approximately 9 hours, training on 2 NVIDIA V100 (32GB), and inference on 1 NVIDIA V100 (32GB) (Zhou et al., 2023). It does not provide explicit mathematical loss definitions, optimizer type, weight decay, image resolution, exact trainable parameter count, or a detailed split strategy (Zhou et al., 2023). A later bias-audit paper describes SkinGPT-4 in MiniGPT-4 terms as a vision-LLM with a ViT image encoder, Q-Former, alignment layer, and Llama-2-13B text decoder, reinforcing the characterization of SkinGPT-4 as a MiniGPT-4-style dermatology VLM (Nijjer et al., 28 Sep 2025).
3. Tasks, outputs, and empirical evaluation
SkinGPT-4 is evaluated in the original paper through a clinical-review protocol rather than standard benchmark classification metrics (Zhou et al., 2023). The evaluation set contains 150 real-life cases, with 10 cases per disease group across 15 groups, and certified dermatologists review the outputs generated in response to four prompts: description of the skin disease, additional visual features, cause explanation, and treatment or medication recommendation (Zhou et al., 2023). The assessed items include diagnostic correctness or relevance, informativeness, usefulness, physician utility, patient utility, privacy implications of local deployment, and willingness to use the system (Zhou et al., 2023).
The reported numbers reflect expert agreement rather than exact-label accuracy. The paper states that 78.76% of SkinGPT-4’s diagnoses were evaluated as correct or relevant by certified dermatologists, with 73.13% “strongly agree” and 5.63% “agree” (Zhou et al., 2023). Causes and potential treatments were judged 80.63% informative and 83.13% useful, while dermatologists felt the system could help doctors in diagnosis at 85% and help patients understand their disease better at 81.25% (Zhou et al., 2023). Agreement on the value of local deployment for privacy protection was 91.88%, and willingness to use SkinGPT-4 was 75% (Zhou et al., 2023). The paper also notes response times of seconds, contrasted with online dermatologist consultation waiting times of minutes, although no exact latency values are reported (Zhou et al., 2023).
The task surface exposed to users is broader than diagnosis alone. The system is designed to support image-based dermatology diagnosis, lesion-feature description, disease-category prediction, explanation of likely cause, treatment and medication recommendation, and interactive follow-up dialogue (Zhou et al., 2023). At the same time, the original paper does not describe a formal differential-diagnosis list, calibrated uncertainty estimate, or abstention mechanism (Zhou et al., 2023).
Later comparative work measures SkinGPT-4 in more conventional classification settings, but these comparisons are reimplementations rather than the original evaluation protocol. DermETAS-SNA, for example, reports an author-implemented SkinGPT-4 baseline with overall accuracy 52.92%, precision 54.57%, recall 46.83%, F1-score 48.51%, and MCC 0.50 on a 23-class DermNet-based task (Oruganty et al., 9 Dec 2025). That study argues the comparison is meaningful but not fully standardized, since exact split details and exhaustive replication settings are not fully documented (Oruganty et al., 9 Dec 2025). This indicates that SkinGPT-4’s published evidence base is strongest for clinician-reviewed interactive output quality, whereas later papers often treat it as a benchmark baseline under different task definitions (Zhou et al., 2023, Oruganty et al., 9 Dec 2025).
4. Reliability, bias, and safety concerns
The most important limitation of SkinGPT-4 as a clinical concept is the separation between fluent dermatology-like output and fully trustworthy multimodal reasoning. The original paper itself does not provide formal safety filters, dangerous-treatment suppression rules, escalation policies, adverse-event monitoring, or calibration analysis (Zhou et al., 2023). It also does not report subgroup fairness, external validation across institutions, or prospective clinical deployment (Zhou et al., 2023).
A later mixed-methods audit focuses directly on skin-tone bias in SkinGPT-4 and finds substantial disparities across Fitzpatrick types (Nijjer et al., 28 Sep 2025). Using SCIN and dermatologist evaluation over 300 cases from six diseases—eczema, allergic-contact dermatitis, psoriasis, tinea, urticaria, and others—the study reports an average demographic parity of 0.10 across Fitzpatrick types and lightest-versus-darkest disparities of 0.10 for diagnostic accuracy, 0.10 for informativity, 0.11 for physician utility, and 0.15 for patient utility (Nijjer et al., 28 Sep 2025). The paper attributes weaker performance on darker skin to underrepresentation in training data and broader imbalance in dermatology datasets, and it reports hallucinations in artifacts and anatomy at a rate of 17.8% (Nijjer et al., 28 Sep 2025).
The same study also shows that SkinGPT-4’s pretrained backbone can be repurposed into more controlled classifiers with better fairness properties (Nijjer et al., 28 Sep 2025). By freezing the vision transformer layers and appending an MLP head, with oversampling and fairness-aware analysis, the customized models achieve average F1, precision, and AUROC of 0.75, 0.78, and 0.78 across visually similar disease pairs; one fine-tuned setting reaches accuracy 0.8095, AUPRC 0.8613, AUROC 0.9082, F1 0.8000, precision 0.8083, and recall 0.8095 (Nijjer et al., 28 Sep 2025). The paper’s practical conclusion is that targeted supervised adaptation of the SkinGPT-4 backbone may be safer and fairer than unrestricted generative diagnosis (Nijjer et al., 28 Sep 2025).
A broader caution comes from work on multimodal GPT-4 vision in medicine. In NEJM Image Challenges, GPT-4V reaches dermatology accuracy of 94.1% versus physician accuracy 79.4%, but hidden flaws remain in 18.8% of dermatology cases where the final answer is correct (Jin et al., 2024). Because SkinGPT-4 belongs to the same class of dermatology-facing image-language systems, this is not direct evidence about SkinGPT-4 itself; however, it strongly indicates that benchmark success can coexist with flawed image comprehension, brittle rationale quality, and “correct answer for the wrong reasons” in skin-image reasoning (Jin et al., 2024). This suggests that high apparent performance for SkinGPT-4-style systems should not be equated with faithful lesion understanding unless morphology description, knowledge recall, and reasoning quality are audited separately.
5. Related systems and methodological descendants
Later literature often treats SkinGPT-4 as a baseline, a predecessor, or a reference point for alternative dermatology AI architectures.
| System | Main focus | Relation to SkinGPT-4 |
|---|---|---|
| DermPrompt (Vashisht et al., 2024) | GPT-4V retrieval, reranking, and alignment pipeline | SkinGPT-4-like teledermatology workflow |
| Skin-SOAP (Kamal et al., 7 Aug 2025) | Weakly supervised SOAP-note generation | Explicitly narrower than SkinGPT-4 |
| SkinGPT-X (Chen et al., 27 Mar 2026) | Multi-agent diagnosis with self-evolving memory | Related SkinGPT-style extension |
| DermETAS-SNA (Oruganty et al., 9 Dec 2025) | ETAS ViT + StackNet + RAG explanations | Direct comparator against SkinGPT-4 |
| SkinGPT-R1 (Shen et al., 19 Nov 2025) | Adapter-only dermatology reasoning CoT | SkinGPT-style reasoning-centric model |
DermPrompt decomposes dermatology answer generation into retrieval, reranking, and aligner modules, with GPT-4V acting as a candidate proposer rather than a retriever over an external index (Vashisht et al., 2024). In validation, context-dependent Naive CoT retrieval using image plus brief history achieves retrieval accuracy 0.851063, while the best standard reranker—Medical Guidelines with image and context—reaches Top-2 0.617021 and Top-1 0.531915 (Vashisht et al., 2024). The paper’s Multi-Agent Conversation framework then improves top-1 accuracy from 0.53333 for MG-GR to 0.73333 on a 15-example subset with 3–5 candidates (Vashisht et al., 2024). Relative to SkinGPT-4, this shifts emphasis from end-to-end fine-tuning toward staged prompt engineering and critique-based differential diagnosis.
Skin-SOAP moves in a different direction. It is a weakly supervised multimodal framework for generating structured SOAP notes from lesion images and sparse clinical text, using PAD-UFES-20, clinical-caption generation from 26 structured features, retrieval over authoritative skin-cancer sources, and QLoRA fine-tuning of Vision-LLaMA 3.2 (Kamal et al., 7 Aug 2025). The paper explicitly contrasts its workflow-oriented documentation scope with SkinGPT-4’s broader dermatological diagnostic reasoning, treating SkinGPT-4 as a multimodal LLM for diagnosis while presenting Skin-SOAP as a note-generation system (Kamal et al., 7 Aug 2025).
SkinGPT-X is presented not as SkinGPT-4 itself but as a related, more structured system for fine-grained and rare-disease diagnosis (Chen et al., 27 Mar 2026). It decomposes diagnosis into a Vision Agent, a PanDerm-based pre-diagnosis agent, textbook retrieval from an Oxford Handbook–derived Skin Handbook, a self-evolving memory called EvoDerma-Mem, and a Case-Review Agent (Chen et al., 27 Mar 2026). The paper reports a +9.6% accuracy improvement on DDI31 and a +13% weighted F1 gain on Dermnet over the state-of-the-art model, as well as gains on a 498-category Dermnet benchmark and an eight-class rare skin disease dataset of 564 samples (Chen et al., 27 Mar 2026). The paper cites prior SkinGPT-4 work and treats SkinGPT-X as part of the same conceptual lineage, but it does not define a formal version lineage (Chen et al., 27 Mar 2026).
DermETAS-SNA reinterprets the design problem as one of specialized visual classification plus grounded explanation (Oruganty et al., 9 Dec 2025). It uses Dermatology-focused Evolutionary Transformer Architecture Search on SKINCON, 23 one-vs-all disease classifiers on DermNet, a StackNet meta-classifier, and a DERM-RAG explanation layer backed by Gemini 2.5 Pro over roughly 5,000 pages of dermatology textbooks (Oruganty et al., 9 Dec 2025). On the authors’ 23-class benchmark, StackNet reports F1-score 56.30% versus 48.51% for SkinGPT-4, and the response-generation layer receives 92% agreement from eight licensed medical doctors versus 48.2% for SkinGPT-4 (Oruganty et al., 9 Dec 2025). The paper interprets the difference as evidence for modular specialization, one-vs-all training, and grounded retrieval (Oruganty et al., 9 Dec 2025).
SkinGPT-R1 shifts the emphasis further toward explicit reasoning traces (Shen et al., 19 Nov 2025). Built on Vision-R1-7B with frozen-backbone adapter-only dual distillation, it uses DermCoT, DermEval, and DermBench to make dermatologic chain of thought explicit, step by step, and verifiable (Shen et al., 19 Nov 2025). On DermBench, across 14 models, SkinGPT-R1 achieves an average score of 4.031 out of 5 and improves the average score over Vision-R1 by about 41% (Shen et al., 19 Nov 2025). The paper cites SkinGPT-4 as prior related work but does not present SkinGPT-R1 as a formally defined version upgrade (Shen et al., 19 Nov 2025).
6. Historical significance and research trajectory
SkinGPT-4 occupies an early and influential position in dermatology-focused multimodal language modeling because it reframed skin-image AI from a label-only prediction problem into an interactive visual-language task that joins morphology description, disease suggestion, explanation, and treatment-oriented text (Zhou et al., 2023). Its most distinctive contribution is not a novel backbone but a domain-adaptation recipe: first align image features to dermatologic clinical concepts, then fine-tune on doctors’ notes so that the model can speak in medically meaningful natural language about skin disease (Zhou et al., 2023).
Subsequent work suggests that the original SkinGPT-4 formulation opened several diverging research directions. One direction keeps the interactive dermatology-assistant objective but replaces monolithic generation with modular retrieval, reranking, critique, and memory mechanisms (Vashisht et al., 2024, Chen et al., 27 Mar 2026). A second direction constrains the output format toward workflow-specific clinical documentation, as in SOAP-note generation (Kamal et al., 7 Aug 2025). A third direction treats SkinGPT-4 mainly as a reusable visual backbone that can be adapted into fairer or more accurate classifiers under stronger task constraints (Nijjer et al., 28 Sep 2025). A fourth direction makes chain-of-thought reasoning itself the primary object of optimization and evaluation (Shen et al., 19 Nov 2025).
This suggests that SkinGPT-4’s long-term importance lies less in any single reported metric than in the design space it helped define. The central questions that later papers repeatedly revisit are whether dermatology systems should be end-to-end or staged, whether explanation should be free-form or grounded, whether accuracy claims are trustworthy without rationale auditing, and whether broad multimodal generation should yield to narrower but safer specialist modules (Vashisht et al., 2024, Jin et al., 2024, Nijjer et al., 28 Sep 2025, Oruganty et al., 9 Dec 2025). In that sense, SkinGPT-4 marks an early transition from conventional dermatology classifiers toward clinically styled multimodal assistants, while later work increasingly treats transparency, fairness, and workflow modularity as first-class requirements rather than secondary refinements.
