DermoGPT: Multimodal Dermatology LLM

• DermoGPT is a family of advanced multimodal large language models designed for dermatological reasoning, integrating image analysis, clinical feature extraction, and fairness-aware learning.
• The platform leverages morphology-anchored datasets, structured instruction trajectories, and multi-stage optimization techniques including supervised fine-tuning and reinforcement learning for high-fidelity diagnostic performance.
• It offers robust clinical applications with comprehensive benchmarking, open-weight implementation, and targeted bias mitigation strategies to enhance dermatological decision support.

DermoGPT is a family of advanced multimodal LLMs (MLLMs) specifically architected for dermatological reasoning, diagnosis, and clinically grounded decision support. Building directly on vision-LLM (VLM) innovations, DermoGPT integrates image analysis, structured clinical feature extraction, chain-of-thought reasoning, and fairness-aware learning within a unified, end-to-end framework. The platform incorporates large-scale, morphology-anchored instruction datasets, rigorous benchmarking protocols, and reinforcement learning objectives tuned for visual-inference consistency. Its open-weight implementations are paired with public benchmarks and datasets, facilitating both research reproducibility and clinical translation (Ru et al., 5 Jan 2026).

1. Morphology-Anchored Data and Benchmarking

DermoGPT’s supervised and reinforcement learning objectives are anchored by the DermoInstruct and DermoBench resources. DermoInstruct comprises over 211,000 unique dermatology images paired with 772,675 expert-validated instruction trajectories, supporting five core task formats: free-text morphological description, structured attribute labeling, chain-of-thought (CoT) explanation, flat (multi-class) diagnosis, and multi-turn hierarchical diagnosis. Ontology induction and multi-stage annotation pipelines ensure patient-level de-duplication, attribute extraction, and explicit mapping to clinical taxonomies. These data formats mirror the progressive stages of specialist diagnostic reasoning: visual description, attribute encoding, stepwise deduction, and final taxonomic classification (Ru et al., 5 Jan 2026).

DermoBench encompasses 11 tasks across four axes: morphology, diagnosis (including out-of-distribution and hierarchical subtasks), clinical reasoning, and skin-type fairness. The open-ended case subset receives iterative annotation and line-by-line revision by board-certified dermatologists, with sanity checks yielding fidelity scores between 3.88–4.60 (Likert 0–5). Human performance baselines are collected for every task. Evaluation encompasses both standard accuracy and LLM-as-judge–based fidelity metrics, generating robust, clinically interpretable performance surfaces (Ru et al., 5 Jan 2026).

2. Model Families and Training Paradigms

2.1 Core Architecture

The baseline DermoGPT model is initialized from a vision-language transformer such as Qwen3-VL-8B-Instruct, featuring a high-capacity vision encoder (“vision tower”), multimodal token merger, and a large autoregressive transformer decoder. All key training and inference pipelines utilize BF16 precision and enable batched attention with FlashAttention (Ru et al., 5 Jan 2026).

2.2 Multi-Stage Optimization

Supervised Fine-Tuning (SFT). DermoGPT utilizes low-rank LoRA adapters (e.g., rank 64, α=64) optimized on multi-task cross-entropy across all instruction formats. The backbone LLM remains frozen during SFT, facilitating data-efficient adaptation (Ru et al., 5 Jan 2026).

Reinforcement Learning—MAVIC. Following SFT, Morphologically-Anchored Visual-Inference-Consistent (MAVIC) RL tuning is employed. MAVIC utilizes Group Relative Policy Optimization (GRPO, group size K=8) with a composite group reward: $$R = R_{\rm acc} + \lambda_{\rm hier}\,S_{\rm hier} + \lambda_{\rm morph}\,g(S_{\rm hier})\,S_{\rm morph} + R_{\rm fmt}$$ where $R_{\rm acc}$ is MCQA accuracy, $S_{\rm hier}$ is Wu–Palmer taxonomy similarity, $S_{\rm morph}$ is a PMI-weighted Tversky score over bottleneck attributes, $g(\cdot)$ is a gating function activating morphology reward only at high semantic similarity, and $R_{\rm fmt}$ enforces output format validity. LoRA adapters are fine-tuned (rank 16, α=32), with backbone weights frozen (Ru et al., 5 Jan 2026).

Test-Time Adaptation—CCT. Confidence-Consistency Test-time Adaptation (CCT) aggregates multiple stochastic decoding rollouts, weighting token choices by both confidence margin and inter-sample consistency. The approach provably suppresses outlier completions on the prediction simplex, offering robustness to skin-type shifts and acquisition noise (Ru et al., 5 Jan 2026).

2.3 Adapter-Based and Dual-Distillation Extensions

Adapter-only fine-tuning and dual-distillation (e.g., SkinGPT-R1) combine a visual distillation head (mapping patch features to dermatology-aware embeddings) and a vocabulary bias adapter in the language decoder, both trained while freezing backbone parameters. Supervised loss schedules balance chain-of-thought (CoT) supervision and vision distillation, leveraging dense, certified CoT datasets such as DermCoT and ablation-validated utility on DermBench and zero-shot clinical benchmarks (Shen et al., 19 Nov 2025).

3. Clinical Feature Extraction and Diagnostic Reasoning

DermoGPT systems ingest clinical images (e.g., dermoscopy, photographic, or multimodal inputs) and extract expert-level visual concepts:

• Color: erythematous, hyperpigmented, hypopigmented, purpuric, etc.
• Morphology: scale, crust, ulceration, papule, plaque, nodule, pustule, telangiectasia.
• Distribution: linear, annular, dermatomal, acral, intertriginous.

Models generate free-text clinical feature descriptions and structured concept bottleneck attributes (e.g., JSON-encoded labels), supporting both direct morphological interpretation and interpretable downstream reasoning. Reasoning proceeds via explicit CoT generation, leading to provisional and hierarchical diagnoses, differential rankings with confidence scores, and tailored explanation or next-step recommendations (Ru et al., 5 Jan 2026, Zhou et al., 2023).

4. Fairness, Skin Tone Bias, and Hallucination Controls

Performance across Fitzpatrick skin types I–VI is addressed via:

• Oversampling of darker-skin examples in mini-batch construction
• Weighted cross-entropy loss and L2 regularization to enforce class and demographic parity
• Adversarial debiasing techniques to suppress skin-tone–predictive subspaces
• Artifact and anatomy hallucination detection by auxiliary tasks or post-processing QA

DermoGPT achieves demographic parity gaps as low as 0.05 on custom-tuned variants (default up to 0.12 in naive settings) and hallucination rates under 5% with best practices, compared to baseline rates of ~17.8% (and up to 22% on Fitzpatrick V–VI). Disparities remain most pronounced in cases with subtle color cues (e.g., eczema, tinea). Continuous monitoring of fairness metrics (DP, EO) and cross-site dermatologist review anchor the validation protocol (Nijjer et al., 28 Sep 2025).

5. Structured Knowledge Representation and Graph-Based Approaches

Advanced DermoGPT systems incorporate structured reasoning via clinical knowledge graphs (KGs), emulating dermatologist logic:

• Nodes represent morphologic features and diagnoses (e.g., for melanoma: ATP-PN, BWV, IR-VS, etc.)
• Directed edges encode conditional probabilities and attribute co-occurrence
• Multi-scale adjacency and adaptive receptive-path convolution (e.g., ARFP over K=3) propagate structure through graph convolutional layers
• Gradient diagnostic strategies (GD-DDW) jointly optimize attribute prediction and diagnosis, enforcing clinical proportionality via data-driven learned weights (Wang et al., 2024)

Graph-aware architectures provide interpretable diagnosis scores, allow model explanations referencing internal KG paths, and facilitate multi-modality feature fusion with explicit weighting parameters (e.g., dermoscopic vs. photographic channels). On the EDRA and ISIC2017/2018 datasets, graph-constrained DermoGPTs achieve AUC values up to 85% and often outperform contemporary CNN or unconstrained MLLM baselines (Wang et al., 2024).

6. Modular Prompt Engineering and Multi-Agent Reasoning

DermoGPT pipelines may combine:

• A retrieval module (e.g., GPT-4V) yielding a candidate set via stepwise feature extraction and naive CoT
• A re-ranker using medical guidelines–grounded CoT or multi-agent conversation (MAC) for differential refinement and evidence-based critique
• A final Aligner module for converging on standardized, clinician-approved report templates

The MAC paradigm orchestrates several specialized “agent” LLMs (e.g., “specialist,” “coordinator,” “admin”), with evidence/critique exchanges yielding top-1 diagnosis accuracies of up to 0.73 versus 0.53 for best single-agent CoT. Performance improvements are statistically significant (e.g., p < 0.05, paired t-tests) across 56-case validation scenarios on the MEDIQA-M3G benchmark (Vashisht et al., 2024).

7. Evaluation, Clinical Implications, and Limitations

DermoGPT achieves state-of-the-art performance on DermoBench, substantially narrowing the human-AI gap (+13.49 points for reasoning, +14.67 for diagnosis, +5.27 for OOD accuracy, and matching humans on fairness). However, a non-trivial gap persists in fine-grained morphological narrative, and current benchmarks lack longitudinal context and active model-expert feedback. All models are subject to dataset bias, evolving clinical guidelines, and regulatory and privacy constraints regarding medical image data. Future work targets full-parameter RL optimization, incorporation of patient history metadata, extension to multi-image reasoning, and dynamic knowledge-graph augmentation (Ru et al., 5 Jan 2026).

---

Selected Papers Referenced:

• "DermoGPT: Open Weights and Open Data for Morphology-Grounded Dermatological Reasoning MLLMs" (Ru et al., 5 Jan 2026)
• "Adapter-Only Dual Distillation for Efficient Dermatology Reasoning" (Shen et al., 19 Nov 2025)
• "Adapting LLMs to Mitigate Skin Tone Biases in Clinical Dermatology Tasks" (Nijjer et al., 28 Sep 2025)
• "GRPO++: Enhancing Dermatological Reasoning under Low Resource Settings" (Swapnil et al., 23 Sep 2025)
• "UMass-BioNLP at MEDIQA-M3G 2024: DermPrompt — A Systematic Exploration of Prompt Engineering with GPT-4V for Dermatological Diagnosis" (Vashisht et al., 2024)
• "AI-Enhanced 7-Point Checklist for Melanoma Detection Using Clinical Knowledge Graphs and Data-Driven Quantification" (Wang et al., 2024)
• "SkinGPT-4: An Interactive Dermatology Diagnostic System with Visual LLM" (Zhou et al., 2023)