SkinGPT-X: Multi-agent Diagnostic System

• SkinGPT-X is a multimodal, memory-augmented multi-agent system that integrates visual, textual, and historical case data for dermatological diagnosis.
• It decomposes the diagnostic workflow into specialized agents using retrieval-augmented generation and explicit case memory to refine clinical guidelines.
• Experimental evaluations show significant gains in accuracy and interpretability, addressing challenges in rare skin disease diagnosis and fine-grained classification.

Searching arXiv for the specified papers and closely related work to ground the article in current literature. Tool call: arxiv_search(query="(Chen et al., 27 Mar 2026) OR \"SkinGPT-X\" dermatological diagnosis multi-agent", max_results=5, sort_by="relevance") SkinGPT-X is a multimodal collaborative multi-agent system for dermatological diagnosis integrated with a self-evolving dermatological memory mechanism. It is designed to simulate the diagnostic workflow of dermatologists, decompose clinical reasoning into modular agents, maintain an explicit memory of prior cases, and continually refine its clinical knowledge. Within the reported formulation, SkinGPT-X is positioned against two limitations: monolithic LLMs frequently struggle with fine-grained, large-scale multi-class diagnostic tasks and rare skin disease diagnosis owing to training data sparsity, and existing multi-agent systems are primarily concentrated on Visual Question Answering and conversational tasks while relying on static knowledge bases that restrict adaptability in complex real-world clinical settings (Chen et al., 27 Mar 2026).

1. Problem setting and antecedents

SkinGPT-X addresses dermatological diagnosis as a multimodal reasoning problem in which visual evidence, textbook criteria, historical cases, and iteratively updated guidelines are jointly audited before final determination. The central design premise is that transparent and trustworthy diagnostics require explicit decomposition of the diagnostic process rather than a single monolithic generator (Chen et al., 27 Mar 2026).

A relevant antecedent is SkinGPT-4, described as an interactive dermatology diagnostic system powered by a visual LLM. SkinGPT-4 leverages a fine-tuned version of MiniGPT-4 trained on 52,929 publicly available and proprietary images together with clinical concepts and doctors’ notes, and its roadmap toward SkinGPT-X includes integrating structured clinical data, adopting next-generation open LLMs, implementing federated or on-device continual learning, and expanding prospective validation (Zhou et al., 2023). This suggests a transition from an interactive V-LLM centered on image-to-report generation toward a collaborative, memory-based architecture with explicit auditability.

Within this progression, SkinGPT-X is distinguished by two additions that are structurally central rather than auxiliary: a retrieval-augmented diagnostic workflow and a self-evolving memory. The stated objective is not only classification performance, but also interpretability, traceability, and management of complex and rare dermatological cases (Chen et al., 27 Mar 2026).

2. System architecture and diagnostic workflow

SkinGPT-X is built around four specialized agents plus a retrieval-augmented generation module and a self-evolving memory. Its high-level workflow simulates how a dermatologist gathers visual clues, consults textbooks, reviews past cases, and synthesizes a final report (Chen et al., 27 Mar 2026).

Component	Input	Task / implementation
Vision Agent	raw clinical image	extract objective morphological findings; implemented with Qwen3-VL, constrained to factual image description
Pre-Diagnosis Agent	image embedding	propose top-5 candidate diseases with confidence scores; implemented as a fine-tuned foundation model (PanDerm)
RAG Module	each candidate label	retrieve canonical diagnostic criteria from the Skin Handbook (Oxford Handbook of Medical Dermatology)
Case-Review Agent	visual findings, candidate list, textbook standards, self-evolved guidelines, and top-5 similar historical cases	cross-verify all evidence in a five-stage review protocol; implemented with Qwen3-A30 in a chain-of-thought style
Summarization Agent	triggered when $\ge N_{thresh}$ new cases accumulate for a category	distill new findings into updated diagnostic guidelines

Operationally, the Vision Agent first extracts morphological observations, while the Pre-Diagnosis Agent generates a ranked list of candidate diseases in parallel. For each candidate, the RAG module fetches textbook standards, and the memory module retrieves top-5 visually similar cases together with the latest evolved guidelines. The Case-Review Agent then audits all sources to produce a validated diagnosis and a structured report. Confirmed cases are appended to the agent memory, and when sufficient new cases exist the Summarization Agent updates the guidelines, closing the self-evolving loop (Chen et al., 27 Mar 2026).

The five-stage review protocol used by the Case-Review Agent is explicitly specified as visual feature validation, guidelines cross-check, empirical alignment, conflict resolution, and final determination. This staged audit is central to the system’s claim of traceable clinical reasoning (Chen et al., 27 Mar 2026).

3. EvoDerma-Mem and self-evolving knowledge synthesis

The self-evolving dermatological memory of SkinGPT-X is termed EvoDerma-Mem. It comprises a graph database of triplets

$M_i = \langle z_i,\;K_i,\;D_i\rangle$

where $z_i \in \mathbb{R}^d$ is the visual embedding of case $i$, $K_i$ is the textual key findings, and $D_i$ is the confirmed diagnosis (Chen et al., 27 Mar 2026).

During inference, the query image $I$ is encoded via a frozen feature extractor $\Phi$ into $z_q = \Phi(I)$. Retrieval is based on cosine similarity,

$$\text{sim}(z_q,z_i)=\frac{z_q\cdot z_i}{\|z_q\|\|z_i\|}$$

and the system retrieves the Top-K most similar cases. In the reported workflow, these retrieved cases are subsequently audited together with textbook standards and evolved guidelines by the Case-Review Agent (Chen et al., 27 Mar 2026).

Knowledge synthesis is handled by the Summarization Agent. For a disease category $M_i = \langle z_i,\;K_i,\;D_i\rangle$0, let $M_i = \langle z_i,\;K_i,\;D_i\rangle$1 denote the set of all findings accumulated up to iteration $M_i = \langle z_i,\;K_i,\;D_i\rangle$2. The guidelines are updated as

$M_i = \langle z_i,\;K_i,\;D_i\rangle$3

where $M_i = \langle z_i,\;K_i,\;D_i\rangle$4 merges prior guidelines and novel findings, resolving contradictions. The update is triggered when sufficient new cases accumulate for a category, expressed in the system description through the threshold condition involving $M_i = \langle z_i,\;K_i,\;D_i\rangle$5 (Chen et al., 27 Mar 2026).

An incremental embedding update rule is also provided: $M_i = \langle z_i,\;K_i,\;D_i\rangle$6 This rule is presented as applicable if incremental embedding update is desired. A plausible implication is that SkinGPT-X separates immutable case storage from adaptive category-level synthesis: instance retrieval operates over embeddings and triplets, while guideline evolution operates over accumulated findings and contradiction resolution (Chen et al., 27 Mar 2026).

4. Interpretability, traceability, and visualization

Interpretability in SkinGPT-X is implemented through explicit logging and structured intermediate representations rather than post hoc explanation alone. Every intermediate output is logged, including visual findings, the pre-diagnosis list, retrieved textbook criteria, retrieved cases, and evolved guidelines. The Case-Review Agent emits a structured “chain-of-thought” report partitioned into the five review stages, with explicit references to which features or historical cases drove each inference (Chen et al., 27 Mar 2026).

The visualization module is described for deployment in a clinical dashboard. It highlights in the image the regions corresponding to key findings, including bounding or segmentation overlays for findings such as “scale” or “vascular dots.” Retrieved historical cases are displayed side by side with their salient features annotated, allowing clinicians to compare morphology directly. The evolution of guidelines is visualized as a timeline bubble chart in which bubble size or intensity indicates the magnitude of change in the guideline at each iteration (Chen et al., 27 Mar 2026).

These mechanisms situate transparency at three levels. First, there is visual transparency through localized highlighting of findings. Second, there is evidentiary transparency through side-by-side retrieval and textbook grounding. Third, there is temporal transparency through visualization of guideline evolution. The reported design therefore treats interpretability as a property of the full diagnostic workflow rather than of a single model output (Chen et al., 27 Mar 2026).

5. Experimental design and reported performance

SkinGPT-X was evaluated in a three-tier framework comprising benchmarking against four state-of-the-art LLMs, fine-grained multi-class classification, and rare skin disease diagnosis. The comparative LLM benchmark used MedGemma, Hulu-Med, Qwen3-VL, and PanDerm on four public datasets: DDI31, Dermnet, HAM10000, and Fitzpatrick-17k. Fine-grained evaluation used Dermnet498, covering 498 sub-classes. Rare disease evaluation used the 564-sample RSDD with 8 rare categories (Chen et al., 27 Mar 2026).

Evaluation setting	Dataset(s)	Reported gains
Comparative LLM benchmark	DDI31, Dermnet, HAM10000, Fitzpatrick-17k	DDI31: +9.6 pp accuracy over the next best model; Dermnet: +13 pp Weighted F1 over the state-of-the-art model
Fine-grained classification	Dermnet498	+5.4 pp ACC, +8.2 pp MCC, +5.2 pp Weighted F1, +5.5 pp Kappa
Rare skin disease diagnosis	RSDD, 564 samples, 8 rare categories	+9.8 pp accuracy, +7.1 pp Weighted F1, +10 pp Cohen’s Kappa over the fine-tuned baseline

Five metrics were used, with three defined explicitly: $M_i = \langle z_i,\;K_i,\;D_i\rangle$7 All key quantitative gains are reported as satisfying $M_i = \langle z_i,\;K_i,\;D_i\rangle$8 in two-sided t-tests (Chen et al., 27 Mar 2026).

The detailed comparative results include DDI31 accuracy of $M_i = \langle z_i,\;K_i,\;D_i\rangle$9 versus $z_i \in \mathbb{R}^d$0 for the next best model, Dermnet Weighted F1 of $z_i \in \mathbb{R}^d$1 versus $z_i \in \mathbb{R}^d$2, and on the Rare Skin Disease Dataset a $z_i \in \mathbb{R}^d$3 pp accuracy improvement, $z_i \in \mathbb{R}^d$4 pp Weighted F1 improvement, and $z_i \in \mathbb{R}^d$5 pp Cohen’s Kappa improvement over the fine-tuned baseline. The abstract further states that SkinGPT-X demonstrates state-of-the-art performance with a $z_i \in \mathbb{R}^d$6 accuracy improvement on DDI31 and a $z_i \in \mathbb{R}^d$7 weighted F1 gain on Dermnet over the state-of-the-art model (Chen et al., 27 Mar 2026).

6. Clinical significance, limitations, and adjacent directions

The reported discussion attributes three principal effects to SkinGPT-X: it addresses data sparsity in rare conditions by maintaining an explicit, self-evolving memory of real clinical cases; it yields superior fine-grained classification through iterative refinement of guidelines; and it offers clinical-grade interpretability through modular decomposition and traceable evidence aggregation. The multi-agent decomposition is said to mirror real multidisciplinary consultation and reduce “black-box” risk (Chen et al., 27 Mar 2026).

Several deployment considerations and limitations are also specified. Orchestrating multiple agents and RAG incurs longer inference latency, and future work is directed toward optimizing communication protocols and agent parallelization. Image acquisition variability can bias retrieval, so robust domain-adaptation techniques are needed to ensure cross-center generalizability. Memory curation is a further constraint: uncurated or noisy cases could degrade guidelines, and ongoing work aims to add quality filters and human-in-the-loop validation (Chen et al., 27 Mar 2026).

The stated future extensions are to incorporate structured EHR data, including demographics and lab results, into the RAG and memory modules; enable real-time two-way clinician feedback to correct or refine memory entries; and extend the framework to other dermatology tasks such as lesion segmentation and treatment recommendation, as well as to other medical specialties (Chen et al., 27 Mar 2026).

Adjacent work points to one additional direction. SkinDualGen describes prompt-driven diffusion with simultaneous image-mask generation in skin lesions and frames on-the-fly synthetic case generation to rebalance rare disease classes as a possible component of a future SkinGPT-X, together with simultaneous image, segmentation, and report generation. This is presented as an implication for SkinGPT-X rather than as part of the reported SkinGPT-X system itself (Xu, 26 Jul 2025).