Derm1M-AgentAug: Enhanced Dermatology VLP Dataset

• Derm1M-AgentAug is a MAGEN-augmented dermatology vision-language dataset that selectively improves low-quality image–text pairs with multi-aspect, clinically grounded descriptions.
• It employs a multi-agent pipeline—integrating Captioning, Summary, and Verification Agents—to generate validated diagnostic captions using structured disease ontologies.
• Serving as the backbone for frameworks like O-MAKE, Derm1M-AgentAug yields significant gains in zero-shot classification, long-tail recognition, and cross-modal retrieval.

Derm1M-AgentAug is the MAGEN-augmented version of the original Derm1M dermatology vision-language dataset. It preserves the same image set while systematically enhancing low-quality image–text pairs through a multi-agent data generation pipeline that replaces sparse or noisy captions with multi-aspect, clinically grounded descriptions and verified disease labels. Within the corresponding pretraining framework, Derm1M-AgentAug serves as the central corpus for all retrained baselines and for the ontology-based multi-aspect knowledge-enhanced method O-MAKE, supplying higher-quality, knowledge-rich supervision for dermatology vision-language pretraining (Li et al., 3 Dec 2025).

1. Definition, scope, and corpus composition

Derm1M-AgentAug is defined as the MAGEN-augmented version of the original Derm1M dermatology VLP dataset. The underlying corpus contains 403,563 image–text pairs collected from textbooks, PubMed, social media, YouTube, and related web resources, and is organized by a 4-level dermatology ontology with 130 clinical concepts and more than 390 skin diseases or conditions. Derm1M-AgentAug keeps the same 403,563 images and one caption per image, but selectively replaces or enriches some captions through MAGEN.

The augmentation policy is selective rather than exhaustive. Low-quality image–text pairs are identified by computing cosine similarity between image and text embeddings using DermLIP-PanDerm; pairs with similarity below 0.7 are sent to MAGEN. This low-quality subset contains 183,934 of 403,563 pairs, approximately 45% of the dataset. After processing, 133,930 pairs receive knowledge-enriched descriptions with verified disease labels, while 50,004 pairs are judged “No definitive diagnosis” by the Verification Agent; in those cases, the Captioning Agent’s descriptions are retained. The remaining approximately 55% of the corpus consists of original Derm1M high-similarity pairs kept unchanged (Li et al., 3 Dec 2025).

The resulting captions are explicitly multi-aspect. When MAGEN can determine a diagnosis confidently, captions typically contain disease diagnosis, morphological descriptors, anatomical site, pattern or distribution, and sometimes symptoms, associated context, or management hints when present in the source material. This makes Derm1M-AgentAug not merely a relabeled dataset, but a corpus in which the textual side is restructured to expose diagnostically useful content that ordinary web captions often omit.

2. MAGEN: multi-agent construction pipeline

Derm1M-AgentAug is produced by MAGEN, a multi-agent data generation system composed of a dermatology foundation model, a Captioning Agent, a Summary Agent, and a Verification Agent. The pipeline begins with PanDerm v2, which computes a visual embedding from image $I$ and performs zero-shot classification against 371 disease class text embeddings, outputting the top-5 differential diagnoses as diagnostic priors.

These priors are passed to a Captioning Agent implemented as a LLaVA-style MLLM. Its architecture combines PanDerm v2’s image encoder, Qwen3-14B as the LLM, and a LLaVA v1.5 style multimodal connector. The Captioning Agent is trained first by vision–language alignment on original Derm1M and then by instruction fine-tuning on Derm1M-Instruct, which contains 98,460 GPT-4o-generated captioning instructions derived from Derm1M captions. Its function is to produce an initial clinical caption emphasizing morphology, preliminary diagnosis suggestions, and some reasoning.

The Summary Agent constructs the external knowledge base used for verification. For each of the 371 diseases in Derm1M’s ontology, it collects approximately 600 tokens of disease information via Google AI search and uses Qwen2.5-72B-Instruct to summarize that material into a Disease Card of approximately 60–120 tokens. Each Disease Card has fixed fields: NAME, POS, SITES, and MINSET. The example provided for guttate psoriasis lists small, red, scaly, drop-like spots, rapid onset, and streptococcal triggering in the POS field; upper trunk, arms, legs, and scalp in SITES; and a minimal discriminative set centered on small, red, scaly spots with rapid onset.

The final stage is the Verification Agent, implemented with Qwen2.5-VL-72B in a retrieval-augmented configuration. It receives the image, the initial caption, and the top-5 candidate diseases together with their Disease Cards. Its instructed reasoning sequence is explicit: extract morphological claims from the initial caption, cross-check those claims against both the image and the Disease Card fields, determine the best-matching diagnosis among candidates if any, and synthesize a refined caption with corrected morphology, anatomically plausible sites, and either a verified diagnosis label or “No definitive diagnosis.” This verification stage is the decisive mechanism that distinguishes full MAGEN output from naïve recaptioning and is the basis on which the final corpus is named Derm1M-AgentAug (Li et al., 3 Dec 2025).

3. Role in O-MAKE pretraining

Derm1M-AgentAug is the pretraining backbone for O-MAKE, an ontology-based multi-aspect knowledge-enhanced framework. For each image–text pair $(I_i, T_i^r)$, the dataset supplies a raw caption $T_i^r$, from which an LLM extracts an ontology caption $T_i^o$ and a visual concept caption $T_i^c$. In addition, the raw caption is split into sentence-level sub-captions $\{T_i^{\text{sub}_j}\}_{j=1}^N$. These views are embedded as

$$E_i^t = \{(E_i^r, E_i^o, E_i^c), \{E_i^{s_j}\}_{j=1}^N\}.$$

The vision encoder is ViT-B/16, producing patch embeddings $E_i^p$ and a global visual embedding $E_i^v$, while the text encoder is GPT-2 with context length 77.

Two ontology-based mechanisms are central. The first is ontology-aware soft-label learning, where similarity between disease labels $d_i$ and $(I_i, T_i^r)$0 is defined by their shared ontology path: $(I_i, T_i^r)$1 This similarity is blended with one-hot targets through

$(I_i, T_i^r)$2

with $(I_i, T_i^r)$3 and $(I_i, T_i^r)$4. The second is ontology-guided weighting, in which the ontology-caption embedding acts as a semantic anchor for weighting the sub-captions by diagnostic relevance.

Training then combines Multi-Knowledge Image Alignment and Fine-Grained Alignment. The total objective is

$(I_i, T_i^r)$5

with $(I_i, T_i^r)$6. In operational terms, Derm1M-AgentAug is valuable here because its enhanced captions are rich enough to support extraction of hierarchical ontology paths, morphology-centric concept captions, and meaningful sentence-level sub-captions. All retrained models—CLIP, SigLIP, CoCa, and O-MAKE—use this same pretraining dataset. The training configuration reported for this stage is 15 epochs, batch size 2048, image resolution $(I_i, T_i^r)$7, text length truncated to 77 tokens, learning rate $(I_i, T_i^r)$8, warmup of 1500 steps, weight decay 0.1, and $(I_i, T_i^r)$9 for contrastive logits (Li et al., 3 Dec 2025).

4. Annotation schema and knowledge organization

Derm1M-AgentAug inherits its disease labeling structure from Derm1M’s hierarchical dermatology ontology. The ontology has 4 levels, 130 clinical concepts, and more than 390 skin diseases, with 371 diseases used explicitly in PanDerm v2 and in the Disease Card knowledge base. Each disease label has a path in the ontology tree, which is used both to build ontology captions and to compute inter-disease similarity for soft-label learning.

At the sample level, the annotation scheme is deliberately heterogeneous. Each image can be associated with a raw caption, an ontology caption, a visual concept caption, and sentence-level sub-captions. The raw caption is the main free-form description and may include diagnosis, morphology, anatomical site, pattern, and, when available, management or history. The ontology caption encodes hierarchical disease identity. The visual concept caption isolates morphology- and appearance-centric content. The sub-captions act as sentence-level decompositions of the raw description into aspect-specific units such as morphology, location, patient attributes or symptoms, and management remarks.

Disease Cards occupy a related but distinct position. They are not part of Derm1M-AgentAug per se; rather, they are external structured knowledge used by MAGEN’s Verification Agent to refine captions and labels. Each Disease Card contains NAME, POS, SITES, and MINSET. This separation between corpus annotations and auxiliary knowledge base is important: the dataset stores the final augmented captions, whereas the Disease Cards serve as a verification substrate during generation.

A common misconception is to treat Derm1M-AgentAug as a manually reannotated corpus. The paper instead describes the annotations as automatically generated or refined through coordinated agents and LLMs. This suggests a hybrid regime in which ontology structure and disease vocabulary constrain the outputs, while the actual caption enrichment is synthetic and verification-guided rather than manually authored on an image-by-image basis (Li et al., 3 Dec 2025).

5. Quantitative effects on pretraining and transfer

The principal empirical claim about Derm1M-AgentAug is that augmentation quality, not merely caption quantity, drives downstream improvement. In the MAGEN ablation, O-MAKE trained on original Derm1M captions reaches average zero-shot classification accuracy 0.493 across PAD, F17K, SD-128, SNU-134, and Daffodil. Replacing those captions with outputs from the Captioning Agent alone slightly lowers performance to 0.491, indicating that naïve recaptioning does not help. Adding diagnostic priors from PanDerm v2 raises the average to 0.531, a 3.8% absolute gain over baseline. Full MAGEN—Captioning Agent, diagnostic priors, and Verification Agent—reaches 0.544, adding a further 1.3% and defining the final Derm1M-AgentAug configuration.

The same pattern appears in the main zero-shot comparison. Retrained baselines pretrained on Derm1M-AgentAug obtain average accuracies of 0.444 for CLIP, 0.453 for SigLIP, and 0.399 for CoCa. O-MAKE reaches 0.544. Its per-dataset zero-shot accuracies are 0.667 on PAD, 0.371 on F17K, 0.460 on SD-128, 0.390 on SNU-134, and 0.832 on Daffodil. Compared with DermLIP-PanDerm trained on original Derm1M, whose average is 0.492, O-MAKE improves by 5.2% absolute.

Long-tail evaluation further localizes the benefit. On SD-Tails and SNU-Tails, O-MAKE reaches average accuracy 0.508, whereas DermLIP-PanDerm obtains 0.466 and MAKE 0.459. On SD-Tails specifically, O-MAKE attains 0.558 versus 0.443 for the retrained SigLIP baseline, a gain of 11.5%. In cross-modal retrieval on SkinCAP, DermLIP-PanDerm scores 0.407, the best retrained baseline CLIP scores 0.410, and O-MAKE reaches 0.453.

Component ablation within O-MAKE shows that Derm1M-AgentAug supports increasingly structured use of the text signal. Starting from CLIP trained on Derm1M-AgentAug at 0.444 average accuracy, adding multi-knowledge alignment yields 0.472; adding sub-captions yields 0.491; adding fine-grained alignment yields 0.527; adding ontology soft labels yields 0.535; and adding ontology weighting gives the full O-MAKE result of 0.544. The dataset augmentation and the O-MAKE objective are therefore presented as jointly responsible for the reported gains (Li et al., 3 Dec 2025).

6. Relation to dermatology agent systems, limitations, and broader significance

Derm1M-AgentAug sits within a broader dermatology literature on agentic augmentation, ontology-guided supervision, and retrieval-grounded reasoning. In related work, the broader Derm1M program is described as a million-scale dermatology vision-language dataset aligned with a four-level ontology and 130 clinical concepts, and DermLIP is presented as a dermatology-specific retrieval and alignment foundation; this broader substrate suggests how AgentAug-style datasets can be embedded in larger multimodal pipelines (Yan et al., 19 Mar 2025). In another line of work, DermAgent uses Derm1M-derived visual encoders and a 413,210-case retrieval bank inside a Plan–Execute–Reflect system, making case retrieval, guideline retrieval, and critic-driven self-correction central to dermatological reasoning (Liu et al., 14 May 2026). SkinGPT-X extends the agentic pattern further through multimodal collaborative diagnosis with a self-evolving dermatological memory and disease-specific guideline synthesis, offering a concrete blueprint for what a large-scale agent-augmented dermatology system could look like (Chen et al., 27 Mar 2026).

Within that landscape, Derm1M-AgentAug is distinctive because its augmentation target is the pretraining corpus itself. It is not an inference-time multi-agent system, but a training-time reconstruction of supervision quality through selective recaptioning, disease-card verification, and ontology-compatible text decomposition. A plausible implication is that it occupies an intermediate layer between raw web-scale dermatology corpora and fully agentic diagnostic systems: it makes the textual supervision more structured and clinically grounded so that later models can support richer retrieval, reasoning, and explanation.

Its limitations remain those of both web-derived corpora and synthetic captioning. The paper notes web-source bias from PubMed, textbooks, YouTube, and related sources; skin tone imbalance is not explicitly quantified; LLM-based Captioning and Verification Agents may miss subtle morphology, misinterpret rare or highly atypical cases, or generate verbose but partially redundant descriptions; ontology coverage is fixed to 371 or 390-plus known conditions; and GPT-2’s 77-token limit forces truncation during O-MAKE pretraining. These constraints do not negate the corpus’s utility, but they delimit what Derm1M-AgentAug can guarantee as a supervision source. Its significance lies in showing that multi-agent recaptioning, diagnostic priors, and verification can materially improve dermatology vision-language pretraining without changing the image inventory, and that such augmentation can propagate into stronger zero-shot classification, long-tail recognition, and cross-modal retrieval (Li et al., 3 Dec 2025).