RareBench: AI Benchmark for Rare Data

• RareBench is a curated collection of datasets and protocols designed to evaluate AI performance in domains with rare data distributions.
• It underpins rare disease risk screening using stratified sampling and robust metrics, enabling early identification and differential diagnosis.
• It assesses compositional generative models via engineered text-to-image prompts, measuring model alignment on rare attribute-object combinations.

RareBench encompasses a set of large-scale, rigorously curated datasets and benchmarking protocols designed to evaluate artificial intelligence systems in domains characterized by rare data distributions. Predominantly, RareBench underpins advances in rare disease risk screening and compositional generative modeling, providing a standardized foundation for both diagnostic and generative AI assessments. The following sections synthesize the corpus of published research on RareBench, including its design methodologies, dataset structure, partitioning strategies, evaluation metrics, baseline results, and the core challenges that arise in both medical and machine learning contexts.

1. RareBench in Rare Disease Risk Screening

RareBench is central to the “RareAlert” screening pipeline, serving as the primary data foundation for universal early identification of rare diseases from initial clinical presentations (Chen et al., 26 Jan 2026). This dataset comprises 158,666 patient cases sourced from multi-institutional real-world data—including PubMed case reports (mapped via Orphanet taxonomy and MeSH/free-text queries), longitudinal hospital course records, and emergency-department presentations.

Candidate records (n=432,811) underwent stringent de-duplication, English-language filtering, and automated inclusion screening (DeepSeek-V3 based). Inclusion required documented clinical sections (history, exam, diagnostics, diagnosis, treatment/outcomes) and consistency with established medical knowledge. Final triage yielded 38,737 rare-disease cases (spanning >7,000 conditions) and 119,929 non-rare controls, with disease categorization determined by Orphanet string matching, FAISS-embedded retrieval, GPT-5 validation, and human correction.

All 33 Orphanet disease categories are represented, with rare case densities reflecting the empirical “long tail” of clinical frequency—i.e., 1–10 cases per condition.

2. Data Structure, Partitioning, and Representation

Each instance in RareBench corresponds to a primary-care clinical vignette, comprising verbatim free-text demographics, medical history, and physical exam findings. No laboratory, imaging, or genetic data is included at this screening stage. Inputs are supplied directly to LLM prompts without manual feature engineering. Model outputs conform to a fixed JSON schema: numeric risk score (0–100), five top-ranked features with normalized weights (∑=1), and structured clinical explanation.

The dataset adheres to an 80%/20% stratified rare/non-rare split, producing a development set (126,933 cases) and a held-out test set (31,733 cases). Within development, stratified k-fold cross-validation (typically 5-fold) maintains the 24% rare-disease prevalence. This ensures unbiased performance estimates and guards against overfitting given the extreme class imbalance.

3. Benchmarking Tasks and Evaluation Metrics

Within the rare disease domain, RareBench defines a binary classification objective—discriminating rare from non-rare cases at the primary visit, using the following metrics:

• ROC-AUC ($$\text{AUC} = \int_0^1 \mathrm{TPR}(\mathrm{FPR}^{-1}(t))\,dt$$): quantifies probability ranking performance.
• Sensitivity ($\mathrm{TP}/(\mathrm{TP}+\mathrm{FN})$), Specificity ($\mathrm{TN}/(\mathrm{TN}+\mathrm{FP})$): measure true positive and true negative rates.
• PPV, NPV, Accuracy, Balanced Accuracy
• $F_1$ and $F_2$ scores: harmonic mean of precision/recall, emphasizing recall in $F_2$.
• Matthews Correlation Coefficient:

$$\mathrm{MCC} = \frac{\mathrm{TP}\cdot\mathrm{TN}-\mathrm{FP}\cdot\mathrm{FN}} {\sqrt{(\mathrm{TP}+\mathrm{FP})(\mathrm{TP}+\mathrm{FN})(\mathrm{TN}+\mathrm{FP})(\mathrm{TN}+\mathrm{FN})}}$$

• Brier score, Expected Calibration Error (ECE)

These metrics provide joint assessment of discrimination, calibration, and classification.

4. RareBench in Generative Modeling

RareBench also denotes a test set for compositional text-to-image (T2I) model evaluation (Park et al., 2024). Here, it comprises 320 prompts (200 single-object, 120 multi-object) explicitly engineered to test generation on rare attribute–object compositions—for example, “zebra-striped duck,” “horned lion and hairy frog.” Prompts are sourced by GPT-4o generation and refined by human annotators to maximize contextual rareness (98.1% flagged as rare). Categories span property, shape, texture, action, and complex mixtures for single objects, and concatenation/relation/complexity schemes for multi-object inputs.

RareBench generative evaluation employs GPT-4o and human raters to score output images on a rubric-mapped scale to percentage alignment. No train/test split is used; the entire set is for evaluation only.

5. Baseline Results and Comparative Analyses

Medical Screening

ROC-AUC performance for individual LLMs spans 0.679 (GPT-40-mini) to 0.909 (GPT-5), with Qwen3-235B at 0.859. Classical ensemble learning (CatBoost on ten LLM scores) achieves 0.912 AUC. RareAlert (Qwen3-4B, multi-LLM aligned) attains 0.917 AUC, with sensitivity of 0.778, specificity of 0.935, accuracy of 0.892, and MCC of 0.668—outperforming all competing models.

Generative Modeling

On RareBench, R2F (rare-to-frequent LLM-guided diffusion (Park et al., 2024)) demonstrates improvement over SD3.0, FLUX, PixArt-α, both for single-object (80.6% alignment, +17.4 pp over baseline) and multi-object (67.5%, +8.8 pp). Notably, largest gains materialize for property and texture categories, with persistent challenges in densely intertwined multi-object scenes.

Rare Disease Differential Diagnosis

RareBench (medicine) enables four-tiered evaluation: phenotype extraction, rare disease rule-in/rule-out, rare/common discrimination, and multiclass diagnosis. Dynamic IC-weighted retrieval of few-shot prompts, leveraging 17,232 HPO terms and a merged OMIM/Orphanet/CCRD graph, boosts LLM performance in differential diagnosis. On 2185 rare-disease cases spanning 421 diseases, GPT-4 zero-shot achieves 32.3% hit@1, 58.9% hit@10; dynamic few-shot prompts double hit@1 for non-GPT4 models (+108%), surpassing physician accuracy (raw EHR hit@1: 45.3%, GPT-4 on phenotypes: 52.0%) (Chen et al., 2024).

6. Design Challenges and Limitations

Constructing RareBench required resolution of several domain-specific challenges:

• Class imbalance (~24% rare), necessitating stratified sampling and cost-sensitive thresholds.
• Heterogeneous text narrative quality across sources; standardized prompt and automated filtering.
• Inclusion of 119,929 non-rare controls to accurately estimate false-positive rates (often missing from prior rare-disease datasets).
• Complex multi-step labeling: exact string matching, embeddings-based retrieval (FAISS), GPT-validation, human verification.
• Privacy constraints: models, such as RareAlert, optimized for on-premise deployment, avoiding transmission of PHI.
• Retrospective and geolinguistic bias: records are historical and English-only; generalizability may require recalibration/validation in other healthcare systems.

In the generative modeling arena, remaining open issues concern compositional scene complexity and fine-grained attribute-object binding.

7. Scientific and Clinical Significance

RareBench establishes benchmarks at two disciplinary frontiers: LLM-augmented rare disease care and LLM-grounded compositional generation. Its medical impact lies in enabling universal early risk screening, comparative model evaluation, and AI-assisted differential diagnosis, supplying both open-source datasets and prompting protocols. In generative modeling, RareBench supplies a high-rareness testbed for rigorous evaluation of T2I systems.

A plausible implication is that systematic rare-phenomenon benchmarks such as RareBench catalyze progress toward automated decision support in both clinical medicine and multimodal generative AI, revealing latent model weaknesses and informing future research in multimodal architectures, uncertainty calibration, domain adaptation, and knowledge graph integration. External validation and ongoing dataset augmentation are necessary to address representational and distributional biases intrinsic to rare data domains.