DiverseHumans Testset Benchmark

Updated 14 July 2026

DiverseHumans Testset is a benchmark framework that evaluates explicit variations in human identity, appearance, geometry, and behavior across multiple modalities.
The framework measures critical indicators such as person-count correctness, unique face accuracy, and identity spread using metrics like Count Accuracy, UFA, and GIS.
It enables cross-modal evaluation by adapting protocols for multi-human image generation, 3D shape retrieval, facial mesh assessments, and behavioral diversity analyses.

Searching arXiv for the specified benchmark and closely related papers. DiverseHumans Testset is a label used for benchmark constructions intended to evaluate whether models remain robust under explicit variation in human identity, appearance, geometry, or behavior. The supplied literature suggests that the name is used most concretely for a multi-human text-to-image benchmark introduced with DisCo, while related work also applies or extends the label to non-rigid 3D human-shape retrieval, diverse 3D facial meshes, controlled face subsets derived from Diversity in Faces, and cross-country behavior-modeling splits derived from DiversityOne (Borse et al., 1 Oct 2025, Pickup et al., 2020). A plausible implication is that the term denotes a benchmark pattern rather than a single universally fixed artifact: explicit curation of human diversity, coupled with protocols that isolate failure modes obscured by aggregate averages.

1. Nomenclature and referential scope

Within the supplied sources, “DiverseHumans Testset” does not refer to one canonical dataset. In the generative-model literature, it is the primary benchmark used to stress-test multi-human image generation under explicit diversity requirements. In 3D geometry and shape analysis, the same label is attached to a human-body retrieval benchmark spanning real, synthetic, and FAUST meshes. Elsewhere, the term appears as a construction target or evaluation template built from larger corpora such as 3D2M, Diversity in Faces, or DiversityOne (Borse et al., 1 Oct 2025, Dasgupta, 2024, Merler et al., 2019, Busso et al., 5 Feb 2025).

Context	Scale	Distinguishing features
Multi-human image generation	1,200 prompts; 8–16 images per prompt	2–7 people; four diversity-tag variants; Count Accuracy, UFA, GIS, HPSv2
Non-rigid 3D human retrieval	Over 900 meshes across Real, Synthetic, and FAUST	Disjoint training/test splits; subject labels; dense correspondence information
3D facial mesh evaluation	188 facial meshes	45 ethnicities; ages 16 to 65; 68 landmarks; Wavefront OBJ
DiF-derived face subset recipe	970,000 source face crops; example subset of 50,000	Ten coding schemes; 336 strata; Shannon and Simpson balancing
Cross-country smartphone sensing	782 participants in 8 countries	26 sensor modalities; 350K+ self-reports; LOCO evaluation

This multiplicity matters methodologically. A benchmark centered on identity duplication in image synthesis measures a different object than one centered on non-rigid retrieval, craniofacial diversity, or cross-country behavioral inference. Consequently, numerical results are not directly comparable across these instantiations even when the same label is used.

2. Canonical formulation in multi-human generation

The most specific and operationally complete instance in the supplied material is the DiverseHumans Testset introduced with DisCo for multi-human text-to-image generation (Borse et al., 1 Oct 2025). It contains 1,200 distinct prompts organized into six bands of 200 prompts each, corresponding to scenes with exactly 2, 3, 4, 5, 6, or 7 people. Each prompt further carries one of four diversity-specification tags, with 25% of the prompts in each category: no explicit diversity instruction, a generic “diverse faces” instruction, a single ethnicity specification such as “all people are Indian,” or individual-by-person ethnicity assignments such as “one White, one Black, one Asian, …”.

For evaluation, each prompt is sampled with $M=8\text{–}16$ images under identical random seeds across all methods, yielding on the order of 10,000–20,000 test images per method. The benchmark is explicitly framed as an out-of-distribution test: prompts are drawn from contexts, settings, and activities different from those seen during the 30,000-prompt training corpus.

The benchmark targets a failure regime specific to compositional human generation. The reported problem is that text-to-image systems may duplicate faces, merge identities, or miscount people when prompts specify several humans. The dataset construction therefore binds person count and diversity constraints together, rather than treating them as separable axes of evaluation. This design makes within-image identity collapse, cross-sample identity repetition, and person-count errors directly measurable.

A closely related benchmark is MultiHuman-Testbench, which is not itself named DiverseHumans Testset but addresses the same evaluation problem with 1,800 samples, 5,550 unique human face images, curated prompts covering simple and complex actions, and human-selected pose-conditioning images (Borse et al., 25 Jun 2025). This parallel development suggests that multi-human generation has become a distinct benchmark niche in which identity diversity, count control, and semantic action correctness must be measured jointly.

3. Metrics and evaluation protocol

On the DisCo benchmark, four primary metrics are reported, all as percentages except the renormalized perceptual score (Borse et al., 1 Oct 2025). Count Accuracy measures the fraction of generated images containing exactly the number of faces requested by the prompt. With target count $N_{\text{target}}(c)$ , generated-image set $\mathcal{X}$ , and detected face count $F(x)=|D(x)|$ , it is defined as

$\mathrm{CountAccuracy} = 100\cdot \frac{1}{|\mathcal{X}|} \sum_{x\in\mathcal{X}} \mathbf{1}\{F(x)=N_{\text{target}}(c)\}.$

Unique Face Accuracy (UFA) measures the fraction of images in which all detected faces are pairwise distinct identities. If $\{f_i\}$ are $\ell_2$ -normalized ArcFace embeddings and cosine similarity is $s(u,v)=u^\top v$ , a duplicate pair exists whenever

$\exists\, i\neq j \text{ such that } s(f_i,f_j)\ge \kappa_{\text{dup}},$

with $\kappa_{\text{dup}}=0.5$ . The metric is then

$N_{\text{target}}(c)$ 0

Global Identity Spread (GIS) extends identity diversity from the image level to the whole test split. All face embeddings from all generated images are clustered with single-linkage clustering at the same threshold $N_{\text{target}}(c)$ 1. If this yields $N_{\text{target}}(c)$ 2 distinct clusters across prompts with total requested faces $N_{\text{target}}(c)$ 3, then

$N_{\text{target}}(c)$ 4

HPSv2 is a perceptual-quality and human-preference score reported on a 0–100 scale after renormalization.

The evaluation stack is fixed. Face detection uses RetinaFace with confidence threshold 0.7. Each crop is passed through an ArcFace encoder to obtain $N_{\text{target}}(c)$ 5-normalized embeddings $N_{\text{target}}(c)$ 6. Duplicate detection is based on cosine similarity $N_{\text{target}}(c)$ 7, and GIS uses single-linkage clustering at that same threshold. Final metrics are averaged over every prompt-to-images group with 8–16 samples per prompt.

Alternative instantiations of DiverseHumans-style evaluation adopt different metric families. The 3D2M facial-mesh setting specifies gender and ethnicity proportions, Shannon entropy for ethnicity and gender, Simpson’s index, point-to-point RMS error, point-to-plane error, and normalized landmark error. The non-rigid 3D human retrieval setting emphasizes Precision@ $N_{\text{target}}(c)$ 8, Recall@ $N_{\text{target}}(c)$ 9, mAP, NN, 1-T, 2-T, E-M, and DCG. The DiF-derived subset recipe likewise uses Shannon and Simpson diversity and evenness over discretized facial coding schemes (Dasgupta, 2024, Pickup et al., 2020, Merler et al., 2019). This suggests that “diversity” is benchmark-specific: identity distinctness in image synthesis is not the same quantity as balanced demographic coverage, geometric variation, or retrieval separability.

4. Reported empirical behavior

The DisCo paper reports quantitative results on DiverseHumans for proprietary baselines, open-source baselines, and two DisCo variants using Flux-Dev and Krea-Dev backbones (Borse et al., 1 Oct 2025).

Model	CountAcc / UFA / GIS	HPSv2
Gemini-Nanobanana	72.3 / 57.2 / 42.7	31.9
GPT-Image-1	90.5 / 85.1 / 89.8	33.4
HiDream	57.9 / 32.3 / 16.2	32.2
Qwen-Image	79.8 / 49.0 / 45.9	32.6
OmniGen2	63.3 / 32.3 / 28.7	33.4
DreamO	70.5 / 31.8 / 35.2	32.0
SD3.5	55.3 / 69.1 / 72.5	28.1
Flux-Dev	70.8 / 48.2 / 50.5	31.7
Krea-Dev	73.6 / 45.8 / 50.6	31.2
DisCo(Flux)	92.4 / 98.6 / 98.3	33.4
DisCo(Krea)	83.5 / 89.7 / 90.6	32.2

DisCo(Flux) achieves 92.4% Count Accuracy, 98.6% UFA, and 98.3% GIS. DisCo(Krea) reaches 83.5%, 89.7%, and 90.6%, respectively. The reported interpretation is that these models substantially outperform both open-source backbones and proprietary systems on identity-diversity metrics while keeping HPSv2 competitive. The same results table also contains an Action score and an Average score; a methodological caveat is stated explicitly for Gemini-Nanobanana, whose Action score is marked as potentially inflated because the same LLM both generated and judged the images.

The paper further reports a crowding analysis across requested group sizes from 2 through 7. Baseline methods exhibit a steep drop-off in count and identity accuracy beyond 4 people, often falling below 50% UFA for 6–7 people. DisCo(Flux), by contrast, maintains $\mathcal{X}$ 0 UFA and $\mathcal{X}$ 1 Count Accuracy even in the most crowded scenes.

The qualitative findings are framed as an “identity crisis” in baseline models. As the number of people increases, diffusion-based generators are reported to reuse the same face multiple times, merge distinct individuals into one, or generate fewer people than requested. Visual comparisons are said to show that DisCo produces distinct facial features for each person, plausible group layouts, and correct person counts in every scene. Across all four diversity-tag variants, DisCo is reported to sustain near-perfect UFA and GIS, at 98–100% across the board, whereas baselines degrade substantially under explicit ethnicity constraints.

Outside the DisCo setting, the supplied literature shows several other ways to instantiate a DiverseHumans-style benchmark. The 3D2M dataset is presented as a diverse-humans test set for facial reconstruction and landmark localization, comprising 188 3D facial meshes with 73 from female candidates and 114 from male candidates, spanning 45 distinct self-reported ethnicities across nine world regions and ages 16 to 65 (Dasgupta, 2024). Each mesh is stored as a Wavefront OBJ file with approximately 17,000 vertices, per-vertex normals, and shared pose and scale conventions inherited from FaceBuilderHead. Each is accompanied by 68 landmarks using the dlib 68-landmark schema, obtained by automatic 2D landmark detection on a frontal render and stored in a per-mesh keypoints vector. Suggested evaluation follows either an 80/20 train/test split or 5-fold cross-validation, with reconstruction assessed by point-to-point RMS error and landmark localization by mean Euclidean error normalized by inter-ocular distance.

In non-rigid 3D human-shape retrieval, the supplied description associated with Pickup et al. applies the DiverseHumans label to a benchmark comprising three collections—Real, Synthetic, and FAUST—with separate training splits, class labels defined by subject identity, and dense correspondence information (Pickup et al., 2020). The Real dataset uses 400 test meshes from 40 subjects and 100 training meshes from 10 other subjects; the Synthetic dataset uses 300 test meshes of 15 shapes and 45 training meshes of 9 additional subjects; the FAUST dataset uses 200 test scans and 100 training scans. Evaluation is defined as ranking gallery meshes by decreasing similarity to a query and reporting Precision@ $\mathcal{X}$ 2, Recall@ $\mathcal{X}$ 3, mAP, NN, 1-T, 2-T, E-M, and DCG. This formulation treats diversity as variation in body shape, pose, and capture modality rather than demographic composition or generated-image identity distinctness.

The supplied description for “Diversity in Faces” does not introduce a released DiverseHumans benchmark directly, but it gives an explicit recipe for carving a smaller, highly controlled, fully balanced DiverseHumans test set out of 970,000 face crops annotated with ten facial coding schemes (Merler et al., 2019). The recipe selects core dimensions such as age bin, gender, skin color, and symmetry quartile, producing $\mathcal{X}$ 4 strata; for an example target size $\mathcal{X}$ 5, approximately 150 samples per stratum are proposed. It then recommends edge-case injection by oversampling the top and bottom 1% of continuous measures and recomputing Shannon and Simpson diversity and evenness after sampling.

A similar constructional logic appears in the DiversityOne blueprint, where a DiverseHumans Testset is framed as a stratified evaluation split over 782 college-student participants in eight countries, with 26 smartphone sensor modalities and more than 350,000 self-reports (Busso et al., 5 Feb 2025). Proposed protocols include country-stratified train/test splits, temporal blocking to avoid leakage, leave-one-country-out evaluation for domain generalization, and leave-one-gender-out or leave-one-age-group-out tests for cross-demographic robustness. Here, “human diversity” is operationalized through country, gender, age, and psychosocial heterogeneity rather than face or body appearance.

6. Methodological issues, ambiguities, and significance

Several interpretive issues recur across the supplied literature. First, benchmark identity is ambiguous. Some sources describe fully specified released resources, whereas others provide blueprints or recipes for deriving a DiverseHumans test set from an upstream dataset. This suggests caution when citing “DiverseHumans Testset” without the accompanying domain, task, and source paper.

Second, diversity guarantees are defined differently across modalities. In the 3D2M facial-mesh setting, diversity is stated to be ensured “by design” through explicit curation of 73 female versus 114 male candidates, 45 ethnicities drawn from nine regions, and age span 16–65, with no formal statistical tests such as $\mathcal{X}$ 6 or ANOVA reported (Dasgupta, 2024). In the DisCo benchmark, diversity is instead operationalized through prompt tags and measured via embedding-based distinctness, count correctness, and clustering. In the DiF-derived recipe, diversity becomes a balancing problem over coding-scheme strata and tail coverage. These are not interchangeable notions.

Third, some source summaries themselves contain caveats. The 3D2M description notes that 73 female plus 114 male candidates sum to 187, although the abstract and summary state 188 total; it also states that the paper does not explicitly specify a license, and that users should check the repository README or contact the author for commercial-use terms (Dasgupta, 2024). The DisCo results table flags Gemini’s Action score as potentially inflated because the same LLM both generated and judged the images (Borse et al., 1 Oct 2025). DiversityOne reports average compliance of roughly 20 prompts per day for about 40% of participants and an overall dropout of about 35% from day 1 to day 28, which is material for any benchmark built on those logs (Busso et al., 5 Feb 2025).

Fourth, allied benchmarks show that metric design changes conclusions. MultiHuman-Testbench evaluates multi-human generation with Count Accuracy, Hungarian ID Similarity, HPS, and MLLM-based Action QA, and it introduces regional isolation and segmentation-plus-Hungarian matching to improve ID retention (Borse et al., 25 Jun 2025). A plausible implication is that benchmark outcomes in this area depend not only on model quality but also on the evaluator stack: detector choice, embedding model, matching or clustering threshold, use of regional priors, and whether action correctness is scored by an external MLLM.

Taken together, the DiverseHumans Testset concept marks an increasingly common research move: replacing undifferentiated average-case evaluation with benchmarks that force explicit accounting of heterogeneity among humans. In multi-human image generation, this takes the form of identity diversity and person-count correctness; in 3D geometry, it takes the form of shape, pose, and correspondence variability; in face datasets, it becomes balanced sampling over coding schemes; and in behavioral sensing, it appears as cross-country and cross-demographic generalization. The unifying significance is methodological rather than nominal: these benchmarks are designed to reveal whether a system that performs well on average still fails when confronted with structured human diversity.