Pick-a-Pic v2: Open Dataset for Text-to-Image Research

• Pick-a-Pic v2 dataset is a large-scale human-preference resource for text-to-image research, featuring over 1M paired judgment examples.
• It employs a rigorous data collection pipeline with freeform prompts, detailed metadata, and stratified splits to ensure reproducibility and diversity.
• The dataset supports PickScore, a CLIP-based regression model that achieves superior preference prediction and improved human-model correlation.

The Pick-a-Pic v2 dataset is an open human-preference dataset for text-to-image generation research, designed to address the scarcity and inaccessibility of large-scale datasets documenting real user choices regarding synthetic image outputs. Building upon its initial deployment, Pick-a-Pic v2 aims to provide an expanded, more diverse, and methodologically rigorous foundation for training and evaluating scoring models, benchmarking text-to-image systems, and developing human-aligned generation workflows. The resource is publicly available, adheres to reproducibility standards, and is licensed for open use with attribution (Kirstain et al., 2023).

1. Dataset Composition and Structure

Pick-a-Pic v1 comprises over 584,000 pairwise user judgments on images generated by text-to-image models from 35,000+ unique freeform prompts. The v2 extension targets a scale of at least 1,000,000 preference examples, broadening the dataset’s representativity and robustness. In both iterations, prompts are entirely user-written (without templating in v1), yielding a prompt length mean of approximately 8 tokens (standard deviation ≈ 4; 10th–90th quantile [4, 12] tokens).

Prompt types in v1 were distributed as follows:

• Fantasy/Illustration: ~40%
• Photorealistic/Product Photo: ~35%
• Portrait/Character Art: ~15%
• Miscellaneous/Other: ~10%

Each judgment instance is structured as a JSON object with defined fields capturing prompt metadata, user/session anonymized IDs, timestamps, image references, selected preference (1, 2, or 0 for tie), text-to-image model identifiers, guidance scale values, and optionally, random seed values for image reproducibility. The dataset directory includes an “images” folder organized by model and guidance scale, and “data” files (train.jsonl, val.jsonl, test.jsonl) containing the annotation records.

Statistic	v1 Value	v2 Target
Unique prompts	35,000+	—
Pairwise judgments	583,747 (train+val+test)	≥ 1,000,000
Unique users	6,394 (all), 4,375 (train set)	Broader/global

The dataset is distributed under the CC BY 4.0 license, requiring attribution for use (Kirstain et al., 2023).

2. Data Collection Pipeline

Data are collected through a web application requiring OAuth authentication via Gmail or Discord. Users may submit free-form prompts which are then interpreted by backbone generative models (e.g., Stable Diffusion 2.1, Dreamlike Photoreal 2.0, SD XL variants) to produce two candidate images per prompt. The user interface presents these images side by side, prompting the user to indicate a preference or declare no preference (tie). Upon selection, the less-preferred image is replaced with another sample from the same prompt, maintaining the prompt context until edited or cleared by the user.

Key aspects of the protocol include:

• Judgments per user in v1: median 75, with an upper cap at 1,000.
• No synthetic prompt templating in v1; prompt creation is solely user-driven.
• Generative model and guidance scale selection are randomly interleaved from a predefined set.
• Explicit NSFW filtering via regex, along with bans for inappropriate or rapid-fire responses.
• Data splits by prompt: 1,000 prompts are held out (500 val, 500 test), each unique to a user; all remaining prompts form the training set.

Instruction to users: “Write the prompt that best describes your desired image. Click the image you prefer, or ‘No preference’ if equal.”

3. Scoring Model: PickScore

To operationalize human preferences, the dataset supports the training of “PickScore”, a CLIP-based regression scoring function. The architecture uses a CLIP-Haugmented (CLIP-H) text encoder and image encoder to map prompts and images into joint embedding space. The score for a prompt–image pair $(x, y)$ is calculated by

$$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$

where $T$ is a learnable temperature scalar. For pairwise preference distributions $p = (p_1, p_2) \in \{(1,0), (0,1), (0.5,0.5)\}$ and model outputs $$\hat p_i = \frac{\exp(s(x, y_i))}{\exp(s(x, y_1)) + \exp(s(x, y_2))}$$, the loss function is

$$L_\text{pref}(x, y_1, y_2, p) = \sum_{i=1}^2 p_i (\log p_i - \log \hat p_i)$$

Examples are reweighted to counteract prompt-frequency biases by $w(x) = 1/\sqrt{\#\text{examples with prompt } x}$.

The PickScore between two candidate images for a prompt is given by

$$\mathrm{PickScore}(x, y_i, y_j) = \sigma(s(x, y_i) - s(x, y_j))$$

where $\sigma$ denotes the standard logistic sigmoid.

Training details include initialization from CLIP-H weights, AdamW optimizer with learning rate $3\times10^{-6}$ and weight decay $$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$0, batch size 128, training on 8×A100 GPUs (<1 hour), with model selection by validation set accuracy in the no-tie scenario.

4. Evaluation Metrics and Results

PickScore’s predictive performance is rigorously quantified using preference prediction accuracy, model–human correlation statistics, and Elo rating correlations on held-out test data. Results indicate:

• Preference prediction accuracy (test set): PickScore $$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$1 over 3 seeds; compared with human experts ($$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$2), HPS ($$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$3), ImageReward ($$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$4), CLIP-H zero-shot ($$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$5), and random or aesthetic predictors ($$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$6).
• Spearman’s $$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$7 for ranking 9 models on 100 MS-COCO prompts: human vs. FID ($$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$8); human vs. PickScore ($$s(x, y) = \langle E_\text{txt}(x), E_\text{img}(y) \rangle / T$$9).
• Elo correlation (mean $T$0 std over 50 shuffles) on the Pick-a-Pic test split:

Metric	$T$1 (mean $T$2 std)
CLIP-H	0.313 ± 0.075
ImageReward	0.492 ± 0.086
HPS	0.670 ± 0.071
PickScore (v1)	0.790 ± 0.054

These results support the use of PickScore as a superior metric for human-aligned evaluation and model selection, and the use of Pick-a-Pic prompts over traditional datasets such as MS-COCO for text-to-image benchmarking (Kirstain et al., 2023).

5. Guidelines and Extensions for v2

Pick-a-Pic v2 is designed to address limitations and further enhance the scope and fidelity of the dataset:

• Scale: Expand to at least 1,000,000 preference examples and broaden the user base (multi-lingual, global).
• Prompt Diversity: Categories now formally tagged, and prompt-templating tiers (controlled vs. creative) introduced for spectrum coverage.
• Label Modalities: Collect non-pairwise signals—$T$3-way comparisons ($T$4), and fine-grained ratings (1–5 scale).
• User Metadata: Addition of opt-in demographic attributes (age bracket, region, expertise).
• Auxiliary Logging: Capture behavioral signals such as decision time and click heatmaps.
• Schema Extensions: New metadata fields: rating type ({pairwise, k-way, scalar}), rating value ($T$5 or $T$6 for non-pairwise), prompt_category tags, image_quality flags (NSFW_flag, blur_score, face_count), explicit provenance.
• Data Integrity: Enforced stratified splits and no prompt leakage by prompt_id.

Additional best practices include Apache Arrow (.parquet) support for scalable loading, stratified sampling scripts, fully open codebases for web-app, data handling, and scoring, inclusion of a “Responsible Use” statement, and continued release under CC BY 4.0 or Open Data Commons licensing. These measures are intended to support reproducibility, facilitate downstream research, and ensure ethical use.

6. Applications and Impact

Pick-a-Pic and its derived scoring model PickScore are recommended for:

• Training human-aligned ranking and scoring models.
• Automatic evaluation of text-to-image generation models in a manner highly correlated with aggregated human preferences.
• Model selection and benchmarking, with documentation showing improved correlation to expert judgments relative to prior metrics.
• Ranking-based image generation workflows, where preference-aligned models boost the quality and relevance of outputs.
• Expanding the scope of prompts and user responses, which is critical for generalizable evaluation frameworks.

A plausible implication is that, by offering both diversity and scale, Pick-a-Pic v2 can alleviate overfitting to legacy image-text datasets and support a new standard in empirical comparison and ablation of generative approaches (Kirstain et al., 2023).

7. Access, Licensing, and Resources

• Dataset v1 is available at https://huggingface.co/datasets/yuvalkirstain/pickapic_v1.
• Dataset v2 (in preparation) to be hosted at https://huggingface.co/datasets/yuvalkirstain/pickapic_v2.
• Model checkpoints: https://huggingface.co/yuvalkirstain/PickScore_v1.
• Source code and detailed documentation: https://github.com/yuvalkirstain/PickScore.

All releases adhere to CC BY 4.0 licensing, include explicit guidance on responsible use, and support standardized, open, and reproducible research practices (Kirstain et al., 2023).