Pick-a-Pic: Open Dataset for Text-to-Image Preferences

• Pick-a-Pic is an open dataset capturing genuine user preferences through iterative pairwise image comparisons with natural, diverse prompts.
• It employs a unique web app interface with tie options to reflect authentic user intent and overcome limitations of fixed evaluation rubrics.
• Paired with PickScore, a CLIP-based predictor, the framework outperforms traditional metrics like FID in aligning evaluations with human judgments.

Searching arXiv for recent and directly relevant papers on Pick-a-Pic and related text-to-image preference evaluation. Pick-a-Pic is an open dataset of human preferences for text-to-image generation built from real text-to-image users interacting with a public web app rather than from paid crowd workers following a fixed rubric. Each example contains a text prompt, two generated images conditioned on that prompt, and a preference label indicating which image the user preferred, or a tie if neither image was clearly better. The dataset was introduced together with PickScore, a CLIP-based scoring function trained to predict these preferences, and was positioned as infrastructure for preference modeling, evaluation, and reranking in modern text-to-image systems (Kirstain et al., 2023).

1. Origin and conceptual scope

Pick-a-Pic was created to address a specific gap in text-to-image research: the lack of a large, public dataset of genuine user preferences over modern model outputs. The paper frames this as a structural limitation of the field, since the ability to collect such data is usually restricted to companies. In response, the authors built a web app through which users could both generate images and express their preferences, thereby producing a large, open dataset of prompts and pairwise judgments (Kirstain et al., 2023).

A defining property of the dataset is that its labels are intended to reflect “natural” preferences. The authors distinguish these judgments from artificial annotations collected under externally imposed evaluation rubrics. In Pick-a-Pic, users had a real intent, wrote their own prompts, and selected images in pursuit of outputs they actually wanted. This makes the dataset not merely a benchmark of image quality, but a record of intrinsically motivated interaction with text-to-image systems (Kirstain et al., 2023).

This design also makes Pick-a-Pic relevant to multiple subproblems at once. It is a source of prompts, a source of pairwise preferences, a basis for training a reward-like scoring model, and a testbed for comparing automatic evaluation metrics against human rankings. A plausible implication is that the dataset occupies a similar role for text-to-image preference alignment that instruction and preference corpora occupy for LLM reward modeling, although the paper itself restricts its concrete claims to text-to-image generation and evaluation.

2. Data collection through the web app

The Pick-a-Pic web app implements a repeated pairwise-comparison loop. A user writes a prompt, the system generates two images for that prompt, the user picks the preferred image or selects a tie, the rejected image is replaced with a fresh generated image, and the loop continues until the prompt is edited or cleared (Kirstain et al., 2023). Because the same prompt can remain active across multiple iterations, the interface yields many pairwise comparisons for a single prompt over time.

The collection protocol was not fixed from the outset. During piloting, the authors experimented with three annotation interfaces: four images without ties, two images without ties, and two images with ties. They report that the two-image interface with ties gave the best engagement and inter-rater agreement, and it therefore became the final collection format (Kirstain et al., 2023). The presence of an explicit tie label is consequential: it preserves cases in which the user does not meaningfully prefer one image over the other rather than forcing an arbitrary binary choice.

To attract real users, the app was promoted through Twitter, Facebook, Discord, and Reddit. The paper also emphasizes safety and quality-control measures, including mandatory authentication via Gmail or Discord, explicit consent to share interactions in a public dataset, anonymization of user IDs, bans for NSFW generation, simultaneous multi-instance abuse, or suspiciously fast judgments, NSFW phrase filtering, and per-user interaction limits that were initially capped at 1000 and later increased (Kirstain et al., 2023).

The dataset contains examples produced by multiple image-generation backbones: Stable Diffusion 2.1, Dreamlike Photoreal 2.0, and Stable Diffusion XL variants. The authors further note that these models were sampled using different classifier-free guidance scales, so the resulting preferences reflect both model-level and decoding-level variation (Kirstain et al., 2023).

3. Dataset structure and statistical profile

At the level of an individual record, Pick-a-Pic consists of a prompt, two generated images, and one of three labels: image 1 preferred, image 2 preferred, or tie (Kirstain et al., 2023). The paper stresses that these are pairwise preference signals rather than absolute ratings.

The reported corpus statistics are given at two levels. Across the app as a whole, the paper reports 968,965 rankings collected overall, 66,798 prompts, and 6,394 users. For the version used in the paper’s experiments, after NSFW filtering, the dataset contained 583,747 training examples, 500 validation examples, 500 test examples, 37,523 training prompts, and 4,375 distinct users in training. The split is prompt-disjoint: no prompt appears across train, validation, and test. To create validation and test, the authors first sampled 1,000 prompts, each from a unique user, then split those prompts between the two sets; all remaining examples whose prompts were not in validation or test were assigned to training (Kirstain et al., 2023).

The paper also describes Pick-a-Pic more broadly as containing over 500,000 examples in the version used in the paper, with 35,000 distinct prompts, and notes that the dataset is updated over time and that a newer version later surpassed one million examples (Kirstain et al., 2023). These multiple counts reflect different snapshots of the resource.

Prompt diversity is central to the paper’s argument for the dataset’s value. The appendix contrasts Pick-a-Pic prompts with MS-COCO captions. Examples from Pick-a-Pic include “forest with ruins, photo,” “A panda bear as a mad scientist,” “A galactic eldritch squid towering over the planet Earth, stars, galaxies and nebulas in the background...,” and “Giant ice cream cone melting and creating a river through a city,” whereas MS-COCO examples are described as mundane everyday photo captions such as “A man riding a bike past a train traveling along tracks” (Kirstain et al., 2023). The authors use this contrast to argue that Pick-a-Pic is more representative of real text-to-image use.

4. PickScore: preference prediction model

PickScore is the scoring model trained on Pick-a-Pic to predict which of two images a user would prefer for a given prompt. Architecturally, it follows a CLIP-like design: a transformer text encoder maps the prompt $x$ to a vector, a transformer image encoder maps image $y$ to a vector, and the score is a scaled inner product,

$$s(x,y) = T \cdot E_{\text{txt}}(x) \cdot E_{\text{img}}(y),$$

with a learned temperature parameter $T$ (Kirstain et al., 2023).

Training uses a preference-matching objective inspired by the reward model in InstructGPT. For a prompt $x$, images $y_1$ and $y_2$, and preference distribution $p$, the labels are encoded as $p=[1,0]$ if $y_1$ is preferred, $y$0 if $y$1 is preferred, and $y$2 for a tie. The model predicts

$y$3

and minimizes

$y$4

Because multiple examples can share a prompt, the losses are weighted inversely by prompt frequency so that repeated prompts do not dominate optimization (Kirstain et al., 2023).

The paper reports that the authors fine-tuned CLIP-H for 4,000 steps using learning rate $y$5, batch size 128, 500-step warmup, linear decay, and 8 A100 GPUs, with total training time under an hour. Model selection used validation accuracy without ties, measured every 100 steps (Kirstain et al., 2023). They also tried adding in-batch negatives in a CLIP-like way, but report that this hurt performance; the appendix gives 65.2% test accuracy for that alternative objective, below the main method (Kirstain et al., 2023).

The tie-aware evaluation metric awards 1 point for matching the user label, 0.5 points if one prediction is tie and the other is not, and 0 points otherwise. Under this metric, the reported preference-prediction results are: Random 56.8%, Aesthetics predictor 56.8%, CLIP-H 60.8%, ImageReward 61.1%, HPS 66.7%, Human expert annotators 68.0%, and PickScore 70.5% (Kirstain et al., 2023). The paper describes this as “superhuman” performance, but immediately qualifies the statement: PickScore is better than the third-party annotators used in evaluation, not better than the original users whose preferences define the labels (Kirstain et al., 2023). This qualification addresses a likely misconception about the meaning of the term.

5. Evaluation role and comparison with existing metrics

A major claim of the paper is that PickScore correlates better with human judgments than other automatic metrics and should therefore be used for future text-to-image evaluation (Kirstain et al., 2023). The recommendation has two components. First, evaluation should use prompts from Pick-a-Pic rather than MS-COCO because the former reflect actual text-to-image usage. Second, evaluation should use PickScore rather than alternatives such as CLIP-H, aesthetics scores, ImageReward, HPS, or FID (Kirstain et al., 2023).

The paper is especially critical of MS-COCO as a prompt source. MS-COCO captions are descriptions of ordinary photographs, whereas contemporary text-to-image usage often involves imagination, stylization, surreal composition, or fictional content. The paper argues that this mismatch makes Pick-a-Pic prompts a more realistic benchmark (Kirstain et al., 2023).

The comparison with FID is one of the strongest empirical points in the paper. In a human evaluation study using 100 MS-COCO captions and outputs from nine model/configuration variants—Stable Diffusion 1.5, Stable Diffusion 2.1, and Dreamlike Photoreal 2.0, each with classifier-free guidance scales 3, 6, and 9—the correlation between PickScore rankings and human rankings is reported as 0.917, whereas the correlation between FID and human rankings is reported as $y$6 (Kirstain et al., 2023). The paper interprets this negative correlation as evidence that FID can be misleading for text-to-image model selection because it measures similarity to a distribution of real images and ignores the prompt.

For metrics that do not rely on ground-truth images, the paper evaluates ranking quality on the Pick-a-Pic test set using real user preferences as ground truth. Over 45 model variants and 14,000 collected preferences, the Elo-based correlations are reported as PickScore $y$7, HPS $y$8, ImageReward $y$9, and CLIP-H $$s(x,y) = T \cdot E_{\text{txt}}(x) \cdot E_{\text{img}}(y),$$0 (Kirstain et al., 2023). These results motivate the authors’ recommendation that PickScore become the default automatic metric for future text-to-image model evaluation.

6. Reranking, practical use, and broader significance

PickScore is presented not only as an evaluator but also as a practical reranker for improving outputs from existing text-to-image models. The reranking protocol is simple: generate many candidates for each prompt, score them with PickScore, and choose the highest-scoring image (Kirstain et al., 2023).

In the reported experiment, the authors used 100 prompts from Pick-a-Pic and generated 100 images per prompt with Dreamlike Photoreal 2.0 at CFG 7.5. These 100 candidates combined 5 random seeds with 20 prompt templates, including a null template $$s(x,y) = T \cdot E_{\text{txt}}(x) \cdot E_{\text{img}}(y),$$1 and prompt-expanding templates such as “breathtaking [prompt]. award-winning, professional, highly detailed” (Kirstain et al., 2023). The image selected by PickScore was then compared with images selected by CLIP-H, an aesthetics predictor, random choice, and a null-template control.

The reported human win rates for the PickScore-selected image are 71.4% versus random seed plus null template, 82.0% versus random seed plus random template, 85.1% versus the aesthetics predictor, and 71.3% versus CLIP-H (Kirstain et al., 2023). These results support the paper’s claim that PickScore is useful not only for offline evaluation but also for practical post hoc selection.

Taken together, Pick-a-Pic and PickScore are presented as a foundation for evaluation and alignment in text-to-image generation (Kirstain et al., 2023). The dataset contributes open access to real user preferences, prompt-disjoint train/validation/test splits, and a prompt distribution that differs markedly from legacy caption corpora. PickScore contributes a preference model trained directly on those signals and shown to outperform prior automatic metrics on preference prediction and ranking correlation. This suggests a shift from evaluation based on proxy image-distribution similarity toward evaluation grounded in user preference data.