Palette-Text-Image Datasets Overview

• Palette-text-image datasets are multimodal corpora that pair textual descriptions with color structures and images, enabling tasks like text-to-palette generation and image colorization.
• They span distinct regimes from manually curated text–palette annotations to automatically derived color and style supervision, supporting varied creative and synthetic applications.
• Evaluation methods use metrics such as CIEDE2000, EMD, and CLIP similarity to assess semantic accuracy, palette diversity, and overall artistic quality.

Palette-text-image datasets are multimodal corpora in which language is paired with image data and, either explicitly or by derivation, with color structure such as palettes or histograms. In the literature represented here, that category spans at least three distinct regimes: manually curated text–palette supervision for semantic color modeling, automatically derived palette or histogram supervision layered onto large image–text corpora, and adjacent text–image datasets in which style, layout, or artistic rendering play the role that explicit palettes play in color-conditioned generation. Together, these datasets support tasks including text-to-palette generation, palette-guided colorization, palette-conditioned diffusion, poem-to-painting generation, multimodal poster text rendering, and dense long-text image generation (Bahng et al., 2018, Aharoni et al., 2 Sep 2025, Li et al., 2021, Gao et al., 2023, Wang et al., 11 Feb 2025).

1. Scope and dataset regimes

The term “palette-text-image dataset” is not used uniformly across the cited works. In "Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation" (Bahng et al., 2018), the supervision is explicit: a text description is paired with a five-color palette, and that palette can then guide grayscale image colorization. In "Palette Aligned Image Diffusion" (Aharoni et al., 2 Sep 2025), by contrast, the paper does not introduce a manually annotated “palette-text-image” benchmark in the usual sense; it constructs a curated image–text corpus and derives palette and histogram supervision automatically from each image. Paint4Poem, PosterT80K, and TextAtlas5M further broaden the design space by pairing text with paintings, poster regions, backgrounds, or long-form layout-rich images rather than with authored palettes (Li et al., 2021, Gao et al., 2023, Wang et al., 11 Feb 2025).

Dataset	Core paired signals	Scale notes
PAT	text descriptions + five-color palettes	10,183 pairs
Curated dataset for Palette-Adapter	image–text pairs + derived palette/histogram supervision	2.4M images
Paint4Poem	poem-painting, caption-painting, poem-painting	301, 3,648, and 89,204 pairs
PosterT80K	poster background + text content + bounding box	117,624 posters with text
TextAtlas5M	long-text images from mixed real + synthetic subsets	5M images; 3000-sample TextAtlasEval

A plausible implication is that palette-text-image supervision should be understood functionally rather than narrowly. In some datasets, the palette is the primary annotation; in others, color, style, or layout is extracted or inferred from images and used as a conditioning signal. That distinction matters because it determines whether the dataset captures human color semantics directly, image statistics indirectly, or broader multimodal structure.

2. Manually curated text–palette supervision: PAT

PAT, short for Palette-and-Text, was introduced to support a task that the paper defines as mapping rich natural-language text to multiple plausible color palettes and then using those palettes to guide grayscale image colorization (Bahng et al., 2018). The dataset was created by refining palette names crawled from color-hex.com. The raw crawl contained 47,665 palette-text pairs; after cleaning and filtering, the final dataset contained 10,183 pairs. The cleaning process used four annotators, and a pair was kept if at least 3 out of 4 annotators agreed that the text matched the palette. The authors explicitly did not require unanimity because they wanted to preserve subjectivity in color semantics. After annotation, spelling errors and punctuation were manually corrected.

PAT contains 10,183 text–palette pairs, 5 colors per palette, and 4,312 unique words. Each palette color is represented in CIE Lab space, so one palette is represented as $y \in \mathbb{R}^{15}$. The text side supports single words, phrases, and sentences / longer descriptions, including inputs such as “autumn,” “vibrant,” “midsummer to autumn,” and “autumn breeze and falling leaves.” This made PAT broader than earlier color datasets that mapped text to a single color, relied on a small controlled vocabulary, or only supported single-word inputs (Bahng et al., 2018).

PAT is central to the Text2Colors pipeline. The first module, TPN, maps text to a palette; the second, PCN, colorizes a grayscale image using a palette. The pipeline is trained separately rather than end-to-end. The dataset thus supports both semantic color modeling and downstream palette-conditioned image synthesis. Conditioning augmentation from StackGAN is used in TPN to model the fact that one text can map to many valid palettes, which is why the paper emphasizes multimodality rather than one-to-one color prediction (Bahng et al., 2018).

The experimental analysis associated with PAT makes the dataset’s role unusually explicit. Without conditioning augmentation, multimodality is 0; with conditioning augmentation, TPN produces multiple palettes for the same text. In the user study for text-to-palette matching, TPN achieved a 56.2% fooling rate, compared with 39.6% for the Heer and Stone baseline, and the paper reports that people preferred the generated palettes over ground truth palettes. The paper also notes limitations: the dataset reflects the biases of the color-hex.com community, some palette names are poetic while others are literal, unknown words can cause failures, and the dataset remains limited compared with general NLP corpora (Bahng et al., 2018).

3. Derived palette supervision and color-balanced curation

"Palette Aligned Image Diffusion" relocates the palette-text-image problem from manual annotation to automatic extraction and dataset balancing (Aharoni et al., 2 Sep 2025). The paper states that it does not introduce a manually annotated palette-text-image benchmark in the usual sense. Instead, it trains on a curated 2.4M-image dataset built from a 2M-image subset of LAION-Art and 400K additional images from LAION-2B-en. The additional 400K images are sampled to emphasize colors from low-valued histogram bins in LAION-Art.

This curation is motivated by a strong color imbalance. The paper shows that LAION-Art is heavily color-biased: the 100 most common colors account for 87% of the total color population, while the 30 rarest colors account for only 0.008253% in the main figure; the supplementary histogram analysis likewise reports that the top 100 bins account for 87%, while the 100 rarest bins are only 0.0756%. The dataset design therefore treats color-space coverage as a first-order concern rather than an incidental property of image–text pretraining (Aharoni et al., 2 Sep 2025).

For each training image, the model uses three linked signals: the original caption, the full image color histogram, and an extracted palette from the same image. Palettes are extracted with median-cut palette extraction via Pylette, with up to 8 colors per image. The model alternates between conditioning on a sparse palette, a full histogram, or no condition, with equal probability between palette and histogram conditions during training, plus a condition-type scalar. This is intended to teach the model that a palette is a coarse, sparse summary of the full image color distribution rather than a strict pixel-level target (Aharoni et al., 2 Sep 2025).

A central formal move is to reinterpret a palette as a sparse histogram. The paper discretizes HSV space into bins $(34, 12, 10)$. The image histogram is formed by projecting pixels into those bins; the palette histogram is formed by projecting each palette color into the corresponding bin with uniform probabilities over selected colors. The paper also introduces two scalar controls for palette ambiguity: histogram entropy,

$Entropy(H)=-\sum_{i=1}^{\#bins} h_i\,log_2\,h_i,$

and palette-to-histogram distance, measured with the Quadratic-Chi Histogram Distance using clipped and sharpened CIEDE2000 as the base color distance. A further mechanism allows a negative palette under CFG:

$$\hat{\epsilon}_{\theta}(x_{t},c^\pm ,t) = w\epsilon_{\theta}(x_{t},c^+,t)+(1-w)\epsilon_{\theta}(x_{t},c^-,t),$$

where $c^+$ is the positive palette and $c^-$ is the negative palette (Aharoni et al., 2 Sep 2025).

The evaluation protocol also reflects the dataset philosophy. Testing uses COCO 2017 val, filtered from 5000 images to the first 1000 captions that do not mention colors, so that palette conditioning is not confounded by explicit color words. Metrics include EMD, MusiQ, and CLIP similarity, and the user study compares against 40 designer-extracted palettes from PicMonkey with color-neutral prompts. This suggests that, in derived palette supervision, dataset design is inseparable from the problem of disentangling text semantics from explicit color language (Aharoni et al., 2 Sep 2025).

4. Artistic text–image corpora as color and style analogues

Paint4Poem extends the discussion from palettes to artistic style and semantic grounding in poem-to-image generation (Li et al., 2021). The paper introduces the task of artistic visualization of classical Chinese poems, where the goal is to generate paintings of a certain artistic style for classical Chinese poems, especially the style of Feng Zikai. It states that this is the first dataset for artistic visualization of classical Chinese poems.

Paint4Poem is split into three parts. Zikai-Poem contains 301 manually collected high-quality poem-painting pairs from Feng Zikai, with a split of 256 training pairs and 75 test pairs and metadata including PoemID, PoemText, PoemTitle, PoemDynasty, PoemAuthor, Explanation, Commentary, and PaintingID. Zikai-Caption contains 3,648 caption-painting pairs, manually collected from Feng Zikai’s paintings; captions are typically 2–3 words or short Chinese phrases. TCP-Poem contains 89,204 poem-painting pairs, collected automatically from the web in traditional Chinese painting style. The latter is built by collecting poem segments, searching with Baidu using the poem plus the keyword “traditional Chinese painting,” downloading the top 10 candidate images, and then selecting the image most likely to be in traditional Chinese painting style using a classifier based on VGG-19, Gram matrix style features, PCA, SVM, and binary cross-entropy loss; the reported classifier accuracy is 96.7% (Li et al., 2021).

The split is explicitly functional. Zikai-Caption is intended to help the model learn Feng Zikai’s artistic style, because it contains many more paintings by the artist than Zikai-Poem. TCP-Poem is intended to help the model learn semantic alignment between poems and paintings at scale. The dataset analysis reinforces that division. Zikai-Poem covers 162 poets, spans all ancient Chinese dynasties, and covers 82% of the 1024 Shi imageries under the ShiXueHanYing taxonomy. Style similarity to Zikai-Poem, measured by GE and LP, is 0.83 / 0.58 for Zikai-Poem, 0.64 / 0.57 for Zikai-Caption, and 0.45 / 0.49 for TCP-Poem. Feng Zikai’s style is described as succinct stroking, restrained coloring, and empty-emphasizing composition (Li et al., 2021).

The most important dataset finding is that semantic relevance is difficult. For 90% of paintings in Zikai-Poem, less than 20% of the poem’s imageries are represented in the painting. In a manual analysis of 24 TCP-Poem pairs, only 5 were semantically relevant. The benchmark with AttnGAN and MirrorGAN shows that the models can generate paintings with good pictorial quality and mimic Feng Zikai’s style, but the reflection of poem semantics is limited. This suggests that in artistic text–image datasets, stylistic relevance and semantic completeness can diverge sharply (Li et al., 2021).

5. Poster and dense-text datasets beyond explicit palettes

PosterT80K and TextAtlas5M are not palette datasets in the narrow sense, but they are part of the same broader trajectory toward richer multimodal conditioning (Gao et al., 2023, Wang et al., 11 Feb 2025). PosterT80K was constructed for Text Image Generation for Posters (TIGER), where the task is to generate a rendered text region conditioned on the poster background image, the text content, and the text position / bounding box. The raw collection contains 165,494 poster images from Chinese e-commerce websites; after filtering out posters without text, 117,624 images remain, split into 106,009 training posters and 11,615 test posters. The appendix further reports 148,891 text images in training and 16,603 text images in test. Each poster image is collected or resized at 513 × 750. Annotation is lightweight and sentence-level: text string $c$ and bounding box $p=(x,y,w,h)$. Background images are created by removing text using Self-Supervised Text Erasing, yielding paired original poster, erased background, text content, and bounding box (Gao et al., 2023).

PosterT80K is explicitly designed for weak supervision, because detailed style attributes such as font, color, and outline are difficult to annotate reliably. The dataset supports style learning from the overall poster background and a position-conditioned local background, and semantic guidance from text encoded with Chinese-CLIP. The paper evaluates on FID, SSIM, and PSNR, and notes that erased-background artifacts have only a slight impact on performance. A plausible implication is that, for poster design, palette-like appearance control emerges from background context rather than from separate color annotations (Gao et al., 2023).

TextAtlas5M addresses a different failure mode: existing text-to-image datasets are not sufficient for long-text rendering (Wang et al., 11 Feb 2025). It contains 5 million images from 10 subsets spanning synthetic and real data, including CleanTextSynth, TextVisionBlend, StyledTextSynth, PPT2Details, PPT2Structured, Paper2Text, CoverBook, LongWordsSubset-A, LongWordsSubset-M, and TextScenesHQ. The paper reports an average OCR-derived token length of 148.82, compared with 26.36 for TextCaps, 13.75 for SynthText, 16.13 for Marion10M, 9.92 for AnyWords3M, and 21.21 for RenderedText. It also introduces TextAtlasEval, a 3000-sample human-improved test set across 3 data domains: StyledTextSynth, TextScenesHQ, and TextVisionBlend (Wang et al., 11 Feb 2025).

The relevance to palette-text-image datasets lies in multimodal specification. TextAtlas5M integrates scene description, rendered text, layout, OCR-based filtering, and, in some subsets, bounding boxes and image captions. The evaluation uses FID, CLIP Score, OCR Accuracy, OCR F1, and CER, and the paper reports that the benchmark is challenging even for GPT-4o + DALL·E 3. This broadens the field from palette adherence to the more general problem of controlling visual generation with structured textual and appearance constraints (Wang et al., 11 Feb 2025).

6. Evaluation, misconceptions, and dataset design implications

Across these datasets, evaluation protocols reveal what each corpus is intended to teach. PAT emphasizes semantic color matching, palette diversity, and multimodality using CIEDE2000-based measures and user studies (Bahng et al., 2018). The Palette-Adapter work emphasizes EMD for color alignment, MusiQ for no-reference quality, and CLIP similarity for prompt alignment, with filtered color-neutral prompts to isolate palette conditioning (Aharoni et al., 2 Sep 2025). Paint4Poem evaluates pictorial quality, stylistic relevance, and semantic relevance using Inception Score, Precision@1, GE, and LP, plus a human study with 5 subjects and a task difficulty rating of 9/10 (Li et al., 2021). PosterT80K uses FID, SSIM, and PSNR (Gao et al., 2023), while TextAtlasEval uses FID, CLIP Score, OCR Accuracy, OCR F1, and CER (Wang et al., 11 Feb 2025).

A common misconception is that palette-text-image datasets are necessarily manually annotated at scale. The cited work shows three alternatives. PAT is manually curated and explicitly semantic (Bahng et al., 2018). The Palette-Adapter paper derives palette and histogram supervision automatically from image–text data and compensates with color-balanced sampling (Aharoni et al., 2 Sep 2025). PosterT80K relies on weak supervision with erased backgrounds and sentence-level boxes rather than explicit color labels (Gao et al., 2023). Paint4Poem combines small curated target data with larger, noisier auxiliary data (Li et al., 2021). TextAtlas5M mixes synthetic generation, real-image collection, OCR filtering, LLM captioning, and human refinement (Wang et al., 11 Feb 2025).

Several limitations recur. PAT inherits the subjectivity and vocabulary limits of crowd-created palette names, and unknown words cause failures (Bahng et al., 2018). Palette-conditioned diffusion faces ambiguity and instability because a palette alone is underdetermined; the paper’s entropy and palette-to-histogram distance controls are direct responses to that problem (Aharoni et al., 2 Sep 2025). Paint4Poem shows that stylistic resemblance can be learned even when semantic reflection remains limited (Li et al., 2021). PosterT80K depends on text-erasing quality and lightweight annotation (Gao et al., 2023). TextAtlas5M shows that long-text rendering still suffers from duplicated words, missing letters, incomplete text, layout mismatches, poor interleaving, weak low-contrast performance, and poor handling of artistic fonts (Wang et al., 11 Feb 2025).

The datasets also imply several design principles. One is to balance the color distribution deliberately and include rare colors intentionally (Aharoni et al., 2 Sep 2025). Another is to separate style learning from semantic alignment when the target data are small or highly curated (Li et al., 2021). A third is to use color-neutral prompts or filtered captions when evaluating palette adherence (Aharoni et al., 2 Sep 2025). A fourth is that multimodal generation datasets increasingly treat appearance, layout, and dense text as joint control variables rather than independent annotations (Gao et al., 2023, Wang et al., 11 Feb 2025). Taken together, these works show that palette-text-image datasets have evolved from compact human-authored text–palette resources into broader, heterogeneous multimodal corpora in which color, style, and layout are all learned as controllable structure.