---
title: Palette-Text-Image Datasets Overview
url: https://www.emergentmind.com/topics/palette-text-image-datasets
type: topic
---

# Palette-Text-Image Datasets Overview

Palette-text-image datasets are multimodal corpora in which language is paired with image data and, either explicitly or by derivation, with color structure such as palettes or histograms. In the literature represented here, that category spans at least three distinct regimes: manually curated text–palette supervision for semantic color modeling, automatically derived palette or histogram supervision layered onto large image–text corpora, and adjacent text–image datasets in which style, layout, or artistic rendering play the role that explicit palettes play in color-conditioned generation. Together, these datasets support tasks including text-to-palette generation, palette-guided colorization, palette-conditioned diffusion, poem-to-painting generation, multimodal poster text rendering, and dense long-text image generation [1804.04128] [2509.02000] [2109.11682] [2308.04733] [2502.07870].

## 1. Scope and dataset regimes

The term “palette-text-image dataset” is not used uniformly across the cited works. In "Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation" [1804.04128], the supervision is explicit: a text description is paired with a five-color palette, and that palette can then guide grayscale image colorization. In "Palette Aligned Image Diffusion" [2509.02000], by contrast, the paper does **not** introduce a manually annotated “palette-text-image” benchmark in the usual sense; it constructs a curated image–text corpus and derives palette and histogram supervision automatically from each image. Paint4Poem, PosterT80K, and TextAtlas5M further broaden the design space by pairing text with paintings, poster regions, backgrounds, or long-form layout-rich images rather than with authored palettes [2109.11682] [2308.04733] [2502.07870].

| Dataset | Core paired signals | Scale notes |
|---|---|---|
| PAT | text descriptions + five-color palettes | 10,183 pairs |
| Curated dataset for Palette-Adapter | image–text pairs + derived palette/histogram supervision | 2.4M images |
| Paint4Poem | poem-painting, caption-painting, poem-painting | 301, 3,648, and 89,204 pairs |
| PosterT80K | poster background + text content + bounding box | 117,624 posters with text |
| TextAtlas5M | long-text images from mixed real + synthetic subsets | 5M images; 3000-sample TextAtlasEval |

A plausible implication is that palette-text-image supervision should be understood functionally rather than narrowly. In some datasets, the palette is the primary annotation; in others, color, style, or layout is extracted or inferred from images and used as a conditioning signal. That distinction matters because it determines whether the dataset captures human color semantics directly, image statistics indirectly, or broader multimodal structure.

## 2. Manually curated text–palette supervision: PAT

PAT, short for Palette-and-Text, was introduced to support a task that the paper defines as mapping rich natural-language text to multiple plausible color palettes and then using those palettes to guide grayscale image colorization [1804.04128]. The dataset was created by refining palette names crawled from color-hex.com. The raw crawl contained **47,665 palette-text pairs**; after cleaning and filtering, the final dataset contained **10,183 pairs**. The cleaning process used **four annotators**, and a pair was kept if **at least 3 out of 4 annotators** agreed that the text matched the palette. The authors explicitly did not require unanimity because they wanted to preserve subjectivity in color semantics. After annotation, spelling errors and punctuation were manually corrected.

PAT contains **10,183 text–palette pairs**, **5 colors per palette**, and **4,312 unique words**. Each palette color is represented in **CIE Lab space**, so one palette is represented as $y \in \mathbb{R}^{15}$. The text side supports **single words**, **phrases**, and **sentences / longer descriptions**, including inputs such as “autumn,” “vibrant,” “midsummer to autumn,” and “autumn breeze and falling leaves.” This made PAT broader than earlier color datasets that mapped text to a single color, relied on a small controlled vocabulary, or only supported single-word inputs [1804.04128].

PAT is central to the Text2Colors pipeline. The first module, TPN, maps text to a palette; the second, PCN, colorizes a grayscale image using a palette. The pipeline is trained separately rather than end-to-end. The dataset thus supports both semantic color modeling and downstream palette-conditioned image synthesis. Conditioning augmentation from StackGAN is used in TPN to model the fact that one text can map to many valid palettes, which is why the paper emphasizes multimodality rather than one-to-one color prediction [1804.04128].

The experimental analysis associated with PAT makes the dataset’s role unusually explicit. Without conditioning augmentation, multimodality is **0**; with conditioning augmentation, TPN produces **multiple palettes for the same text**. In the user study for text-to-palette matching, TPN achieved a **56.2%** fooling rate, compared with **39.6%** for the Heer and Stone baseline, and the paper reports that people preferred the generated palettes over ground truth palettes. The paper also notes limitations: the dataset reflects the biases of the color-hex.com community, some palette names are poetic while others are literal, unknown words can cause failures, and the dataset remains limited compared with general NLP corpora [1804.04128].

## 3. Derived palette supervision and color-balanced curation

"Palette Aligned Image Diffusion" relocates the palette-text-image problem from manual annotation to automatic extraction and dataset balancing [2509.02000]. The paper states that it does **not** introduce a manually annotated palette-text-image benchmark in the usual sense. Instead, it trains on a **curated 2.4M-image dataset** built from a **2M-image subset of LAION-Art** and **400K additional images from LAION-2B-en**. The additional 400K images are sampled to emphasize colors from low-valued histogram bins in LAION-Art.

This curation is motivated by a strong color imbalance. The paper shows that LAION-Art is heavily color-biased: the **100 most common colors account for 87%** of the total color population, while the **30 rarest colors account for only 0.008253%** in the main figure; the supplementary histogram analysis likewise reports that the **top 100 bins account for 87%**, while the **100 rarest bins are only 0.0756%**. The dataset design therefore treats color-space coverage as a first-order concern rather than an incidental property of image–text pretraining [2509.02000].

For each training image, the model uses three linked signals: the original caption, the full image color histogram, and an extracted palette from the same image. Palettes are extracted with **median-cut palette extraction** via **Pylette**, with **up to 8 colors** per image. The model alternates between conditioning on a **sparse palette**, a **full histogram**, or **no condition**, with equal probability between palette and histogram conditions during training, plus a condition-type scalar. This is intended to teach the model that a palette is a coarse, sparse summary of the full image color distribution rather than a strict pixel-level target [2509.02000].

A central formal move is to reinterpret a palette as a sparse histogram. The paper discretizes HSV space into bins $(34, 12, 10)$. The image histogram is formed by projecting pixels into those bins; the palette histogram is formed by projecting each palette color into the corresponding bin with uniform probabilities over selected colors. The paper also introduces two scalar controls for palette ambiguity: histogram entropy,
$$
Entropy(H)=-\sum_{i=1}^{\#bins} h_i\,log_2\,h_i,
$$
and palette-to-histogram distance, measured with the **Quadratic-Chi Histogram Distance** using **clipped and sharpened CIEDE2000** as the base color distance. A further mechanism allows a negative palette under CFG:
$$
\hat{\epsilon}_{\theta}(x_{t},c^\pm ,t) = w\epsilon_{\theta}(x_{t},c^+,t)+(1-w)\epsilon_{\theta}(x_{t},c^-,t),
$$
where $c^+$ is the positive palette and $c^-$ is the negative palette [2509.02000].

The evaluation protocol also reflects the dataset philosophy. Testing uses **COCO 2017 val**, filtered from 5000 images to the **first 1000** captions that **do not mention colors**, so that palette conditioning is not confounded by explicit color words. Metrics include **EMD**, **MusiQ**, and **CLIP similarity**, and the user study compares against **40 designer-extracted palettes** from PicMonkey with color-neutral prompts. This suggests that, in derived palette supervision, dataset design is inseparable from the problem of disentangling text semantics from explicit color language [2509.02000].

## 4. Artistic text–image corpora as color and style analogues

Paint4Poem extends the discussion from palettes to artistic style and semantic grounding in poem-to-image generation [2109.11682]. The paper introduces the task of **artistic visualization of classical Chinese poems**, where the goal is to generate paintings of a certain artistic style for classical Chinese poems, especially the style of **Feng Zikai**. It states that this is the **first dataset for artistic visualization of classical Chinese poems**.

Paint4Poem is split into three parts. **Zikai-Poem** contains **301 manually collected high-quality poem-painting pairs** from Feng Zikai, with a split of **256 training pairs** and **75 test pairs** and metadata including PoemID, PoemText, PoemTitle, PoemDynasty, PoemAuthor, Explanation, Commentary, and PaintingID. **Zikai-Caption** contains **3,648 caption-painting pairs**, manually collected from Feng Zikai’s paintings; captions are typically **2–3 words** or short Chinese phrases. **TCP-Poem** contains **89,204 poem-painting pairs**, collected automatically from the web in traditional Chinese painting style. The latter is built by collecting poem segments, searching with Baidu using the poem plus the keyword “traditional Chinese painting,” downloading the top 10 candidate images, and then selecting the image most likely to be in traditional Chinese painting style using a classifier based on **VGG-19**, **Gram matrix** style features, **PCA**, **SVM**, and binary cross-entropy loss; the reported classifier accuracy is **96.7%** [2109.11682].

The split is explicitly functional. **Zikai-Caption** is intended to help the model learn **Feng Zikai’s artistic style**, because it contains many more paintings by the artist than Zikai-Poem. **TCP-Poem** is intended to help the model learn **semantic alignment between poems and paintings** at scale. The dataset analysis reinforces that division. Zikai-Poem covers **162 poets**, spans all ancient Chinese dynasties, and covers **82% of the 1024 Shi imageries** under the ShiXueHanYing taxonomy. Style similarity to Zikai-Poem, measured by **GE** and **LP**, is **0.83 / 0.58** for Zikai-Poem, **0.64 / 0.57** for Zikai-Caption, and **0.45 / 0.49** for TCP-Poem. Feng Zikai’s style is described as **succinct stroking**, **restrained coloring**, and **empty-emphasizing composition** [2109.11682].

The most important dataset finding is that semantic relevance is difficult. For **90% of paintings** in Zikai-Poem, less than **20%** of the poem’s imageries are represented in the painting. In a manual analysis of **24** TCP-Poem pairs, only **5** were semantically relevant. The benchmark with **AttnGAN** and **MirrorGAN** shows that the models can generate paintings with good pictorial quality and mimic Feng Zikai’s style, but the reflection of poem semantics is limited. This suggests that in artistic text–image datasets, stylistic relevance and semantic completeness can diverge sharply [2109.11682].

## 5. Poster and dense-text datasets beyond explicit palettes

PosterT80K and TextAtlas5M are not palette datasets in the narrow sense, but they are part of the same broader trajectory toward richer multimodal conditioning [2308.04733] [2502.07870]. PosterT80K was constructed for **Text Image Generation for Posters (TIGER)**, where the task is to generate a rendered text region conditioned on the **poster background image**, the **text content**, and the **text position / bounding box**. The raw collection contains **165,494** poster images from Chinese e-commerce websites; after filtering out posters without text, **117,624** images remain, split into **106,009 training posters** and **11,615 test posters**. The appendix further reports **148,891 text images** in training and **16,603 text images** in test. Each poster image is collected or resized at **513 × 750**. Annotation is lightweight and sentence-level: text string $c$ and bounding box $p=(x,y,w,h)$. Background images are created by removing text using **Self-Supervised Text Erasing**, yielding paired original poster, erased background, text content, and bounding box [2308.04733].

PosterT80K is explicitly designed for **weak supervision**, because detailed style attributes such as font, color, and outline are difficult to annotate reliably. The dataset supports style learning from the overall poster background and a position-conditioned local background, and semantic guidance from text encoded with **Chinese-CLIP**. The paper evaluates on **FID**, **SSIM**, and **PSNR**, and notes that erased-background artifacts have only a slight impact on performance. A plausible implication is that, for poster design, palette-like appearance control emerges from background context rather than from separate color annotations [2308.04733].

TextAtlas5M addresses a different failure mode: existing text-to-image datasets are not sufficient for **long-text rendering** [2502.07870]. It contains **5 million** images from **10 subsets** spanning synthetic and real data, including CleanTextSynth, TextVisionBlend, StyledTextSynth, PPT2Details, PPT2Structured, Paper2Text, CoverBook, LongWordsSubset-A, LongWordsSubset-M, and TextScenesHQ. The paper reports an average OCR-derived token length of **148.82**, compared with **26.36** for TextCaps, **13.75** for SynthText, **16.13** for Marion10M, **9.92** for AnyWords3M, and **21.21** for RenderedText. It also introduces **TextAtlasEval**, a **3000-sample** human-improved test set across **3 data domains**: StyledTextSynth, TextScenesHQ, and TextVisionBlend [2502.07870].

The relevance to palette-text-image datasets lies in multimodal specification. TextAtlas5M integrates scene description, rendered text, layout, OCR-based filtering, and, in some subsets, bounding boxes and image captions. The evaluation uses **FID**, **CLIP Score**, **OCR Accuracy**, **OCR F1**, and **CER**, and the paper reports that the benchmark is challenging even for **GPT-4o + DALL·E 3**. This broadens the field from palette adherence to the more general problem of controlling visual generation with structured textual and appearance constraints [2502.07870].

## 6. Evaluation, misconceptions, and dataset design implications

Across these datasets, evaluation protocols reveal what each corpus is intended to teach. PAT emphasizes semantic color matching, palette diversity, and multimodality using **CIEDE2000**-based measures and user studies [1804.04128]. The Palette-Adapter work emphasizes **EMD** for color alignment, **MusiQ** for no-reference quality, and **CLIP similarity** for prompt alignment, with filtered color-neutral prompts to isolate palette conditioning [2509.02000]. Paint4Poem evaluates **pictorial quality**, **stylistic relevance**, and **semantic relevance** using **Inception Score**, **Precision@1**, **GE**, and **LP**, plus a human study with **5 subjects** and a task difficulty rating of **9/10** [2109.11682]. PosterT80K uses **FID**, **SSIM**, and **PSNR** [2308.04733], while TextAtlasEval uses **FID**, **CLIP Score**, **OCR Accuracy**, **OCR F1**, and **CER** [2502.07870].

A common misconception is that palette-text-image datasets are necessarily manually annotated at scale. The cited work shows three alternatives. PAT is manually curated and explicitly semantic [1804.04128]. The Palette-Adapter paper derives palette and histogram supervision automatically from image–text data and compensates with color-balanced sampling [2509.02000]. PosterT80K relies on weak supervision with erased backgrounds and sentence-level boxes rather than explicit color labels [2308.04733]. Paint4Poem combines small curated target data with larger, noisier auxiliary data [2109.11682]. TextAtlas5M mixes synthetic generation, real-image collection, OCR filtering, LLM captioning, and human refinement [2502.07870].

Several limitations recur. PAT inherits the subjectivity and vocabulary limits of crowd-created palette names, and unknown words cause failures [1804.04128]. Palette-conditioned diffusion faces ambiguity and instability because a palette alone is underdetermined; the paper’s entropy and palette-to-histogram distance controls are direct responses to that problem [2509.02000]. Paint4Poem shows that stylistic resemblance can be learned even when semantic reflection remains limited [2109.11682]. PosterT80K depends on text-erasing quality and lightweight annotation [2308.04733]. TextAtlas5M shows that long-text rendering still suffers from duplicated words, missing letters, incomplete text, layout mismatches, poor interleaving, weak low-contrast performance, and poor handling of artistic fonts [2502.07870].

The datasets also imply several design principles. One is to balance the color distribution deliberately and include rare colors intentionally [2509.02000]. Another is to separate style learning from semantic alignment when the target data are small or highly curated [2109.11682]. A third is to use color-neutral prompts or filtered captions when evaluating palette adherence [2509.02000]. A fourth is that multimodal generation datasets increasingly treat appearance, layout, and dense text as joint control variables rather than independent annotations [2308.04733] [2502.07870]. Taken together, these works show that palette-text-image datasets have evolved from compact human-authored text–palette resources into broader, heterogeneous multimodal corpora in which color, style, and layout are all learned as controllable structure.

Source: https://www.emergentmind.com/topics/palette-text-image-datasets