---
title: 'PAlt: Pinterest Alt-Text Dataset'
url: https://www.emergentmind.com/topics/palt
type: topic
---

# PAlt: Pinterest Alt-Text Dataset

Searching arXiv for relevant papers on “PAlt”.
Using the arXiv search tool to verify the primary source and related context.
PAlt is a Pinterest-domain alt-text dataset introduced alongside TAlt in the paper "MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation" [2510.00647]. It was created to support context-sensitive alt-text generation, where a model must produce concise, context-relevant descriptions of images rather than generic long-form captions. In the paper’s design, PAlt is not a standalone artifact but one branch of a paired dataset construction spanning two social-media domains: Twitter through TAlt and Pinterest through PAlt. Its distinctive role is to provide cleaned supervised triplets, verified preference pairs, and a held-out evaluation set for multimodal preference optimization in a setting where user-written alt-text is scarce, noisy, and strongly conditioned on surrounding post context [2510.00647].

## 1. Definition, scope, and motivation

PAlt serves three functions in the experimental pipeline: a supervised dataset for initial fine-tuning, a preference dataset for preference optimization, and a test set for evaluation. The paper motivates its construction through three linked observations. First, existing alt-text data is scarce; for Pinterest specifically, only **0.68%** of encountered samples included alt-text. Second, user-provided alt-text is noisy, because annotation standards on social media are inconsistent in quality, completeness, grammar, and salience selection. Third, alt-text is context-sensitive in a way ordinary image captions are not: it should be concise and grounded in both the image and the surrounding post-text, while avoiding merely repeating that context [2510.00647].

These design premises distinguish PAlt from a conventional caption corpus. Its basic supervised unit is not simply image plus text, but **context, image, alt-text**. This enables both domain adaptation to social-media alt-text and the construction of preference-learning instances that depend on response quality, image grounding, and contextual alignment.

A common misconception is to treat PAlt as a manually curated set of ideal captions. The paper instead presents it as a resource built from user-written Pinterest alt-text and then refined through filtering, grammar correction, preference construction, and model-assisted verification. Another misconception is to treat it as a response-only preference dataset. The paper’s formulation is broader: PAlt provides the aligned multimodal structure from which single-preference, paired-preference, and multi-preference objectives are instantiated.

## 2. Dataset construction pipeline

PAlt is sourced from Pinterest by scraping posts with images, post-text, and alt-text. Because Pinterest does not provide an API, the collection procedure used **100,000 commonly used Wikipedia words as search queries** to find web pages and collect posts. This yielded over **22 million posts**, from which approximately **150,000 rough samples** containing image, post-text, and alt-text were retained before later processing [2510.00647].

The pipeline then applies a sequence of normalization and quality-control steps. Non-static images such as GIFs are removed, as are non-English samples, samples with hashtags, and samples whose alt-text has fewer than **5 words**. Deduplication uses **CLIP** to identify duplicates based on the representation similarity of post-text or image embeddings across samples, and the paper additionally states that “visual match clusters” were determined based on pixel overlap. Person names are replaced with **`[person]`** using named entity recognition tools, with the stated rationale that predicting a person’s identity from an image would involve facial recognition and is out of scope.

A second stage applies grammar correction to **post-text and alt-text** using **Gemini 1.5 Flash**. The stated intent is to make as few modifications as possible and only correct grammar issues. As a result, the supervised PAlt data is not raw platform text, but grammatically corrected user context and user alt-text.

A third stage constructs the preference subset. The chosen alt-text \(y_w\) is the human-preferred alt-text derived from the platform data after cleaning and correction. The rejected alt-text \(y_l\) is generated by **Gemini 1.5 Flash** from the prompt:

\[
\text{Context: } \{context\} \ \text{Best alt-text: } \{alt\_text\} \ \text{Suboptimal alt-text:}
\]

The paper states that **temperature sampling and manual filtering** are used to reduce bias from relying on a single generator model and to ensure that the rejected response is clearly worse in accuracy and coherence without imposing a stylistic template. Preference-pair verification is then performed with Gemini as evaluator by comparing chosen and rejected responses in both orders; a pair is kept only if the chosen response is consistently preferred before and after swapping positions. If not, the rejected candidate is regenerated and re-evaluated, up to **three attempts**.

## 3. Internal structure and data splits

PAlt is a suite of three related subsets rather than a single homogeneous file. The paper explicitly reports three Pinterest-specific splits: **PAlt Training**, **PAlt Preference**, and **PAlt Test** [2510.00647].

| Subset | Size | Role |
|---|---:|---|
| PAlt Training | 101,715 | Supervised fine-tuning |
| PAlt Preference | 8,212 | Preference optimization |
| PAlt Test | 1,702 | Evaluation |

For supervised learning, each example is a triplet \((c, m, y)\) consisting of **context / post-text** \(c\), **image** \(m\), and **alt-text** \(y\). For preference optimization, a sample minimally contains a prompt \(x\), chosen image \(m_w\), chosen context \(c_w\), chosen alt-text \(y_w\), and rejected alt-text \(y_l\). During training, additional rejected modalities are constructed: rejected context \(c_l\), obtained by randomly selecting context from another sample, and rejected image \(m_l\), created from the chosen image using a transformation or another strategy. By implication, the test set contains image, context, and reference alt-text.

The paper does **not** report a separate validation split for PAlt, and it does **not** provide PAlt-specific average text lengths, category breakdowns, topic distributions, or preference-dimension frequency counts. It also does not describe Pinterest board metadata, pin titles, comments, or other side information beyond post-text/context and image. This suggests that PAlt’s operative representation is intentionally narrow: context-conditioned visual description rather than broader social-media metadata modeling.

## 4. Preference structure and relation to MCM-DPO

PAlt is technically central because it supports the preference construction required by MCM-DPO. The paper does not describe PAlt as containing manually labeled annotations for seven separate preference dimensions. Instead, it provides chosen and rejected responses plus aligned image-context-alt-text triples from which seven preference objectives are instantiated [2510.00647].

The seven preference dimensions are grouped into three modules. The **Single Preference Optimization** module contains **Responsive Preference Optimization (RPO)**, **Visual Preference Optimization (VPO)**, and **Contextual Preference Optimization (CPO)**. The **Pairwise Preference Optimization** module contains **Visual and Responsive Preference Optimization (VRPO)**, **Contextual and Responsive Preference Optimization (CRPO)**, and **Visual and Contextual Preference Optimization (VCPO)**. The **Multi-Preference Optimization Module** contributes \(\mathcal{L}_{MTPO}\), which jointly contrasts chosen and rejected image, context, and response tuples.

The overall objective is

\[
\mathcal{L}_{MCM-DPO} = \lambda * \mathcal{L}_{RPO} + \alpha * (\mathcal{L}_{VPO} + \mathcal{L}_{CPO}) + \gamma * (\mathcal{L}_{VRPO} + \mathcal{L}_{CRPO} + \mathcal{L}_{VCPO} + \mathcal{L}_{MTPO}).
\]

The best reported weights are \(\lambda = 1\), \(\alpha = 0.5\), and \(\gamma = 0.2\).

This structure clarifies what PAlt contributes. The human Pinterest alt-text provides \(y_w\); Gemini-generated suboptimal alt-text provides \(y_l\); random context replacement provides \(c_l\); and transformed or replaced images provide \(m_l\). A plausible implication is that PAlt functions less as a fixed label repository than as a scaffold for constructing cross-modal preference tuples over response, context, and image.

## 5. Experimental role and benchmark behavior

PAlt is used throughout the paper’s full training and evaluation pipeline. In the supervised stage, the base model **LLaVA-1.6 (7B)** is fine-tuned on the combined **202K** supervised samples from PAlt and TAlt, of which PAlt contributes **101,715** examples. In the alignment stage, DPO and MCM-DPO are applied to the SFT model using the PAlt and TAlt preference datasets, with PAlt contributing **8,212** preference examples. In evaluation, **PAlt-Test** with **1,702** examples is one of the two primary benchmarks, and many detailed analyses are conducted specifically on PAlt, including rejected-image strategy, diffusion noise-step tuning, component ablations, hyperparameter study, representation visualization, and case-focused discussion [2510.00647].

On **PAlt (Pinterest)**, the paper reports the following results:

- **LLaVA-1.6 (7B)**: ROUGE-L 17.32, BLEU4 2.46, METEOR 31.46, CIDEr 40.23  
- **+SFT**: 24.63, 7.63, 41.13, 99.27  
- **+DPO**: 32.71, 11.11, 40.84, 157.73  
- **+mDPO**: 36.63, 12.90, 41.37, 183.89  
- **+MCM-DPO**: **39.54**, **14.48**, 40.60, **207.98**

These results are used in the paper to support several conclusions: generic MLLMs perform poorly on Pinterest alt-text generation; supervised adaptation helps substantially; preference optimization helps more than SFT alone; and MCM-DPO outperforms both DPO and mDPO on PAlt. The paper also reports that the best PAlt result comes from training the vision encoder during both SFT and alignment, with MCM-DPO reaching **ROUGE-L 39.54** and **CIDEr 207.98** under that paradigm.

PAlt also serves as the main bed for analyzing synthetic visual negatives. On the rejected-image strategy comparison, the best PAlt results use the **Diffusion** strategy with **39.54 / 14.48 / 40.60 / 207.98**, and the best diffusion noise-step setting is reported as \(T=700\). In the ablation study, removing the multi-preference term reduces performance from **ROUGE-L 39.54, CIDEr 207.98** to **ROUGE-L 33.60, CIDEr 165.18**, while removing contextual preference also harms performance strongly. This reinforces the paper’s central claim that PAlt’s context field is not incidental but essential.

## 6. Limitations, caveats, and release conditions

The paper identifies several caveats that are specific to PAlt or directly affect its interpretation. The clearest is rarity and selection bias: only **0.68%** of encountered Pinterest samples contained alt-text, and the authors explicitly leave investigation of biases such as **cultural background** and **gender** for future work. A second caveat is reliance on model-assisted curation. Grammar correction, rejected-alt generation, and quality verification all use Gemini, so the preference subset is partly synthetic and model-mediated even though temperature sampling and manual filtering were introduced to reduce bias [2510.00647].

Further limitations are documentary rather than algorithmic. The paper does not provide a formal annotation manual for what counts as ideal alt-text beyond the operational pipeline. It does not report detailed descriptive statistics such as content-domain distributions, demographic distributions, average context length, or linguistic statistics. It also does not specify a separate validation split in the paper text.

These constraints matter for downstream use. PAlt is well suited for studying preference-based multimodal alignment in context-sensitive alt-text generation, but less suited for auditing platform-wide distributions or deriving a normative taxonomy of alt-text quality from explicit human guidelines. This suggests that the dataset’s strongest contribution lies in supervised-plus-preference training infrastructure rather than exhaustive sociolinguistic characterization.

The paper states that code and data are released at **`https://github.com/LVUGAI/MCM-DPO`**, and the appendix states that the dataset is licensed under **CC BY-NC 4.0** for non-commercial use. The same appendix also states that all data, code, and models are for research use only and must follow LLaMA/Vicuna license terms.

## 7. Significance within alt-text research

PAlt’s importance in the paper derives from the way it operationalizes alt-text generation as a multimodal, context-conditioned preference-learning problem rather than a standard captioning task. Its base supervised unit is a triplet of context, image, and alt-text, and its preference subset supports response-, image-, and context-level contrasts. In that sense, PAlt is the dataset that makes the paper’s multifaceted objective feasible on Pinterest data [2510.00647].

This role also clarifies what makes PAlt distinct from earlier resources. It is not merely a set of user-written alt-text examples, and it is not merely a benchmark test set. It combines real user-written alt-text, associated textual context, grammatically corrected supervised triplets, verified chosen/rejected response pairs, and the aligned multimodal structure required to instantiate single-preference, paired-preference, and multi-preference optimization. The paper’s practical takeaway is therefore specific: PAlt fills a gap left by prior alt-text work by providing a Pinterest-domain resource structured for both supervised adaptation and cross-modal preference learning.

Source: https://www.emergentmind.com/topics/palt