PAlt: Pinterest Alt-Text Dataset

• PAlt is a Pinterest-domain alt-text dataset that integrates context, image, and alt-text triplets to support precise, context-sensitive caption generation.
• It is constructed via rigorous filtering, grammar correction, and preference pairing, resulting in distinct splits for supervised fine-tuning, preference optimization, and evaluation.
• Experimental results, including ROUGE-L up to 39.54 and CIDEr of 207.98, demonstrate its effectiveness in enhancing multimodal preference learning.

Searching arXiv for relevant papers on “PAlt”. Using the arXiv search tool to verify the primary source and related context. PAlt is a Pinterest-domain alt-text dataset introduced alongside TAlt in the paper "MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation" (Fu et al., 1 Oct 2025). It was created to support context-sensitive alt-text generation, where a model must produce concise, context-relevant descriptions of images rather than generic long-form captions. In the paper’s design, PAlt is not a standalone artifact but one branch of a paired dataset construction spanning two social-media domains: Twitter through TAlt and Pinterest through PAlt. Its distinctive role is to provide cleaned supervised triplets, verified preference pairs, and a held-out evaluation set for multimodal preference optimization in a setting where user-written alt-text is scarce, noisy, and strongly conditioned on surrounding post context (Fu et al., 1 Oct 2025).

1. Definition, scope, and motivation

PAlt serves three functions in the experimental pipeline: a supervised dataset for initial fine-tuning, a preference dataset for preference optimization, and a test set for evaluation. The paper motivates its construction through three linked observations. First, existing alt-text data is scarce; for Pinterest specifically, only 0.68% of encountered samples included alt-text. Second, user-provided alt-text is noisy, because annotation standards on social media are inconsistent in quality, completeness, grammar, and salience selection. Third, alt-text is context-sensitive in a way ordinary image captions are not: it should be concise and grounded in both the image and the surrounding post-text, while avoiding merely repeating that context (Fu et al., 1 Oct 2025).

These design premises distinguish PAlt from a conventional caption corpus. Its basic supervised unit is not simply image plus text, but context, image, alt-text. This enables both domain adaptation to social-media alt-text and the construction of preference-learning instances that depend on response quality, image grounding, and contextual alignment.

A common misconception is to treat PAlt as a manually curated set of ideal captions. The paper instead presents it as a resource built from user-written Pinterest alt-text and then refined through filtering, grammar correction, preference construction, and model-assisted verification. Another misconception is to treat it as a response-only preference dataset. The paper’s formulation is broader: PAlt provides the aligned multimodal structure from which single-preference, paired-preference, and multi-preference objectives are instantiated.

2. Dataset construction pipeline

PAlt is sourced from Pinterest by scraping posts with images, post-text, and alt-text. Because Pinterest does not provide an API, the collection procedure used 100,000 commonly used Wikipedia words as search queries to find web pages and collect posts. This yielded over 22 million posts, from which approximately 150,000 rough samples containing image, post-text, and alt-text were retained before later processing (Fu et al., 1 Oct 2025).

The pipeline then applies a sequence of normalization and quality-control steps. Non-static images such as GIFs are removed, as are non-English samples, samples with hashtags, and samples whose alt-text has fewer than 5 words. Deduplication uses CLIP to identify duplicates based on the representation similarity of post-text or image embeddings across samples, and the paper additionally states that “visual match clusters” were determined based on pixel overlap. Person names are replaced with [person] using named entity recognition tools, with the stated rationale that predicting a person’s identity from an image would involve facial recognition and is out of scope.

A second stage applies grammar correction to post-text and alt-text using Gemini 1.5 Flash. The stated intent is to make as few modifications as possible and only correct grammar issues. As a result, the supervised PAlt data is not raw platform text, but grammatically corrected user context and user alt-text.

A third stage constructs the preference subset. The chosen alt-text $y_w$ is the human-preferred alt-text derived from the platform data after cleaning and correction. The rejected alt-text $y_l$ is generated by Gemini 1.5 Flash from the prompt:

$$\text{Context: } \{context\} \ \text{Best alt-text: } \{alt\_text\} \ \text{Suboptimal alt-text:}$$

The paper states that temperature sampling and manual filtering are used to reduce bias from relying on a single generator model and to ensure that the rejected response is clearly worse in accuracy and coherence without imposing a stylistic template. Preference-pair verification is then performed with Gemini as evaluator by comparing chosen and rejected responses in both orders; a pair is kept only if the chosen response is consistently preferred before and after swapping positions. If not, the rejected candidate is regenerated and re-evaluated, up to three attempts.

3. Internal structure and data splits

PAlt is a suite of three related subsets rather than a single homogeneous file. The paper explicitly reports three Pinterest-specific splits: PAlt Training, PAlt Preference, and PAlt Test (Fu et al., 1 Oct 2025).

Subset	Size	Role
PAlt Training	101,715	Supervised fine-tuning
PAlt Preference	8,212	Preference optimization
PAlt Test	1,702	Evaluation

For supervised learning, each example is a triplet $(c, m, y)$ consisting of context / post-text $c$, image $m$, and alt-text $y$. For preference optimization, a sample minimally contains a prompt $x$, chosen image $m_w$, chosen context $c_w$, chosen alt-text $y_l$0, and rejected alt-text $y_l$1. During training, additional rejected modalities are constructed: rejected context $y_l$2, obtained by randomly selecting context from another sample, and rejected image $y_l$3, created from the chosen image using a transformation or another strategy. By implication, the test set contains image, context, and reference alt-text.

The paper does not report a separate validation split for PAlt, and it does not provide PAlt-specific average text lengths, category breakdowns, topic distributions, or preference-dimension frequency counts. It also does not describe Pinterest board metadata, pin titles, comments, or other side information beyond post-text/context and image. This suggests that PAlt’s operative representation is intentionally narrow: context-conditioned visual description rather than broader social-media metadata modeling.

4. Preference structure and relation to MCM-DPO

PAlt is technically central because it supports the preference construction required by MCM-DPO. The paper does not describe PAlt as containing manually labeled annotations for seven separate preference dimensions. Instead, it provides chosen and rejected responses plus aligned image-context-alt-text triples from which seven preference objectives are instantiated (Fu et al., 1 Oct 2025).

The seven preference dimensions are grouped into three modules. The Single Preference Optimization module contains Responsive Preference Optimization (RPO), Visual Preference Optimization (VPO), and Contextual Preference Optimization (CPO). The Pairwise Preference Optimization module contains Visual and Responsive Preference Optimization (VRPO), Contextual and Responsive Preference Optimization (CRPO), and Visual and Contextual Preference Optimization (VCPO). The Multi-Preference Optimization Module contributes $y_l$4, which jointly contrasts chosen and rejected image, context, and response tuples.

The overall objective is

$y_l$5

The best reported weights are $y_l$6, $y_l$7, and $y_l$8.

This structure clarifies what PAlt contributes. The human Pinterest alt-text provides $y_l$9; Gemini-generated suboptimal alt-text provides $$\text{Context: } \{context\} \ \text{Best alt-text: } \{alt\_text\} \ \text{Suboptimal alt-text:}$$0; random context replacement provides $$\text{Context: } \{context\} \ \text{Best alt-text: } \{alt\_text\} \ \text{Suboptimal alt-text:}$$1; and transformed or replaced images provide $$\text{Context: } \{context\} \ \text{Best alt-text: } \{alt\_text\} \ \text{Suboptimal alt-text:}$$2. A plausible implication is that PAlt functions less as a fixed label repository than as a scaffold for constructing cross-modal preference tuples over response, context, and image.

5. Experimental role and benchmark behavior

PAlt is used throughout the paper’s full training and evaluation pipeline. In the supervised stage, the base model LLaVA-1.6 (7B) is fine-tuned on the combined 202K supervised samples from PAlt and TAlt, of which PAlt contributes 101,715 examples. In the alignment stage, DPO and MCM-DPO are applied to the SFT model using the PAlt and TAlt preference datasets, with PAlt contributing 8,212 preference examples. In evaluation, PAlt-Test with 1,702 examples is one of the two primary benchmarks, and many detailed analyses are conducted specifically on PAlt, including rejected-image strategy, diffusion noise-step tuning, component ablations, hyperparameter study, representation visualization, and case-focused discussion (Fu et al., 1 Oct 2025).

On PAlt (Pinterest), the paper reports the following results:

• LLaVA-1.6 (7B): ROUGE-L 17.32, BLEU4 2.46, METEOR 31.46, CIDEr 40.23
• +SFT: 24.63, 7.63, 41.13, 99.27
• +DPO: 32.71, 11.11, 40.84, 157.73
• +mDPO: 36.63, 12.90, 41.37, 183.89
• +MCM-DPO: 39.54, 14.48, 40.60, 207.98

These results are used in the paper to support several conclusions: generic MLLMs perform poorly on Pinterest alt-text generation; supervised adaptation helps substantially; preference optimization helps more than SFT alone; and MCM-DPO outperforms both DPO and mDPO on PAlt. The paper also reports that the best PAlt result comes from training the vision encoder during both SFT and alignment, with MCM-DPO reaching ROUGE-L 39.54 and CIDEr 207.98 under that paradigm.

PAlt also serves as the main bed for analyzing synthetic visual negatives. On the rejected-image strategy comparison, the best PAlt results use the Diffusion strategy with 39.54 / 14.48 / 40.60 / 207.98, and the best diffusion noise-step setting is reported as $$\text{Context: } \{context\} \ \text{Best alt-text: } \{alt\_text\} \ \text{Suboptimal alt-text:}$$3. In the ablation study, removing the multi-preference term reduces performance from ROUGE-L 39.54, CIDEr 207.98 to ROUGE-L 33.60, CIDEr 165.18, while removing contextual preference also harms performance strongly. This reinforces the paper’s central claim that PAlt’s context field is not incidental but essential.

6. Limitations, caveats, and release conditions

The paper identifies several caveats that are specific to PAlt or directly affect its interpretation. The clearest is rarity and selection bias: only 0.68% of encountered Pinterest samples contained alt-text, and the authors explicitly leave investigation of biases such as cultural background and gender for future work. A second caveat is reliance on model-assisted curation. Grammar correction, rejected-alt generation, and quality verification all use Gemini, so the preference subset is partly synthetic and model-mediated even though temperature sampling and manual filtering were introduced to reduce bias (Fu et al., 1 Oct 2025).

Further limitations are documentary rather than algorithmic. The paper does not provide a formal annotation manual for what counts as ideal alt-text beyond the operational pipeline. It does not report detailed descriptive statistics such as content-domain distributions, demographic distributions, average context length, or linguistic statistics. It also does not specify a separate validation split in the paper text.

These constraints matter for downstream use. PAlt is well suited for studying preference-based multimodal alignment in context-sensitive alt-text generation, but less suited for auditing platform-wide distributions or deriving a normative taxonomy of alt-text quality from explicit human guidelines. This suggests that the dataset’s strongest contribution lies in supervised-plus-preference training infrastructure rather than exhaustive sociolinguistic characterization.

The paper states that code and data are released at https://github.com/LVUGAI/MCM-DPO, and the appendix states that the dataset is licensed under CC BY-NC 4.0 for non-commercial use. The same appendix also states that all data, code, and models are for research use only and must follow LLaMA/Vicuna license terms.

7. Significance within alt-text research

PAlt’s importance in the paper derives from the way it operationalizes alt-text generation as a multimodal, context-conditioned preference-learning problem rather than a standard captioning task. Its base supervised unit is a triplet of context, image, and alt-text, and its preference subset supports response-, image-, and context-level contrasts. In that sense, PAlt is the dataset that makes the paper’s multifaceted objective feasible on Pinterest data (Fu et al., 1 Oct 2025).

This role also clarifies what makes PAlt distinct from earlier resources. It is not merely a set of user-written alt-text examples, and it is not merely a benchmark test set. It combines real user-written alt-text, associated textual context, grammatically corrected supervised triplets, verified chosen/rejected response pairs, and the aligned multimodal structure required to instantiate single-preference, paired-preference, and multi-preference optimization. The paper’s practical takeaway is therefore specific: PAlt fills a gap left by prior alt-text work by providing a Pinterest-domain resource structured for both supervised adaptation and cross-modal preference learning.