---
title: 'TAlt: Twitter Alt-text Dataset'
url: https://www.emergentmind.com/topics/talt
type: topic
---

# TAlt: Twitter Alt-text Dataset

to=arxiv_search.search 񎔽json
{"query":"2510.00647 MCM-DPO Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation TAlt", "max_results": 5, "sort_by":"relevance"}{"query":"Twitter alt-text dataset Srivatsan 2024 371K alt-text images tweets arXiv", "max_results": 5, "sort_by":"relevance"}
to=arxiv_search.search  qq彩票json
{"query":"alt-text generation arXiv Twitter alt-text dataset MCM-DPO", "max_results": 10, "sort_by":"relevance"}
to=arxiv_search.search  手机版天天中彩票ेसjson
{"query":"2510.00647", "max_results": 5, "sort_by":"relevance"}
TAlt, short for **Twitter Alt-text**, is a multimodal dataset for **context-aware alt-text generation** introduced in “MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation” [2510.00647]. It was constructed to support automatic generation of concise, context-relevant image descriptions for accessibility, especially for blind and low-vision users consuming social-media content. In the formulation used by the paper, each sample contains a prompt \(x\), an image \(m\), a context \(c\), and a response \(y\). TAlt is not a generic image-caption corpus: it is a Twitter-domain resource in which each example couples an image with surrounding post text and an alt-text description, and it also includes a structured preference subset for alignment-style training.

## 1. Definition and corpus structure

TAlt is one of two datasets introduced alongside **PAlt** in the MCM-DPO framework, with the two sources chosen to cover different social-media ecosystems. TAlt is the Twitter-domain component, whereas PAlt is sourced from Pinterest. The paper uses both supervised corpora jointly for supervised fine-tuning and uses domain-specific preference data and test sets for alignment and evaluation, which makes TAlt both a training resource and a benchmark [2510.00647].

The paper reports three TAlt components.

| Component | Role | Size |
|---|---|---:|
| Training (supervised) set | SFT corpus | 101,078 |
| Preference set | DPO / MCM-DPO | 9,938 |
| Test set | Evaluation | 1,688 |

These statistics place TAlt inside the paper’s broader combined total of **202K annotated alt-text samples** and **18K preference-annotated samples** across TAlt and PAlt. The reported Twitter evaluation split is named **TAlt-Test**. The paper does not report a separate validation split for TAlt, does not give per-category distributions, and does not provide image-taxonomy statistics. It also explicitly filters to **English**, so the released TAlt is effectively **English-only**.

At the supervised level, TAlt examples are described as **triplets of (context, image, alt-text)**. In the paper’s broader multimodal notation, the prompt \(x\) is part of the model input interface rather than a separately enumerated dataset field.

## 2. Data sources and construction pipeline

TAlt was created from a pre-existing Twitter alt-text resource rather than from direct platform collection from scratch. The paper states that it leverages the publicly available Twitter alt-text dataset from Srivatsan et al. (2024), described as containing **371K pairs of alt-text, images, and tweets** collected via the Twitter API. TAlt is derived from that corpus through filtering, cleanup, grammar correction, and later preference construction [2510.00647].

The construction pipeline is given in three stages: **(1) collection of user-written data, (2) grammar correction, and (3) preference collection**. For TAlt, the explicit filtering criteria are the following: removal of **non-static images** such as GIFs, removal of **non-English** samples, exclusion of samples with **hashtags**, and enforcement of a minimum alt-text length of **at least 5 words**. No additional rules such as toxicity filtering, OCR filtering, or image-safety filtering are reported.

After filtering, the dataset is **deduplicated** “to ensure the diversity of the collected samples.” The paper states that **CLIP** is used to filter duplicate data based on representation similarity of post-text or image embeddings across samples, and it also states that “visual match clusters” are determined based on pixel overlap. The exact thresholds are not reported.

A further normalization step is **person name replacement**. Named entity recognition tools are used to identify person names and replace them with the token **“[person]”**. The stated reason is that predicting identity from an image would verge into facial recognition and is outside scope. This implies that TAlt is not a verbatim archive of raw Twitter alt-text; it is a processed dataset whose textual fields have been normalized for privacy and task-boundary reasons.

## 3. Text normalization and preference annotation

The paper treats user-authored social-media text as noisy and irregular, and it therefore applies **grammar correction** to textual fields including both **post-text** and **alt-text**. The model used for this step is **Gemini 1.5 Flash**. According to the appendix summary reported in the dataset description, Gemini is prompted to detect whether grammatical errors exist and, if so, output a minimally modified corrected text. Consequently, TAlt’s supervised annotations are not raw user text exactly as posted; they are **user-written alt-texts that have been grammar-corrected by Gemini** [2510.00647].

The preference subset extends the annotation schema beyond supervised triplets. The paper defines chosen and rejected variants over response, image, and context:
\[
(y_w, y_l), \quad (m_w, m_l), \quad (c_w, c_l).
\]
For TAlt preference data, the chosen alt-text \(y_w\) comes from human-written alt-text, while the rejected alt-text \(y_l\) is generated with Gemini. Rejected contexts \(c_l\) are formed by randomly sampling context from other training samples. Rejected images \(m_l\) are synthetic perturbations or alternatives, with later analysis considering diffusion noise, black images, crops, rotations, and random unrelated images.

The rejected-response construction pipeline uses **Gemini 1.5 Flash** together with **temperature sampling** and **manual filtering**. The quality-verification stage also uses Gemini as evaluator, with candidate order swapped to reduce position bias. A pair is retained only if Gemini consistently chooses the preferred response before and after swapping; otherwise Gemini regenerates the rejected alt-text and re-evaluates, for up to **three attempts**. The paper does not report inter-annotator agreement, human agreement rates, or audit statistics such as Cohen’s kappa or Krippendorff’s alpha.

The paper’s operative notion of good alt-text is not given as a formal rubric, but it is described as **concise, context-aware, and focused on key information**. During preference construction, rejected alt-texts are manually filtered and quality-checked so that they are “clearly inferior in **accuracy and coherence**, without enforcing stylistic patterns.”

## 4. Modeling role in MCM-DPO

TAlt is tightly coupled to the paper’s learning formulation because the proposed method, **MCM-DPO**, requires both standard supervision and structured multimodal preference signals. For a sample with prompt \(x\), image \(m\), context \(c\), and response \(y\), the paper first gives multimodal DPO as
\[
\mathcal{L_{MDPO} = -\mathbb{E}\bigl[\log \sigma(r(x, m_w, y_w)- r(x, m_w, y_l))\bigr]
\]
and equivalently
\[
\mathcal{L_{MDPO} =  -\log \sigma \bigg(\beta\log \frac{\pi_{\theta} (y_w|m_w,x)}{\pi_\text{ref} (y_w|m_w,x)}- \beta\log \frac{\pi_{\theta} (y_l|m_w,x)}{\pi_\text{ref} (y_l|m_w,x)}\bigg).
\]
TAlt supplies the chosen/rejected tuples needed to instantiate these objectives [2510.00647].

The paper then decomposes the learning signal into seven preference losses spanning responses, images, and contexts. The response-only objective is
\[
\mathcal{L_{RPO} = -\mathbb{E}\bigl[\log \sigma(r(x, m_w, c_w, y_w)- r(x, m_w, c_w, y_l))\bigr],
\]
the visual preference objective is
\[
\mathcal{L_{VPO} =  -\mathbb{E}\bigl[\log \sigma(r(x, m_w, c_w, y_w)- r(x, m_l, c_w, y_w))\bigr],
\]
and the contextual preference objective is
\[
\mathcal{L_{CPO} =  -\mathbb{E}\bigl[\log \sigma(r(x, m_w, c_w, y_w)- r(x, m_w, c_l, y_w))\bigr].
\]
Pairwise objectives are also defined:
\[
\mathcal{L_{VRPO} =  -\mathbb{E}\bigl[\log \sigma(r(x, m_w, c_w, y_w)- r(x, m_l, c_w, y_l))\bigr],
\]
\[
\mathcal{L_{CRPO} =  -\mathbb{E}\bigl[\log \sigma(r(x, m_w, c_w, y_w) - r(x, m_w, c_l, y_l))\bigr],
\]
\[
\mathcal{L_{VCPO} =  -\mathbb{E}\bigl[\log \sigma(r(x, m_w, c_w, y_w) - r(x, m_l, c_l, y_w))\bigr],
\]
together with the full three-way objective
\[
\mathcal{L_{MTPO} =  -\mathbb{E}\bigl[\log \sigma(r(x, m_w, c_w, y_w) - r(x, m_l, c_l, y_l))\bigr].
\]

The overall MCM-DPO objective is given as
\[
\mathcal{L_{MCM-DPO} = \lambda * \mathcal{L_{RPO} + \alpha * (\mathcal{L_{VPO} +\mathcal{L_{CPO})  + \gamma * (\mathcal{L_{VRPO} + \mathcal{L_{CRPO} +\mathcal{L_{VCPO} + \mathcal{L_{MTPO}).
\]
The paper notes a typesetting issue, but the intended interpretation is a weighted sum of the seven losses with best hyperparameters \(\lambda=1\), \(\alpha=0.5\), and \(\gamma=0.2\). In this framework, TAlt is not merely a benchmark; it is the Twitter-domain source of the multimodal tuples needed for response, visual, contextual, pairwise, and multi-preference optimization.

## 5. Experimental use and benchmark performance

TAlt is used in all three major experimental roles in the paper: **supervised fine-tuning**, **preference optimization**, and **evaluation**. The supervised TAlt and PAlt training sets are mixed to fine-tune **LLaVA-1.6** for one epoch, while the TAlt preference set is used for alignment experiments and **TAlt-Test** is used for evaluation [2510.00647].

The evaluated baseline models on TAlt-Test include **InstructBLIP (13B), Chameleon, Qwen-VL-Chat (9.6B), MiniCPM-V 2.0 (2.8B), MiniCPM-Llama3-V 2.5 (8B), LLaVA-NEXT-Vicuna (13B), LLaVA-NEXT (34B), VIP-LLaVA (13B), and LLaVA-1.6 (7B)**. The paper’s central controlled comparison is within the LLaVA-1.6 backbone under the sequence base \(\rightarrow\) +SFT \(\rightarrow\) +DPO \(\rightarrow\) +mDPO \(\rightarrow\) +MCM-DPO.

On **TAlt (Twitter)**, the main reported scores for LLaVA-1.6 variants are:
- **LLaVA-1.6 (7B):** ROUGE-L 20.71, BLEU4 4.86, METEOR 21.31, CIDEr 49.85
- **+SFT:** 33.83, 14.90, 33.29, 144.64
- **+DPO:** 35.32, 15.87, 36.32, 146.32
- **+mDPO:** 35.74, 15.78, 35.77, 153.15
- **+MCM-DPO:** **36.32, 16.25, 36.62, 158.74**

Thus, on TAlt the best-performing method in the main table is **MCM-DPO**, which improves over both SFT and plain DPO. Relative to SFT, MCM-DPO improves ROUGE-L by about **2.49** points and CIDEr by about **14.10** points. Relative to DPO, it improves ROUGE-L by **1.00** and CIDEr by **12.42**. The training-paradigm analysis further reports that the best TAlt configuration is **+MCM-DPO with the visual encoder trainable in both SFT and alignment**, yielding the same best score tuple: **ROUGE-L 36.32, BLEU4 16.25, METEOR 36.62, CIDEr 158.74**.

Human evaluation is reported only in aggregate across the alt-text setting rather than separately for TAlt. Under the selected paradigm, **MCM-DPO outperformed DPO by 29% (39% vs. 10%)**, and paradigm 4 outperformed paradigm 1 by 21% (32% vs. 11%). A plausible implication is that the structured preference data supplied by TAlt contributes not only to lexical-overlap gains but also to comparative human preference, although the paper does not publish TAlt-only human-evaluation figures.

## 6. Scope, caveats, and naming ambiguities

TAlt has several explicit limitations. It is sourced from social-media content, so it inherits the noise and inconsistency of user-authored alt-text. The paper acknowledges possible dataset bias and notes that investigation of biases such as **cultural background** and **gender** is left for future work. Other reported concerns include dependence on automated grammar correction, dependence on Gemini for rejected-response generation and verification, English-only filtering, sparse source coverage because only a small fraction of posts contain alt-text, and the absence of inter-annotator agreement statistics [2510.00647].

Access and reuse are constrained. The paper states that **code and data are released** at the project repository, and the appendix states that the dataset is licensed under **CC BY-NC 4.0**, allowing **non-commercial use only**. It also states that data, code, and models are for **research use only** and must additionally follow **LLaMA and Vicuna license terms** for model usage.

A recurrent source of confusion is nomenclature. In current arXiv usage, **TAlt** in an explicit sense refers to **Twitter Alt-text** from the MCM-DPO work [2510.00647]. It is distinct from similarly named but unrelated methods and systems, including **TAAL** for medical-image active learning [2301.06624], **TALAN** for targeted post-training of large language models [2606.06902], and **TALON** for spacecraft pose estimation [2605.31217] or speculative decoding [2601.07353]. It is also unrelated to the use of “T” in vector-like top-quark searches [2408.08789]. For accessibility and multimodal generation research, however, TAlt denotes a Twitter-domain alt-text dataset whose main significance lies in combining supervised triplets with multimodal preference structure for alignment-oriented training.

Source: https://www.emergentmind.com/topics/talt