TAlt: Twitter Alt-text Dataset

• TAlt is a Twitter-focused alt-text dataset composed of (context, image, alt-text) triplets with an added preference subset for alignment training.
• It undergoes a rigorous pipeline including filtering, deduplication, grammar correction via Gemini 1.5 Flash, and normalization to ensure clean, context-aware annotations.
• When used in the MCM-DPO framework, TAlt significantly improves performance metrics like ROUGE-L and CIDEr, demonstrating its value for accessibility and multimodal research.

to=arxiv_search.search 񎔽json {"query":"(Fu et al., 1 Oct 2025) MCM-DPO Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation TAlt", "max_results": 5, "sort_by":"relevance"}{"query":"Twitter alt-text dataset Srivatsan 2024 371K alt-text images tweets arXiv", "max_results": 5, "sort_by":"relevance"} to=arxiv_search.search qq彩票json {"query":"alt-text generation arXiv Twitter alt-text dataset MCM-DPO", "max_results": 10, "sort_by":"relevance"} to=arxiv_search.search 手机版天天中彩票ेसjson {"query":"(Fu et al., 1 Oct 2025)", "max_results": 5, "sort_by":"relevance"} TAlt, short for Twitter Alt-text, is a multimodal dataset for context-aware alt-text generation introduced in “MCM-DPO: Multifaceted Cross-Modal Direct Preference Optimization for Alt-text Generation” (Fu et al., 1 Oct 2025). It was constructed to support automatic generation of concise, context-relevant image descriptions for accessibility, especially for blind and low-vision users consuming social-media content. In the formulation used by the paper, each sample contains a prompt $x$, an image $m$, a context $c$, and a response $y$. TAlt is not a generic image-caption corpus: it is a Twitter-domain resource in which each example couples an image with surrounding post text and an alt-text description, and it also includes a structured preference subset for alignment-style training.

1. Definition and corpus structure

TAlt is one of two datasets introduced alongside PAlt in the MCM-DPO framework, with the two sources chosen to cover different social-media ecosystems. TAlt is the Twitter-domain component, whereas PAlt is sourced from Pinterest. The paper uses both supervised corpora jointly for supervised fine-tuning and uses domain-specific preference data and test sets for alignment and evaluation, which makes TAlt both a training resource and a benchmark (Fu et al., 1 Oct 2025).

The paper reports three TAlt components.

Component	Role	Size
Training (supervised) set	SFT corpus	101,078
Preference set	DPO / MCM-DPO	9,938
Test set	Evaluation	1,688

These statistics place TAlt inside the paper’s broader combined total of 202K annotated alt-text samples and 18K preference-annotated samples across TAlt and PAlt. The reported Twitter evaluation split is named TAlt-Test. The paper does not report a separate validation split for TAlt, does not give per-category distributions, and does not provide image-taxonomy statistics. It also explicitly filters to English, so the released TAlt is effectively English-only.

At the supervised level, TAlt examples are described as triplets of (context, image, alt-text). In the paper’s broader multimodal notation, the prompt $x$ is part of the model input interface rather than a separately enumerated dataset field.

2. Data sources and construction pipeline

TAlt was created from a pre-existing Twitter alt-text resource rather than from direct platform collection from scratch. The paper states that it leverages the publicly available Twitter alt-text dataset from Srivatsan et al. (2024), described as containing 371K pairs of alt-text, images, and tweets collected via the Twitter API. TAlt is derived from that corpus through filtering, cleanup, grammar correction, and later preference construction (Fu et al., 1 Oct 2025).

The construction pipeline is given in three stages: (1) collection of user-written data, (2) grammar correction, and (3) preference collection. For TAlt, the explicit filtering criteria are the following: removal of non-static images such as GIFs, removal of non-English samples, exclusion of samples with hashtags, and enforcement of a minimum alt-text length of at least 5 words. No additional rules such as toxicity filtering, OCR filtering, or image-safety filtering are reported.

After filtering, the dataset is deduplicated “to ensure the diversity of the collected samples.” The paper states that CLIP is used to filter duplicate data based on representation similarity of post-text or image embeddings across samples, and it also states that “visual match clusters” are determined based on pixel overlap. The exact thresholds are not reported.

A further normalization step is person name replacement. Named entity recognition tools are used to identify person names and replace them with the token “[person]”. The stated reason is that predicting identity from an image would verge into facial recognition and is outside scope. This implies that TAlt is not a verbatim archive of raw Twitter alt-text; it is a processed dataset whose textual fields have been normalized for privacy and task-boundary reasons.

3. Text normalization and preference annotation

The paper treats user-authored social-media text as noisy and irregular, and it therefore applies grammar correction to textual fields including both post-text and alt-text. The model used for this step is Gemini 1.5 Flash. According to the appendix summary reported in the dataset description, Gemini is prompted to detect whether grammatical errors exist and, if so, output a minimally modified corrected text. Consequently, TAlt’s supervised annotations are not raw user text exactly as posted; they are user-written alt-texts that have been grammar-corrected by Gemini (Fu et al., 1 Oct 2025).

The preference subset extends the annotation schema beyond supervised triplets. The paper defines chosen and rejected variants over response, image, and context: $(y_w, y_l), \quad (m_w, m_l), \quad (c_w, c_l).$ For TAlt preference data, the chosen alt-text $y_w$ comes from human-written alt-text, while the rejected alt-text $y_l$ is generated with Gemini. Rejected contexts $c_l$ are formed by randomly sampling context from other training samples. Rejected images $m_l$ are synthetic perturbations or alternatives, with later analysis considering diffusion noise, black images, crops, rotations, and random unrelated images.

The rejected-response construction pipeline uses Gemini 1.5 Flash together with temperature sampling and manual filtering. The quality-verification stage also uses Gemini as evaluator, with candidate order swapped to reduce position bias. A pair is retained only if Gemini consistently chooses the preferred response before and after swapping; otherwise Gemini regenerates the rejected alt-text and re-evaluates, for up to three attempts. The paper does not report inter-annotator agreement, human agreement rates, or audit statistics such as Cohen’s kappa or Krippendorff’s alpha.

The paper’s operative notion of good alt-text is not given as a formal rubric, but it is described as concise, context-aware, and focused on key information. During preference construction, rejected alt-texts are manually filtered and quality-checked so that they are “clearly inferior in accuracy and coherence, without enforcing stylistic patterns.”

4. Modeling role in MCM-DPO

TAlt is tightly coupled to the paper’s learning formulation because the proposed method, MCM-DPO, requires both standard supervision and structured multimodal preference signals. For a sample with prompt $m$0, image $m$1, context $m$2, and response $m$3, the paper first gives multimodal DPO as

$m$4

and equivalently

$m$5

TAlt supplies the chosen/rejected tuples needed to instantiate these objectives (Fu et al., 1 Oct 2025).

The paper then decomposes the learning signal into seven preference losses spanning responses, images, and contexts. The response-only objective is

$m$6

the visual preference objective is

$m$7

and the contextual preference objective is

$m$8

Pairwise objectives are also defined: $m$9

$c$0

$c$1

together with the full three-way objective

$c$2

The overall MCM-DPO objective is given as

$c$3

The paper notes a typesetting issue, but the intended interpretation is a weighted sum of the seven losses with best hyperparameters $c$4, $c$5, and $c$6. In this framework, TAlt is not merely a benchmark; it is the Twitter-domain source of the multimodal tuples needed for response, visual, contextual, pairwise, and multi-preference optimization.

5. Experimental use and benchmark performance

TAlt is used in all three major experimental roles in the paper: supervised fine-tuning, preference optimization, and evaluation. The supervised TAlt and PAlt training sets are mixed to fine-tune LLaVA-1.6 for one epoch, while the TAlt preference set is used for alignment experiments and TAlt-Test is used for evaluation (Fu et al., 1 Oct 2025).

The evaluated baseline models on TAlt-Test include InstructBLIP (13B), Chameleon, Qwen-VL-Chat (9.6B), MiniCPM-V 2.0 (2.8B), MiniCPM-Llama3-V 2.5 (8B), LLaVA-NEXT-Vicuna (13B), LLaVA-NEXT (34B), VIP-LLaVA (13B), and LLaVA-1.6 (7B). The paper’s central controlled comparison is within the LLaVA-1.6 backbone under the sequence base $c$7 +SFT $c$8 +DPO $c$9 +mDPO $y$0 +MCM-DPO.

On TAlt (Twitter), the main reported scores for LLaVA-1.6 variants are:

• LLaVA-1.6 (7B): ROUGE-L 20.71, BLEU4 4.86, METEOR 21.31, CIDEr 49.85
• +SFT: 33.83, 14.90, 33.29, 144.64
• +DPO: 35.32, 15.87, 36.32, 146.32
• +mDPO: 35.74, 15.78, 35.77, 153.15
• +MCM-DPO: 36.32, 16.25, 36.62, 158.74

Thus, on TAlt the best-performing method in the main table is MCM-DPO, which improves over both SFT and plain DPO. Relative to SFT, MCM-DPO improves ROUGE-L by about 2.49 points and CIDEr by about 14.10 points. Relative to DPO, it improves ROUGE-L by 1.00 and CIDEr by 12.42. The training-paradigm analysis further reports that the best TAlt configuration is +MCM-DPO with the visual encoder trainable in both SFT and alignment, yielding the same best score tuple: ROUGE-L 36.32, BLEU4 16.25, METEOR 36.62, CIDEr 158.74.

Human evaluation is reported only in aggregate across the alt-text setting rather than separately for TAlt. Under the selected paradigm, MCM-DPO outperformed DPO by 29% (39% vs. 10%), and paradigm 4 outperformed paradigm 1 by 21% (32% vs. 11%). A plausible implication is that the structured preference data supplied by TAlt contributes not only to lexical-overlap gains but also to comparative human preference, although the paper does not publish TAlt-only human-evaluation figures.

6. Scope, caveats, and naming ambiguities

TAlt has several explicit limitations. It is sourced from social-media content, so it inherits the noise and inconsistency of user-authored alt-text. The paper acknowledges possible dataset bias and notes that investigation of biases such as cultural background and gender is left for future work. Other reported concerns include dependence on automated grammar correction, dependence on Gemini for rejected-response generation and verification, English-only filtering, sparse source coverage because only a small fraction of posts contain alt-text, and the absence of inter-annotator agreement statistics (Fu et al., 1 Oct 2025).

Access and reuse are constrained. The paper states that code and data are released at the project repository, and the appendix states that the dataset is licensed under CC BY-NC 4.0, allowing non-commercial use only. It also states that data, code, and models are for research use only and must additionally follow LLaMA and Vicuna license terms for model usage.

A recurrent source of confusion is nomenclature. In current arXiv usage, TAlt in an explicit sense refers to Twitter Alt-text from the MCM-DPO work (Fu et al., 1 Oct 2025). It is distinct from similarly named but unrelated methods and systems, including TAAL for medical-image active learning (Gaillochet et al., 2023), TALAN for targeted post-training of LLMs (Zhang et al., 5 Jun 2026), and TALON for spacecraft pose estimation (Ali et al., 29 May 2026) or speculative decoding (Liu et al., 12 Jan 2026). It is also unrelated to the use of “T” in vector-like top-quark searches (Collaboration, 2024). For accessibility and multimodal generation research, however, TAlt denotes a Twitter-domain alt-text dataset whose main significance lies in combining supervised triplets with multimodal preference structure for alignment-oriented training.